Jul 16 22:33:44.521747 ip-10-0-140-146 systemd[1]: Starting Kubernetes Kubelet...
Jul 16 22:33:45.151944 ip-10-0-140-146 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 22:33:45.151944 ip-10-0-140-146 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 16 22:33:45.151944 ip-10-0-140-146 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 22:33:45.151944 ip-10-0-140-146 kubenswrapper[2573]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 16 22:33:45.151944 ip-10-0-140-146 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 22:33:45.151944 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.915814    2573 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 16 22:33:45.151944 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921825    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 22:33:45.151944 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921838    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 22:33:45.151944 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921842    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 22:33:45.151944 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921845    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 22:33:45.151944 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921848    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 22:33:45.151944 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921850    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 22:33:45.151944 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921853    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 22:33:45.151944 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921856    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 22:33:44.978860 ip-10-0-140-146 systemd[1]: Started Kubernetes Kubelet.
Jul 16 22:33:45.167357 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921859    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 22:33:45.167357 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921861    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 22:33:45.167357 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921864    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 22:33:45.167357 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921867    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 22:33:45.167357 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921870    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 22:33:45.167357 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921873    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 22:33:45.167357 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921875    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 22:33:45.167357 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921878    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 22:33:45.167357 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921881    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 22:33:45.167357 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921884    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 22:33:45.167357 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921887    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 22:33:45.167357 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921889    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 22:33:45.167357 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921892    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 22:33:45.167357 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921895    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 22:33:45.167357 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921897    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 22:33:45.167357 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921909    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 22:33:45.167357 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921912    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 22:33:45.167357 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921915    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 22:33:45.167357 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921917    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 22:33:45.167357 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921920    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 22:33:45.168103 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921923    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 22:33:45.168103 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921926    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 22:33:45.168103 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921928    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 22:33:45.168103 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921931    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 22:33:45.168103 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921933    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 22:33:45.168103 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921936    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 22:33:45.168103 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921938    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 22:33:45.168103 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921941    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 22:33:45.168103 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921943    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 22:33:45.168103 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921946    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 22:33:45.168103 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921949    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 22:33:45.168103 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921952    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 22:33:45.168103 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921954    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 22:33:45.168103 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921957    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 22:33:45.168103 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921959    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 22:33:45.168103 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921962    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 22:33:45.168103 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921964    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 22:33:45.168103 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921967    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 22:33:45.168103 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921969    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 22:33:45.168103 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921972    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 22:33:45.168810 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921974    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 22:33:45.168810 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921977    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 22:33:45.168810 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921979    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 22:33:45.168810 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921982    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 22:33:45.168810 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921984    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 22:33:45.168810 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921987    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 22:33:45.168810 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921989    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 22:33:45.168810 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921992    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 22:33:45.168810 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921995    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 22:33:45.168810 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.921998    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 22:33:45.168810 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922001    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 22:33:45.168810 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922003    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 22:33:45.168810 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922006    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 22:33:45.168810 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922009    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 22:33:45.168810 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922011    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 22:33:45.168810 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922016    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 22:33:45.168810 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922020    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 22:33:45.168810 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922023    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 22:33:45.168810 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922025    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 22:33:45.168810 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922028    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 22:33:45.169387 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922031    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 22:33:45.169387 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922039    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 22:33:45.169387 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922042    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 22:33:45.169387 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922044    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 22:33:45.169387 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922047    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 22:33:45.169387 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922050    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 22:33:45.169387 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922054    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 22:33:45.169387 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922058    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 22:33:45.169387 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922061    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 22:33:45.169387 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922064    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 22:33:45.169387 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922067    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 22:33:45.169387 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922070    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 22:33:45.169387 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922074    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 22:33:45.169387 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922077    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 22:33:45.169387 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922080    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 22:33:45.169387 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922083    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 22:33:45.169387 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922086    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 22:33:45.169387 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922089    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 22:33:45.169387 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922091    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 22:33:45.169997 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922095    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 22:33:45.169997 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922098    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 22:33:45.169997 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922100    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 22:33:45.169997 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922103    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 22:33:45.169997 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922106    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 22:33:45.169997 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922108    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 22:33:45.169997 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922111    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 22:33:45.169997 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922114    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 22:33:45.169997 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922116    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 22:33:45.169997 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922119    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 22:33:45.169997 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922122    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 22:33:45.169997 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922124    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 22:33:45.169997 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922807    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 22:33:45.169997 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922816    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 22:33:45.169997 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922820    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 22:33:45.169997 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922824    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 22:33:45.169997 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922827    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 22:33:45.169997 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922830    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 22:33:45.169997 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922833    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 22:33:45.170567 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922836    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 22:33:45.170567 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922838    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 22:33:45.170567 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922841    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 22:33:45.170567 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922844    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 22:33:45.170567 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922846    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 22:33:45.170567 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922849    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 22:33:45.170567 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922851    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 22:33:45.170567 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922854    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 22:33:45.170567 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922856    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 22:33:45.170567 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922859    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 22:33:45.170567 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922863    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 22:33:45.170567 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922865    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 22:33:45.170567 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922868    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 22:33:45.170567 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922871    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 22:33:45.170567 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922875    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 22:33:45.170567 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922878    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 22:33:45.170567 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922881    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 22:33:45.170567 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922883    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 22:33:45.170567 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922886    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 22:33:45.170567 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922888    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 22:33:45.171217 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922891    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 22:33:45.171217 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922893    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 22:33:45.171217 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922896    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 22:33:45.171217 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922898    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 22:33:45.171217 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922901    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 22:33:45.171217 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922904    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 22:33:45.171217 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922906    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 22:33:45.171217 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922910    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 22:33:45.171217 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922912    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 22:33:45.171217 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922914    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 22:33:45.171217 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922917    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 22:33:45.171217 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922920    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 22:33:45.171217 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922922    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 22:33:45.171217 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922924    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 22:33:45.171217 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922927    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 22:33:45.171217 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922929    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 22:33:45.171217 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922932    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 22:33:45.171217 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922935    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 22:33:45.171217 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922938    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 22:33:45.171217 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922940    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 22:33:45.171817 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922943    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 22:33:45.171817 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922945    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 22:33:45.171817 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922948    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 22:33:45.171817 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922952    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 22:33:45.171817 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922955    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 22:33:45.171817 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922958    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 22:33:45.171817 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922960    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 22:33:45.171817 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922963    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 22:33:45.171817 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922966    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 22:33:45.171817 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922968    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 22:33:45.171817 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922971    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 22:33:45.171817 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922974    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 22:33:45.171817 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922977    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 22:33:45.171817 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922981    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 22:33:45.171817 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922984    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 22:33:45.171817 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922986    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 22:33:45.171817 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922989    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 22:33:45.171817 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922993    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 22:33:45.171817 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922996    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 22:33:45.172406 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.922999    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 22:33:45.172406 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923002    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 22:33:45.172406 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923005    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 22:33:45.172406 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923008    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 22:33:45.172406 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923011    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 22:33:45.172406 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923014    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 22:33:45.172406 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923017    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 22:33:45.172406 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923020    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 22:33:45.172406 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923023    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 22:33:45.172406 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923026    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 22:33:45.172406 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923029    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 22:33:45.172406 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923032    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 22:33:45.172406 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923034    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 22:33:45.172406 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923037    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 22:33:45.172406 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923039    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 22:33:45.172406 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923042    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 22:33:45.172406 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923044    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 22:33:45.172406 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923047    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 22:33:45.172406 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923050    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 22:33:45.172406 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923052    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 22:33:45.173035 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923055    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 22:33:45.173035 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923058    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 22:33:45.173035 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923060    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 22:33:45.173035 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923062    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 22:33:45.173035 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923065    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 22:33:45.173035 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923068    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 22:33:45.173035 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923070    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 22:33:45.173035 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923073    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 22:33:45.173035 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923075    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 22:33:45.173035 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923078    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 22:33:45.173035 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923081    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 22:33:45.173035 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923083    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 22:33:45.173035 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923086    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 22:33:45.173035 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923434    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 22:33:45.173035 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923438    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 22:33:45.173035 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923441    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 22:33:45.173035 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923444    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 22:33:45.173035 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923447    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 22:33:45.173035 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923449    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 22:33:45.173035 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923452    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 22:33:45.173671 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923454    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 22:33:45.173671 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923457    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 22:33:45.173671 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923460    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 22:33:45.173671 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923462    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 22:33:45.173671 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923465    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 22:33:45.173671 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923468    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 22:33:45.173671 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923470    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 22:33:45.173671 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923473    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 22:33:45.173671 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923475    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 22:33:45.173671 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923478    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 22:33:45.173671 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923481    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 22:33:45.173671 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923503    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 22:33:45.173671 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923508    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 22:33:45.173671 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923510    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 22:33:45.173671 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923513    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 22:33:45.173671 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923517    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 22:33:45.173671 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923519    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 22:33:45.173671 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923521    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 22:33:45.173671 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923524    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 22:33:45.173671 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923528    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 22:33:45.174263 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923531    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 22:33:45.174263 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923534    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 22:33:45.174263 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923536    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 22:33:45.174263 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923539    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 22:33:45.174263 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923542    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 22:33:45.174263 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923544    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 22:33:45.174263 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923547    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 22:33:45.174263 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923549    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 22:33:45.174263 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923552    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 22:33:45.174263 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923555    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 22:33:45.174263 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923558    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 22:33:45.174263 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923560    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 22:33:45.174263 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923563    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 22:33:45.174263 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923565    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 22:33:45.174263 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923568    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 22:33:45.174263 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923570    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 22:33:45.174263 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923573    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 22:33:45.174263 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923575    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 22:33:45.174263 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923579    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 22:33:45.174822 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923581    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 22:33:45.174822 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923584    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 22:33:45.174822 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923587    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 22:33:45.174822 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923590    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 22:33:45.174822 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923592    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 22:33:45.174822 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923595    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 22:33:45.174822 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923597    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 22:33:45.174822 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923600    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 22:33:45.174822 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923602    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 22:33:45.174822 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923605    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 22:33:45.174822 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923607    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 22:33:45.174822 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923610    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 22:33:45.174822 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923614    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 22:33:45.174822 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923617    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 22:33:45.174822 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923621    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 22:33:45.174822 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923624    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 22:33:45.174822 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923626    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 22:33:45.174822 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923630    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 22:33:45.174822 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923633    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 22:33:45.174822 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923635    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 22:33:45.175335 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923638    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 22:33:45.175335 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923641    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 22:33:45.175335 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923643    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 22:33:45.175335 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923646    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 22:33:45.175335 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923648    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 22:33:45.175335 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923652    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 22:33:45.175335 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923654    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 22:33:45.175335 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923657    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 22:33:45.175335 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923659    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 22:33:45.175335 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923662    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 22:33:45.175335 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923664    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 22:33:45.175335 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923667    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 22:33:45.175335 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923669    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 22:33:45.175335 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923674    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 22:33:45.175335 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923677    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 22:33:45.175335 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923680    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 22:33:45.175335 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923683    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 22:33:45.175335 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923686    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 22:33:45.175335 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923689    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 22:33:45.175335 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923692    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 22:33:45.175908 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923696    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 22:33:45.175908 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923699    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 22:33:45.175908 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923702    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 22:33:45.175908 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923705    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 22:33:45.175908 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923709    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 22:33:45.175908 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923712    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 22:33:45.175908 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923715    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 22:33:45.175908 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923717    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 22:33:45.175908 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923720    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 22:33:45.175908 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923725    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 22:33:45.175908 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923728    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 22:33:45.175908 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923730    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 22:33:45.175908 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.923733    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 22:33:45.175908 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923798    2573 flags.go:64] FLAG: --address="0.0.0.0"
Jul 16 22:33:45.175908 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923805    2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 16 22:33:45.175908 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923811    2573 flags.go:64] FLAG: --anonymous-auth="true"
Jul 16 22:33:45.175908 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923815    2573 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 16 22:33:45.175908 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923819    2573 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 16 22:33:45.175908 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923822    2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 16 22:33:45.175908 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923826    2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 16 22:33:45.175908 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923830    2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 16 22:33:45.176448 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923833    2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 16 22:33:45.176448 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923836    2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 16 22:33:45.176448 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923839    2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 16 22:33:45.176448 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923842    2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 16 22:33:45.176448 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923845    2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 16 22:33:45.176448 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923848    2573 flags.go:64] FLAG: --cgroup-root=""
Jul 16 22:33:45.176448 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923851    2573 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 16 22:33:45.176448 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923855    2573 flags.go:64] FLAG: --client-ca-file=""
Jul 16 22:33:45.176448 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923858    2573 flags.go:64] FLAG: --cloud-provider="external"
Jul 16 22:33:45.176448 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923861    2573 flags.go:64] FLAG: --cluster-dns="[]"
Jul 16 22:33:45.176448 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923866    2573 flags.go:64] FLAG: --cluster-domain=""
Jul 16 22:33:45.176448 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923868    2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 16 22:33:45.176448 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923872    2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 16 22:33:45.176448 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923875    2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 16 22:33:45.176448 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923878    2573 flags.go:64] FLAG: --container-log-max-files="5"
Jul 16 22:33:45.176448 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923883    2573 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 16 22:33:45.176448 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923886    2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 16 22:33:45.176448 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923889    2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 16 22:33:45.176448 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923892    2573 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 16 22:33:45.176448 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923895    2573 flags.go:64] FLAG: --contention-profiling="false"
Jul 16 22:33:45.176448 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923899    2573 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 16 22:33:45.176448 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923902    2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 16 22:33:45.176448 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923905    2573 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 16 22:33:45.176448 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923908    2573 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 16 22:33:45.177429 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923912    2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 16 22:33:45.177429 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923915    2573 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 16 22:33:45.177429 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923918    2573 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 16 22:33:45.177429 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923921    2573 flags.go:64] FLAG: --enable-load-reader="false"
Jul 16 22:33:45.177429 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923924    2573 flags.go:64] FLAG: --enable-server="true"
Jul 16 22:33:45.177429 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923927    2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 16 22:33:45.177429 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923931    2573 flags.go:64] FLAG: --event-burst="100"
Jul 16 22:33:45.177429 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923935    2573 flags.go:64] FLAG: --event-qps="50"
Jul 16 22:33:45.177429 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923937    2573 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 16 22:33:45.177429 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923941    2573 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 16 22:33:45.177429 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923943    2573 flags.go:64] FLAG: --eviction-hard=""
Jul 16 22:33:45.177429 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923947    2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 16 22:33:45.177429 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923950    2573 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 16 22:33:45.177429 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923953    2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 16 22:33:45.177429 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923956    2573 flags.go:64] FLAG: --eviction-soft=""
Jul 16 22:33:45.177429 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923959    2573 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 16 22:33:45.177429 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923961    2573 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 16 22:33:45.177429 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923965    2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 16 22:33:45.177429 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923968    2573 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 16 22:33:45.177429 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923971    2573 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 16 22:33:45.177429 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923974    2573 flags.go:64] FLAG: --fail-swap-on="true"
Jul 16 22:33:45.177429 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923977    2573 flags.go:64] FLAG: --feature-gates=""
Jul 16 22:33:45.177429 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923980    2573 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 16 22:33:45.177429 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923985    2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 16 22:33:45.177429 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923988    2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 16 22:33:45.178577 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923991    2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 16 22:33:45.178577 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923994    2573 flags.go:64] FLAG: --healthz-port="10248"
Jul 16 22:33:45.178577 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.923997    2573 flags.go:64] FLAG: --help="false"
Jul 16 22:33:45.178577 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924001    2573 flags.go:64] FLAG: --hostname-override="ip-10-0-140-146.ec2.internal"
Jul 16 22:33:45.178577 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924005    2573 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 16 22:33:45.178577 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924008    2573 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 16 22:33:45.178577 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924011    2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 16 22:33:45.178577 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924014    2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 16 22:33:45.178577 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924017    2573 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 16 22:33:45.178577 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924020    2573 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 16 22:33:45.178577 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924023    2573 flags.go:64] FLAG: --image-service-endpoint=""
Jul 16 22:33:45.178577 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924026    2573 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 16 22:33:45.178577 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924029    2573 flags.go:64] FLAG: --kube-api-burst="100"
Jul 16 22:33:45.178577 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924032    2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 16 22:33:45.178577 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924036    2573 flags.go:64] FLAG: --kube-api-qps="50"
Jul 16 22:33:45.178577 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924038    2573 flags.go:64] FLAG: --kube-reserved=""
Jul 16 22:33:45.178577 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924041    2573 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 16 22:33:45.178577 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924044    2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 16 22:33:45.178577 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924047    2573 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 16 22:33:45.178577 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924050    2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 16 22:33:45.178577 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924053    2573 flags.go:64] FLAG: --lock-file=""
Jul 16 22:33:45.178577 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924056    2573 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 16 22:33:45.178577 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924059    2573 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 16 22:33:45.178577 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924062    2573 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 16 22:33:45.179736 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924073    2573 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 16 22:33:45.179736 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924077    2573 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 16 22:33:45.179736 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924079    2573 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 16 22:33:45.179736 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924082    2573 flags.go:64] FLAG: --logging-format="text"
Jul 16 22:33:45.179736 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924085    2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 16 22:33:45.179736 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924088    2573 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 16 22:33:45.179736 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924092    2573 flags.go:64] FLAG: --manifest-url=""
Jul 16 22:33:45.179736 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924095    2573 flags.go:64] FLAG: --manifest-url-header=""
Jul 16 22:33:45.179736 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924100    2573 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 16 22:33:45.179736 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924103    2573 flags.go:64] FLAG: --max-open-files="1000000"
Jul 16 22:33:45.179736 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924106    2573 flags.go:64] FLAG: --max-pods="110"
Jul 16 22:33:45.179736 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924110    2573 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 16 22:33:45.179736 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924113    2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 16 22:33:45.179736 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924116    2573 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 16 22:33:45.179736 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924119    2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 16 22:33:45.179736 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924122    2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 16 22:33:45.179736 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924125    2573 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 16 22:33:45.179736 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924129    2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 16 22:33:45.179736 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924136    2573 flags.go:64] FLAG: --node-status-max-images="50"
Jul 16 22:33:45.179736 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924139    2573 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 16 22:33:45.179736 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924142    2573 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 16 22:33:45.179736 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924145    2573 flags.go:64] FLAG: --pod-cidr=""
Jul 16 22:33:45.179736 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924148    2573 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 16 22:33:45.183074 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924153    2573 flags.go:64] FLAG: --pod-manifest-path=""
Jul 16 22:33:45.183074 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924156    2573 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 16 22:33:45.183074 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924159    2573 flags.go:64] FLAG: --pods-per-core="0"
Jul 16 22:33:45.183074 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924162    2573 flags.go:64] FLAG: --port="10250"
Jul 16 22:33:45.183074 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924165    2573 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 16 22:33:45.183074 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924168    2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a98371aa6aff91ce"
Jul 16 22:33:45.183074 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924171    2573 flags.go:64] FLAG: --qos-reserved=""
Jul 16 22:33:45.183074 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924174    2573 flags.go:64] FLAG: --read-only-port="10255"
Jul 16 22:33:45.183074 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924177    2573 flags.go:64] FLAG: --register-node="true"
Jul 16 22:33:45.183074 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924180    2573 flags.go:64] FLAG: --register-with-taints=""
Jul 16 22:33:45.183074 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924183    2573 flags.go:64] FLAG: --registry-burst="10"
Jul 16 22:33:45.183074 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924186    2573 flags.go:64] FLAG: --registry-qps="5"
Jul 16 22:33:45.183074 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924189    2573 flags.go:64] FLAG: --reserved-cpus=""
Jul 16 22:33:45.183074 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924192    2573 flags.go:64] FLAG: --reserved-memory=""
Jul 16 22:33:45.183074 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924196    2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 16 22:33:45.183074 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924199    2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 16 22:33:45.183074 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924203    2573 flags.go:64] FLAG: --rotate-certificates="false"
Jul 16 22:33:45.183074 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924206    2573 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 16 22:33:45.183074 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924209    2573 flags.go:64] FLAG: --runonce="false"
Jul 16 22:33:45.183074 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924212    2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 16 22:33:45.183074 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924216    2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 16 22:33:45.183074 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924219    2573 flags.go:64] FLAG: --seccomp-default="false"
Jul 16 22:33:45.183074 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924222    2573 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 16 22:33:45.183074 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924225    2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 16 22:33:45.183074 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924228    2573 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 16 22:33:45.183074 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924231    2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 16 22:33:45.184411 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924234    2573 flags.go:64] FLAG: --storage-driver-password="root"
Jul 16 22:33:45.184411 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924237    2573 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 16 22:33:45.184411 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924240    2573 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 16 22:33:45.184411 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924243    2573 flags.go:64] FLAG: --storage-driver-user="root"
Jul 16 22:33:45.184411 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924246    2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 16 22:33:45.184411 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924249    2573 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 16 22:33:45.184411 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924252    2573 flags.go:64] FLAG: --system-cgroups=""
Jul 16 22:33:45.184411 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924255    2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 16 22:33:45.184411 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924260    2573 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 16 22:33:45.184411 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924263    2573 flags.go:64] FLAG: --tls-cert-file=""
Jul 16 22:33:45.184411 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924265    2573 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 16 22:33:45.184411 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924269    2573 flags.go:64] FLAG: --tls-min-version=""
Jul 16 22:33:45.184411 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924272    2573 flags.go:64] FLAG: --tls-private-key-file=""
Jul 16 22:33:45.184411 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924275    2573 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 16 22:33:45.184411 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924278    2573 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 16 22:33:45.184411 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924281    2573 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 16 22:33:45.184411 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924284    2573 flags.go:64] FLAG: --v="2"
Jul 16 22:33:45.184411 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924288    2573 flags.go:64] FLAG: --version="false"
Jul 16 22:33:45.184411 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924292    2573 flags.go:64] FLAG: --vmodule=""
Jul 16 22:33:45.184411 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924295    2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 16 22:33:45.184411 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.924299    2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 16 22:33:45.184411 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924409    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 22:33:45.184411 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924416    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 22:33:45.184411 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924419    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 22:33:45.268389 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924423    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 22:33:45.268389 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924426    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 22:33:45.268389 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924430    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 22:33:45.268389 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924433    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 22:33:45.268389 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924436    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 22:33:45.268389 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924439    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 22:33:45.268389 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924442    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 22:33:45.268389 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924445    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 22:33:45.268389 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924448    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 22:33:45.268389 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924451    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 22:33:45.268389 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924453    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 22:33:45.268389 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924456    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 22:33:45.268389 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924458    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 22:33:45.268389 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924461    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 22:33:45.268389 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924465    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 22:33:45.268389 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924468    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 22:33:45.268389 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924471    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 22:33:45.268389 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924473    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 22:33:45.268389 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924476    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 22:33:45.269307 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924478    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 22:33:45.269307 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924481    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 22:33:45.269307 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924509    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 22:33:45.269307 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924513    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 22:33:45.269307 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924516    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 22:33:45.269307 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924518    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 22:33:45.269307 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924521    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 22:33:45.269307 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924524    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 22:33:45.269307 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924527    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 22:33:45.269307 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924529    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 22:33:45.269307 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924532    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 22:33:45.269307 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924536    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 22:33:45.269307 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924539    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 22:33:45.269307 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924543    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 22:33:45.269307 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924546    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 22:33:45.269307 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924550    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 22:33:45.269307 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924553    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 22:33:45.269307 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924556    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 22:33:45.269307 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924559    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 22:33:45.269307 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924562    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 22:33:45.270192 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924564    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 22:33:45.270192 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924567    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 22:33:45.270192 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924569    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 22:33:45.270192 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924572    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 22:33:45.270192 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924575    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 22:33:45.270192 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924577    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 22:33:45.270192 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924580    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 22:33:45.270192 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924582    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 22:33:45.270192 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924585    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 22:33:45.270192 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924587    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 22:33:45.270192 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924590    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 22:33:45.270192 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924593    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 22:33:45.270192 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924595    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 22:33:45.270192 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924597    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 22:33:45.270192 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924600    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 22:33:45.270192 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924602    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 22:33:45.270192 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924606    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 22:33:45.270192 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924608    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 22:33:45.270192 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924611    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 22:33:45.270192 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924613    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 22:33:45.271091 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924616    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 22:33:45.271091 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924619    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 22:33:45.271091 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924629    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 22:33:45.271091 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924633    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 22:33:45.271091 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924636    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 22:33:45.271091 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924638    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 22:33:45.271091 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924641    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 22:33:45.271091 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924645    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 22:33:45.271091 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924647    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 22:33:45.271091 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924650    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 22:33:45.271091 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924653    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 22:33:45.271091 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924655    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 22:33:45.271091 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924658    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 22:33:45.271091 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924660    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 22:33:45.271091 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924663    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 22:33:45.271091 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924666    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 22:33:45.271091 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924668    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 22:33:45.271091 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924671    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 22:33:45.271091 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924673    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 22:33:45.271091 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924676    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 22:33:45.271982 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924678    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 22:33:45.271982 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924681    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 22:33:45.271982 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924684    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 22:33:45.271982 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924686    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 22:33:45.271982 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924688    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 22:33:45.271982 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924691    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 22:33:45.271982 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924694    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 22:33:45.271982 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924696    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 22:33:45.271982 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924698    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 22:33:45.271982 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924701    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 22:33:45.271982 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924704    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 22:33:45.271982 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924708    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 22:33:45.271982 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924710    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 22:33:45.271982 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924713    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 22:33:45.271982 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924716    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 22:33:45.271982 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924720    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 22:33:45.271982 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.924722    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 22:33:45.272877 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.925374    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 22:33:45.272877 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.931856    2573 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 16 22:33:45.272877 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.931873    2573 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 16 22:33:45.272877 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.932461    2573 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 16 22:33:45.272877 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.932468    2573 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 16 22:33:45.272877 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932544    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 22:33:45.272877 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932549    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 22:33:45.272877 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932552    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 22:33:45.272877 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932556    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 22:33:45.272877 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932559    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 22:33:45.272877 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932562    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 22:33:45.272877 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932564    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 22:33:45.272877 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932567    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 22:33:45.272877 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932570    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 22:33:45.272877 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932573    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 22:33:45.273760 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932576    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 22:33:45.273760 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932579    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 22:33:45.273760 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932582    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 22:33:45.273760 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932586    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 22:33:45.273760 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932588    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 22:33:45.273760 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932591    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 22:33:45.273760 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932594    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 22:33:45.273760 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932597    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 22:33:45.273760 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932599    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 22:33:45.273760 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932602    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 22:33:45.273760 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932604    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 22:33:45.273760 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932607    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 22:33:45.273760 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932609    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 22:33:45.273760 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932613    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 22:33:45.273760 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932617    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 22:33:45.273760 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932625    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 22:33:45.273760 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932629    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 22:33:45.273760 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932631    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 22:33:45.273760 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932634    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 22:33:45.274635 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932643    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 22:33:45.274635 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932646    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 22:33:45.274635 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932650    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 22:33:45.274635 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932654    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 22:33:45.274635 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932657    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 22:33:45.274635 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932661    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 22:33:45.274635 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932664    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 22:33:45.274635 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932667    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 22:33:45.274635 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932670    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 22:33:45.274635 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932673    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 22:33:45.274635 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932676    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 22:33:45.274635 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932678    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 22:33:45.274635 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932681    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 22:33:45.274635 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932684    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 22:33:45.274635 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932687    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 22:33:45.274635 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932689    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 22:33:45.274635 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932692    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 22:33:45.274635 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932694    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 22:33:45.274635 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932697    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 22:33:45.274635 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932699    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 22:33:45.275735 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932702    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 22:33:45.275735 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932704    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 22:33:45.275735 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932707    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 22:33:45.275735 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932709    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 22:33:45.275735 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932712    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 22:33:45.275735 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932714    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 22:33:45.275735 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932717    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 22:33:45.275735 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932719    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 22:33:45.275735 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932722    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 22:33:45.275735 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932724    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 22:33:45.275735 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932728    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 22:33:45.275735 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932731    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 22:33:45.275735 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932734    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 22:33:45.275735 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932744    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 22:33:45.275735 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932749    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 22:33:45.275735 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932753    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 22:33:45.275735 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932756    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 22:33:45.275735 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932758    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 22:33:45.275735 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932761    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 22:33:45.276625 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932764    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 22:33:45.276625 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932766    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 22:33:45.276625 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932769    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 22:33:45.276625 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932771    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 22:33:45.276625 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932774    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 22:33:45.276625 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932777    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 22:33:45.276625 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932779    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 22:33:45.276625 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932782    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 22:33:45.276625 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932785    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 22:33:45.276625 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932787    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 22:33:45.276625 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932790    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 22:33:45.276625 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932792    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 22:33:45.276625 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932795    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 22:33:45.276625 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932797    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 22:33:45.276625 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932800    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 22:33:45.276625 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932802    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 22:33:45.276625 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932805    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 22:33:45.276625 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932807    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 22:33:45.276625 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932809    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 22:33:45.276625 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932812    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 22:33:45.277534 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932815    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 22:33:45.277534 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932817    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 22:33:45.277534 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932820    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 22:33:45.277534 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932822    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 22:33:45.277534 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932825    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 22:33:45.277534 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932827    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 22:33:45.277534 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932830    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 22:33:45.277534 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932839    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 22:33:45.277534 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932843    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 22:33:45.277534 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932846    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 22:33:45.277534 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932848    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 22:33:45.277534 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.932853    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 22:33:45.277534 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932988    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 22:33:45.277534 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932993    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 22:33:45.277534 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932997    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 22:33:45.278204 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.932999    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 22:33:45.278204 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933003    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 22:33:45.278204 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933005    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 22:33:45.278204 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933008    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 22:33:45.278204 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933011    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 22:33:45.278204 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933014    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 22:33:45.278204 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933016    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 22:33:45.278204 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933018    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 22:33:45.278204 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933021    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 22:33:45.278204 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933024    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 22:33:45.278204 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933026    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 22:33:45.278204 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933029    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 22:33:45.278204 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933031    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 22:33:45.278204 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933034    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 22:33:45.278204 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933036    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 22:33:45.278204 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933039    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 22:33:45.278204 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933042    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 22:33:45.278204 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933044    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 22:33:45.278204 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933046    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 22:33:45.278204 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933049    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 22:33:45.279106 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933051    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 22:33:45.279106 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933054    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 22:33:45.279106 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933056    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 22:33:45.279106 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933059    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 22:33:45.279106 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933062    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 22:33:45.279106 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933071    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 22:33:45.279106 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933075    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 22:33:45.279106 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933077    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 22:33:45.279106 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933080    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 22:33:45.279106 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933082    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 22:33:45.279106 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933084    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 22:33:45.279106 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933088    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 22:33:45.279106 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933090    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 22:33:45.279106 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933093    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 22:33:45.279106 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933095    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 22:33:45.279106 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933098    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 22:33:45.279106 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933100    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 22:33:45.279106 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933103    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 22:33:45.279106 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933106    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 22:33:45.279983 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933110    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 22:33:45.279983 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933113    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 22:33:45.279983 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933115    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 22:33:45.279983 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933118    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 22:33:45.279983 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933120    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 22:33:45.279983 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933123    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 22:33:45.279983 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933125    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 22:33:45.279983 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933128    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 22:33:45.279983 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933130    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 22:33:45.279983 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933133    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 22:33:45.279983 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933137    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 22:33:45.279983 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933140    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 22:33:45.279983 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933143    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 22:33:45.279983 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933145    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 22:33:45.279983 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933148    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 22:33:45.279983 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933151    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 22:33:45.279983 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933153    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 22:33:45.279983 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933157    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 22:33:45.279983 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933159    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 22:33:45.279983 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933169    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 22:33:45.280881 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933171    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 22:33:45.280881 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933174    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 22:33:45.280881 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933177    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 22:33:45.280881 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933180    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 22:33:45.280881 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933182    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 22:33:45.280881 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933185    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 22:33:45.280881 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933188    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 22:33:45.280881 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933191    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 22:33:45.280881 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933194    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 22:33:45.280881 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933197    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 22:33:45.280881 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933199    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 22:33:45.280881 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933202    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 22:33:45.280881 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933204    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 22:33:45.280881 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933207    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 22:33:45.280881 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933210    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 22:33:45.280881 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933212    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 22:33:45.280881 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933214    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 22:33:45.280881 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933217    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 22:33:45.280881 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933219    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 22:33:45.281765 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933222    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 22:33:45.281765 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933225    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 22:33:45.281765 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933227    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 22:33:45.281765 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933229    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 22:33:45.281765 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933232    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 22:33:45.281765 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933234    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 22:33:45.281765 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933237    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 22:33:45.281765 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933240    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 22:33:45.281765 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933242    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 22:33:45.281765 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933244    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 22:33:45.281765 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933248    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 22:33:45.281765 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933250    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 22:33:45.281765 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933254    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 22:33:45.281765 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933262    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 22:33:45.281765 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933265    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 22:33:45.281765 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933268    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 22:33:45.281765 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933270    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 22:33:45.281765 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:44.933273    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 22:33:45.282911 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.933278    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 22:33:45.282911 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.933390    2573 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 16 22:33:45.282911 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.935216    2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 16 22:33:45.282911 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.936343    2573 server.go:1019] "Starting client certificate rotation"
Jul 16 22:33:45.282911 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.936447    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 16 22:33:45.282911 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.936474    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 16 22:33:45.282911 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.938981    2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 16 22:33:45.282911 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.940874    2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 16 22:33:45.282911 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.948880    2573 log.go:25] "Validated CRI v1 runtime API"
Jul 16 22:33:45.282911 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.950795    2573 log.go:25] "Validated CRI v1 image API"
Jul 16 22:33:45.282911 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.952157    2573 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 16 22:33:45.282911 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.954278    2573 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 883b59c3-3343-4b55-aab1-9a0b49d07da4:/dev/nvme0n1p4 f58dea04-9d29-4365-a809-ff3d08f0d1ee:/dev/nvme0n1p3]
Jul 16 22:33:45.283530 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.954292    2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 16 22:33:45.283733 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.960114    2573 manager.go:216] Machine: {Timestamp:2026-07-16 22:33:44.958895237 +0000 UTC m=+0.318906768 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103639 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23dd97d486016ec07bc313635388c8 SystemUUID:ec23dd97-d486-016e-c07b-c313635388c8 BootID:e2d8ee55-4748-49a0-9aeb-43d05c626eee Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a6:cb:83:39:d5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a6:cb:83:39:d5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3a:9d:e3:fe:48:78 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 16 22:33:45.283733 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.960208    2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 16 22:33:45.283733 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.960272    2573 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 16 22:33:45.283733 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.963092    2573 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 16 22:33:45.283733 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.963119    2573 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-146.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 16 22:33:45.283733 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.964251    2573 topology_manager.go:138] "Creating topology manager with none policy"
Jul 16 22:33:45.283733 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.964262    2573 container_manager_linux.go:309] "Creating device plugin manager"
Jul 16 22:33:45.283733 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.964269    2573 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 16 22:33:45.284387 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.964917    2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 16 22:33:45.284387 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.965978    2573 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 16 22:33:45.284387 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.967156    2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 16 22:33:45.284387 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.968614    2573 state_mem.go:36] "Initialized new in-memory state store"
Jul 16 22:33:45.284387 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.968724    2573 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 16 22:33:45.284387 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.968750    2573 kubelet.go:486] "Attempting to sync node with API server"
Jul 16 22:33:45.284387 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.968759    2573 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 16 22:33:45.284387 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.968776    2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 16 22:33:45.284387 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.968787    2573 kubelet.go:398] "Adding apiserver pod source"
Jul 16 22:33:45.284387 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.968804    2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 16 22:33:45.284387 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.969149    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 16 22:33:45.284387 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.969164    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 16 22:33:45.284387 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.971254    2573 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 16 22:33:45.284387 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.973118    2573 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 16 22:33:45.284387 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.973145    2573 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 16 22:33:45.284387 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.974328    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 16 22:33:45.284387 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.974342    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 16 22:33:45.284387 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.974348    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 16 22:33:45.284387 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.974353    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 16 22:33:45.284387 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.974363    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 16 22:33:45.284387 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.974371    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 16 22:33:45.284387 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.974384    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 16 22:33:45.284387 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.974389    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 16 22:33:45.284387 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.974395    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 16 22:33:45.284387 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.974400    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 16 22:33:45.284387 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.974413    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 16 22:33:45.284387 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.974418    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 16 22:33:45.284387 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.975132    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 16 22:33:45.284387 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.975144    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 16 22:33:45.284387 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.978112    2573 server.go:1268] "Started kubelet"
Jul 16 22:33:45.284387 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.978216    2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 16 22:33:45.284387 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.978377    2573 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 16 22:33:45.284387 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.978656    2573 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 16 22:33:45.284387 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.979840    2573 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 16 22:33:45.284387 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.981019    2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-146.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 16 22:33:45.284387 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:44.981282    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 16 22:33:45.284387 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:44.981313    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-146.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 16 22:33:45.284387 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.981345    2573 server.go:310] "Adding debug handlers to kubelet server"
Jul 16 22:33:45.284387 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:44.985233    2573 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 16 22:33:45.289811 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:44.984808    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-146.ec2.internal.18c2e5f0d86cff95  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-146.ec2.internal,UID:ip-10-0-140-146.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-146.ec2.internal,},FirstTimestamp:2026-07-16 22:33:44.978071445 +0000 UTC m=+0.338083066,LastTimestamp:2026-07-16 22:33:44.978071445 +0000 UTC m=+0.338083066,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-146.ec2.internal,}"
Jul 16 22:33:45.289811 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.986813    2573 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 16 22:33:45.289811 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.986817    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 16 22:33:45.289811 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:44.986884    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-146.ec2.internal\" not found"
Jul 16 22:33:45.289811 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.987478    2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 16 22:33:45.289811 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.987479    2573 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 16 22:33:45.289811 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.987542    2573 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 16 22:33:45.289811 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.987638    2573 reconstruct.go:97] "Volume reconstruction finished"
Jul 16 22:33:45.289811 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.987647    2573 reconciler.go:29] "Reconciler: start to sync state"
Jul 16 22:33:45.289811 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.987911    2573 factory.go:153] Registering CRI-O factory
Jul 16 22:33:45.289811 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.987926    2573 factory.go:223] Registration of the crio container factory successfully
Jul 16 22:33:45.289811 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.987974    2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 16 22:33:45.289811 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.987984    2573 factory.go:55] Registering systemd factory
Jul 16 22:33:45.289811 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.987993    2573 factory.go:223] Registration of the systemd container factory successfully
Jul 16 22:33:45.289811 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.988013    2573 factory.go:103] Registering Raw factory
Jul 16 22:33:45.289811 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.988027    2573 manager.go:1196] Started watching for new ooms in manager
Jul 16 22:33:45.289811 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.988868    2573 manager.go:349] Starting recovery of all containers
Jul 16 22:33:45.289811 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:44.990851    2573 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-146.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 16 22:33:45.289811 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:44.990948    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 16 22:33:45.289811 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.996690    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nlfmq"
Jul 16 22:33:45.289811 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:44.999172    2573 manager.go:354] Recovery completed
Jul 16 22:33:45.289811 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:45.000563    2573 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 16 22:33:45.289811 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.003251    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 22:33:45.289811 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.003396    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nlfmq"
Jul 16 22:33:45.289811 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.006007    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-146.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 22:33:45.289811 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.006037    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-146.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 22:33:45.289811 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.006051    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-146.ec2.internal" event="NodeHasSufficientPID"
Jul 16 22:33:45.289811 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.006533    2573 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 16 22:33:45.289811 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.006544    2573 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 16 22:33:45.289811 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.006546    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 22:33:45.289811 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.006560    2573 state_mem.go:36] "Initialized new in-memory state store"
Jul 16 22:33:45.289811 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.007649    2573 policy_none.go:49] "None policy: Start"
Jul 16 22:33:45.289811 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.007661    2573 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 16 22:33:45.291361 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.007670    2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 16 22:33:45.291361 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:45.007996    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-146.ec2.internal.18c2e5f0da17783b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-146.ec2.internal,UID:ip-10-0-140-146.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-146.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-146.ec2.internal,},FirstTimestamp:2026-07-16 22:33:45.006020667 +0000 UTC m=+0.366032179,LastTimestamp:2026-07-16 22:33:45.006020667 +0000 UTC m=+0.366032179,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-146.ec2.internal,}"
Jul 16 22:33:45.291361 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.008473    2573 policy_none.go:47] "Start"
Jul 16 22:33:45.291361 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.008881    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-146.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 22:33:45.291361 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.008904    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-146.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 22:33:45.291361 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.008917    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-146.ec2.internal" event="NodeHasSufficientPID"
Jul 16 22:33:45.291361 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:45.087683    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-146.ec2.internal\" not found"
Jul 16 22:33:45.291361 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.160023    2573 manager.go:337] "Starting Device Plugin manager"
Jul 16 22:33:45.291361 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:45.160161    2573 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 16 22:33:45.291361 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.160171    2573 server.go:87] "Starting device plugin registration server"
Jul 16 22:33:45.291361 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.160375    2573 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 16 22:33:45.291361 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.160402    2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 16 22:33:45.291361 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.160571    2573 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 16 22:33:45.291361 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.160662    2573 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 16 22:33:45.291361 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.160671    2573 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 16 22:33:45.291361 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:45.161099    2573 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 16 22:33:45.291361 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:45.161141    2573 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-146.ec2.internal\" not found"
Jul 16 22:33:45.291361 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.161225    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 16 22:33:45.291361 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.162910    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 16 22:33:45.291361 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.162944    2573 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 16 22:33:45.291361 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.162964    2573 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 16 22:33:45.291361 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:45.163000    2573 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 16 22:33:45.291361 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.165632    2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 22:33:45.291361 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:45.196951    2573 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-146.ec2.internal\" not found" node="ip-10-0-140-146.ec2.internal"
Jul 16 22:33:45.291361 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.261067    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 22:33:45.291361 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.261825    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-146.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 22:33:45.291361 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.261847    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-146.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 22:33:45.291361 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.261861    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-146.ec2.internal" event="NodeHasSufficientPID"
Jul 16 22:33:45.291361 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.261884    2573 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-146.ec2.internal"
Jul 16 22:33:45.291361 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.263974    2573 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-146.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-146.ec2.internal"]
Jul 16 22:33:45.291361 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.264022    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 22:33:45.291361 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.271390    2573 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-146.ec2.internal"
Jul 16 22:33:45.403896 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:45.271406    2573 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-146.ec2.internal\": node \"ip-10-0-140-146.ec2.internal\" not found"
Jul 16 22:33:45.403896 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.289122    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70417f5efada6062acead97c2af4fbd0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-146.ec2.internal\" (UID: \"70417f5efada6062acead97c2af4fbd0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-146.ec2.internal"
Jul 16 22:33:45.403896 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.289142    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/70417f5efada6062acead97c2af4fbd0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-146.ec2.internal\" (UID: \"70417f5efada6062acead97c2af4fbd0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-146.ec2.internal"
Jul 16 22:33:45.403896 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.300731    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-146.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 22:33:45.403896 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.300751    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-146.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 22:33:45.403896 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.300760    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-146.ec2.internal" event="NodeHasSufficientPID"
Jul 16 22:33:45.403896 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.302547    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 22:33:45.403896 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.302658    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-146.ec2.internal"
Jul 16 22:33:45.403896 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.302687    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 22:33:45.403896 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.303282    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-146.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 22:33:45.403896 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.303289    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-146.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 22:33:45.403896 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.303309    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-146.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 22:33:45.403896 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.303320    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-146.ec2.internal" event="NodeHasSufficientPID"
Jul 16 22:33:45.403896 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.303310    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-146.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 22:33:45.403896 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.303390    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-146.ec2.internal" event="NodeHasSufficientPID"
Jul 16 22:33:45.403896 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.305193    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-146.ec2.internal"
Jul 16 22:33:45.403896 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.305213    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 22:33:45.403896 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.305832    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-146.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 22:33:45.403896 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.305848    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-146.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 22:33:45.403896 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.305857    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-146.ec2.internal" event="NodeHasSufficientPID"
Jul 16 22:33:45.403896 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:45.324562    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-146.ec2.internal\" not found"
Jul 16 22:33:45.403896 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:45.336089    2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-146.ec2.internal\" not found" node="ip-10-0-140-146.ec2.internal"
Jul 16 22:33:45.403896 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:45.340245    2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-146.ec2.internal\" not found" node="ip-10-0-140-146.ec2.internal"
Jul 16 22:33:45.403896 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.389543    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/70417f5efada6062acead97c2af4fbd0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-146.ec2.internal\" (UID: \"70417f5efada6062acead97c2af4fbd0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-146.ec2.internal"
Jul 16 22:33:45.403896 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.389564    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70417f5efada6062acead97c2af4fbd0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-146.ec2.internal\" (UID: \"70417f5efada6062acead97c2af4fbd0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-146.ec2.internal"
Jul 16 22:33:45.405198 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.389581    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f66c4d021c76588d605e3a72206506df-config\") pod \"kube-apiserver-proxy-ip-10-0-140-146.ec2.internal\" (UID: \"f66c4d021c76588d605e3a72206506df\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-146.ec2.internal"
Jul 16 22:33:45.405198 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.389633    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/70417f5efada6062acead97c2af4fbd0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-146.ec2.internal\" (UID: \"70417f5efada6062acead97c2af4fbd0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-146.ec2.internal"
Jul 16 22:33:45.405198 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.389639    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70417f5efada6062acead97c2af4fbd0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-146.ec2.internal\" (UID: \"70417f5efada6062acead97c2af4fbd0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-146.ec2.internal"
Jul 16 22:33:45.501920 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:45.424596    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-146.ec2.internal\" not found"
Jul 16 22:33:45.501920 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.489686    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f66c4d021c76588d605e3a72206506df-config\") pod \"kube-apiserver-proxy-ip-10-0-140-146.ec2.internal\" (UID: \"f66c4d021c76588d605e3a72206506df\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-146.ec2.internal"
Jul 16 22:33:45.501920 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.489722    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f66c4d021c76588d605e3a72206506df-config\") pod \"kube-apiserver-proxy-ip-10-0-140-146.ec2.internal\" (UID: \"f66c4d021c76588d605e3a72206506df\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-146.ec2.internal"
Jul 16 22:33:47.002646 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:45.524926    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-146.ec2.internal\" not found"
Jul 16 22:33:47.002646 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:45.625449    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-146.ec2.internal\" not found"
Jul 16 22:33:47.002646 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.638597    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-146.ec2.internal"
Jul 16 22:33:47.002646 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:45.642405    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-146.ec2.internal"
Jul 16 22:33:47.106274 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:47.076213    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-146.ec2.internal\" not found"
Jul 16 22:33:47.213814 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:47.154992    2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 22:33:47.213814 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:47.176548    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-146.ec2.internal\" not found"
Jul 16 22:33:47.344144 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:47.277044    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-146.ec2.internal\" not found"
Jul 16 22:33:47.344144 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:47.286195    2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 16 22:33:47.344144 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:47.286309    2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 16 22:33:47.344144 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:47.286354    2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 16 22:33:47.443263 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:47.356430    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 22:28:46 +0000 UTC" deadline="2027-12-14 15:07:15.667500697 +0000 UTC"
Jul 16 22:33:47.443263 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:47.356450    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12376h33m28.311053745s"
Jul 16 22:33:47.443263 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:47.377277    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-146.ec2.internal\" not found"
Jul 16 22:33:47.528060 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:47.477740    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-146.ec2.internal\" not found"
Jul 16 22:33:47.610887 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:47.578395    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-146.ec2.internal\" not found"
Jul 16 22:33:47.758037 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:47.678854    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-146.ec2.internal\" not found"
Jul 16 22:33:47.922809 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:47.779378    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-146.ec2.internal\" not found"
Jul 16 22:33:47.922809 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:47.806038    2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 22:33:47.922809 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:47.807219    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70417f5efada6062acead97c2af4fbd0.slice/crio-3b5ba3f2e421a98c4e467a6b2d6e1545f009e86d964a0f019f4cddf489f08688 WatchSource:0}: Error finding container 3b5ba3f2e421a98c4e467a6b2d6e1545f009e86d964a0f019f4cddf489f08688: Status 404 returned error can't find the container with id 3b5ba3f2e421a98c4e467a6b2d6e1545f009e86d964a0f019f4cddf489f08688
Jul 16 22:33:47.922809 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:47.807737    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf66c4d021c76588d605e3a72206506df.slice/crio-30244c673fbaa32c646cb7d3261550ec378d700cbe7501350344db41879257a4 WatchSource:0}: Error finding container 30244c673fbaa32c646cb7d3261550ec378d700cbe7501350344db41879257a4: Status 404 returned error can't find the container with id 30244c673fbaa32c646cb7d3261550ec378d700cbe7501350344db41879257a4
Jul 16 22:33:47.922809 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:47.812421    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 22:33:47.922809 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:47.837258    2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-146.ec2.internal"
Jul 16 22:33:47.922809 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:47.839211    2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 22:33:47.922809 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:47.850106    2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 22:33:47.922809 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:47.851501    2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-146.ec2.internal"
Jul 16 22:33:47.922809 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:47.860310    2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 22:33:48.241089 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.133913    2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 22:33:48.320222 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.319976    2573 apiserver.go:52] "Watching apiserver"
Jul 16 22:33:48.330803 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.330610    2573 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 16 22:33:48.331081 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.331033    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-146.ec2.internal","openshift-multus/multus-additional-cni-plugins-rtgnk","openshift-network-diagnostics/network-check-target-7nqtn","openshift-ovn-kubernetes/ovnkube-node-r9jm4","kube-system/konnectivity-agent-466rj","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m44bd","openshift-cluster-node-tuning-operator/tuned-bv2kv","openshift-dns/node-resolver-8bst9","openshift-image-registry/node-ca-x4th8","openshift-multus/multus-85dmh","openshift-multus/network-metrics-daemon-6xfmc","openshift-network-operator/iptables-alerter-k9pz9","kube-system/global-pull-secret-syncer-kxpfx","kube-system/kube-apiserver-proxy-ip-10-0-140-146.ec2.internal"]
Jul 16 22:33:48.334465 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.333686    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bv2kv"
Jul 16 22:33:48.336285 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.336145    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 16 22:33:48.336574 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.336551    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fpg4c\""
Jul 16 22:33:48.336908 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.336890    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 16 22:33:48.338820 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.338523    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kxpfx"
Jul 16 22:33:48.338820 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.338523    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 16 22:33:48.338820 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:48.338644    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kxpfx" podUID="c8d410fb-20ac-4175-a4b4-38d5b44636d6"
Jul 16 22:33:48.338820 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.338679    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nqtn"
Jul 16 22:33:48.338820 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:48.338738    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7nqtn" podUID="bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef"
Jul 16 22:33:48.340609 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.340591    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.342556 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.342539    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-466rj"
Jul 16 22:33:48.345482 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.345462    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 16 22:33:48.346402 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.345715    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 16 22:33:48.346402 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.345726    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7gbsf\""
Jul 16 22:33:48.346402 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.345938    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 16 22:33:48.346402 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.345947    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m44bd"
Jul 16 22:33:48.346402 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.346114    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 16 22:33:48.346693 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.346576    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 16 22:33:48.347508 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.346811    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 16 22:33:48.347508 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.346866    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vn57h\""
Jul 16 22:33:48.347508 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.346879    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 16 22:33:48.347508 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.346818    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 16 22:33:48.347508 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.347320    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-x4th8"
Jul 16 22:33:48.348674 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.348633    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/791625c5-dbf2-4e78-b2ad-08ef2a62a57f-var-lib-kubelet\") pod \"tuned-bv2kv\" (UID: \"791625c5-dbf2-4e78-b2ad-08ef2a62a57f\") " pod="openshift-cluster-node-tuning-operator/tuned-bv2kv"
Jul 16 22:33:48.348771 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.348689    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c8d410fb-20ac-4175-a4b4-38d5b44636d6-kubelet-config\") pod \"global-pull-secret-syncer-kxpfx\" (UID: \"c8d410fb-20ac-4175-a4b4-38d5b44636d6\") " pod="kube-system/global-pull-secret-syncer-kxpfx"
Jul 16 22:33:48.348771 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.348713    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-host-cni-bin\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.348771 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.348762    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c8d410fb-20ac-4175-a4b4-38d5b44636d6-dbus\") pod \"global-pull-secret-syncer-kxpfx\" (UID: \"c8d410fb-20ac-4175-a4b4-38d5b44636d6\") " pod="kube-system/global-pull-secret-syncer-kxpfx"
Jul 16 22:33:48.348924 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.348797    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-host-slash\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.348924 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.348825    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-run-openvswitch\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.348924 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.348854    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-log-socket\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.348924 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.348889    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-host-cni-netd\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.348924 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.348915    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/791625c5-dbf2-4e78-b2ad-08ef2a62a57f-etc-sysconfig\") pod \"tuned-bv2kv\" (UID: \"791625c5-dbf2-4e78-b2ad-08ef2a62a57f\") " pod="openshift-cluster-node-tuning-operator/tuned-bv2kv"
Jul 16 22:33:48.349178 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.348940    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/791625c5-dbf2-4e78-b2ad-08ef2a62a57f-etc-systemd\") pod \"tuned-bv2kv\" (UID: \"791625c5-dbf2-4e78-b2ad-08ef2a62a57f\") " pod="openshift-cluster-node-tuning-operator/tuned-bv2kv"
Jul 16 22:33:48.349178 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.348964    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/791625c5-dbf2-4e78-b2ad-08ef2a62a57f-lib-modules\") pod \"tuned-bv2kv\" (UID: \"791625c5-dbf2-4e78-b2ad-08ef2a62a57f\") " pod="openshift-cluster-node-tuning-operator/tuned-bv2kv"
Jul 16 22:33:48.349178 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.348994    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/791625c5-dbf2-4e78-b2ad-08ef2a62a57f-host\") pod \"tuned-bv2kv\" (UID: \"791625c5-dbf2-4e78-b2ad-08ef2a62a57f\") " pod="openshift-cluster-node-tuning-operator/tuned-bv2kv"
Jul 16 22:33:48.349178 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.349035    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/791625c5-dbf2-4e78-b2ad-08ef2a62a57f-etc-tuned\") pod \"tuned-bv2kv\" (UID: \"791625c5-dbf2-4e78-b2ad-08ef2a62a57f\") " pod="openshift-cluster-node-tuning-operator/tuned-bv2kv"
Jul 16 22:33:48.349178 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.349065    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/791625c5-dbf2-4e78-b2ad-08ef2a62a57f-tmp\") pod \"tuned-bv2kv\" (UID: \"791625c5-dbf2-4e78-b2ad-08ef2a62a57f\") " pod="openshift-cluster-node-tuning-operator/tuned-bv2kv"
Jul 16 22:33:48.349178 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.349090    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-host-run-netns\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.349178 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.349115    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-run-systemd\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.349178 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.349139    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/791625c5-dbf2-4e78-b2ad-08ef2a62a57f-sys\") pod \"tuned-bv2kv\" (UID: \"791625c5-dbf2-4e78-b2ad-08ef2a62a57f\") " pod="openshift-cluster-node-tuning-operator/tuned-bv2kv"
Jul 16 22:33:48.349178 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.349161    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-var-lib-openvswitch\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.349630 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.349184    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-host-run-ovn-kubernetes\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.349630 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.349220    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 16 22:33:48.349630 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.349219    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/506f904a-3272-4586-8528-146884e728f9-ovn-node-metrics-cert\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.349630 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.349285    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/791625c5-dbf2-4e78-b2ad-08ef2a62a57f-run\") pod \"tuned-bv2kv\" (UID: \"791625c5-dbf2-4e78-b2ad-08ef2a62a57f\") " pod="openshift-cluster-node-tuning-operator/tuned-bv2kv"
Jul 16 22:33:48.349630 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.349308    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjgdw\" (UniqueName: \"kubernetes.io/projected/bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef-kube-api-access-vjgdw\") pod \"network-check-target-7nqtn\" (UID: \"bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef\") " pod="openshift-network-diagnostics/network-check-target-7nqtn"
Jul 16 22:33:48.349630 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.349331    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/506f904a-3272-4586-8528-146884e728f9-env-overrides\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.349630 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.349353    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mscr\" (UniqueName: \"kubernetes.io/projected/506f904a-3272-4586-8528-146884e728f9-kube-api-access-5mscr\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.349630 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.349376    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/791625c5-dbf2-4e78-b2ad-08ef2a62a57f-etc-kubernetes\") pod \"tuned-bv2kv\" (UID: \"791625c5-dbf2-4e78-b2ad-08ef2a62a57f\") " pod="openshift-cluster-node-tuning-operator/tuned-bv2kv"
Jul 16 22:33:48.349630 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.349408    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/791625c5-dbf2-4e78-b2ad-08ef2a62a57f-etc-sysctl-d\") pod \"tuned-bv2kv\" (UID: \"791625c5-dbf2-4e78-b2ad-08ef2a62a57f\") " pod="openshift-cluster-node-tuning-operator/tuned-bv2kv"
Jul 16 22:33:48.349630 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.349438    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/791625c5-dbf2-4e78-b2ad-08ef2a62a57f-etc-sysctl-conf\") pod \"tuned-bv2kv\" (UID: \"791625c5-dbf2-4e78-b2ad-08ef2a62a57f\") " pod="openshift-cluster-node-tuning-operator/tuned-bv2kv"
Jul 16 22:33:48.349630 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.349461    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfdt4\" (UniqueName: \"kubernetes.io/projected/791625c5-dbf2-4e78-b2ad-08ef2a62a57f-kube-api-access-qfdt4\") pod \"tuned-bv2kv\" (UID: \"791625c5-dbf2-4e78-b2ad-08ef2a62a57f\") " pod="openshift-cluster-node-tuning-operator/tuned-bv2kv"
Jul 16 22:33:48.349630 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.349484    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/791625c5-dbf2-4e78-b2ad-08ef2a62a57f-etc-modprobe-d\") pod \"tuned-bv2kv\" (UID: \"791625c5-dbf2-4e78-b2ad-08ef2a62a57f\") " pod="openshift-cluster-node-tuning-operator/tuned-bv2kv"
Jul 16 22:33:48.349630 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.349520    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c8d410fb-20ac-4175-a4b4-38d5b44636d6-original-pull-secret\") pod \"global-pull-secret-syncer-kxpfx\" (UID: \"c8d410fb-20ac-4175-a4b4-38d5b44636d6\") " pod="kube-system/global-pull-secret-syncer-kxpfx"
Jul 16 22:33:48.349630 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.349544    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-host-kubelet\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.350359 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.349568    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-etc-openvswitch\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.350359 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.349602    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.350359 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.349627    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-systemd-units\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.350359 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.349653    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 16 22:33:48.350359 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.349655    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-run-ovn\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.350359 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.349748    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-node-log\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.350359 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.349773    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/506f904a-3272-4586-8528-146884e728f9-ovnkube-config\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.350359 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.349777    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 16 22:33:48.350359 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.349796    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/506f904a-3272-4586-8528-146884e728f9-ovnkube-script-lib\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.350359 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.350066    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8bst9"
Jul 16 22:33:48.350359 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.350156    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8pkjh\""
Jul 16 22:33:48.350948 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.350491    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 16 22:33:48.350948 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.350519    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 16 22:33:48.350948 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.350617    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-85sfg\""
Jul 16 22:33:48.350948 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.350715    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 16 22:33:48.351997 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.351972    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 16 22:33:48.352324 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.352306    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rcqq4\""
Jul 16 22:33:48.352537 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.352519    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 16 22:33:48.352615 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.352587    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rtgnk"
Jul 16 22:33:48.353333 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.353151    2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 16 22:33:48.355433 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.355056    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 16 22:33:48.355433 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.355161    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 16 22:33:48.355433 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.355373    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.355611 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.355496    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 16 22:33:48.355835 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.355775    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 16 22:33:48.355835 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.355774    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 16 22:33:48.356066 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.356048    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-774t7\""
Jul 16 22:33:48.357581 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.357525    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 16 22:33:48.357850 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.357570    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6c898\""
Jul 16 22:33:48.358391 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.358131    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xfmc"
Jul 16 22:33:48.358391 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:48.358196    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xfmc" podUID="9b332999-321a-4e3f-99bf-19517a268e28"
Jul 16 22:33:48.361780 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.361321    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k9pz9"
Jul 16 22:33:48.364188 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.363825    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 16 22:33:48.365408 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.365392    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 16 22:33:48.365500 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.365416    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5v7lw\""
Jul 16 22:33:48.365669 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.365647    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 16 22:33:48.381675 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.381658    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vv5xv"
Jul 16 22:33:48.390295 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.389954    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vv5xv"
Jul 16 22:33:48.439050 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.439020    2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 16 22:33:48.450290 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.450249    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-var-lib-openvswitch\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.450404 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.450336    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-var-lib-openvswitch\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.450404 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.450388    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-host-run-ovn-kubernetes\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.450518 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.450420    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3aefec9e-d46f-4718-89eb-43a524eb0fcf-host-run-netns\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.450518 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.450444    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3aefec9e-d46f-4718-89eb-43a524eb0fcf-host-var-lib-cni-multus\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.450518 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.450467    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fpn6\" (UniqueName: \"kubernetes.io/projected/1d026b73-92ba-465b-bc31-5843395ae3e4-kube-api-access-7fpn6\") pod \"node-ca-x4th8\" (UID: \"1d026b73-92ba-465b-bc31-5843395ae3e4\") " pod="openshift-image-registry/node-ca-x4th8"
Jul 16 22:33:48.450518 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.450491    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/791625c5-dbf2-4e78-b2ad-08ef2a62a57f-run\") pod \"tuned-bv2kv\" (UID: \"791625c5-dbf2-4e78-b2ad-08ef2a62a57f\") " pod="openshift-cluster-node-tuning-operator/tuned-bv2kv"
Jul 16 22:33:48.450518 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.450513    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7144d772-5219-4c76-b0a6-950c4e94a235-agent-certs\") pod \"konnectivity-agent-466rj\" (UID: \"7144d772-5219-4c76-b0a6-950c4e94a235\") " pod="kube-system/konnectivity-agent-466rj"
Jul 16 22:33:48.450741 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.450536    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmvth\" (UniqueName: \"kubernetes.io/projected/15f7c683-2d93-427c-abb8-c345d44fdb68-kube-api-access-nmvth\") pod \"aws-ebs-csi-driver-node-m44bd\" (UID: \"15f7c683-2d93-427c-abb8-c345d44fdb68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m44bd"
Jul 16 22:33:48.450741 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.450560    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d026b73-92ba-465b-bc31-5843395ae3e4-host\") pod \"node-ca-x4th8\" (UID: \"1d026b73-92ba-465b-bc31-5843395ae3e4\") " pod="openshift-image-registry/node-ca-x4th8"
Jul 16 22:33:48.450741 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.450584    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2da4fabd-5fae-4a74-a228-7262f2a3e2c8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rtgnk\" (UID: \"2da4fabd-5fae-4a74-a228-7262f2a3e2c8\") " pod="openshift-multus/multus-additional-cni-plugins-rtgnk"
Jul 16 22:33:48.450741 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.450608    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/791625c5-dbf2-4e78-b2ad-08ef2a62a57f-etc-sysctl-d\") pod \"tuned-bv2kv\" (UID: \"791625c5-dbf2-4e78-b2ad-08ef2a62a57f\") " pod="openshift-cluster-node-tuning-operator/tuned-bv2kv"
Jul 16 22:33:48.450741 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.450630    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/268b78b0-10f6-44ce-a822-a124771df729-tmp-dir\") pod \"node-resolver-8bst9\" (UID: \"268b78b0-10f6-44ce-a822-a124771df729\") " pod="openshift-dns/node-resolver-8bst9"
Jul 16 22:33:48.450741 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.450661    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3aefec9e-d46f-4718-89eb-43a524eb0fcf-host-var-lib-cni-bin\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.450741 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.450686    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/791625c5-dbf2-4e78-b2ad-08ef2a62a57f-etc-modprobe-d\") pod \"tuned-bv2kv\" (UID: \"791625c5-dbf2-4e78-b2ad-08ef2a62a57f\") " pod="openshift-cluster-node-tuning-operator/tuned-bv2kv"
Jul 16 22:33:48.450741 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.450709    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c8d410fb-20ac-4175-a4b4-38d5b44636d6-original-pull-secret\") pod \"global-pull-secret-syncer-kxpfx\" (UID: \"c8d410fb-20ac-4175-a4b4-38d5b44636d6\") " pod="kube-system/global-pull-secret-syncer-kxpfx"
Jul 16 22:33:48.450741 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.450731    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-host-kubelet\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.452273 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.450770    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/15f7c683-2d93-427c-abb8-c345d44fdb68-sys-fs\") pod \"aws-ebs-csi-driver-node-m44bd\" (UID: \"15f7c683-2d93-427c-abb8-c345d44fdb68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m44bd"
Jul 16 22:33:48.452273 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.450793    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-run-ovn\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.452273 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.450826    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/506f904a-3272-4586-8528-146884e728f9-ovnkube-config\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.452273 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.450850    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/15f7c683-2d93-427c-abb8-c345d44fdb68-device-dir\") pod \"aws-ebs-csi-driver-node-m44bd\" (UID: \"15f7c683-2d93-427c-abb8-c345d44fdb68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m44bd"
Jul 16 22:33:48.452273 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.450890    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/268b78b0-10f6-44ce-a822-a124771df729-hosts-file\") pod \"node-resolver-8bst9\" (UID: \"268b78b0-10f6-44ce-a822-a124771df729\") " pod="openshift-dns/node-resolver-8bst9"
Jul 16 22:33:48.452273 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.450914    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2da4fabd-5fae-4a74-a228-7262f2a3e2c8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rtgnk\" (UID: \"2da4fabd-5fae-4a74-a228-7262f2a3e2c8\") " pod="openshift-multus/multus-additional-cni-plugins-rtgnk"
Jul 16 22:33:48.452273 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.450939    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c8d410fb-20ac-4175-a4b4-38d5b44636d6-kubelet-config\") pod \"global-pull-secret-syncer-kxpfx\" (UID: \"c8d410fb-20ac-4175-a4b4-38d5b44636d6\") " pod="kube-system/global-pull-secret-syncer-kxpfx"
Jul 16 22:33:48.452273 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.450962    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-host-cni-bin\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.452273 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.450983    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3aefec9e-d46f-4718-89eb-43a524eb0fcf-cnibin\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.452273 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.451005    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2da4fabd-5fae-4a74-a228-7262f2a3e2c8-system-cni-dir\") pod \"multus-additional-cni-plugins-rtgnk\" (UID: \"2da4fabd-5fae-4a74-a228-7262f2a3e2c8\") " pod="openshift-multus/multus-additional-cni-plugins-rtgnk"
Jul 16 22:33:48.452273 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.451031    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c8d410fb-20ac-4175-a4b4-38d5b44636d6-dbus\") pod \"global-pull-secret-syncer-kxpfx\" (UID: \"c8d410fb-20ac-4175-a4b4-38d5b44636d6\") " pod="kube-system/global-pull-secret-syncer-kxpfx"
Jul 16 22:33:48.452273 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.451039    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-host-run-ovn-kubernetes\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.452273 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.451055    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-run-openvswitch\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.452273 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.451080    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-log-socket\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.452980 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.451100    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/15f7c683-2d93-427c-abb8-c345d44fdb68-etc-selinux\") pod \"aws-ebs-csi-driver-node-m44bd\" (UID: \"15f7c683-2d93-427c-abb8-c345d44fdb68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m44bd"
Jul 16 22:33:48.452980 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.451120    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/791625c5-dbf2-4e78-b2ad-08ef2a62a57f-etc-systemd\") pod \"tuned-bv2kv\" (UID: \"791625c5-dbf2-4e78-b2ad-08ef2a62a57f\") " pod="openshift-cluster-node-tuning-operator/tuned-bv2kv"
Jul 16 22:33:48.452980 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.451139    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/791625c5-dbf2-4e78-b2ad-08ef2a62a57f-host\") pod \"tuned-bv2kv\" (UID: \"791625c5-dbf2-4e78-b2ad-08ef2a62a57f\") " pod="openshift-cluster-node-tuning-operator/tuned-bv2kv"
Jul 16 22:33:48.452980 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.451171    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-host-run-netns\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.452980 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.451185    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/791625c5-dbf2-4e78-b2ad-08ef2a62a57f-run\") pod \"tuned-bv2kv\" (UID: \"791625c5-dbf2-4e78-b2ad-08ef2a62a57f\") " pod="openshift-cluster-node-tuning-operator/tuned-bv2kv"
Jul 16 22:33:48.452980 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.451195    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/506f904a-3272-4586-8528-146884e728f9-ovn-node-metrics-cert\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.452980 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.451218    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3aefec9e-d46f-4718-89eb-43a524eb0fcf-etc-kubernetes\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.452980 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.451273    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxn7q\" (UniqueName: \"kubernetes.io/projected/3aefec9e-d46f-4718-89eb-43a524eb0fcf-kube-api-access-rxn7q\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.452980 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.451332    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2da4fabd-5fae-4a74-a228-7262f2a3e2c8-os-release\") pod \"multus-additional-cni-plugins-rtgnk\" (UID: \"2da4fabd-5fae-4a74-a228-7262f2a3e2c8\") " pod="openshift-multus/multus-additional-cni-plugins-rtgnk"
Jul 16 22:33:48.452980 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.451336    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/791625c5-dbf2-4e78-b2ad-08ef2a62a57f-etc-sysctl-d\") pod \"tuned-bv2kv\" (UID: \"791625c5-dbf2-4e78-b2ad-08ef2a62a57f\") " pod="openshift-cluster-node-tuning-operator/tuned-bv2kv"
Jul 16 22:33:48.452980 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.451333    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-host-cni-bin\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.452980 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.451403    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-run-openvswitch\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.452980 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.451412    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c8d410fb-20ac-4175-a4b4-38d5b44636d6-kubelet-config\") pod \"global-pull-secret-syncer-kxpfx\" (UID: \"c8d410fb-20ac-4175-a4b4-38d5b44636d6\") " pod="kube-system/global-pull-secret-syncer-kxpfx"
Jul 16 22:33:48.452980 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.451421    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-host-kubelet\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.453801 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.451438    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-log-socket\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.453801 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.451456    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-run-ovn\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.453801 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.451466    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-host-run-netns\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.453801 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.451467    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/791625c5-dbf2-4e78-b2ad-08ef2a62a57f-etc-modprobe-d\") pod \"tuned-bv2kv\" (UID: \"791625c5-dbf2-4e78-b2ad-08ef2a62a57f\") " pod="openshift-cluster-node-tuning-operator/tuned-bv2kv"
Jul 16 22:33:48.453801 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.451530    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/791625c5-dbf2-4e78-b2ad-08ef2a62a57f-host\") pod \"tuned-bv2kv\" (UID: \"791625c5-dbf2-4e78-b2ad-08ef2a62a57f\") " pod="openshift-cluster-node-tuning-operator/tuned-bv2kv"
Jul 16 22:33:48.453801 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:48.451535    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 22:33:48.453801 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.451572    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/791625c5-dbf2-4e78-b2ad-08ef2a62a57f-etc-systemd\") pod \"tuned-bv2kv\" (UID: \"791625c5-dbf2-4e78-b2ad-08ef2a62a57f\") " pod="openshift-cluster-node-tuning-operator/tuned-bv2kv"
Jul 16 22:33:48.453801 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:48.451603    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8d410fb-20ac-4175-a4b4-38d5b44636d6-original-pull-secret podName:c8d410fb-20ac-4175-a4b4-38d5b44636d6 nodeName:}" failed. No retries permitted until 2026-07-16 22:33:48.951583294 +0000 UTC m=+2.961845962 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c8d410fb-20ac-4175-a4b4-38d5b44636d6-original-pull-secret") pod "global-pull-secret-syncer-kxpfx" (UID: "c8d410fb-20ac-4175-a4b4-38d5b44636d6") : object "kube-system"/"original-pull-secret" not registered
Jul 16 22:33:48.453801 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.451644    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3aefec9e-d46f-4718-89eb-43a524eb0fcf-hostroot\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.453801 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.451674    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3aefec9e-d46f-4718-89eb-43a524eb0fcf-multus-daemon-config\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.453801 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.451715    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68qgq\" (UniqueName: \"kubernetes.io/projected/2da4fabd-5fae-4a74-a228-7262f2a3e2c8-kube-api-access-68qgq\") pod \"multus-additional-cni-plugins-rtgnk\" (UID: \"2da4fabd-5fae-4a74-a228-7262f2a3e2c8\") " pod="openshift-multus/multus-additional-cni-plugins-rtgnk"
Jul 16 22:33:48.453801 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.451751    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjgdw\" (UniqueName: \"kubernetes.io/projected/bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef-kube-api-access-vjgdw\") pod \"network-check-target-7nqtn\" (UID: \"bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef\") " pod="openshift-network-diagnostics/network-check-target-7nqtn"
Jul 16 22:33:48.453801 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.451754    2573 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 16 22:33:48.453801 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.451787    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/506f904a-3272-4586-8528-146884e728f9-env-overrides\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.454462 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.451904    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mscr\" (UniqueName: \"kubernetes.io/projected/506f904a-3272-4586-8528-146884e728f9-kube-api-access-5mscr\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.454462 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.451940    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3aefec9e-d46f-4718-89eb-43a524eb0fcf-system-cni-dir\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.454462 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.451947    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c8d410fb-20ac-4175-a4b4-38d5b44636d6-dbus\") pod \"global-pull-secret-syncer-kxpfx\" (UID: \"c8d410fb-20ac-4175-a4b4-38d5b44636d6\") " pod="kube-system/global-pull-secret-syncer-kxpfx"
Jul 16 22:33:48.454462 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.451989    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/506f904a-3272-4586-8528-146884e728f9-ovnkube-config\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.454462 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.451989    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2ada79d5-4ef5-489e-88af-07f0108282da-host-slash\") pod \"iptables-alerter-k9pz9\" (UID: \"2ada79d5-4ef5-489e-88af-07f0108282da\") " pod="openshift-network-operator/iptables-alerter-k9pz9"
Jul 16 22:33:48.454462 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.452071    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/791625c5-dbf2-4e78-b2ad-08ef2a62a57f-etc-kubernetes\") pod \"tuned-bv2kv\" (UID: \"791625c5-dbf2-4e78-b2ad-08ef2a62a57f\") " pod="openshift-cluster-node-tuning-operator/tuned-bv2kv"
Jul 16 22:33:48.454462 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.452105    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/791625c5-dbf2-4e78-b2ad-08ef2a62a57f-etc-sysctl-conf\") pod \"tuned-bv2kv\" (UID: \"791625c5-dbf2-4e78-b2ad-08ef2a62a57f\") " pod="openshift-cluster-node-tuning-operator/tuned-bv2kv"
Jul 16 22:33:48.454462 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.452153    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfdt4\" (UniqueName: \"kubernetes.io/projected/791625c5-dbf2-4e78-b2ad-08ef2a62a57f-kube-api-access-qfdt4\") pod \"tuned-bv2kv\" (UID: \"791625c5-dbf2-4e78-b2ad-08ef2a62a57f\") " pod="openshift-cluster-node-tuning-operator/tuned-bv2kv"
Jul 16 22:33:48.454462 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.452153    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/791625c5-dbf2-4e78-b2ad-08ef2a62a57f-etc-kubernetes\") pod \"tuned-bv2kv\" (UID: \"791625c5-dbf2-4e78-b2ad-08ef2a62a57f\") " pod="openshift-cluster-node-tuning-operator/tuned-bv2kv"
Jul 16 22:33:48.454462 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.452186    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/15f7c683-2d93-427c-abb8-c345d44fdb68-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m44bd\" (UID: \"15f7c683-2d93-427c-abb8-c345d44fdb68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m44bd"
Jul 16 22:33:48.454462 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.452269    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3aefec9e-d46f-4718-89eb-43a524eb0fcf-multus-cni-dir\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.454462 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.452273    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/791625c5-dbf2-4e78-b2ad-08ef2a62a57f-etc-sysctl-conf\") pod \"tuned-bv2kv\" (UID: \"791625c5-dbf2-4e78-b2ad-08ef2a62a57f\") " pod="openshift-cluster-node-tuning-operator/tuned-bv2kv"
Jul 16 22:33:48.454462 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.452285    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3aefec9e-d46f-4718-89eb-43a524eb0fcf-os-release\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.454462 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.452325    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1d026b73-92ba-465b-bc31-5843395ae3e4-serviceca\") pod \"node-ca-x4th8\" (UID: \"1d026b73-92ba-465b-bc31-5843395ae3e4\") " pod="openshift-image-registry/node-ca-x4th8"
Jul 16 22:33:48.455099 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.452360    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-etc-openvswitch\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.455099 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.452396    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-etc-openvswitch\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.455099 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.452403    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.455099 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.452438    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7144d772-5219-4c76-b0a6-950c4e94a235-konnectivity-ca\") pod \"konnectivity-agent-466rj\" (UID: \"7144d772-5219-4c76-b0a6-950c4e94a235\") " pod="kube-system/konnectivity-agent-466rj"
Jul 16 22:33:48.455099 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.452444    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/506f904a-3272-4586-8528-146884e728f9-env-overrides\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.455099 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.452468    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3aefec9e-d46f-4718-89eb-43a524eb0fcf-multus-socket-dir-parent\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.455099 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.452497    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3aefec9e-d46f-4718-89eb-43a524eb0fcf-multus-conf-dir\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.455099 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.452482    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.455099 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.452521    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfcj6\" (UniqueName: \"kubernetes.io/projected/9b332999-321a-4e3f-99bf-19517a268e28-kube-api-access-qfcj6\") pod \"network-metrics-daemon-6xfmc\" (UID: \"9b332999-321a-4e3f-99bf-19517a268e28\") " pod="openshift-multus/network-metrics-daemon-6xfmc"
Jul 16 22:33:48.455099 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.452604    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-systemd-units\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.455099 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.452628    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-node-log\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.455099 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.452630    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-systemd-units\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.455099 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.452665    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/506f904a-3272-4586-8528-146884e728f9-ovnkube-script-lib\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.455099 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.452670    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-node-log\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.455816 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.452702    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3aefec9e-d46f-4718-89eb-43a524eb0fcf-host-run-multus-certs\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.455816 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.452731    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2da4fabd-5fae-4a74-a228-7262f2a3e2c8-cnibin\") pod \"multus-additional-cni-plugins-rtgnk\" (UID: \"2da4fabd-5fae-4a74-a228-7262f2a3e2c8\") " pod="openshift-multus/multus-additional-cni-plugins-rtgnk"
Jul 16 22:33:48.455816 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.452756    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2da4fabd-5fae-4a74-a228-7262f2a3e2c8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rtgnk\" (UID: \"2da4fabd-5fae-4a74-a228-7262f2a3e2c8\") " pod="openshift-multus/multus-additional-cni-plugins-rtgnk"
Jul 16 22:33:48.455816 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.452781    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2ada79d5-4ef5-489e-88af-07f0108282da-iptables-alerter-script\") pod \"iptables-alerter-k9pz9\" (UID: \"2ada79d5-4ef5-489e-88af-07f0108282da\") " pod="openshift-network-operator/iptables-alerter-k9pz9"
Jul 16 22:33:48.455816 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.452806    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/791625c5-dbf2-4e78-b2ad-08ef2a62a57f-var-lib-kubelet\") pod \"tuned-bv2kv\" (UID: \"791625c5-dbf2-4e78-b2ad-08ef2a62a57f\") " pod="openshift-cluster-node-tuning-operator/tuned-bv2kv"
Jul 16 22:33:48.455816 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.452829    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/15f7c683-2d93-427c-abb8-c345d44fdb68-registration-dir\") pod \"aws-ebs-csi-driver-node-m44bd\" (UID: \"15f7c683-2d93-427c-abb8-c345d44fdb68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m44bd"
Jul 16 22:33:48.455816 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.452852    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3aefec9e-d46f-4718-89eb-43a524eb0fcf-host-var-lib-kubelet\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.455816 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.452875    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b332999-321a-4e3f-99bf-19517a268e28-metrics-certs\") pod \"network-metrics-daemon-6xfmc\" (UID: \"9b332999-321a-4e3f-99bf-19517a268e28\") " pod="openshift-multus/network-metrics-daemon-6xfmc"
Jul 16 22:33:48.455816 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.452898    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgm4j\" (UniqueName: \"kubernetes.io/projected/2ada79d5-4ef5-489e-88af-07f0108282da-kube-api-access-dgm4j\") pod \"iptables-alerter-k9pz9\" (UID: \"2ada79d5-4ef5-489e-88af-07f0108282da\") " pod="openshift-network-operator/iptables-alerter-k9pz9"
Jul 16 22:33:48.455816 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.452925    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-host-slash\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.455816 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.452947    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-host-cni-netd\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.455816 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.452971    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9dxq\" (UniqueName: \"kubernetes.io/projected/268b78b0-10f6-44ce-a822-a124771df729-kube-api-access-t9dxq\") pod \"node-resolver-8bst9\" (UID: \"268b78b0-10f6-44ce-a822-a124771df729\") " pod="openshift-dns/node-resolver-8bst9"
Jul 16 22:33:48.455816 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.452992    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2da4fabd-5fae-4a74-a228-7262f2a3e2c8-cni-binary-copy\") pod \"multus-additional-cni-plugins-rtgnk\" (UID: \"2da4fabd-5fae-4a74-a228-7262f2a3e2c8\") " pod="openshift-multus/multus-additional-cni-plugins-rtgnk"
Jul 16 22:33:48.456585 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.453019    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/791625c5-dbf2-4e78-b2ad-08ef2a62a57f-etc-sysconfig\") pod \"tuned-bv2kv\" (UID: \"791625c5-dbf2-4e78-b2ad-08ef2a62a57f\") " pod="openshift-cluster-node-tuning-operator/tuned-bv2kv"
Jul 16 22:33:48.456585 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.453041    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/791625c5-dbf2-4e78-b2ad-08ef2a62a57f-lib-modules\") pod \"tuned-bv2kv\" (UID: \"791625c5-dbf2-4e78-b2ad-08ef2a62a57f\") " pod="openshift-cluster-node-tuning-operator/tuned-bv2kv"
Jul 16 22:33:48.456585 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.453065    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/791625c5-dbf2-4e78-b2ad-08ef2a62a57f-etc-tuned\") pod \"tuned-bv2kv\" (UID: \"791625c5-dbf2-4e78-b2ad-08ef2a62a57f\") " pod="openshift-cluster-node-tuning-operator/tuned-bv2kv"
Jul 16 22:33:48.456585 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.453086    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/791625c5-dbf2-4e78-b2ad-08ef2a62a57f-tmp\") pod \"tuned-bv2kv\" (UID: \"791625c5-dbf2-4e78-b2ad-08ef2a62a57f\") " pod="openshift-cluster-node-tuning-operator/tuned-bv2kv"
Jul 16 22:33:48.456585 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.453111    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/506f904a-3272-4586-8528-146884e728f9-ovnkube-script-lib\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.456585 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.453109    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-run-systemd\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.456585 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.453190    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/791625c5-dbf2-4e78-b2ad-08ef2a62a57f-var-lib-kubelet\") pod \"tuned-bv2kv\" (UID: \"791625c5-dbf2-4e78-b2ad-08ef2a62a57f\") " pod="openshift-cluster-node-tuning-operator/tuned-bv2kv"
Jul 16 22:33:48.456585 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.453194    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/15f7c683-2d93-427c-abb8-c345d44fdb68-socket-dir\") pod \"aws-ebs-csi-driver-node-m44bd\" (UID: \"15f7c683-2d93-427c-abb8-c345d44fdb68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m44bd"
Jul 16 22:33:48.456585 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.453259    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-run-systemd\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.456585 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.453307    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-host-cni-netd\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.456585 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.453347    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/506f904a-3272-4586-8528-146884e728f9-host-slash\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.456585 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.453353    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3aefec9e-d46f-4718-89eb-43a524eb0fcf-cni-binary-copy\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.456585 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.453382    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3aefec9e-d46f-4718-89eb-43a524eb0fcf-host-run-k8s-cni-cncf-io\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.456585 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.453419    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/791625c5-dbf2-4e78-b2ad-08ef2a62a57f-lib-modules\") pod \"tuned-bv2kv\" (UID: \"791625c5-dbf2-4e78-b2ad-08ef2a62a57f\") " pod="openshift-cluster-node-tuning-operator/tuned-bv2kv"
Jul 16 22:33:48.456585 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.453427    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/791625c5-dbf2-4e78-b2ad-08ef2a62a57f-sys\") pod \"tuned-bv2kv\" (UID: \"791625c5-dbf2-4e78-b2ad-08ef2a62a57f\") " pod="openshift-cluster-node-tuning-operator/tuned-bv2kv"
Jul 16 22:33:48.457304 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.453527    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/791625c5-dbf2-4e78-b2ad-08ef2a62a57f-sys\") pod \"tuned-bv2kv\" (UID: \"791625c5-dbf2-4e78-b2ad-08ef2a62a57f\") " pod="openshift-cluster-node-tuning-operator/tuned-bv2kv"
Jul 16 22:33:48.457304 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.453530    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/791625c5-dbf2-4e78-b2ad-08ef2a62a57f-etc-sysconfig\") pod \"tuned-bv2kv\" (UID: \"791625c5-dbf2-4e78-b2ad-08ef2a62a57f\") " pod="openshift-cluster-node-tuning-operator/tuned-bv2kv"
Jul 16 22:33:48.457304 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.456288    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/506f904a-3272-4586-8528-146884e728f9-ovn-node-metrics-cert\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.457304 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.456562    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/791625c5-dbf2-4e78-b2ad-08ef2a62a57f-etc-tuned\") pod \"tuned-bv2kv\" (UID: \"791625c5-dbf2-4e78-b2ad-08ef2a62a57f\") " pod="openshift-cluster-node-tuning-operator/tuned-bv2kv"
Jul 16 22:33:48.457304 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.456663    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/791625c5-dbf2-4e78-b2ad-08ef2a62a57f-tmp\") pod \"tuned-bv2kv\" (UID: \"791625c5-dbf2-4e78-b2ad-08ef2a62a57f\") " pod="openshift-cluster-node-tuning-operator/tuned-bv2kv"
Jul 16 22:33:48.462321 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:48.462300    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 22:33:48.462425 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:48.462330    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 22:33:48.462425 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:48.462345    2573 projected.go:196] Error preparing data for projected volume kube-api-access-vjgdw for pod openshift-network-diagnostics/network-check-target-7nqtn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 22:33:48.462425 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:48.462411    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef-kube-api-access-vjgdw podName:bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef nodeName:}" failed. No retries permitted until 2026-07-16 22:33:48.962393082 +0000 UTC m=+2.972655740 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vjgdw" (UniqueName: "kubernetes.io/projected/bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef-kube-api-access-vjgdw") pod "network-check-target-7nqtn" (UID: "bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 22:33:48.465601 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.465549    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfdt4\" (UniqueName: \"kubernetes.io/projected/791625c5-dbf2-4e78-b2ad-08ef2a62a57f-kube-api-access-qfdt4\") pod \"tuned-bv2kv\" (UID: \"791625c5-dbf2-4e78-b2ad-08ef2a62a57f\") " pod="openshift-cluster-node-tuning-operator/tuned-bv2kv"
Jul 16 22:33:48.465802 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.465785    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mscr\" (UniqueName: \"kubernetes.io/projected/506f904a-3272-4586-8528-146884e728f9-kube-api-access-5mscr\") pod \"ovnkube-node-r9jm4\" (UID: \"506f904a-3272-4586-8528-146884e728f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.520844 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.520760    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-146.ec2.internal" event={"ID":"70417f5efada6062acead97c2af4fbd0","Type":"ContainerStarted","Data":"3b5ba3f2e421a98c4e467a6b2d6e1545f009e86d964a0f019f4cddf489f08688"}
Jul 16 22:33:48.522095 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.522043    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-146.ec2.internal" event={"ID":"f66c4d021c76588d605e3a72206506df","Type":"ContainerStarted","Data":"30244c673fbaa32c646cb7d3261550ec378d700cbe7501350344db41879257a4"}
Jul 16 22:33:48.554990 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.553965    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgm4j\" (UniqueName: \"kubernetes.io/projected/2ada79d5-4ef5-489e-88af-07f0108282da-kube-api-access-dgm4j\") pod \"iptables-alerter-k9pz9\" (UID: \"2ada79d5-4ef5-489e-88af-07f0108282da\") " pod="openshift-network-operator/iptables-alerter-k9pz9"
Jul 16 22:33:48.554990 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.554009    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9dxq\" (UniqueName: \"kubernetes.io/projected/268b78b0-10f6-44ce-a822-a124771df729-kube-api-access-t9dxq\") pod \"node-resolver-8bst9\" (UID: \"268b78b0-10f6-44ce-a822-a124771df729\") " pod="openshift-dns/node-resolver-8bst9"
Jul 16 22:33:48.554990 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.554036    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2da4fabd-5fae-4a74-a228-7262f2a3e2c8-cni-binary-copy\") pod \"multus-additional-cni-plugins-rtgnk\" (UID: \"2da4fabd-5fae-4a74-a228-7262f2a3e2c8\") " pod="openshift-multus/multus-additional-cni-plugins-rtgnk"
Jul 16 22:33:48.554990 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.554066    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/15f7c683-2d93-427c-abb8-c345d44fdb68-socket-dir\") pod \"aws-ebs-csi-driver-node-m44bd\" (UID: \"15f7c683-2d93-427c-abb8-c345d44fdb68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m44bd"
Jul 16 22:33:48.554990 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.554098    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3aefec9e-d46f-4718-89eb-43a524eb0fcf-cni-binary-copy\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.554990 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.554122    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3aefec9e-d46f-4718-89eb-43a524eb0fcf-host-run-k8s-cni-cncf-io\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.554990 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.554149    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3aefec9e-d46f-4718-89eb-43a524eb0fcf-host-run-netns\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.554990 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.554171    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3aefec9e-d46f-4718-89eb-43a524eb0fcf-host-var-lib-cni-multus\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.554990 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.554199    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fpn6\" (UniqueName: \"kubernetes.io/projected/1d026b73-92ba-465b-bc31-5843395ae3e4-kube-api-access-7fpn6\") pod \"node-ca-x4th8\" (UID: \"1d026b73-92ba-465b-bc31-5843395ae3e4\") " pod="openshift-image-registry/node-ca-x4th8"
Jul 16 22:33:48.554990 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.554224    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7144d772-5219-4c76-b0a6-950c4e94a235-agent-certs\") pod \"konnectivity-agent-466rj\" (UID: \"7144d772-5219-4c76-b0a6-950c4e94a235\") " pod="kube-system/konnectivity-agent-466rj"
Jul 16 22:33:48.554990 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.554265    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmvth\" (UniqueName: \"kubernetes.io/projected/15f7c683-2d93-427c-abb8-c345d44fdb68-kube-api-access-nmvth\") pod \"aws-ebs-csi-driver-node-m44bd\" (UID: \"15f7c683-2d93-427c-abb8-c345d44fdb68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m44bd"
Jul 16 22:33:48.554990 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.554294    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d026b73-92ba-465b-bc31-5843395ae3e4-host\") pod \"node-ca-x4th8\" (UID: \"1d026b73-92ba-465b-bc31-5843395ae3e4\") " pod="openshift-image-registry/node-ca-x4th8"
Jul 16 22:33:48.554990 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.554319    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2da4fabd-5fae-4a74-a228-7262f2a3e2c8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rtgnk\" (UID: \"2da4fabd-5fae-4a74-a228-7262f2a3e2c8\") " pod="openshift-multus/multus-additional-cni-plugins-rtgnk"
Jul 16 22:33:48.554990 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.554344    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/268b78b0-10f6-44ce-a822-a124771df729-tmp-dir\") pod \"node-resolver-8bst9\" (UID: \"268b78b0-10f6-44ce-a822-a124771df729\") " pod="openshift-dns/node-resolver-8bst9"
Jul 16 22:33:48.556274 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.554375    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3aefec9e-d46f-4718-89eb-43a524eb0fcf-host-var-lib-cni-bin\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.556274 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.554416    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/15f7c683-2d93-427c-abb8-c345d44fdb68-sys-fs\") pod \"aws-ebs-csi-driver-node-m44bd\" (UID: \"15f7c683-2d93-427c-abb8-c345d44fdb68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m44bd"
Jul 16 22:33:48.556274 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.554455    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/15f7c683-2d93-427c-abb8-c345d44fdb68-device-dir\") pod \"aws-ebs-csi-driver-node-m44bd\" (UID: \"15f7c683-2d93-427c-abb8-c345d44fdb68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m44bd"
Jul 16 22:33:48.556274 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.554481    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/268b78b0-10f6-44ce-a822-a124771df729-hosts-file\") pod \"node-resolver-8bst9\" (UID: \"268b78b0-10f6-44ce-a822-a124771df729\") " pod="openshift-dns/node-resolver-8bst9"
Jul 16 22:33:48.556274 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.554507    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2da4fabd-5fae-4a74-a228-7262f2a3e2c8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rtgnk\" (UID: \"2da4fabd-5fae-4a74-a228-7262f2a3e2c8\") " pod="openshift-multus/multus-additional-cni-plugins-rtgnk"
Jul 16 22:33:48.556274 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.554535    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3aefec9e-d46f-4718-89eb-43a524eb0fcf-cnibin\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.556274 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.554559    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2da4fabd-5fae-4a74-a228-7262f2a3e2c8-system-cni-dir\") pod \"multus-additional-cni-plugins-rtgnk\" (UID: \"2da4fabd-5fae-4a74-a228-7262f2a3e2c8\") " pod="openshift-multus/multus-additional-cni-plugins-rtgnk"
Jul 16 22:33:48.556274 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.554587    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/15f7c683-2d93-427c-abb8-c345d44fdb68-etc-selinux\") pod \"aws-ebs-csi-driver-node-m44bd\" (UID: \"15f7c683-2d93-427c-abb8-c345d44fdb68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m44bd"
Jul 16 22:33:48.556274 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.554616    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3aefec9e-d46f-4718-89eb-43a524eb0fcf-etc-kubernetes\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.556274 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.554640    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxn7q\" (UniqueName: \"kubernetes.io/projected/3aefec9e-d46f-4718-89eb-43a524eb0fcf-kube-api-access-rxn7q\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.556274 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.554666    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2da4fabd-5fae-4a74-a228-7262f2a3e2c8-os-release\") pod \"multus-additional-cni-plugins-rtgnk\" (UID: \"2da4fabd-5fae-4a74-a228-7262f2a3e2c8\") " pod="openshift-multus/multus-additional-cni-plugins-rtgnk"
Jul 16 22:33:48.556274 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.554694    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3aefec9e-d46f-4718-89eb-43a524eb0fcf-hostroot\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.556274 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.554719    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3aefec9e-d46f-4718-89eb-43a524eb0fcf-multus-daemon-config\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.556274 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.554744    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68qgq\" (UniqueName: \"kubernetes.io/projected/2da4fabd-5fae-4a74-a228-7262f2a3e2c8-kube-api-access-68qgq\") pod \"multus-additional-cni-plugins-rtgnk\" (UID: \"2da4fabd-5fae-4a74-a228-7262f2a3e2c8\") " pod="openshift-multus/multus-additional-cni-plugins-rtgnk"
Jul 16 22:33:48.556941 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.554812    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3aefec9e-d46f-4718-89eb-43a524eb0fcf-system-cni-dir\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.556941 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.554824    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3aefec9e-d46f-4718-89eb-43a524eb0fcf-host-run-netns\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.556941 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.554843    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/15f7c683-2d93-427c-abb8-c345d44fdb68-device-dir\") pod \"aws-ebs-csi-driver-node-m44bd\" (UID: \"15f7c683-2d93-427c-abb8-c345d44fdb68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m44bd"
Jul 16 22:33:48.556941 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.554849    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2ada79d5-4ef5-489e-88af-07f0108282da-host-slash\") pod \"iptables-alerter-k9pz9\" (UID: \"2ada79d5-4ef5-489e-88af-07f0108282da\") " pod="openshift-network-operator/iptables-alerter-k9pz9"
Jul 16 22:33:48.556941 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.554886    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2ada79d5-4ef5-489e-88af-07f0108282da-host-slash\") pod \"iptables-alerter-k9pz9\" (UID: \"2ada79d5-4ef5-489e-88af-07f0108282da\") " pod="openshift-network-operator/iptables-alerter-k9pz9"
Jul 16 22:33:48.556941 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.554888    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/15f7c683-2d93-427c-abb8-c345d44fdb68-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m44bd\" (UID: \"15f7c683-2d93-427c-abb8-c345d44fdb68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m44bd"
Jul 16 22:33:48.556941 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.554920    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3aefec9e-d46f-4718-89eb-43a524eb0fcf-multus-cni-dir\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.556941 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.554927    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/15f7c683-2d93-427c-abb8-c345d44fdb68-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m44bd\" (UID: \"15f7c683-2d93-427c-abb8-c345d44fdb68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m44bd"
Jul 16 22:33:48.556941 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.554947    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3aefec9e-d46f-4718-89eb-43a524eb0fcf-os-release\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.556941 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.554970    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1d026b73-92ba-465b-bc31-5843395ae3e4-serviceca\") pod \"node-ca-x4th8\" (UID: \"1d026b73-92ba-465b-bc31-5843395ae3e4\") " pod="openshift-image-registry/node-ca-x4th8"
Jul 16 22:33:48.556941 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.554996    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7144d772-5219-4c76-b0a6-950c4e94a235-konnectivity-ca\") pod \"konnectivity-agent-466rj\" (UID: \"7144d772-5219-4c76-b0a6-950c4e94a235\") " pod="kube-system/konnectivity-agent-466rj"
Jul 16 22:33:48.556941 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.555024    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3aefec9e-d46f-4718-89eb-43a524eb0fcf-multus-socket-dir-parent\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.556941 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.555048    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3aefec9e-d46f-4718-89eb-43a524eb0fcf-multus-conf-dir\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.556941 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.555073    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfcj6\" (UniqueName: \"kubernetes.io/projected/9b332999-321a-4e3f-99bf-19517a268e28-kube-api-access-qfcj6\") pod \"network-metrics-daemon-6xfmc\" (UID: \"9b332999-321a-4e3f-99bf-19517a268e28\") " pod="openshift-multus/network-metrics-daemon-6xfmc"
Jul 16 22:33:48.556941 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.555131    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3aefec9e-d46f-4718-89eb-43a524eb0fcf-host-run-multus-certs\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.557679 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.555153    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3aefec9e-d46f-4718-89eb-43a524eb0fcf-host-run-k8s-cni-cncf-io\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.557679 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.555163    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2da4fabd-5fae-4a74-a228-7262f2a3e2c8-cnibin\") pod \"multus-additional-cni-plugins-rtgnk\" (UID: \"2da4fabd-5fae-4a74-a228-7262f2a3e2c8\") " pod="openshift-multus/multus-additional-cni-plugins-rtgnk"
Jul 16 22:33:48.557679 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.555190    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2da4fabd-5fae-4a74-a228-7262f2a3e2c8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rtgnk\" (UID: \"2da4fabd-5fae-4a74-a228-7262f2a3e2c8\") " pod="openshift-multus/multus-additional-cni-plugins-rtgnk"
Jul 16 22:33:48.557679 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.555207    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/15f7c683-2d93-427c-abb8-c345d44fdb68-sys-fs\") pod \"aws-ebs-csi-driver-node-m44bd\" (UID: \"15f7c683-2d93-427c-abb8-c345d44fdb68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m44bd"
Jul 16 22:33:48.557679 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.555216    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2ada79d5-4ef5-489e-88af-07f0108282da-iptables-alerter-script\") pod \"iptables-alerter-k9pz9\" (UID: \"2ada79d5-4ef5-489e-88af-07f0108282da\") " pod="openshift-network-operator/iptables-alerter-k9pz9"
Jul 16 22:33:48.557679 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.555268    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d026b73-92ba-465b-bc31-5843395ae3e4-host\") pod \"node-ca-x4th8\" (UID: \"1d026b73-92ba-465b-bc31-5843395ae3e4\") " pod="openshift-image-registry/node-ca-x4th8"
Jul 16 22:33:48.557679 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.554891    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3aefec9e-d46f-4718-89eb-43a524eb0fcf-host-var-lib-cni-multus\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.557679 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.555328    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/15f7c683-2d93-427c-abb8-c345d44fdb68-registration-dir\") pod \"aws-ebs-csi-driver-node-m44bd\" (UID: \"15f7c683-2d93-427c-abb8-c345d44fdb68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m44bd"
Jul 16 22:33:48.557679 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.555353    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3aefec9e-d46f-4718-89eb-43a524eb0fcf-host-var-lib-kubelet\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.557679 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.555393    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b332999-321a-4e3f-99bf-19517a268e28-metrics-certs\") pod \"network-metrics-daemon-6xfmc\" (UID: \"9b332999-321a-4e3f-99bf-19517a268e28\") " pod="openshift-multus/network-metrics-daemon-6xfmc"
Jul 16 22:33:48.557679 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.555430    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/15f7c683-2d93-427c-abb8-c345d44fdb68-registration-dir\") pod \"aws-ebs-csi-driver-node-m44bd\" (UID: \"15f7c683-2d93-427c-abb8-c345d44fdb68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m44bd"
Jul 16 22:33:48.557679 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:48.555497    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 22:33:48.557679 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:48.555545    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b332999-321a-4e3f-99bf-19517a268e28-metrics-certs podName:9b332999-321a-4e3f-99bf-19517a268e28 nodeName:}" failed. No retries permitted until 2026-07-16 22:33:49.055528986 +0000 UTC m=+3.065791642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9b332999-321a-4e3f-99bf-19517a268e28-metrics-certs") pod "network-metrics-daemon-6xfmc" (UID: "9b332999-321a-4e3f-99bf-19517a268e28") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 22:33:48.557679 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.555588    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2da4fabd-5fae-4a74-a228-7262f2a3e2c8-cni-binary-copy\") pod \"multus-additional-cni-plugins-rtgnk\" (UID: \"2da4fabd-5fae-4a74-a228-7262f2a3e2c8\") " pod="openshift-multus/multus-additional-cni-plugins-rtgnk"
Jul 16 22:33:48.558380 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.555637    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3aefec9e-d46f-4718-89eb-43a524eb0fcf-host-var-lib-kubelet\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.558380 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.555688    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/15f7c683-2d93-427c-abb8-c345d44fdb68-socket-dir\") pod \"aws-ebs-csi-driver-node-m44bd\" (UID: \"15f7c683-2d93-427c-abb8-c345d44fdb68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m44bd"
Jul 16 22:33:48.558380 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.555782    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3aefec9e-d46f-4718-89eb-43a524eb0fcf-etc-kubernetes\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.558380 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.555894    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/268b78b0-10f6-44ce-a822-a124771df729-hosts-file\") pod \"node-resolver-8bst9\" (UID: \"268b78b0-10f6-44ce-a822-a124771df729\") " pod="openshift-dns/node-resolver-8bst9"
Jul 16 22:33:48.558380 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.555941    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2da4fabd-5fae-4a74-a228-7262f2a3e2c8-system-cni-dir\") pod \"multus-additional-cni-plugins-rtgnk\" (UID: \"2da4fabd-5fae-4a74-a228-7262f2a3e2c8\") " pod="openshift-multus/multus-additional-cni-plugins-rtgnk"
Jul 16 22:33:48.558380 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.555939    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3aefec9e-d46f-4718-89eb-43a524eb0fcf-cnibin\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.558380 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.555993    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2ada79d5-4ef5-489e-88af-07f0108282da-iptables-alerter-script\") pod \"iptables-alerter-k9pz9\" (UID: \"2ada79d5-4ef5-489e-88af-07f0108282da\") " pod="openshift-network-operator/iptables-alerter-k9pz9"
Jul 16 22:33:48.558380 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.556057    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/15f7c683-2d93-427c-abb8-c345d44fdb68-etc-selinux\") pod \"aws-ebs-csi-driver-node-m44bd\" (UID: \"15f7c683-2d93-427c-abb8-c345d44fdb68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m44bd"
Jul 16 22:33:48.558380 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.556060    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3aefec9e-d46f-4718-89eb-43a524eb0fcf-hostroot\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.558380 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.556109    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3aefec9e-d46f-4718-89eb-43a524eb0fcf-system-cni-dir\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.558380 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.556209    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/268b78b0-10f6-44ce-a822-a124771df729-tmp-dir\") pod \"node-resolver-8bst9\" (UID: \"268b78b0-10f6-44ce-a822-a124771df729\") " pod="openshift-dns/node-resolver-8bst9"
Jul 16 22:33:48.558380 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.556277    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3aefec9e-d46f-4718-89eb-43a524eb0fcf-cni-binary-copy\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.558380 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.556297    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2da4fabd-5fae-4a74-a228-7262f2a3e2c8-os-release\") pod \"multus-additional-cni-plugins-rtgnk\" (UID: \"2da4fabd-5fae-4a74-a228-7262f2a3e2c8\") " pod="openshift-multus/multus-additional-cni-plugins-rtgnk"
Jul 16 22:33:48.558380 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.556281    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2da4fabd-5fae-4a74-a228-7262f2a3e2c8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rtgnk\" (UID: \"2da4fabd-5fae-4a74-a228-7262f2a3e2c8\") " pod="openshift-multus/multus-additional-cni-plugins-rtgnk"
Jul 16 22:33:48.558380 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.556532    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2da4fabd-5fae-4a74-a228-7262f2a3e2c8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rtgnk\" (UID: \"2da4fabd-5fae-4a74-a228-7262f2a3e2c8\") " pod="openshift-multus/multus-additional-cni-plugins-rtgnk"
Jul 16 22:33:48.559062 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.556591    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3aefec9e-d46f-4718-89eb-43a524eb0fcf-multus-socket-dir-parent\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.559062 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.556613    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3aefec9e-d46f-4718-89eb-43a524eb0fcf-multus-cni-dir\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.559062 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.556696    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3aefec9e-d46f-4718-89eb-43a524eb0fcf-os-release\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.559062 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.556734    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3aefec9e-d46f-4718-89eb-43a524eb0fcf-multus-conf-dir\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.559062 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.556792    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3aefec9e-d46f-4718-89eb-43a524eb0fcf-host-run-multus-certs\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.559062 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.556803    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7144d772-5219-4c76-b0a6-950c4e94a235-konnectivity-ca\") pod \"konnectivity-agent-466rj\" (UID: \"7144d772-5219-4c76-b0a6-950c4e94a235\") " pod="kube-system/konnectivity-agent-466rj"
Jul 16 22:33:48.559062 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.556912    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2da4fabd-5fae-4a74-a228-7262f2a3e2c8-cnibin\") pod \"multus-additional-cni-plugins-rtgnk\" (UID: \"2da4fabd-5fae-4a74-a228-7262f2a3e2c8\") " pod="openshift-multus/multus-additional-cni-plugins-rtgnk"
Jul 16 22:33:48.559062 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.557198    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3aefec9e-d46f-4718-89eb-43a524eb0fcf-multus-daemon-config\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.559062 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.557372    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2da4fabd-5fae-4a74-a228-7262f2a3e2c8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rtgnk\" (UID: \"2da4fabd-5fae-4a74-a228-7262f2a3e2c8\") " pod="openshift-multus/multus-additional-cni-plugins-rtgnk"
Jul 16 22:33:48.559062 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.557430    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3aefec9e-d46f-4718-89eb-43a524eb0fcf-host-var-lib-cni-bin\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.559062 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.557604    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1d026b73-92ba-465b-bc31-5843395ae3e4-serviceca\") pod \"node-ca-x4th8\" (UID: \"1d026b73-92ba-465b-bc31-5843395ae3e4\") " pod="openshift-image-registry/node-ca-x4th8"
Jul 16 22:33:48.559062 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.558513    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7144d772-5219-4c76-b0a6-950c4e94a235-agent-certs\") pod \"konnectivity-agent-466rj\" (UID: \"7144d772-5219-4c76-b0a6-950c4e94a235\") " pod="kube-system/konnectivity-agent-466rj"
Jul 16 22:33:48.565124 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.565099    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9dxq\" (UniqueName: \"kubernetes.io/projected/268b78b0-10f6-44ce-a822-a124771df729-kube-api-access-t9dxq\") pod \"node-resolver-8bst9\" (UID: \"268b78b0-10f6-44ce-a822-a124771df729\") " pod="openshift-dns/node-resolver-8bst9"
Jul 16 22:33:48.565942 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.565888    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmvth\" (UniqueName: \"kubernetes.io/projected/15f7c683-2d93-427c-abb8-c345d44fdb68-kube-api-access-nmvth\") pod \"aws-ebs-csi-driver-node-m44bd\" (UID: \"15f7c683-2d93-427c-abb8-c345d44fdb68\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m44bd"
Jul 16 22:33:48.566564 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.566465    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxn7q\" (UniqueName: \"kubernetes.io/projected/3aefec9e-d46f-4718-89eb-43a524eb0fcf-kube-api-access-rxn7q\") pod \"multus-85dmh\" (UID: \"3aefec9e-d46f-4718-89eb-43a524eb0fcf\") " pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.566564 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.566515    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fpn6\" (UniqueName: \"kubernetes.io/projected/1d026b73-92ba-465b-bc31-5843395ae3e4-kube-api-access-7fpn6\") pod \"node-ca-x4th8\" (UID: \"1d026b73-92ba-465b-bc31-5843395ae3e4\") " pod="openshift-image-registry/node-ca-x4th8"
Jul 16 22:33:48.568013 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.567957    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfcj6\" (UniqueName: \"kubernetes.io/projected/9b332999-321a-4e3f-99bf-19517a268e28-kube-api-access-qfcj6\") pod \"network-metrics-daemon-6xfmc\" (UID: \"9b332999-321a-4e3f-99bf-19517a268e28\") " pod="openshift-multus/network-metrics-daemon-6xfmc"
Jul 16 22:33:48.568684 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.568661    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68qgq\" (UniqueName: \"kubernetes.io/projected/2da4fabd-5fae-4a74-a228-7262f2a3e2c8-kube-api-access-68qgq\") pod \"multus-additional-cni-plugins-rtgnk\" (UID: \"2da4fabd-5fae-4a74-a228-7262f2a3e2c8\") " pod="openshift-multus/multus-additional-cni-plugins-rtgnk"
Jul 16 22:33:48.569327 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.569309    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgm4j\" (UniqueName: \"kubernetes.io/projected/2ada79d5-4ef5-489e-88af-07f0108282da-kube-api-access-dgm4j\") pod \"iptables-alerter-k9pz9\" (UID: \"2ada79d5-4ef5-489e-88af-07f0108282da\") " pod="openshift-network-operator/iptables-alerter-k9pz9"
Jul 16 22:33:48.652221 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.652197    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bv2kv"
Jul 16 22:33:48.661472 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.661452    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:33:48.663091 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:48.663062    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod791625c5_dbf2_4e78_b2ad_08ef2a62a57f.slice/crio-1e197259a233fa48301b4e912a2faa2621521a853649544882d7d52f0c28d87f WatchSource:0}: Error finding container 1e197259a233fa48301b4e912a2faa2621521a853649544882d7d52f0c28d87f: Status 404 returned error can't find the container with id 1e197259a233fa48301b4e912a2faa2621521a853649544882d7d52f0c28d87f
Jul 16 22:33:48.671572 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:48.671545    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod506f904a_3272_4586_8528_146884e728f9.slice/crio-5d20cd4a5c56646ad9ce015d484db9c1273ed0d18ae13fff4cf1909e32120e2e WatchSource:0}: Error finding container 5d20cd4a5c56646ad9ce015d484db9c1273ed0d18ae13fff4cf1909e32120e2e: Status 404 returned error can't find the container with id 5d20cd4a5c56646ad9ce015d484db9c1273ed0d18ae13fff4cf1909e32120e2e
Jul 16 22:33:48.672476 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.672456    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rtgnk"
Jul 16 22:33:48.679941 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.679856    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-466rj"
Jul 16 22:33:48.681860 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:48.681836    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2da4fabd_5fae_4a74_a228_7262f2a3e2c8.slice/crio-b9d945d871b26731ed74ecf0de6b92c95eba1e564d4d2cdb52c9d69d8983c7ed WatchSource:0}: Error finding container b9d945d871b26731ed74ecf0de6b92c95eba1e564d4d2cdb52c9d69d8983c7ed: Status 404 returned error can't find the container with id b9d945d871b26731ed74ecf0de6b92c95eba1e564d4d2cdb52c9d69d8983c7ed
Jul 16 22:33:48.688503 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.688483    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m44bd"
Jul 16 22:33:48.689212 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:48.689191    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7144d772_5219_4c76_b0a6_950c4e94a235.slice/crio-7c993a8d93151bf5be4a2a2d549ae3e4d9d82b3c2217b16cdb8ce08379ca0dc4 WatchSource:0}: Error finding container 7c993a8d93151bf5be4a2a2d549ae3e4d9d82b3c2217b16cdb8ce08379ca0dc4: Status 404 returned error can't find the container with id 7c993a8d93151bf5be4a2a2d549ae3e4d9d82b3c2217b16cdb8ce08379ca0dc4
Jul 16 22:33:48.696886 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:48.696866    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15f7c683_2d93_427c_abb8_c345d44fdb68.slice/crio-57764576e9589d7d89107f891d03e65629d88111b77df97d504c4049915f730e WatchSource:0}: Error finding container 57764576e9589d7d89107f891d03e65629d88111b77df97d504c4049915f730e: Status 404 returned error can't find the container with id 57764576e9589d7d89107f891d03e65629d88111b77df97d504c4049915f730e
Jul 16 22:33:48.696970 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.696914    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-x4th8"
Jul 16 22:33:48.705141 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.705119    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8bst9"
Jul 16 22:33:48.712900 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.712880    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-85dmh"
Jul 16 22:33:48.719439 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.719367    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k9pz9"
Jul 16 22:33:48.958506 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:48.958424    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c8d410fb-20ac-4175-a4b4-38d5b44636d6-original-pull-secret\") pod \"global-pull-secret-syncer-kxpfx\" (UID: \"c8d410fb-20ac-4175-a4b4-38d5b44636d6\") " pod="kube-system/global-pull-secret-syncer-kxpfx"
Jul 16 22:33:48.958653 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:48.958547    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 22:33:48.958653 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:48.958609    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8d410fb-20ac-4175-a4b4-38d5b44636d6-original-pull-secret podName:c8d410fb-20ac-4175-a4b4-38d5b44636d6 nodeName:}" failed. No retries permitted until 2026-07-16 22:33:49.958592592 +0000 UTC m=+3.968855253 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c8d410fb-20ac-4175-a4b4-38d5b44636d6-original-pull-secret") pod "global-pull-secret-syncer-kxpfx" (UID: "c8d410fb-20ac-4175-a4b4-38d5b44636d6") : object "kube-system"/"original-pull-secret" not registered
Jul 16 22:33:49.059377 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:49.059348    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjgdw\" (UniqueName: \"kubernetes.io/projected/bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef-kube-api-access-vjgdw\") pod \"network-check-target-7nqtn\" (UID: \"bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef\") " pod="openshift-network-diagnostics/network-check-target-7nqtn"
Jul 16 22:33:49.059525 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:49.059399    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b332999-321a-4e3f-99bf-19517a268e28-metrics-certs\") pod \"network-metrics-daemon-6xfmc\" (UID: \"9b332999-321a-4e3f-99bf-19517a268e28\") " pod="openshift-multus/network-metrics-daemon-6xfmc"
Jul 16 22:33:49.059525 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:49.059495    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 22:33:49.059525 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:49.059508    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 22:33:49.059687 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:49.059530    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 22:33:49.059687 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:49.059545    2573 projected.go:196] Error preparing data for projected volume kube-api-access-vjgdw for pod openshift-network-diagnostics/network-check-target-7nqtn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 22:33:49.059687 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:49.059558    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b332999-321a-4e3f-99bf-19517a268e28-metrics-certs podName:9b332999-321a-4e3f-99bf-19517a268e28 nodeName:}" failed. No retries permitted until 2026-07-16 22:33:50.059542044 +0000 UTC m=+4.069804692 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9b332999-321a-4e3f-99bf-19517a268e28-metrics-certs") pod "network-metrics-daemon-6xfmc" (UID: "9b332999-321a-4e3f-99bf-19517a268e28") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 22:33:49.059687 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:49.059600    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef-kube-api-access-vjgdw podName:bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef nodeName:}" failed. No retries permitted until 2026-07-16 22:33:50.059579784 +0000 UTC m=+4.069842427 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vjgdw" (UniqueName: "kubernetes.io/projected/bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef-kube-api-access-vjgdw") pod "network-check-target-7nqtn" (UID: "bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 22:33:49.393385 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:49.393299    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 22:28:48 +0000 UTC" deadline="2028-03-01 22:25:49.127220639 +0000 UTC"
Jul 16 22:33:49.393385 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:49.393337    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14255h51m59.733888772s"
Jul 16 22:33:49.525701 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:49.525624    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bv2kv" event={"ID":"791625c5-dbf2-4e78-b2ad-08ef2a62a57f","Type":"ContainerStarted","Data":"1e197259a233fa48301b4e912a2faa2621521a853649544882d7d52f0c28d87f"}
Jul 16 22:33:49.527731 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:49.527608    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m44bd" event={"ID":"15f7c683-2d93-427c-abb8-c345d44fdb68","Type":"ContainerStarted","Data":"57764576e9589d7d89107f891d03e65629d88111b77df97d504c4049915f730e"}
Jul 16 22:33:49.529565 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:49.529468    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-466rj" event={"ID":"7144d772-5219-4c76-b0a6-950c4e94a235","Type":"ContainerStarted","Data":"7c993a8d93151bf5be4a2a2d549ae3e4d9d82b3c2217b16cdb8ce08379ca0dc4"}
Jul 16 22:33:49.532056 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:49.532030    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rtgnk" event={"ID":"2da4fabd-5fae-4a74-a228-7262f2a3e2c8","Type":"ContainerStarted","Data":"b9d945d871b26731ed74ecf0de6b92c95eba1e564d4d2cdb52c9d69d8983c7ed"}
Jul 16 22:33:49.534296 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:49.534188    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4" event={"ID":"506f904a-3272-4586-8528-146884e728f9","Type":"ContainerStarted","Data":"5d20cd4a5c56646ad9ce015d484db9c1273ed0d18ae13fff4cf1909e32120e2e"}
Jul 16 22:33:49.668000 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:49.667658    2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 22:33:49.959906 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:49.959843    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aefec9e_d46f_4718_89eb_43a524eb0fcf.slice/crio-10c518090d59008c9bb0b4d1c907491abd6724e11a1b453ce15e0544ae1e3ed8 WatchSource:0}: Error finding container 10c518090d59008c9bb0b4d1c907491abd6724e11a1b453ce15e0544ae1e3ed8: Status 404 returned error can't find the container with id 10c518090d59008c9bb0b4d1c907491abd6724e11a1b453ce15e0544ae1e3ed8
Jul 16 22:33:49.966668 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:49.966455    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c8d410fb-20ac-4175-a4b4-38d5b44636d6-original-pull-secret\") pod \"global-pull-secret-syncer-kxpfx\" (UID: \"c8d410fb-20ac-4175-a4b4-38d5b44636d6\") " pod="kube-system/global-pull-secret-syncer-kxpfx"
Jul 16 22:33:49.966668 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:49.966591    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 22:33:49.966668 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:49.966642    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8d410fb-20ac-4175-a4b4-38d5b44636d6-original-pull-secret podName:c8d410fb-20ac-4175-a4b4-38d5b44636d6 nodeName:}" failed. No retries permitted until 2026-07-16 22:33:51.966624544 +0000 UTC m=+5.976887204 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c8d410fb-20ac-4175-a4b4-38d5b44636d6-original-pull-secret") pod "global-pull-secret-syncer-kxpfx" (UID: "c8d410fb-20ac-4175-a4b4-38d5b44636d6") : object "kube-system"/"original-pull-secret" not registered
Jul 16 22:33:49.970098 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:33:49.969962    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod268b78b0_10f6_44ce_a822_a124771df729.slice/crio-0d961a451452a2e580a91787d0a003bc3799a3b22186ffb8e429bef789f87021 WatchSource:0}: Error finding container 0d961a451452a2e580a91787d0a003bc3799a3b22186ffb8e429bef789f87021: Status 404 returned error can't find the container with id 0d961a451452a2e580a91787d0a003bc3799a3b22186ffb8e429bef789f87021
Jul 16 22:33:50.068004 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:50.067622    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b332999-321a-4e3f-99bf-19517a268e28-metrics-certs\") pod \"network-metrics-daemon-6xfmc\" (UID: \"9b332999-321a-4e3f-99bf-19517a268e28\") " pod="openshift-multus/network-metrics-daemon-6xfmc"
Jul 16 22:33:50.068004 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:50.067708    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjgdw\" (UniqueName: \"kubernetes.io/projected/bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef-kube-api-access-vjgdw\") pod \"network-check-target-7nqtn\" (UID: \"bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef\") " pod="openshift-network-diagnostics/network-check-target-7nqtn"
Jul 16 22:33:50.068004 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:50.067862    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 22:33:50.068004 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:50.067879    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 22:33:50.068004 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:50.067891    2573 projected.go:196] Error preparing data for projected volume kube-api-access-vjgdw for pod openshift-network-diagnostics/network-check-target-7nqtn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 22:33:50.068004 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:50.067942    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 22:33:50.074741 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:50.067958    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef-kube-api-access-vjgdw podName:bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef nodeName:}" failed. No retries permitted until 2026-07-16 22:33:52.067932209 +0000 UTC m=+6.078194864 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vjgdw" (UniqueName: "kubernetes.io/projected/bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef-kube-api-access-vjgdw") pod "network-check-target-7nqtn" (UID: "bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 22:33:50.074741 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:50.074343    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b332999-321a-4e3f-99bf-19517a268e28-metrics-certs podName:9b332999-321a-4e3f-99bf-19517a268e28 nodeName:}" failed. No retries permitted until 2026-07-16 22:33:52.074311249 +0000 UTC m=+6.084573906 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9b332999-321a-4e3f-99bf-19517a268e28-metrics-certs") pod "network-metrics-daemon-6xfmc" (UID: "9b332999-321a-4e3f-99bf-19517a268e28") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 22:33:50.393862 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:50.393769    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 22:28:48 +0000 UTC" deadline="2027-12-19 08:09:29.985491515 +0000 UTC"
Jul 16 22:33:50.393862 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:50.393831    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12489h35m39.591665226s"
Jul 16 22:33:50.513338 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:50.512990    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kxpfx"
Jul 16 22:33:50.513338 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:50.513093    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kxpfx" podUID="c8d410fb-20ac-4175-a4b4-38d5b44636d6"
Jul 16 22:33:50.513338 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:50.513333    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nqtn"
Jul 16 22:33:50.513610 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:50.513423    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7nqtn" podUID="bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef"
Jul 16 22:33:50.513610 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:50.513524    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xfmc"
Jul 16 22:33:50.513717 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:50.513609    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xfmc" podUID="9b332999-321a-4e3f-99bf-19517a268e28"
Jul 16 22:33:50.540705 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:50.540675    2573 generic.go:356] "Generic (PLEG): container finished" podID="70417f5efada6062acead97c2af4fbd0" containerID="3fe3462a5204d39d1bd0de2c35031b7e7b8620f6f44b3f511ca89fcd614acf48" exitCode=0
Jul 16 22:33:50.540854 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:50.540758    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-146.ec2.internal" event={"ID":"70417f5efada6062acead97c2af4fbd0","Type":"ContainerDied","Data":"3fe3462a5204d39d1bd0de2c35031b7e7b8620f6f44b3f511ca89fcd614acf48"}
Jul 16 22:33:50.543629 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:50.543600    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8bst9" event={"ID":"268b78b0-10f6-44ce-a822-a124771df729","Type":"ContainerStarted","Data":"0d961a451452a2e580a91787d0a003bc3799a3b22186ffb8e429bef789f87021"}
Jul 16 22:33:50.546149 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:50.546124    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-x4th8" event={"ID":"1d026b73-92ba-465b-bc31-5843395ae3e4","Type":"ContainerStarted","Data":"442cd3b84aee0ffc3181a8d0f064d4faea024d894eda1996b21afb19d6353832"}
Jul 16 22:33:50.547901 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:50.547877    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k9pz9" event={"ID":"2ada79d5-4ef5-489e-88af-07f0108282da","Type":"ContainerStarted","Data":"7bf7d589a5e4dda64e761ef6a1da507c2e77161746d4194efa2ea06941125b2e"}
Jul 16 22:33:50.551372 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:50.550957    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-146.ec2.internal" event={"ID":"f66c4d021c76588d605e3a72206506df","Type":"ContainerStarted","Data":"bc05874bfd325da536d9b874fed7cf309784e239e5d5bcadb3768f3f1c703fb3"}
Jul 16 22:33:50.553748 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:50.553698    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-85dmh" event={"ID":"3aefec9e-d46f-4718-89eb-43a524eb0fcf","Type":"ContainerStarted","Data":"10c518090d59008c9bb0b4d1c907491abd6724e11a1b453ce15e0544ae1e3ed8"}
Jul 16 22:33:51.983276 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:51.983223    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c8d410fb-20ac-4175-a4b4-38d5b44636d6-original-pull-secret\") pod \"global-pull-secret-syncer-kxpfx\" (UID: \"c8d410fb-20ac-4175-a4b4-38d5b44636d6\") " pod="kube-system/global-pull-secret-syncer-kxpfx"
Jul 16 22:33:51.983746 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:51.983398    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 22:33:51.983746 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:51.983460    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8d410fb-20ac-4175-a4b4-38d5b44636d6-original-pull-secret podName:c8d410fb-20ac-4175-a4b4-38d5b44636d6 nodeName:}" failed. No retries permitted until 2026-07-16 22:33:55.983441302 +0000 UTC m=+9.993703966 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c8d410fb-20ac-4175-a4b4-38d5b44636d6-original-pull-secret") pod "global-pull-secret-syncer-kxpfx" (UID: "c8d410fb-20ac-4175-a4b4-38d5b44636d6") : object "kube-system"/"original-pull-secret" not registered
Jul 16 22:33:52.083717 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:52.083682    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjgdw\" (UniqueName: \"kubernetes.io/projected/bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef-kube-api-access-vjgdw\") pod \"network-check-target-7nqtn\" (UID: \"bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef\") " pod="openshift-network-diagnostics/network-check-target-7nqtn"
Jul 16 22:33:52.083899 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:52.083743    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b332999-321a-4e3f-99bf-19517a268e28-metrics-certs\") pod \"network-metrics-daemon-6xfmc\" (UID: \"9b332999-321a-4e3f-99bf-19517a268e28\") " pod="openshift-multus/network-metrics-daemon-6xfmc"
Jul 16 22:33:52.083899 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:52.083877    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 22:33:52.084008 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:52.083939    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b332999-321a-4e3f-99bf-19517a268e28-metrics-certs podName:9b332999-321a-4e3f-99bf-19517a268e28 nodeName:}" failed. No retries permitted until 2026-07-16 22:33:56.083913754 +0000 UTC m=+10.094176411 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9b332999-321a-4e3f-99bf-19517a268e28-metrics-certs") pod "network-metrics-daemon-6xfmc" (UID: "9b332999-321a-4e3f-99bf-19517a268e28") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 22:33:52.084383 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:52.084356    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 22:33:52.084471 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:52.084386    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 22:33:52.084471 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:52.084400    2573 projected.go:196] Error preparing data for projected volume kube-api-access-vjgdw for pod openshift-network-diagnostics/network-check-target-7nqtn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 22:33:52.084471 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:52.084446    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef-kube-api-access-vjgdw podName:bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef nodeName:}" failed. No retries permitted until 2026-07-16 22:33:56.08443067 +0000 UTC m=+10.094693326 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vjgdw" (UniqueName: "kubernetes.io/projected/bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef-kube-api-access-vjgdw") pod "network-check-target-7nqtn" (UID: "bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 22:33:52.515439 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:52.515408    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nqtn"
Jul 16 22:33:52.515624 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:52.515534    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7nqtn" podUID="bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef"
Jul 16 22:33:52.515904 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:52.515885    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xfmc"
Jul 16 22:33:52.516160 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:52.515992    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xfmc" podUID="9b332999-321a-4e3f-99bf-19517a268e28"
Jul 16 22:33:52.516160 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:52.516037    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kxpfx"
Jul 16 22:33:52.516160 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:52.516105    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kxpfx" podUID="c8d410fb-20ac-4175-a4b4-38d5b44636d6"
Jul 16 22:33:54.513875 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:54.513361    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nqtn"
Jul 16 22:33:54.513875 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:54.513504    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7nqtn" podUID="bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef"
Jul 16 22:33:54.513875 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:54.513528    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xfmc"
Jul 16 22:33:54.513875 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:54.513681    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xfmc" podUID="9b332999-321a-4e3f-99bf-19517a268e28"
Jul 16 22:33:54.513875 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:54.513731    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kxpfx"
Jul 16 22:33:54.513875 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:54.513823    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kxpfx" podUID="c8d410fb-20ac-4175-a4b4-38d5b44636d6"
Jul 16 22:33:56.017004 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:56.016568    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c8d410fb-20ac-4175-a4b4-38d5b44636d6-original-pull-secret\") pod \"global-pull-secret-syncer-kxpfx\" (UID: \"c8d410fb-20ac-4175-a4b4-38d5b44636d6\") " pod="kube-system/global-pull-secret-syncer-kxpfx"
Jul 16 22:33:56.017004 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:56.016756    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 22:33:56.017004 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:56.016818    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8d410fb-20ac-4175-a4b4-38d5b44636d6-original-pull-secret podName:c8d410fb-20ac-4175-a4b4-38d5b44636d6 nodeName:}" failed. No retries permitted until 2026-07-16 22:34:04.016800048 +0000 UTC m=+18.027062704 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c8d410fb-20ac-4175-a4b4-38d5b44636d6-original-pull-secret") pod "global-pull-secret-syncer-kxpfx" (UID: "c8d410fb-20ac-4175-a4b4-38d5b44636d6") : object "kube-system"/"original-pull-secret" not registered
Jul 16 22:33:56.117032 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:56.116985    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjgdw\" (UniqueName: \"kubernetes.io/projected/bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef-kube-api-access-vjgdw\") pod \"network-check-target-7nqtn\" (UID: \"bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef\") " pod="openshift-network-diagnostics/network-check-target-7nqtn"
Jul 16 22:33:56.117200 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:56.117061    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b332999-321a-4e3f-99bf-19517a268e28-metrics-certs\") pod \"network-metrics-daemon-6xfmc\" (UID: \"9b332999-321a-4e3f-99bf-19517a268e28\") " pod="openshift-multus/network-metrics-daemon-6xfmc"
Jul 16 22:33:56.117278 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:56.117200    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 22:33:56.117391 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:56.117282    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b332999-321a-4e3f-99bf-19517a268e28-metrics-certs podName:9b332999-321a-4e3f-99bf-19517a268e28 nodeName:}" failed. No retries permitted until 2026-07-16 22:34:04.11726048 +0000 UTC m=+18.127523125 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9b332999-321a-4e3f-99bf-19517a268e28-metrics-certs") pod "network-metrics-daemon-6xfmc" (UID: "9b332999-321a-4e3f-99bf-19517a268e28") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 22:33:56.117813 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:56.117697    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 22:33:56.117813 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:56.117723    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 22:33:56.117813 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:56.117736    2573 projected.go:196] Error preparing data for projected volume kube-api-access-vjgdw for pod openshift-network-diagnostics/network-check-target-7nqtn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 22:33:56.117813 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:56.117781    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef-kube-api-access-vjgdw podName:bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef nodeName:}" failed. No retries permitted until 2026-07-16 22:34:04.117767122 +0000 UTC m=+18.128029778 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vjgdw" (UniqueName: "kubernetes.io/projected/bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef-kube-api-access-vjgdw") pod "network-check-target-7nqtn" (UID: "bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 22:33:56.514654 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:56.514085    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xfmc"
Jul 16 22:33:56.514654 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:56.514216    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xfmc" podUID="9b332999-321a-4e3f-99bf-19517a268e28"
Jul 16 22:33:56.514654 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:56.514282    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kxpfx"
Jul 16 22:33:56.514654 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:56.514440    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kxpfx" podUID="c8d410fb-20ac-4175-a4b4-38d5b44636d6"
Jul 16 22:33:56.514654 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:56.514505    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nqtn"
Jul 16 22:33:56.514654 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:56.514577    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7nqtn" podUID="bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef"
Jul 16 22:33:58.513636 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:58.513605    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nqtn"
Jul 16 22:33:58.514100 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:58.513605    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xfmc"
Jul 16 22:33:58.514100 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:58.513733    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7nqtn" podUID="bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef"
Jul 16 22:33:58.514100 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:33:58.513609    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kxpfx"
Jul 16 22:33:58.514100 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:58.513813    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xfmc" podUID="9b332999-321a-4e3f-99bf-19517a268e28"
Jul 16 22:33:58.514100 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:33:58.513892    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kxpfx" podUID="c8d410fb-20ac-4175-a4b4-38d5b44636d6"
Jul 16 22:34:00.513420 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:00.513373    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xfmc"
Jul 16 22:34:00.513856 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:00.513373    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kxpfx"
Jul 16 22:34:00.513856 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:00.513505    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xfmc" podUID="9b332999-321a-4e3f-99bf-19517a268e28"
Jul 16 22:34:00.513856 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:00.513594    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kxpfx" podUID="c8d410fb-20ac-4175-a4b4-38d5b44636d6"
Jul 16 22:34:00.513856 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:00.513387    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nqtn"
Jul 16 22:34:00.513856 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:00.513686    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7nqtn" podUID="bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef"
Jul 16 22:34:02.513968 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:02.513932    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kxpfx"
Jul 16 22:34:02.513968 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:02.513970    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xfmc"
Jul 16 22:34:02.514509 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:02.513932    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nqtn"
Jul 16 22:34:02.514509 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:02.514071    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kxpfx" podUID="c8d410fb-20ac-4175-a4b4-38d5b44636d6"
Jul 16 22:34:02.514509 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:02.514137    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7nqtn" podUID="bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef"
Jul 16 22:34:02.514509 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:02.514228    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xfmc" podUID="9b332999-321a-4e3f-99bf-19517a268e28"
Jul 16 22:34:04.075910 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:04.075877    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c8d410fb-20ac-4175-a4b4-38d5b44636d6-original-pull-secret\") pod \"global-pull-secret-syncer-kxpfx\" (UID: \"c8d410fb-20ac-4175-a4b4-38d5b44636d6\") " pod="kube-system/global-pull-secret-syncer-kxpfx"
Jul 16 22:34:04.076365 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:04.076005    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 22:34:04.076365 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:04.076058    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8d410fb-20ac-4175-a4b4-38d5b44636d6-original-pull-secret podName:c8d410fb-20ac-4175-a4b4-38d5b44636d6 nodeName:}" failed. No retries permitted until 2026-07-16 22:34:20.076043493 +0000 UTC m=+34.086306135 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c8d410fb-20ac-4175-a4b4-38d5b44636d6-original-pull-secret") pod "global-pull-secret-syncer-kxpfx" (UID: "c8d410fb-20ac-4175-a4b4-38d5b44636d6") : object "kube-system"/"original-pull-secret" not registered
Jul 16 22:34:04.176534 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:04.176499    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjgdw\" (UniqueName: \"kubernetes.io/projected/bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef-kube-api-access-vjgdw\") pod \"network-check-target-7nqtn\" (UID: \"bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef\") " pod="openshift-network-diagnostics/network-check-target-7nqtn"
Jul 16 22:34:04.176681 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:04.176548    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b332999-321a-4e3f-99bf-19517a268e28-metrics-certs\") pod \"network-metrics-daemon-6xfmc\" (UID: \"9b332999-321a-4e3f-99bf-19517a268e28\") " pod="openshift-multus/network-metrics-daemon-6xfmc"
Jul 16 22:34:04.176733 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:04.176681    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 22:34:04.176790 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:04.176739    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b332999-321a-4e3f-99bf-19517a268e28-metrics-certs podName:9b332999-321a-4e3f-99bf-19517a268e28 nodeName:}" failed. No retries permitted until 2026-07-16 22:34:20.176721871 +0000 UTC m=+34.186984512 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9b332999-321a-4e3f-99bf-19517a268e28-metrics-certs") pod "network-metrics-daemon-6xfmc" (UID: "9b332999-321a-4e3f-99bf-19517a268e28") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 22:34:04.176790 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:04.176681    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 22:34:04.176896 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:04.176816    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 22:34:04.176896 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:04.176837    2573 projected.go:196] Error preparing data for projected volume kube-api-access-vjgdw for pod openshift-network-diagnostics/network-check-target-7nqtn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 22:34:04.176977 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:04.176900    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef-kube-api-access-vjgdw podName:bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef nodeName:}" failed. No retries permitted until 2026-07-16 22:34:20.176883332 +0000 UTC m=+34.187145990 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vjgdw" (UniqueName: "kubernetes.io/projected/bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef-kube-api-access-vjgdw") pod "network-check-target-7nqtn" (UID: "bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 22:34:04.513124 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:04.513040    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xfmc"
Jul 16 22:34:04.513303 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:04.513041    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nqtn"
Jul 16 22:34:04.513303 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:04.513167    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xfmc" podUID="9b332999-321a-4e3f-99bf-19517a268e28"
Jul 16 22:34:04.513303 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:04.513221    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kxpfx"
Jul 16 22:34:04.513433 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:04.513316    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7nqtn" podUID="bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef"
Jul 16 22:34:04.513433 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:04.513395    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kxpfx" podUID="c8d410fb-20ac-4175-a4b4-38d5b44636d6"
Jul 16 22:34:06.513836 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:06.513795    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xfmc"
Jul 16 22:34:06.514300 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:06.513906    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xfmc" podUID="9b332999-321a-4e3f-99bf-19517a268e28"
Jul 16 22:34:06.514300 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:06.514215    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kxpfx"
Jul 16 22:34:06.514435 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:06.514353    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kxpfx" podUID="c8d410fb-20ac-4175-a4b4-38d5b44636d6"
Jul 16 22:34:06.514435 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:06.514372    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nqtn"
Jul 16 22:34:06.514532 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:06.514438    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7nqtn" podUID="bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef"
Jul 16 22:34:07.590285 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:07.589963    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8bst9" event={"ID":"268b78b0-10f6-44ce-a822-a124771df729","Type":"ContainerStarted","Data":"3bbbeb052d462e270bd45122757d1e6176638ee3ab1fdb6b04591713c3080572"}
Jul 16 22:34:07.591704 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:07.591585    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-x4th8" event={"ID":"1d026b73-92ba-465b-bc31-5843395ae3e4","Type":"ContainerStarted","Data":"0f14f6744ba6ae0b43293a5abcc73e707a17e0e1d72daa9812881d71180df9e4"}
Jul 16 22:34:07.593270 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:07.593246    2573 generic.go:356] "Generic (PLEG): container finished" podID="2da4fabd-5fae-4a74-a228-7262f2a3e2c8" containerID="4650e2ff5c21a3322a05f7ea850e0768cde9814403b64bf9bfb494de606155a4" exitCode=0
Jul 16 22:34:07.593338 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:07.593273    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rtgnk" event={"ID":"2da4fabd-5fae-4a74-a228-7262f2a3e2c8","Type":"ContainerDied","Data":"4650e2ff5c21a3322a05f7ea850e0768cde9814403b64bf9bfb494de606155a4"}
Jul 16 22:34:07.594937 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:07.594904    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bv2kv" event={"ID":"791625c5-dbf2-4e78-b2ad-08ef2a62a57f","Type":"ContainerStarted","Data":"3ae8b9efc6a90a7d4260902665c7e177b85f4078190da7933e33b6382167c2f7"}
Jul 16 22:34:07.596434 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:07.596412    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-85dmh" event={"ID":"3aefec9e-d46f-4718-89eb-43a524eb0fcf","Type":"ContainerStarted","Data":"5225ee94e8ceb5e3f6f9ba115ac5d7b2dd1dbe702647c32b1eb2603d8dc794c1"}
Jul 16 22:34:07.600223 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:07.600168    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-146.ec2.internal" event={"ID":"70417f5efada6062acead97c2af4fbd0","Type":"ContainerStarted","Data":"c92d9f04cc7ec3cab6610b7d3b2382e71f13021bce16dac310a60e00365fb4bc"}
Jul 16 22:34:07.601815 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:07.601772    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m44bd" event={"ID":"15f7c683-2d93-427c-abb8-c345d44fdb68","Type":"ContainerStarted","Data":"449947cc937c2719759e837292d01e6c1f2a25ef4ee49b09e8ce33f16eeb3d6b"}
Jul 16 22:34:07.603384 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:07.603319    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-466rj" event={"ID":"7144d772-5219-4c76-b0a6-950c4e94a235","Type":"ContainerStarted","Data":"3bf2d2c2c87f5ca5cc61aeaac616679e2272e6eaa331641e9ecc2e11eb2eef5e"}
Jul 16 22:34:07.606678 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:07.606638    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-146.ec2.internal" podStartSLOduration=20.606626677 podStartE2EDuration="20.606626677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:33:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:33:50.579629332 +0000 UTC m=+4.589892024" watchObservedRunningTime="2026-07-16 22:34:07.606626677 +0000 UTC m=+21.616889341"
Jul 16 22:34:07.607090 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:07.607060    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-8bst9" podStartSLOduration=4.441131307 podStartE2EDuration="21.607051971s" totalImagesPullingTime="17.165920664s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:33:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:34:07.606588453 +0000 UTC m=+21.616851117" watchObservedRunningTime="2026-07-16 22:34:07.607051971 +0000 UTC m=+21.617314635"
Jul 16 22:34:07.621188 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:07.621152    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-x4th8" podStartSLOduration=4.449070445 podStartE2EDuration="21.621138462s" totalImagesPullingTime="17.172068017s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:33:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:34:07.620991322 +0000 UTC m=+21.631253987" watchObservedRunningTime="2026-07-16 22:34:07.621138462 +0000 UTC m=+21.631401126"
Jul 16 22:34:07.640547 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:07.640509    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-85dmh" podStartSLOduration=4.377318118 podStartE2EDuration="21.640495158s" totalImagesPullingTime="17.26317704s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:33:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:34:07.64014962 +0000 UTC m=+21.650412285" watchObservedRunningTime="2026-07-16 22:34:07.640495158 +0000 UTC m=+21.650757821"
Jul 16 22:34:07.675782 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:07.675737    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-466rj" podStartSLOduration=7.145833432 podStartE2EDuration="21.675723158s" totalImagesPullingTime="14.529889726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:33:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:34:07.675488357 +0000 UTC m=+21.685751021" watchObservedRunningTime="2026-07-16 22:34:07.675723158 +0000 UTC m=+21.685985805"
Jul 16 22:34:07.676176 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:07.676149    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-146.ec2.internal" podStartSLOduration=20.676140656 podStartE2EDuration="20.676140656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:33:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:34:07.659312832 +0000 UTC m=+21.669575500" watchObservedRunningTime="2026-07-16 22:34:07.676140656 +0000 UTC m=+21.686403319"
Jul 16 22:34:07.722090 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:07.722036    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:07.722761 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:07.722729    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bv2kv" podStartSLOduration=3.45415653 podStartE2EDuration="21.722714858s" totalImagesPullingTime="18.268558328s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:33:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:34:07.721356278 +0000 UTC m=+21.731618953" watchObservedRunningTime="2026-07-16 22:34:07.722714858 +0000 UTC m=+21.732977522"
Jul 16 22:34:07.797265 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:07.797228    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:07.902191 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:07.902169    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:08.104023 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:08.103988    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-466rj"
Jul 16 22:34:08.513451 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:08.513425    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xfmc"
Jul 16 22:34:08.513654 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:08.513425    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kxpfx"
Jul 16 22:34:08.513654 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:08.513433    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nqtn"
Jul 16 22:34:08.513654 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:08.513564    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xfmc" podUID="9b332999-321a-4e3f-99bf-19517a268e28"
Jul 16 22:34:08.513825 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:08.513658    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7nqtn" podUID="bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef"
Jul 16 22:34:08.513825 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:08.513743    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kxpfx" podUID="c8d410fb-20ac-4175-a4b4-38d5b44636d6"
Jul 16 22:34:08.607540 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:08.607497    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k9pz9" event={"ID":"2ada79d5-4ef5-489e-88af-07f0108282da","Type":"ContainerStarted","Data":"95562e2d12636b27952cce8397ce32f7b214db0e0db44950d971ac7063b46c6f"}
Jul 16 22:34:08.610649 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:08.610630    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r9jm4_506f904a-3272-4586-8528-146884e728f9/ovn-acl-logging/0.log"
Jul 16 22:34:08.611017 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:08.610991    2573 generic.go:356] "Generic (PLEG): container finished" podID="506f904a-3272-4586-8528-146884e728f9" containerID="d49f233c9e9a5766bcaededafb20ac9741ee521cea993dce5830d8c925c5f819" exitCode=1
Jul 16 22:34:08.611131 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:08.611022    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4" event={"ID":"506f904a-3272-4586-8528-146884e728f9","Type":"ContainerStarted","Data":"495715e0c9822f1df3633f1aac4fd4dcf99c4641199e2bc157012c957be66589"}
Jul 16 22:34:08.611131 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:08.611102    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4" event={"ID":"506f904a-3272-4586-8528-146884e728f9","Type":"ContainerStarted","Data":"55cb64a9c2ec7052c0989366906a9624989f8a30ac7e8c4592482c6897d91b7b"}
Jul 16 22:34:08.611231 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:08.611154    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4" event={"ID":"506f904a-3272-4586-8528-146884e728f9","Type":"ContainerStarted","Data":"8e904e8114ca0033e6e6a18d042182adc9b1411ac7812e1d00466114a237f74c"}
Jul 16 22:34:08.611231 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:08.611202    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4" event={"ID":"506f904a-3272-4586-8528-146884e728f9","Type":"ContainerStarted","Data":"6edba6c9f3679e942867bd6db3443d306687c1f0e90d1edab961720a20e3e882"}
Jul 16 22:34:08.611331 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:08.611267    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4" event={"ID":"506f904a-3272-4586-8528-146884e728f9","Type":"ContainerDied","Data":"d49f233c9e9a5766bcaededafb20ac9741ee521cea993dce5830d8c925c5f819"}
Jul 16 22:34:08.611331 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:08.611319    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4" event={"ID":"506f904a-3272-4586-8528-146884e728f9","Type":"ContainerStarted","Data":"793fa67c78afb049a8a440cd7bbcf1d86185e620db0c25800b45c36b799663aa"}
Jul 16 22:34:08.625217 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:08.625165    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-k9pz9" podStartSLOduration=5.449034856 podStartE2EDuration="22.625150991s" totalImagesPullingTime="17.176116135s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:33:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:34:08.62453818 +0000 UTC m=+22.634800843" watchObservedRunningTime="2026-07-16 22:34:08.625150991 +0000 UTC m=+22.635413654"
Jul 16 22:34:08.759509 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:08.759482    2573 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 16 22:34:08.783374 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:08.783345    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:08.859788 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:08.859751    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:09.137222 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:09.137192    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-466rj"
Jul 16 22:34:09.137741 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:09.137724    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-466rj"
Jul 16 22:34:09.524845 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:09.524678    2573 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T22:34:08.759504652Z","UUID":"1be23d21-bb1c-41a8-906b-a3d971fa1252","Handler":null,"Name":"","Endpoint":""}
Jul 16 22:34:09.527878 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:09.527855    2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 16 22:34:09.528016 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:09.527900    2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 16 22:34:09.616344 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:09.616254    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m44bd" event={"ID":"15f7c683-2d93-427c-abb8-c345d44fdb68","Type":"ContainerStarted","Data":"b1317d9cfda2268cfce75dc190487962419952d0f56c1ffabb4990559fd6bd67"}
Jul 16 22:34:09.617226 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:09.617204    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-466rj"
Jul 16 22:34:09.684457 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:09.684409    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:10.513918 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:10.513888    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xfmc"
Jul 16 22:34:10.514108 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:10.513883    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nqtn"
Jul 16 22:34:10.514108 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:10.514023    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xfmc" podUID="9b332999-321a-4e3f-99bf-19517a268e28"
Jul 16 22:34:10.514108 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:10.514080    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kxpfx"
Jul 16 22:34:10.514316 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:10.514188    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7nqtn" podUID="bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef"
Jul 16 22:34:10.514316 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:10.514304    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kxpfx" podUID="c8d410fb-20ac-4175-a4b4-38d5b44636d6"
Jul 16 22:34:12.151713 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:12.151547    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:12.242998 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:12.242923    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:12.513215 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:12.513148    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kxpfx"
Jul 16 22:34:12.513215 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:12.513170    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nqtn"
Jul 16 22:34:12.513215 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:12.513170    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xfmc"
Jul 16 22:34:12.513464 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:12.513259    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kxpfx" podUID="c8d410fb-20ac-4175-a4b4-38d5b44636d6"
Jul 16 22:34:12.513464 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:12.513356    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7nqtn" podUID="bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef"
Jul 16 22:34:12.513464 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:12.513399    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xfmc" podUID="9b332999-321a-4e3f-99bf-19517a268e28"
Jul 16 22:34:12.623692 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:12.623668    2573 generic.go:356] "Generic (PLEG): container finished" podID="2da4fabd-5fae-4a74-a228-7262f2a3e2c8" containerID="fe3143e090d98bc1a1b53f191d2b8476fe3ba7397161fb9c7bf1037d33d3e402" exitCode=0
Jul 16 22:34:12.623836 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:12.623735    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rtgnk" event={"ID":"2da4fabd-5fae-4a74-a228-7262f2a3e2c8","Type":"ContainerDied","Data":"fe3143e090d98bc1a1b53f191d2b8476fe3ba7397161fb9c7bf1037d33d3e402"}
Jul 16 22:34:12.625879 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:12.625855    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m44bd" event={"ID":"15f7c683-2d93-427c-abb8-c345d44fdb68","Type":"ContainerStarted","Data":"29bbfa15774a5460175885e59d1bbd1b00f2803fadf8651965684f5727c2b174"}
Jul 16 22:34:12.628899 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:12.628882    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r9jm4_506f904a-3272-4586-8528-146884e728f9/ovn-acl-logging/0.log"
Jul 16 22:34:12.629222 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:12.629203    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4" event={"ID":"506f904a-3272-4586-8528-146884e728f9","Type":"ContainerStarted","Data":"d7a6aa429e71e2c97cd80b9146c9fc9e58c6dda775db905a5fbaab8245cebc85"}
Jul 16 22:34:12.669480 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:12.669436    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m44bd" podStartSLOduration=13.783044492 podStartE2EDuration="26.669423958s" totalImagesPullingTime="12.886379466s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:33:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:34:12.669328635 +0000 UTC m=+26.679591298" watchObservedRunningTime="2026-07-16 22:34:12.669423958 +0000 UTC m=+26.679686621"
Jul 16 22:34:14.023644 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:14.023178    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:14.122068 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:14.122049    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:14.513935 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:14.513908    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xfmc"
Jul 16 22:34:14.514034 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:14.514005    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xfmc" podUID="9b332999-321a-4e3f-99bf-19517a268e28"
Jul 16 22:34:14.514034 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:14.513908    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kxpfx"
Jul 16 22:34:14.514157 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:14.514135    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kxpfx" podUID="c8d410fb-20ac-4175-a4b4-38d5b44636d6"
Jul 16 22:34:14.514192 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:14.514174    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nqtn"
Jul 16 22:34:14.514299 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:14.514273    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7nqtn" podUID="bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef"
Jul 16 22:34:14.635392 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:14.635367    2573 generic.go:356] "Generic (PLEG): container finished" podID="2da4fabd-5fae-4a74-a228-7262f2a3e2c8" containerID="1c7764b970fb48e7a225cb4d95d551461d2a6c070b877af5fcea6170b51af49b" exitCode=0
Jul 16 22:34:14.635505 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:14.635450    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rtgnk" event={"ID":"2da4fabd-5fae-4a74-a228-7262f2a3e2c8","Type":"ContainerDied","Data":"1c7764b970fb48e7a225cb4d95d551461d2a6c070b877af5fcea6170b51af49b"}
Jul 16 22:34:14.638859 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:14.638845    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r9jm4_506f904a-3272-4586-8528-146884e728f9/ovn-acl-logging/0.log"
Jul 16 22:34:14.639196 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:14.639178    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4" event={"ID":"506f904a-3272-4586-8528-146884e728f9","Type":"ContainerStarted","Data":"a69473abecd79d582a23a0bf4eb77263b0e979a4a88eaae74827a29517cf1e4d"}
Jul 16 22:34:14.639472 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:14.639444    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:34:14.639555 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:14.639515    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:34:14.639634 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:14.639619    2573 scope.go:117] "RemoveContainer" containerID="d49f233c9e9a5766bcaededafb20ac9741ee521cea993dce5830d8c925c5f819"
Jul 16 22:34:14.655339 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:14.655320    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:34:15.644659 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:15.644633    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r9jm4_506f904a-3272-4586-8528-146884e728f9/ovn-acl-logging/0.log"
Jul 16 22:34:15.645115 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:15.645002    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4" event={"ID":"506f904a-3272-4586-8528-146884e728f9","Type":"ContainerStarted","Data":"9e6b08e207ffcf8b9c852acd9a0c5caf183d7d459410d7f42679dde1c794ec35"}
Jul 16 22:34:15.645232 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:15.645213    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:34:15.659303 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:15.659284    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:34:15.678746 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:15.678703    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4" podStartSLOduration=10.782798751 podStartE2EDuration="29.678689204s" totalImagesPullingTime="18.895890453s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:33:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:34:15.67837104 +0000 UTC m=+29.688633732" watchObservedRunningTime="2026-07-16 22:34:15.678689204 +0000 UTC m=+29.688951868"
Jul 16 22:34:16.179636 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:16.179605    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7nqtn"]
Jul 16 22:34:16.179752 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:16.179736    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nqtn"
Jul 16 22:34:16.179868 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:16.179846    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7nqtn" podUID="bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef"
Jul 16 22:34:16.182817 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:16.182778    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kxpfx"]
Jul 16 22:34:16.182942 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:16.182887    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kxpfx"
Jul 16 22:34:16.183014 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:16.182988    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kxpfx" podUID="c8d410fb-20ac-4175-a4b4-38d5b44636d6"
Jul 16 22:34:16.183393 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:16.183374    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6xfmc"]
Jul 16 22:34:16.183491 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:16.183478    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xfmc"
Jul 16 22:34:16.183594 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:16.183575    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xfmc" podUID="9b332999-321a-4e3f-99bf-19517a268e28"
Jul 16 22:34:16.650203 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:16.650166    2573 generic.go:356] "Generic (PLEG): container finished" podID="2da4fabd-5fae-4a74-a228-7262f2a3e2c8" containerID="38b9ea460b102042bfe9af1c8699c2bae39c46a0a72ddfe3a0588be03ec38121" exitCode=0
Jul 16 22:34:16.650728 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:16.650264    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rtgnk" event={"ID":"2da4fabd-5fae-4a74-a228-7262f2a3e2c8","Type":"ContainerDied","Data":"38b9ea460b102042bfe9af1c8699c2bae39c46a0a72ddfe3a0588be03ec38121"}
Jul 16 22:34:17.512931 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:17.512899    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nqtn"
Jul 16 22:34:17.513198 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:17.512938    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xfmc"
Jul 16 22:34:17.513198 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:17.513006    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7nqtn" podUID="bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef"
Jul 16 22:34:17.513198 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:17.513129    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xfmc" podUID="9b332999-321a-4e3f-99bf-19517a268e28"
Jul 16 22:34:18.512976 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:18.512942    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kxpfx"
Jul 16 22:34:18.513627 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:18.513059    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kxpfx" podUID="c8d410fb-20ac-4175-a4b4-38d5b44636d6"
Jul 16 22:34:18.522916 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:18.522889    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:18.623473 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:18.623423    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:19.513070 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:19.513038    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xfmc"
Jul 16 22:34:19.513499 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:19.513046    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nqtn"
Jul 16 22:34:19.513499 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:19.513143    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6xfmc" podUID="9b332999-321a-4e3f-99bf-19517a268e28"
Jul 16 22:34:19.513499 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:19.513251    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7nqtn" podUID="bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef"
Jul 16 22:34:20.097904 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:20.097867    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c8d410fb-20ac-4175-a4b4-38d5b44636d6-original-pull-secret\") pod \"global-pull-secret-syncer-kxpfx\" (UID: \"c8d410fb-20ac-4175-a4b4-38d5b44636d6\") " pod="kube-system/global-pull-secret-syncer-kxpfx"
Jul 16 22:34:20.098080 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:20.097991    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 22:34:20.098080 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:20.098055    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8d410fb-20ac-4175-a4b4-38d5b44636d6-original-pull-secret podName:c8d410fb-20ac-4175-a4b4-38d5b44636d6 nodeName:}" failed. No retries permitted until 2026-07-16 22:34:52.098036283 +0000 UTC m=+66.108298928 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c8d410fb-20ac-4175-a4b4-38d5b44636d6-original-pull-secret") pod "global-pull-secret-syncer-kxpfx" (UID: "c8d410fb-20ac-4175-a4b4-38d5b44636d6") : object "kube-system"/"original-pull-secret" not registered
Jul 16 22:34:20.198859 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:20.198826    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjgdw\" (UniqueName: \"kubernetes.io/projected/bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef-kube-api-access-vjgdw\") pod \"network-check-target-7nqtn\" (UID: \"bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef\") " pod="openshift-network-diagnostics/network-check-target-7nqtn"
Jul 16 22:34:20.199039 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:20.198886    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b332999-321a-4e3f-99bf-19517a268e28-metrics-certs\") pod \"network-metrics-daemon-6xfmc\" (UID: \"9b332999-321a-4e3f-99bf-19517a268e28\") " pod="openshift-multus/network-metrics-daemon-6xfmc"
Jul 16 22:34:20.199039 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:20.199015    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 22:34:20.199039 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:20.199031    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 22:34:20.199039 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:20.199040    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 22:34:20.199212 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:20.199054    2573 projected.go:196] Error preparing data for projected volume kube-api-access-vjgdw for pod openshift-network-diagnostics/network-check-target-7nqtn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 22:34:20.199212 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:20.199088    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b332999-321a-4e3f-99bf-19517a268e28-metrics-certs podName:9b332999-321a-4e3f-99bf-19517a268e28 nodeName:}" failed. No retries permitted until 2026-07-16 22:34:52.199072767 +0000 UTC m=+66.209335408 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9b332999-321a-4e3f-99bf-19517a268e28-metrics-certs") pod "network-metrics-daemon-6xfmc" (UID: "9b332999-321a-4e3f-99bf-19517a268e28") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 22:34:20.199212 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:20.199101    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef-kube-api-access-vjgdw podName:bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef nodeName:}" failed. No retries permitted until 2026-07-16 22:34:52.199095466 +0000 UTC m=+66.209358108 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vjgdw" (UniqueName: "kubernetes.io/projected/bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef-kube-api-access-vjgdw") pod "network-check-target-7nqtn" (UID: "bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 22:34:20.513648 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:20.513571    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kxpfx"
Jul 16 22:34:20.514078 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:20.513704    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kxpfx" podUID="c8d410fb-20ac-4175-a4b4-38d5b44636d6"
Jul 16 22:34:21.332837 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.332812    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-146.ec2.internal" event="NodeReady"
Jul 16 22:34:21.332973 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.332899    2573 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 16 22:34:21.382077 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.382047    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nbc9n"]
Jul 16 22:34:21.401868 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.401844    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-btltg"]
Jul 16 22:34:21.401997 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.401982    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nbc9n"
Jul 16 22:34:21.404228 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.404200    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 16 22:34:21.404349 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.404224    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 16 22:34:21.404349 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.404261    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-57zhl\""
Jul 16 22:34:21.415950 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.415931    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-btltg"]
Jul 16 22:34:21.416034 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.415957    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nbc9n"]
Jul 16 22:34:21.416034 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.416014    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-btltg"
Jul 16 22:34:21.420470 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.418718    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-m9krj\""
Jul 16 22:34:21.420470 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.418735    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 16 22:34:21.420470 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.418845    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 16 22:34:21.420470 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.418980    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 16 22:34:21.501524 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.501485    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:21.508443 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.508422    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53ae108c-2ec1-4c24-b756-4a899fbcf19d-tmp-dir\") pod \"dns-default-nbc9n\" (UID: \"53ae108c-2ec1-4c24-b756-4a899fbcf19d\") " pod="openshift-dns/dns-default-nbc9n"
Jul 16 22:34:21.508520 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.508451    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93773e2d-fdc2-4715-825e-ea777c6e6916-cert\") pod \"ingress-canary-btltg\" (UID: \"93773e2d-fdc2-4715-825e-ea777c6e6916\") " pod="openshift-ingress-canary/ingress-canary-btltg"
Jul 16 22:34:21.508520 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.508468    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/53ae108c-2ec1-4c24-b756-4a899fbcf19d-config-volume\") pod \"dns-default-nbc9n\" (UID: \"53ae108c-2ec1-4c24-b756-4a899fbcf19d\") " pod="openshift-dns/dns-default-nbc9n"
Jul 16 22:34:21.508520 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.508485    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxsgf\" (UniqueName: \"kubernetes.io/projected/93773e2d-fdc2-4715-825e-ea777c6e6916-kube-api-access-pxsgf\") pod \"ingress-canary-btltg\" (UID: \"93773e2d-fdc2-4715-825e-ea777c6e6916\") " pod="openshift-ingress-canary/ingress-canary-btltg"
Jul 16 22:34:21.508622 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.508608    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt2gd\" (UniqueName: \"kubernetes.io/projected/53ae108c-2ec1-4c24-b756-4a899fbcf19d-kube-api-access-bt2gd\") pod \"dns-default-nbc9n\" (UID: \"53ae108c-2ec1-4c24-b756-4a899fbcf19d\") " pod="openshift-dns/dns-default-nbc9n"
Jul 16 22:34:21.508666 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.508651    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53ae108c-2ec1-4c24-b756-4a899fbcf19d-metrics-tls\") pod \"dns-default-nbc9n\" (UID: \"53ae108c-2ec1-4c24-b756-4a899fbcf19d\") " pod="openshift-dns/dns-default-nbc9n"
Jul 16 22:34:21.513556 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.513541    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xfmc"
Jul 16 22:34:21.513643 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.513541    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nqtn"
Jul 16 22:34:21.516014 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.515996    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jdr86\""
Jul 16 22:34:21.516493 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.516054    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 16 22:34:21.516493 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.516161    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 16 22:34:21.516493 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.516196    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 16 22:34:21.516493 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.516203    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9dnx4\""
Jul 16 22:34:21.559084 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.559062    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:21.609125 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.609079    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53ae108c-2ec1-4c24-b756-4a899fbcf19d-metrics-tls\") pod \"dns-default-nbc9n\" (UID: \"53ae108c-2ec1-4c24-b756-4a899fbcf19d\") " pod="openshift-dns/dns-default-nbc9n"
Jul 16 22:34:21.609125 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.609106    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53ae108c-2ec1-4c24-b756-4a899fbcf19d-tmp-dir\") pod \"dns-default-nbc9n\" (UID: \"53ae108c-2ec1-4c24-b756-4a899fbcf19d\") " pod="openshift-dns/dns-default-nbc9n"
Jul 16 22:34:21.609285 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.609125    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93773e2d-fdc2-4715-825e-ea777c6e6916-cert\") pod \"ingress-canary-btltg\" (UID: \"93773e2d-fdc2-4715-825e-ea777c6e6916\") " pod="openshift-ingress-canary/ingress-canary-btltg"
Jul 16 22:34:21.609285 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.609153    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/53ae108c-2ec1-4c24-b756-4a899fbcf19d-config-volume\") pod \"dns-default-nbc9n\" (UID: \"53ae108c-2ec1-4c24-b756-4a899fbcf19d\") " pod="openshift-dns/dns-default-nbc9n"
Jul 16 22:34:21.609285 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.609171    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxsgf\" (UniqueName: \"kubernetes.io/projected/93773e2d-fdc2-4715-825e-ea777c6e6916-kube-api-access-pxsgf\") pod \"ingress-canary-btltg\" (UID: \"93773e2d-fdc2-4715-825e-ea777c6e6916\") " pod="openshift-ingress-canary/ingress-canary-btltg"
Jul 16 22:34:21.609285 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:21.609233    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 22:34:21.609456 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:21.609308    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ae108c-2ec1-4c24-b756-4a899fbcf19d-metrics-tls podName:53ae108c-2ec1-4c24-b756-4a899fbcf19d nodeName:}" failed. No retries permitted until 2026-07-16 22:34:22.109289487 +0000 UTC m=+36.119552142 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/53ae108c-2ec1-4c24-b756-4a899fbcf19d-metrics-tls") pod "dns-default-nbc9n" (UID: "53ae108c-2ec1-4c24-b756-4a899fbcf19d") : secret "dns-default-metrics-tls" not found
Jul 16 22:34:21.609456 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.609329    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt2gd\" (UniqueName: \"kubernetes.io/projected/53ae108c-2ec1-4c24-b756-4a899fbcf19d-kube-api-access-bt2gd\") pod \"dns-default-nbc9n\" (UID: \"53ae108c-2ec1-4c24-b756-4a899fbcf19d\") " pod="openshift-dns/dns-default-nbc9n"
Jul 16 22:34:21.609561 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.609477    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53ae108c-2ec1-4c24-b756-4a899fbcf19d-tmp-dir\") pod \"dns-default-nbc9n\" (UID: \"53ae108c-2ec1-4c24-b756-4a899fbcf19d\") " pod="openshift-dns/dns-default-nbc9n"
Jul 16 22:34:21.609766 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.609749    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/53ae108c-2ec1-4c24-b756-4a899fbcf19d-config-volume\") pod \"dns-default-nbc9n\" (UID: \"53ae108c-2ec1-4c24-b756-4a899fbcf19d\") " pod="openshift-dns/dns-default-nbc9n"
Jul 16 22:34:21.613065 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.613045    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93773e2d-fdc2-4715-825e-ea777c6e6916-cert\") pod \"ingress-canary-btltg\" (UID: \"93773e2d-fdc2-4715-825e-ea777c6e6916\") " pod="openshift-ingress-canary/ingress-canary-btltg"
Jul 16 22:34:21.617563 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.617535    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxsgf\" (UniqueName: \"kubernetes.io/projected/93773e2d-fdc2-4715-825e-ea777c6e6916-kube-api-access-pxsgf\") pod \"ingress-canary-btltg\" (UID: \"93773e2d-fdc2-4715-825e-ea777c6e6916\") " pod="openshift-ingress-canary/ingress-canary-btltg"
Jul 16 22:34:21.617711 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.617691    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt2gd\" (UniqueName: \"kubernetes.io/projected/53ae108c-2ec1-4c24-b756-4a899fbcf19d-kube-api-access-bt2gd\") pod \"dns-default-nbc9n\" (UID: \"53ae108c-2ec1-4c24-b756-4a899fbcf19d\") " pod="openshift-dns/dns-default-nbc9n"
Jul 16 22:34:21.663977 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.663955    2573 generic.go:356] "Generic (PLEG): container finished" podID="2da4fabd-5fae-4a74-a228-7262f2a3e2c8" containerID="af94458a657b5f98be38d6d2b16483f36534866a8ff570af35dbe3f34c3e049a" exitCode=0
Jul 16 22:34:21.664052 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.663993    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rtgnk" event={"ID":"2da4fabd-5fae-4a74-a228-7262f2a3e2c8","Type":"ContainerDied","Data":"af94458a657b5f98be38d6d2b16483f36534866a8ff570af35dbe3f34c3e049a"}
Jul 16 22:34:21.725012 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.724680    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-btltg"
Jul 16 22:34:21.898178 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:21.898154    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-btltg"]
Jul 16 22:34:21.920007 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:34:21.919983    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93773e2d_fdc2_4715_825e_ea777c6e6916.slice/crio-5539c25e16ad813dfe985af596c898f29c807b7f12d5015434345ef156d46e26 WatchSource:0}: Error finding container 5539c25e16ad813dfe985af596c898f29c807b7f12d5015434345ef156d46e26: Status 404 returned error can't find the container with id 5539c25e16ad813dfe985af596c898f29c807b7f12d5015434345ef156d46e26
Jul 16 22:34:22.067310 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:22.067281    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:22.113314 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:22.113290    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53ae108c-2ec1-4c24-b756-4a899fbcf19d-metrics-tls\") pod \"dns-default-nbc9n\" (UID: \"53ae108c-2ec1-4c24-b756-4a899fbcf19d\") " pod="openshift-dns/dns-default-nbc9n"
Jul 16 22:34:22.113406 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:22.113394    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 22:34:22.113456 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:22.113448    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ae108c-2ec1-4c24-b756-4a899fbcf19d-metrics-tls podName:53ae108c-2ec1-4c24-b756-4a899fbcf19d nodeName:}" failed. No retries permitted until 2026-07-16 22:34:23.113433537 +0000 UTC m=+37.123696179 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/53ae108c-2ec1-4c24-b756-4a899fbcf19d-metrics-tls") pod "dns-default-nbc9n" (UID: "53ae108c-2ec1-4c24-b756-4a899fbcf19d") : secret "dns-default-metrics-tls" not found
Jul 16 22:34:22.167630 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:22.167586    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:22.517344 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:22.517285    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kxpfx"
Jul 16 22:34:22.520075 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:22.520053    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 16 22:34:22.667788 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:22.667751    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-btltg" event={"ID":"93773e2d-fdc2-4715-825e-ea777c6e6916","Type":"ContainerStarted","Data":"5539c25e16ad813dfe985af596c898f29c807b7f12d5015434345ef156d46e26"}
Jul 16 22:34:22.671595 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:22.671567    2573 generic.go:356] "Generic (PLEG): container finished" podID="2da4fabd-5fae-4a74-a228-7262f2a3e2c8" containerID="875784017da0695c932e231d2ba27c076878b47e130aab324bba3a708ffa92e0" exitCode=0
Jul 16 22:34:22.671732 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:22.671610    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rtgnk" event={"ID":"2da4fabd-5fae-4a74-a228-7262f2a3e2c8","Type":"ContainerDied","Data":"875784017da0695c932e231d2ba27c076878b47e130aab324bba3a708ffa92e0"}
Jul 16 22:34:23.122571 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:23.122338    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53ae108c-2ec1-4c24-b756-4a899fbcf19d-metrics-tls\") pod \"dns-default-nbc9n\" (UID: \"53ae108c-2ec1-4c24-b756-4a899fbcf19d\") " pod="openshift-dns/dns-default-nbc9n"
Jul 16 22:34:23.122744 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:23.122490    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 22:34:23.122744 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:23.122686    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ae108c-2ec1-4c24-b756-4a899fbcf19d-metrics-tls podName:53ae108c-2ec1-4c24-b756-4a899fbcf19d nodeName:}" failed. No retries permitted until 2026-07-16 22:34:25.122665921 +0000 UTC m=+39.132928563 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/53ae108c-2ec1-4c24-b756-4a899fbcf19d-metrics-tls") pod "dns-default-nbc9n" (UID: "53ae108c-2ec1-4c24-b756-4a899fbcf19d") : secret "dns-default-metrics-tls" not found
Jul 16 22:34:23.678100 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:23.678058    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rtgnk" event={"ID":"2da4fabd-5fae-4a74-a228-7262f2a3e2c8","Type":"ContainerStarted","Data":"3c3aece1694e2bb71cf7e8ad02a18b0d3cc2747cce4c5bb5fbd586d2c96a9634"}
Jul 16 22:34:23.701095 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:23.701040    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rtgnk" podStartSLOduration=12.365471251 podStartE2EDuration="37.70101636s" totalImagesPullingTime="25.335545109s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:33:46 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:34:23.699657102 +0000 UTC m=+37.709919787" watchObservedRunningTime="2026-07-16 22:34:23.70101636 +0000 UTC m=+37.711279005"
Jul 16 22:34:23.822225 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:23.822194    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:23.924616 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:23.924580    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:24.682216 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:24.682183    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-btltg" event={"ID":"93773e2d-fdc2-4715-825e-ea777c6e6916","Type":"ContainerStarted","Data":"666fe62e9e2f674f5303c60ec702b180a66d805486d8a0cfa7876c9151440151"}
Jul 16 22:34:24.701033 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:24.700984    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-btltg" podStartSLOduration=1.301297295 podStartE2EDuration="3.700970478s" totalImagesPullingTime="2.399673183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:34:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:34:24.699919058 +0000 UTC m=+38.710181722" watchObservedRunningTime="2026-07-16 22:34:24.700970478 +0000 UTC m=+38.711233119"
Jul 16 22:34:25.135960 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:25.135932    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53ae108c-2ec1-4c24-b756-4a899fbcf19d-metrics-tls\") pod \"dns-default-nbc9n\" (UID: \"53ae108c-2ec1-4c24-b756-4a899fbcf19d\") " pod="openshift-dns/dns-default-nbc9n"
Jul 16 22:34:25.136131 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:25.136107    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 22:34:25.136188 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:25.136178    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ae108c-2ec1-4c24-b756-4a899fbcf19d-metrics-tls podName:53ae108c-2ec1-4c24-b756-4a899fbcf19d nodeName:}" failed. No retries permitted until 2026-07-16 22:34:29.136160742 +0000 UTC m=+43.146423385 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/53ae108c-2ec1-4c24-b756-4a899fbcf19d-metrics-tls") pod "dns-default-nbc9n" (UID: "53ae108c-2ec1-4c24-b756-4a899fbcf19d") : secret "dns-default-metrics-tls" not found
Jul 16 22:34:25.778121 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:25.778091    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:29.162248 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:29.162210    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53ae108c-2ec1-4c24-b756-4a899fbcf19d-metrics-tls\") pod \"dns-default-nbc9n\" (UID: \"53ae108c-2ec1-4c24-b756-4a899fbcf19d\") " pod="openshift-dns/dns-default-nbc9n"
Jul 16 22:34:29.162635 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:29.162357    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 22:34:29.162635 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:29.162428    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ae108c-2ec1-4c24-b756-4a899fbcf19d-metrics-tls podName:53ae108c-2ec1-4c24-b756-4a899fbcf19d nodeName:}" failed. No retries permitted until 2026-07-16 22:34:37.162410846 +0000 UTC m=+51.172673488 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/53ae108c-2ec1-4c24-b756-4a899fbcf19d-metrics-tls") pod "dns-default-nbc9n" (UID: "53ae108c-2ec1-4c24-b756-4a899fbcf19d") : secret "dns-default-metrics-tls" not found
Jul 16 22:34:37.215303 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:37.215261    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53ae108c-2ec1-4c24-b756-4a899fbcf19d-metrics-tls\") pod \"dns-default-nbc9n\" (UID: \"53ae108c-2ec1-4c24-b756-4a899fbcf19d\") " pod="openshift-dns/dns-default-nbc9n"
Jul 16 22:34:37.215852 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:37.215423    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 22:34:37.215852 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:37.215498    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ae108c-2ec1-4c24-b756-4a899fbcf19d-metrics-tls podName:53ae108c-2ec1-4c24-b756-4a899fbcf19d nodeName:}" failed. No retries permitted until 2026-07-16 22:34:53.215482372 +0000 UTC m=+67.225745014 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/53ae108c-2ec1-4c24-b756-4a899fbcf19d-metrics-tls") pod "dns-default-nbc9n" (UID: "53ae108c-2ec1-4c24-b756-4a899fbcf19d") : secret "dns-default-metrics-tls" not found
Jul 16 22:34:41.043209 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:41.043176    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:41.143178 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:41.143153    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:42.896955 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:42.896916    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-mtdx8"]
Jul 16 22:34:42.898710 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:42.898692    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-zvtkr"]
Jul 16 22:34:42.898826 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:42.898807    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-mtdx8"
Jul 16 22:34:42.900319 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:42.900298    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-zvtkr"
Jul 16 22:34:42.901594 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:42.901574    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 16 22:34:42.901718 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:42.901648    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 16 22:34:42.902066 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:42.902048    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-4p96d\""
Jul 16 22:34:42.904667 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:42.904649    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 16 22:34:42.906171 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:42.906150    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 16 22:34:42.906298 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:42.906263    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-p84ss\""
Jul 16 22:34:42.906391 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:42.906372    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 16 22:34:42.914009 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:42.913986    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-zvtkr"]
Jul 16 22:34:42.914742 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:42.914723    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-mtdx8"]
Jul 16 22:34:42.999408 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:42.999384    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-zm76k"]
Jul 16 22:34:43.002691 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.002676    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-zm76k"
Jul 16 22:34:43.004689 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.004667    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6669f9958b-wkhdn"]
Jul 16 22:34:43.005866 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.005844    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-pqfr9\""
Jul 16 22:34:43.006520 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.006502    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 16 22:34:43.006805 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.006791    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 16 22:34:43.007333 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.007314    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 16 22:34:43.009129 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.009111    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6669f9958b-wkhdn"
Jul 16 22:34:43.011107 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.011092    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 16 22:34:43.011107 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.011100    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 16 22:34:43.011365 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.011351    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 16 22:34:43.011612 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.011598    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 16 22:34:43.011700 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.011685    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 16 22:34:43.011866 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.011851    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 16 22:34:43.012156 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.012141    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 16 22:34:43.012326 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.012309    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-qjslv\""
Jul 16 22:34:43.013853 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.013822    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-zm76k"]
Jul 16 22:34:43.022806 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.022786    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6669f9958b-wkhdn"]
Jul 16 22:34:43.055185 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.055165    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq8gt\" (UniqueName: \"kubernetes.io/projected/3244b5d6-318c-423d-844c-9d93d94ea9d8-kube-api-access-jq8gt\") pod \"cluster-samples-operator-79b7c55f87-zvtkr\" (UID: \"3244b5d6-318c-423d-844c-9d93d94ea9d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-zvtkr"
Jul 16 22:34:43.055299 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.055212    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f4z4\" (UniqueName: \"kubernetes.io/projected/03851daf-9b83-4d3a-8282-8466d1d4b574-kube-api-access-9f4z4\") pod \"volume-data-source-validator-78cf4c6f6-mtdx8\" (UID: \"03851daf-9b83-4d3a-8282-8466d1d4b574\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-mtdx8"
Jul 16 22:34:43.055299 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.055257    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3244b5d6-318c-423d-844c-9d93d94ea9d8-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-zvtkr\" (UID: \"3244b5d6-318c-423d-844c-9d93d94ea9d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-zvtkr"
Jul 16 22:34:43.156376 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.156328    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-default-certificate\") pod \"router-default-6669f9958b-wkhdn\" (UID: \"4a28ac1e-8676-4819-aae2-68bf9cd4ea37\") " pod="openshift-ingress/router-default-6669f9958b-wkhdn"
Jul 16 22:34:43.156376 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.156356    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-service-ca-bundle\") pod \"router-default-6669f9958b-wkhdn\" (UID: \"4a28ac1e-8676-4819-aae2-68bf9cd4ea37\") " pod="openshift-ingress/router-default-6669f9958b-wkhdn"
Jul 16 22:34:43.156503 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.156396    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6492a043-84e1-4d73-aac5-4239e445d109-serving-cert\") pod \"service-ca-operator-5bcbc94c78-zm76k\" (UID: \"6492a043-84e1-4d73-aac5-4239e445d109\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-zm76k"
Jul 16 22:34:43.156503 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.156457    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6492a043-84e1-4d73-aac5-4239e445d109-config\") pod \"service-ca-operator-5bcbc94c78-zm76k\" (UID: \"6492a043-84e1-4d73-aac5-4239e445d109\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-zm76k"
Jul 16 22:34:43.156503 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.156482    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq8gt\" (UniqueName: \"kubernetes.io/projected/3244b5d6-318c-423d-844c-9d93d94ea9d8-kube-api-access-jq8gt\") pod \"cluster-samples-operator-79b7c55f87-zvtkr\" (UID: \"3244b5d6-318c-423d-844c-9d93d94ea9d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-zvtkr"
Jul 16 22:34:43.156655 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.156527    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f4z4\" (UniqueName: \"kubernetes.io/projected/03851daf-9b83-4d3a-8282-8466d1d4b574-kube-api-access-9f4z4\") pod \"volume-data-source-validator-78cf4c6f6-mtdx8\" (UID: \"03851daf-9b83-4d3a-8282-8466d1d4b574\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-mtdx8"
Jul 16 22:34:43.156655 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.156543    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3244b5d6-318c-423d-844c-9d93d94ea9d8-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-zvtkr\" (UID: \"3244b5d6-318c-423d-844c-9d93d94ea9d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-zvtkr"
Jul 16 22:34:43.156655 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.156567    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-metrics-certs\") pod \"router-default-6669f9958b-wkhdn\" (UID: \"4a28ac1e-8676-4819-aae2-68bf9cd4ea37\") " pod="openshift-ingress/router-default-6669f9958b-wkhdn"
Jul 16 22:34:43.156655 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.156593    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwhkk\" (UniqueName: \"kubernetes.io/projected/6492a043-84e1-4d73-aac5-4239e445d109-kube-api-access-bwhkk\") pod \"service-ca-operator-5bcbc94c78-zm76k\" (UID: \"6492a043-84e1-4d73-aac5-4239e445d109\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-zm76k"
Jul 16 22:34:43.156655 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.156621    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-stats-auth\") pod \"router-default-6669f9958b-wkhdn\" (UID: \"4a28ac1e-8676-4819-aae2-68bf9cd4ea37\") " pod="openshift-ingress/router-default-6669f9958b-wkhdn"
Jul 16 22:34:43.156655 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:43.156636    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 16 22:34:43.156655 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.156648    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p796j\" (UniqueName: \"kubernetes.io/projected/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-kube-api-access-p796j\") pod \"router-default-6669f9958b-wkhdn\" (UID: \"4a28ac1e-8676-4819-aae2-68bf9cd4ea37\") " pod="openshift-ingress/router-default-6669f9958b-wkhdn"
Jul 16 22:34:43.156931 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:43.156707    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3244b5d6-318c-423d-844c-9d93d94ea9d8-samples-operator-tls podName:3244b5d6-318c-423d-844c-9d93d94ea9d8 nodeName:}" failed. No retries permitted until 2026-07-16 22:34:43.656688124 +0000 UTC m=+57.666950786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3244b5d6-318c-423d-844c-9d93d94ea9d8-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-zvtkr" (UID: "3244b5d6-318c-423d-844c-9d93d94ea9d8") : secret "samples-operator-tls" not found
Jul 16 22:34:43.169550 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.169530    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f4z4\" (UniqueName: \"kubernetes.io/projected/03851daf-9b83-4d3a-8282-8466d1d4b574-kube-api-access-9f4z4\") pod \"volume-data-source-validator-78cf4c6f6-mtdx8\" (UID: \"03851daf-9b83-4d3a-8282-8466d1d4b574\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-mtdx8"
Jul 16 22:34:43.169629 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.169551    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq8gt\" (UniqueName: \"kubernetes.io/projected/3244b5d6-318c-423d-844c-9d93d94ea9d8-kube-api-access-jq8gt\") pod \"cluster-samples-operator-79b7c55f87-zvtkr\" (UID: \"3244b5d6-318c-423d-844c-9d93d94ea9d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-zvtkr"
Jul 16 22:34:43.176226 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.176202    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:43.209113 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.209094    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-mtdx8"
Jul 16 22:34:43.257465 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.257438    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6492a043-84e1-4d73-aac5-4239e445d109-config\") pod \"service-ca-operator-5bcbc94c78-zm76k\" (UID: \"6492a043-84e1-4d73-aac5-4239e445d109\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-zm76k"
Jul 16 22:34:43.257559 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.257510    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-metrics-certs\") pod \"router-default-6669f9958b-wkhdn\" (UID: \"4a28ac1e-8676-4819-aae2-68bf9cd4ea37\") " pod="openshift-ingress/router-default-6669f9958b-wkhdn"
Jul 16 22:34:43.257559 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.257540    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwhkk\" (UniqueName: \"kubernetes.io/projected/6492a043-84e1-4d73-aac5-4239e445d109-kube-api-access-bwhkk\") pod \"service-ca-operator-5bcbc94c78-zm76k\" (UID: \"6492a043-84e1-4d73-aac5-4239e445d109\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-zm76k"
Jul 16 22:34:43.257683 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.257571    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-stats-auth\") pod \"router-default-6669f9958b-wkhdn\" (UID: \"4a28ac1e-8676-4819-aae2-68bf9cd4ea37\") " pod="openshift-ingress/router-default-6669f9958b-wkhdn"
Jul 16 22:34:43.257683 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.257600    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p796j\" (UniqueName: \"kubernetes.io/projected/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-kube-api-access-p796j\") pod \"router-default-6669f9958b-wkhdn\" (UID: \"4a28ac1e-8676-4819-aae2-68bf9cd4ea37\") " pod="openshift-ingress/router-default-6669f9958b-wkhdn"
Jul 16 22:34:43.257683 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.257631    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-default-certificate\") pod \"router-default-6669f9958b-wkhdn\" (UID: \"4a28ac1e-8676-4819-aae2-68bf9cd4ea37\") " pod="openshift-ingress/router-default-6669f9958b-wkhdn"
Jul 16 22:34:43.257683 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:43.257655    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 16 22:34:43.257885 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:43.257720    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-metrics-certs podName:4a28ac1e-8676-4819-aae2-68bf9cd4ea37 nodeName:}" failed. No retries permitted until 2026-07-16 22:34:43.757700686 +0000 UTC m=+57.767963339 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-metrics-certs") pod "router-default-6669f9958b-wkhdn" (UID: "4a28ac1e-8676-4819-aae2-68bf9cd4ea37") : secret "router-metrics-certs-default" not found
Jul 16 22:34:43.257885 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:43.257777    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-service-ca-bundle podName:4a28ac1e-8676-4819-aae2-68bf9cd4ea37 nodeName:}" failed. No retries permitted until 2026-07-16 22:34:43.757758148 +0000 UTC m=+57.768020815 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-service-ca-bundle") pod "router-default-6669f9958b-wkhdn" (UID: "4a28ac1e-8676-4819-aae2-68bf9cd4ea37") : configmap references non-existent config key: service-ca.crt
Jul 16 22:34:43.257885 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.257659    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-service-ca-bundle\") pod \"router-default-6669f9958b-wkhdn\" (UID: \"4a28ac1e-8676-4819-aae2-68bf9cd4ea37\") " pod="openshift-ingress/router-default-6669f9958b-wkhdn"
Jul 16 22:34:43.257885 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.257858    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6492a043-84e1-4d73-aac5-4239e445d109-serving-cert\") pod \"service-ca-operator-5bcbc94c78-zm76k\" (UID: \"6492a043-84e1-4d73-aac5-4239e445d109\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-zm76k"
Jul 16 22:34:43.258126 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.258046    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6492a043-84e1-4d73-aac5-4239e445d109-config\") pod \"service-ca-operator-5bcbc94c78-zm76k\" (UID: \"6492a043-84e1-4d73-aac5-4239e445d109\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-zm76k"
Jul 16 22:34:43.260040 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.260017    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-stats-auth\") pod \"router-default-6669f9958b-wkhdn\" (UID: \"4a28ac1e-8676-4819-aae2-68bf9cd4ea37\") " pod="openshift-ingress/router-default-6669f9958b-wkhdn"
Jul 16 22:34:43.260124 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.260058    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6492a043-84e1-4d73-aac5-4239e445d109-serving-cert\") pod \"service-ca-operator-5bcbc94c78-zm76k\" (UID: \"6492a043-84e1-4d73-aac5-4239e445d109\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-zm76k"
Jul 16 22:34:43.260124 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.260097    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-default-certificate\") pod \"router-default-6669f9958b-wkhdn\" (UID: \"4a28ac1e-8676-4819-aae2-68bf9cd4ea37\") " pod="openshift-ingress/router-default-6669f9958b-wkhdn"
Jul 16 22:34:43.266769 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.266744    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwhkk\" (UniqueName: \"kubernetes.io/projected/6492a043-84e1-4d73-aac5-4239e445d109-kube-api-access-bwhkk\") pod \"service-ca-operator-5bcbc94c78-zm76k\" (UID: \"6492a043-84e1-4d73-aac5-4239e445d109\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-zm76k"
Jul 16 22:34:43.266854 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.266825    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p796j\" (UniqueName: \"kubernetes.io/projected/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-kube-api-access-p796j\") pod \"router-default-6669f9958b-wkhdn\" (UID: \"4a28ac1e-8676-4819-aae2-68bf9cd4ea37\") " pod="openshift-ingress/router-default-6669f9958b-wkhdn"
Jul 16 22:34:43.276792 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.276766    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:43.310901 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.310846    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-zm76k"
Jul 16 22:34:43.363654 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.363551    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-mtdx8"]
Jul 16 22:34:43.366157 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:34:43.366133    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03851daf_9b83_4d3a_8282_8466d1d4b574.slice/crio-37843db1accfeed8b94b739a7c5968bad506f64087a30d868cc95ed93f8c5756 WatchSource:0}: Error finding container 37843db1accfeed8b94b739a7c5968bad506f64087a30d868cc95ed93f8c5756: Status 404 returned error can't find the container with id 37843db1accfeed8b94b739a7c5968bad506f64087a30d868cc95ed93f8c5756
Jul 16 22:34:43.483424 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.483395    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-zm76k"]
Jul 16 22:34:43.486564 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:34:43.486541    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6492a043_84e1_4d73_aac5_4239e445d109.slice/crio-f0b5cc6e6e1604c963e89a4ad67319968a9c892c7175847988b09b1a98820465 WatchSource:0}: Error finding container f0b5cc6e6e1604c963e89a4ad67319968a9c892c7175847988b09b1a98820465: Status 404 returned error can't find the container with id f0b5cc6e6e1604c963e89a4ad67319968a9c892c7175847988b09b1a98820465
Jul 16 22:34:43.660799 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.660774    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3244b5d6-318c-423d-844c-9d93d94ea9d8-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-zvtkr\" (UID: \"3244b5d6-318c-423d-844c-9d93d94ea9d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-zvtkr"
Jul 16 22:34:43.660938 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:43.660918    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 16 22:34:43.660993 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:43.660984    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3244b5d6-318c-423d-844c-9d93d94ea9d8-samples-operator-tls podName:3244b5d6-318c-423d-844c-9d93d94ea9d8 nodeName:}" failed. No retries permitted until 2026-07-16 22:34:44.660967228 +0000 UTC m=+58.671229870 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3244b5d6-318c-423d-844c-9d93d94ea9d8-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-zvtkr" (UID: "3244b5d6-318c-423d-844c-9d93d94ea9d8") : secret "samples-operator-tls" not found
Jul 16 22:34:43.726202 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.726128    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-zm76k" event={"ID":"6492a043-84e1-4d73-aac5-4239e445d109","Type":"ContainerStarted","Data":"f0b5cc6e6e1604c963e89a4ad67319968a9c892c7175847988b09b1a98820465"}
Jul 16 22:34:43.727272 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.727250    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-mtdx8" event={"ID":"03851daf-9b83-4d3a-8282-8466d1d4b574","Type":"ContainerStarted","Data":"37843db1accfeed8b94b739a7c5968bad506f64087a30d868cc95ed93f8c5756"}
Jul 16 22:34:43.761562 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.761541    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-metrics-certs\") pod \"router-default-6669f9958b-wkhdn\" (UID: \"4a28ac1e-8676-4819-aae2-68bf9cd4ea37\") " pod="openshift-ingress/router-default-6669f9958b-wkhdn"
Jul 16 22:34:43.761614 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:43.761578    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-service-ca-bundle\") pod \"router-default-6669f9958b-wkhdn\" (UID: \"4a28ac1e-8676-4819-aae2-68bf9cd4ea37\") " pod="openshift-ingress/router-default-6669f9958b-wkhdn"
Jul 16 22:34:43.761706 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:43.761692    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 16 22:34:43.761737 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:43.761709    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-service-ca-bundle podName:4a28ac1e-8676-4819-aae2-68bf9cd4ea37 nodeName:}" failed. No retries permitted until 2026-07-16 22:34:44.761691265 +0000 UTC m=+58.771953922 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-service-ca-bundle") pod "router-default-6669f9958b-wkhdn" (UID: "4a28ac1e-8676-4819-aae2-68bf9cd4ea37") : configmap references non-existent config key: service-ca.crt
Jul 16 22:34:43.761737 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:43.761735    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-metrics-certs podName:4a28ac1e-8676-4819-aae2-68bf9cd4ea37 nodeName:}" failed. No retries permitted until 2026-07-16 22:34:44.761724638 +0000 UTC m=+58.771987291 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-metrics-certs") pod "router-default-6669f9958b-wkhdn" (UID: "4a28ac1e-8676-4819-aae2-68bf9cd4ea37") : secret "router-metrics-certs-default" not found
Jul 16 22:34:44.667751 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:44.667720    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3244b5d6-318c-423d-844c-9d93d94ea9d8-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-zvtkr\" (UID: \"3244b5d6-318c-423d-844c-9d93d94ea9d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-zvtkr"
Jul 16 22:34:44.668177 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:44.667898    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 16 22:34:44.668177 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:44.667975    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3244b5d6-318c-423d-844c-9d93d94ea9d8-samples-operator-tls podName:3244b5d6-318c-423d-844c-9d93d94ea9d8 nodeName:}" failed. No retries permitted until 2026-07-16 22:34:46.667957568 +0000 UTC m=+60.678220209 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3244b5d6-318c-423d-844c-9d93d94ea9d8-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-zvtkr" (UID: "3244b5d6-318c-423d-844c-9d93d94ea9d8") : secret "samples-operator-tls" not found
Jul 16 22:34:44.769178 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:44.769144    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-metrics-certs\") pod \"router-default-6669f9958b-wkhdn\" (UID: \"4a28ac1e-8676-4819-aae2-68bf9cd4ea37\") " pod="openshift-ingress/router-default-6669f9958b-wkhdn"
Jul 16 22:34:44.769358 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:44.769211    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-service-ca-bundle\") pod \"router-default-6669f9958b-wkhdn\" (UID: \"4a28ac1e-8676-4819-aae2-68bf9cd4ea37\") " pod="openshift-ingress/router-default-6669f9958b-wkhdn"
Jul 16 22:34:44.769358 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:44.769311    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 16 22:34:44.769476 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:44.769386    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-metrics-certs podName:4a28ac1e-8676-4819-aae2-68bf9cd4ea37 nodeName:}" failed. No retries permitted until 2026-07-16 22:34:46.769367143 +0000 UTC m=+60.779629798 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-metrics-certs") pod "router-default-6669f9958b-wkhdn" (UID: "4a28ac1e-8676-4819-aae2-68bf9cd4ea37") : secret "router-metrics-certs-default" not found
Jul 16 22:34:44.769476 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:44.769408    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-service-ca-bundle podName:4a28ac1e-8676-4819-aae2-68bf9cd4ea37 nodeName:}" failed. No retries permitted until 2026-07-16 22:34:46.769396362 +0000 UTC m=+60.779659004 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-service-ca-bundle") pod "router-default-6669f9958b-wkhdn" (UID: "4a28ac1e-8676-4819-aae2-68bf9cd4ea37") : configmap references non-existent config key: service-ca.crt
Jul 16 22:34:45.541825 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:45.541788    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:45.736981 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:45.736900    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-zm76k" event={"ID":"6492a043-84e1-4d73-aac5-4239e445d109","Type":"ContainerStarted","Data":"78305322d55ede1cd044f5d60cc6507916dad04fe5c3492efa3f1139c80c745d"}
Jul 16 22:34:45.738482 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:45.738455    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-mtdx8" event={"ID":"03851daf-9b83-4d3a-8282-8466d1d4b574","Type":"ContainerStarted","Data":"8704cfbcf168e228924bdc3b226254e7b2f6f8c8962760c0fe7758398e53b6a7"}
Jul 16 22:34:45.753876 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:45.753832    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-zm76k" podStartSLOduration=1.752178445 podStartE2EDuration="3.753819113s" totalImagesPullingTime="2.001640668s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:34:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:34:45.753452168 +0000 UTC m=+59.763714833" watchObservedRunningTime="2026-07-16 22:34:45.753819113 +0000 UTC m=+59.764081773"
Jul 16 22:34:45.874127 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:45.874095    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:45.962701 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:45.962673    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:46.682182 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:46.682147    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3244b5d6-318c-423d-844c-9d93d94ea9d8-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-zvtkr\" (UID: \"3244b5d6-318c-423d-844c-9d93d94ea9d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-zvtkr"
Jul 16 22:34:46.682340 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:46.682278    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 16 22:34:46.682340 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:46.682332    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3244b5d6-318c-423d-844c-9d93d94ea9d8-samples-operator-tls podName:3244b5d6-318c-423d-844c-9d93d94ea9d8 nodeName:}" failed. No retries permitted until 2026-07-16 22:34:50.682314647 +0000 UTC m=+64.692577294 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3244b5d6-318c-423d-844c-9d93d94ea9d8-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-zvtkr" (UID: "3244b5d6-318c-423d-844c-9d93d94ea9d8") : secret "samples-operator-tls" not found
Jul 16 22:34:46.701587 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:46.701561    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:46.782908 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:46.782877    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-metrics-certs\") pod \"router-default-6669f9958b-wkhdn\" (UID: \"4a28ac1e-8676-4819-aae2-68bf9cd4ea37\") " pod="openshift-ingress/router-default-6669f9958b-wkhdn"
Jul 16 22:34:46.783224 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:46.782926    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-service-ca-bundle\") pod \"router-default-6669f9958b-wkhdn\" (UID: \"4a28ac1e-8676-4819-aae2-68bf9cd4ea37\") " pod="openshift-ingress/router-default-6669f9958b-wkhdn"
Jul 16 22:34:46.783224 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:46.783006    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 16 22:34:46.783224 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:46.783066    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-metrics-certs podName:4a28ac1e-8676-4819-aae2-68bf9cd4ea37 nodeName:}" failed. No retries permitted until 2026-07-16 22:34:50.783053016 +0000 UTC m=+64.793315662 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-metrics-certs") pod "router-default-6669f9958b-wkhdn" (UID: "4a28ac1e-8676-4819-aae2-68bf9cd4ea37") : secret "router-metrics-certs-default" not found
Jul 16 22:34:46.783224 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:46.783079    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-service-ca-bundle podName:4a28ac1e-8676-4819-aae2-68bf9cd4ea37 nodeName:}" failed. No retries permitted until 2026-07-16 22:34:50.783073091 +0000 UTC m=+64.793335733 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-service-ca-bundle") pod "router-default-6669f9958b-wkhdn" (UID: "4a28ac1e-8676-4819-aae2-68bf9cd4ea37") : configmap references non-existent config key: service-ca.crt
Jul 16 22:34:46.830422 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:46.830393    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:47.170085 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:47.170055    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:47.380882 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:47.380844    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:47.477526 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:47.477464    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:47.662822 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:47.662799    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r9jm4"
Jul 16 22:34:47.708173 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:47.708128    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-mtdx8" podStartSLOduration=4.125508656 podStartE2EDuration="5.708113614s" totalImagesPullingTime="1.582604958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:34:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:34:45.76809067 +0000 UTC m=+59.778353333" watchObservedRunningTime="2026-07-16 22:34:47.708113614 +0000 UTC m=+61.718376321"
Jul 16 22:34:47.712941 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:47.712919    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df849ddbc-q4j65"]
Jul 16 22:34:47.715282 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:47.715268    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df849ddbc-q4j65"
Jul 16 22:34:47.718383 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:47.718367    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 16 22:34:47.718701 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:47.718685    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 16 22:34:47.718987 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:47.718971    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 16 22:34:47.719162 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:47.719147    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 16 22:34:47.719450 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:47.719437    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 16 22:34:47.719982 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:47.719968    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 16 22:34:47.721566 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:47.721176    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 16 22:34:47.730727 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:47.730682    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df849ddbc-q4j65"]
Jul 16 22:34:47.845116 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:47.845086    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:47.890627 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:47.890597    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6d255e84-6012-48c3-8ec4-cb68205873a8-hub\") pod \"cluster-proxy-proxy-agent-7df849ddbc-q4j65\" (UID: \"6d255e84-6012-48c3-8ec4-cb68205873a8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df849ddbc-q4j65"
Jul 16 22:34:47.890742 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:47.890629    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d255e84-6012-48c3-8ec4-cb68205873a8-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7df849ddbc-q4j65\" (UID: \"6d255e84-6012-48c3-8ec4-cb68205873a8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df849ddbc-q4j65"
Jul 16 22:34:47.890784 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:47.890737    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6d255e84-6012-48c3-8ec4-cb68205873a8-ca\") pod \"cluster-proxy-proxy-agent-7df849ddbc-q4j65\" (UID: \"6d255e84-6012-48c3-8ec4-cb68205873a8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df849ddbc-q4j65"
Jul 16 22:34:47.890784 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:47.890766    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6d255e84-6012-48c3-8ec4-cb68205873a8-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7df849ddbc-q4j65\" (UID: \"6d255e84-6012-48c3-8ec4-cb68205873a8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df849ddbc-q4j65"
Jul 16 22:34:47.890845 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:47.890819    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6d255e84-6012-48c3-8ec4-cb68205873a8-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7df849ddbc-q4j65\" (UID: \"6d255e84-6012-48c3-8ec4-cb68205873a8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df849ddbc-q4j65"
Jul 16 22:34:47.890914 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:47.890900    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4dj9\" (UniqueName: \"kubernetes.io/projected/6d255e84-6012-48c3-8ec4-cb68205873a8-kube-api-access-s4dj9\") pod \"cluster-proxy-proxy-agent-7df849ddbc-q4j65\" (UID: \"6d255e84-6012-48c3-8ec4-cb68205873a8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df849ddbc-q4j65"
Jul 16 22:34:47.946247 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:47.946204    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:47.991488 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:47.991423    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6d255e84-6012-48c3-8ec4-cb68205873a8-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7df849ddbc-q4j65\" (UID: \"6d255e84-6012-48c3-8ec4-cb68205873a8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df849ddbc-q4j65"
Jul 16 22:34:47.991600 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:47.991509    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4dj9\" (UniqueName: \"kubernetes.io/projected/6d255e84-6012-48c3-8ec4-cb68205873a8-kube-api-access-s4dj9\") pod \"cluster-proxy-proxy-agent-7df849ddbc-q4j65\" (UID: \"6d255e84-6012-48c3-8ec4-cb68205873a8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df849ddbc-q4j65"
Jul 16 22:34:47.991600 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:47.991540    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6d255e84-6012-48c3-8ec4-cb68205873a8-hub\") pod \"cluster-proxy-proxy-agent-7df849ddbc-q4j65\" (UID: \"6d255e84-6012-48c3-8ec4-cb68205873a8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df849ddbc-q4j65"
Jul 16 22:34:47.991600 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:47.991576    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d255e84-6012-48c3-8ec4-cb68205873a8-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7df849ddbc-q4j65\" (UID: \"6d255e84-6012-48c3-8ec4-cb68205873a8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df849ddbc-q4j65"
Jul 16 22:34:47.991755 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:47.991660    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6d255e84-6012-48c3-8ec4-cb68205873a8-ca\") pod \"cluster-proxy-proxy-agent-7df849ddbc-q4j65\" (UID: \"6d255e84-6012-48c3-8ec4-cb68205873a8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df849ddbc-q4j65"
Jul 16 22:34:47.991755 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:47.991687    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6d255e84-6012-48c3-8ec4-cb68205873a8-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7df849ddbc-q4j65\" (UID: \"6d255e84-6012-48c3-8ec4-cb68205873a8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df849ddbc-q4j65"
Jul 16 22:34:47.992463 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:47.992439    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d255e84-6012-48c3-8ec4-cb68205873a8-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7df849ddbc-q4j65\" (UID: \"6d255e84-6012-48c3-8ec4-cb68205873a8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df849ddbc-q4j65"
Jul 16 22:34:47.993986 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:47.993963    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6d255e84-6012-48c3-8ec4-cb68205873a8-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7df849ddbc-q4j65\" (UID: \"6d255e84-6012-48c3-8ec4-cb68205873a8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df849ddbc-q4j65"
Jul 16 22:34:47.994075 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:47.994045    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6d255e84-6012-48c3-8ec4-cb68205873a8-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7df849ddbc-q4j65\" (UID: \"6d255e84-6012-48c3-8ec4-cb68205873a8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df849ddbc-q4j65"
Jul 16 22:34:47.994364 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:47.994340    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6d255e84-6012-48c3-8ec4-cb68205873a8-ca\") pod \"cluster-proxy-proxy-agent-7df849ddbc-q4j65\" (UID: \"6d255e84-6012-48c3-8ec4-cb68205873a8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df849ddbc-q4j65"
Jul 16 22:34:47.994565 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:47.994546    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6d255e84-6012-48c3-8ec4-cb68205873a8-hub\") pod \"cluster-proxy-proxy-agent-7df849ddbc-q4j65\" (UID: \"6d255e84-6012-48c3-8ec4-cb68205873a8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df849ddbc-q4j65"
Jul 16 22:34:48.000188 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:48.000166    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4dj9\" (UniqueName: \"kubernetes.io/projected/6d255e84-6012-48c3-8ec4-cb68205873a8-kube-api-access-s4dj9\") pod \"cluster-proxy-proxy-agent-7df849ddbc-q4j65\" (UID: \"6d255e84-6012-48c3-8ec4-cb68205873a8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df849ddbc-q4j65"
Jul 16 22:34:48.035317 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:48.035294    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df849ddbc-q4j65"
Jul 16 22:34:48.160417 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:48.160389    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df849ddbc-q4j65"]
Jul 16 22:34:48.163334 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:34:48.163295    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d255e84_6012_48c3_8ec4_cb68205873a8.slice/crio-e1a369c659397d81da4ebfad0ba45af6f6196bc91c8952981c60809112b08e89 WatchSource:0}: Error finding container e1a369c659397d81da4ebfad0ba45af6f6196bc91c8952981c60809112b08e89: Status 404 returned error can't find the container with id e1a369c659397d81da4ebfad0ba45af6f6196bc91c8952981c60809112b08e89
Jul 16 22:34:48.746017 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:48.745979    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df849ddbc-q4j65" event={"ID":"6d255e84-6012-48c3-8ec4-cb68205873a8","Type":"ContainerStarted","Data":"e1a369c659397d81da4ebfad0ba45af6f6196bc91c8952981c60809112b08e89"}
Jul 16 22:34:48.980522 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:48.979943    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-9qdl5"]
Jul 16 22:34:48.982860 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:48.982322    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-9qdl5"
Jul 16 22:34:48.985427 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:48.985198    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 16 22:34:48.985427 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:48.985223    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 16 22:34:48.986201 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:48.985970    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 16 22:34:48.986201 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:48.985982    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-t8dkl\""
Jul 16 22:34:48.986201 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:48.985986    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 16 22:34:48.996737 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:48.996655    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-9qdl5"]
Jul 16 22:34:49.020519 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:49.020495    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:49.100717 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:49.100624    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szrwp\" (UniqueName: \"kubernetes.io/projected/ce666ec1-77a0-41bb-89a6-1bf0d5c77794-kube-api-access-szrwp\") pod \"service-ca-58598695f7-9qdl5\" (UID: \"ce666ec1-77a0-41bb-89a6-1bf0d5c77794\") " pod="openshift-service-ca/service-ca-58598695f7-9qdl5"
Jul 16 22:34:49.100865 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:49.100829    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ce666ec1-77a0-41bb-89a6-1bf0d5c77794-signing-key\") pod \"service-ca-58598695f7-9qdl5\" (UID: \"ce666ec1-77a0-41bb-89a6-1bf0d5c77794\") " pod="openshift-service-ca/service-ca-58598695f7-9qdl5"
Jul 16 22:34:49.100934 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:49.100872    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ce666ec1-77a0-41bb-89a6-1bf0d5c77794-signing-cabundle\") pod \"service-ca-58598695f7-9qdl5\" (UID: \"ce666ec1-77a0-41bb-89a6-1bf0d5c77794\") " pod="openshift-service-ca/service-ca-58598695f7-9qdl5"
Jul 16 22:34:49.202154 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:49.202119    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ce666ec1-77a0-41bb-89a6-1bf0d5c77794-signing-key\") pod \"service-ca-58598695f7-9qdl5\" (UID: \"ce666ec1-77a0-41bb-89a6-1bf0d5c77794\") " pod="openshift-service-ca/service-ca-58598695f7-9qdl5"
Jul 16 22:34:49.202335 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:49.202166    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ce666ec1-77a0-41bb-89a6-1bf0d5c77794-signing-cabundle\") pod \"service-ca-58598695f7-9qdl5\" (UID: \"ce666ec1-77a0-41bb-89a6-1bf0d5c77794\") " pod="openshift-service-ca/service-ca-58598695f7-9qdl5"
Jul 16 22:34:49.202335 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:49.202230    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szrwp\" (UniqueName: \"kubernetes.io/projected/ce666ec1-77a0-41bb-89a6-1bf0d5c77794-kube-api-access-szrwp\") pod \"service-ca-58598695f7-9qdl5\" (UID: \"ce666ec1-77a0-41bb-89a6-1bf0d5c77794\") " pod="openshift-service-ca/service-ca-58598695f7-9qdl5"
Jul 16 22:34:49.203207 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:49.203161    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ce666ec1-77a0-41bb-89a6-1bf0d5c77794-signing-cabundle\") pod \"service-ca-58598695f7-9qdl5\" (UID: \"ce666ec1-77a0-41bb-89a6-1bf0d5c77794\") " pod="openshift-service-ca/service-ca-58598695f7-9qdl5"
Jul 16 22:34:49.205526 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:49.205463    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ce666ec1-77a0-41bb-89a6-1bf0d5c77794-signing-key\") pod \"service-ca-58598695f7-9qdl5\" (UID: \"ce666ec1-77a0-41bb-89a6-1bf0d5c77794\") " pod="openshift-service-ca/service-ca-58598695f7-9qdl5"
Jul 16 22:34:49.212203 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:49.212177    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szrwp\" (UniqueName: \"kubernetes.io/projected/ce666ec1-77a0-41bb-89a6-1bf0d5c77794-kube-api-access-szrwp\") pod \"service-ca-58598695f7-9qdl5\" (UID: \"ce666ec1-77a0-41bb-89a6-1bf0d5c77794\") " pod="openshift-service-ca/service-ca-58598695f7-9qdl5"
Jul 16 22:34:49.296686 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:49.296604    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-9qdl5"
Jul 16 22:34:49.447759 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:49.447722    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-9qdl5"]
Jul 16 22:34:49.451766 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:34:49.451738    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce666ec1_77a0_41bb_89a6_1bf0d5c77794.slice/crio-c7ecec8c4f688bb041de8f13a5fcdd3641706b4e3b97678d3992bf4eb77ffd32 WatchSource:0}: Error finding container c7ecec8c4f688bb041de8f13a5fcdd3641706b4e3b97678d3992bf4eb77ffd32: Status 404 returned error can't find the container with id c7ecec8c4f688bb041de8f13a5fcdd3641706b4e3b97678d3992bf4eb77ffd32
Jul 16 22:34:49.751807 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:49.751692    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-9qdl5" event={"ID":"ce666ec1-77a0-41bb-89a6-1bf0d5c77794","Type":"ContainerStarted","Data":"a64f3c76da4289b51fe5def1eea093b69587d3c44984d18dcdfce55900fc673d"}
Jul 16 22:34:49.751807 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:49.751749    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-9qdl5" event={"ID":"ce666ec1-77a0-41bb-89a6-1bf0d5c77794","Type":"ContainerStarted","Data":"c7ecec8c4f688bb041de8f13a5fcdd3641706b4e3b97678d3992bf4eb77ffd32"}
Jul 16 22:34:49.774264 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:49.774167    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-9qdl5" podStartSLOduration=1.774149799 podStartE2EDuration="1.774149799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:34:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:34:49.773890938 +0000 UTC m=+63.784153628" watchObservedRunningTime="2026-07-16 22:34:49.774149799 +0000 UTC m=+63.784412464"
Jul 16 22:34:50.142340 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:50.142299    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:50.716756 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:50.716718    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3244b5d6-318c-423d-844c-9d93d94ea9d8-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-zvtkr\" (UID: \"3244b5d6-318c-423d-844c-9d93d94ea9d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-zvtkr"
Jul 16 22:34:50.716939 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:50.716879    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 16 22:34:50.716998 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:50.716956    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3244b5d6-318c-423d-844c-9d93d94ea9d8-samples-operator-tls podName:3244b5d6-318c-423d-844c-9d93d94ea9d8 nodeName:}" failed. No retries permitted until 2026-07-16 22:34:58.716940243 +0000 UTC m=+72.727202885 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3244b5d6-318c-423d-844c-9d93d94ea9d8-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-zvtkr" (UID: "3244b5d6-318c-423d-844c-9d93d94ea9d8") : secret "samples-operator-tls" not found
Jul 16 22:34:50.818384 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:50.818340    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-service-ca-bundle\") pod \"router-default-6669f9958b-wkhdn\" (UID: \"4a28ac1e-8676-4819-aae2-68bf9cd4ea37\") " pod="openshift-ingress/router-default-6669f9958b-wkhdn"
Jul 16 22:34:50.818561 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:50.818506    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-service-ca-bundle podName:4a28ac1e-8676-4819-aae2-68bf9cd4ea37 nodeName:}" failed. No retries permitted until 2026-07-16 22:34:58.81848273 +0000 UTC m=+72.828745373 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-service-ca-bundle") pod "router-default-6669f9958b-wkhdn" (UID: "4a28ac1e-8676-4819-aae2-68bf9cd4ea37") : configmap references non-existent config key: service-ca.crt
Jul 16 22:34:50.818648 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:50.818582    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-metrics-certs\") pod \"router-default-6669f9958b-wkhdn\" (UID: \"4a28ac1e-8676-4819-aae2-68bf9cd4ea37\") " pod="openshift-ingress/router-default-6669f9958b-wkhdn"
Jul 16 22:34:50.818707 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:50.818692    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 16 22:34:50.818781 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:50.818768    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-metrics-certs podName:4a28ac1e-8676-4819-aae2-68bf9cd4ea37 nodeName:}" failed. No retries permitted until 2026-07-16 22:34:58.818750275 +0000 UTC m=+72.829012934 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-metrics-certs") pod "router-default-6669f9958b-wkhdn" (UID: "4a28ac1e-8676-4819-aae2-68bf9cd4ea37") : secret "router-metrics-certs-default" not found
Jul 16 22:34:51.130065 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:51.130028    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:51.207436 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:51.207398    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:51.562596 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:51.561933    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:51.817348 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:51.817271    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:51.936892 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:51.936820    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:52.027491 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:52.027425    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:52.129568 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:52.129533    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:52.132653 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:52.132624    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c8d410fb-20ac-4175-a4b4-38d5b44636d6-original-pull-secret\") pod \"global-pull-secret-syncer-kxpfx\" (UID: \"c8d410fb-20ac-4175-a4b4-38d5b44636d6\") " pod="kube-system/global-pull-secret-syncer-kxpfx"
Jul 16 22:34:52.135403 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:52.135384    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 16 22:34:52.145670 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:52.145648    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c8d410fb-20ac-4175-a4b4-38d5b44636d6-original-pull-secret\") pod \"global-pull-secret-syncer-kxpfx\" (UID: \"c8d410fb-20ac-4175-a4b4-38d5b44636d6\") " pod="kube-system/global-pull-secret-syncer-kxpfx"
Jul 16 22:34:52.227810 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:52.227783    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kxpfx"
Jul 16 22:34:52.233411 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:52.233387    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjgdw\" (UniqueName: \"kubernetes.io/projected/bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef-kube-api-access-vjgdw\") pod \"network-check-target-7nqtn\" (UID: \"bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef\") " pod="openshift-network-diagnostics/network-check-target-7nqtn"
Jul 16 22:34:52.233513 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:52.233442    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b332999-321a-4e3f-99bf-19517a268e28-metrics-certs\") pod \"network-metrics-daemon-6xfmc\" (UID: \"9b332999-321a-4e3f-99bf-19517a268e28\") " pod="openshift-multus/network-metrics-daemon-6xfmc"
Jul 16 22:34:52.245276 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:52.243839    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 16 22:34:52.245276 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:52.244092    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 16 22:34:52.245276 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:52.244279    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 16 22:34:52.245276 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:52.244340    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b332999-321a-4e3f-99bf-19517a268e28-metrics-certs podName:9b332999-321a-4e3f-99bf-19517a268e28 nodeName:}" failed. No retries permitted until 2026-07-16 22:35:56.244319991 +0000 UTC m=+130.254582649 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9b332999-321a-4e3f-99bf-19517a268e28-metrics-certs") pod "network-metrics-daemon-6xfmc" (UID: "9b332999-321a-4e3f-99bf-19517a268e28") : secret "metrics-daemon-secret" not found
Jul 16 22:34:52.246074 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:52.245964    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:52.256469 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:52.256271    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 16 22:34:52.277394 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:52.277212    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjgdw\" (UniqueName: \"kubernetes.io/projected/bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef-kube-api-access-vjgdw\") pod \"network-check-target-7nqtn\" (UID: \"bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef\") " pod="openshift-network-diagnostics/network-check-target-7nqtn"
Jul 16 22:34:52.416759 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:52.416553    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kxpfx"]
Jul 16 22:34:52.419840 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:34:52.419806    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8d410fb_20ac_4175_a4b4_38d5b44636d6.slice/crio-77c5bd1d087fbffb54c45ac1f75329ce80a5b25f4cd5a734578464835e47a9ba WatchSource:0}: Error finding container 77c5bd1d087fbffb54c45ac1f75329ce80a5b25f4cd5a734578464835e47a9ba: Status 404 returned error can't find the container with id 77c5bd1d087fbffb54c45ac1f75329ce80a5b25f4cd5a734578464835e47a9ba
Jul 16 22:34:52.430998 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:52.430834    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jdr86\""
Jul 16 22:34:52.438363 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:52.438219    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7nqtn"
Jul 16 22:34:52.500809 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:52.500751    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:52.588488 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:52.588459    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7nqtn"]
Jul 16 22:34:52.591390 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:34:52.591354    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbec212bf_6a2b_4c1e_bd0e_405bfe87b5ef.slice/crio-8e53e7e328a705eb5d0b680dec2b8ce6a671263143880f5919e21ada5688fe8c WatchSource:0}: Error finding container 8e53e7e328a705eb5d0b680dec2b8ce6a671263143880f5919e21ada5688fe8c: Status 404 returned error can't find the container with id 8e53e7e328a705eb5d0b680dec2b8ce6a671263143880f5919e21ada5688fe8c
Jul 16 22:34:52.592614 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:52.592586    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:52.657515 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:52.657463    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:52.762675 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:52.762648    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7nqtn" event={"ID":"bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef","Type":"ContainerStarted","Data":"8e53e7e328a705eb5d0b680dec2b8ce6a671263143880f5919e21ada5688fe8c"}
Jul 16 22:34:52.763847 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:52.763814    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kxpfx" event={"ID":"c8d410fb-20ac-4175-a4b4-38d5b44636d6","Type":"ContainerStarted","Data":"77c5bd1d087fbffb54c45ac1f75329ce80a5b25f4cd5a734578464835e47a9ba"}
Jul 16 22:34:52.765608 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:52.765586    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df849ddbc-q4j65" event={"ID":"6d255e84-6012-48c3-8ec4-cb68205873a8","Type":"ContainerStarted","Data":"7e9cf7f6e027841bc47e80794d09deb886809a5a8395ee9bcf661e7b2f5b32e1"}
Jul 16 22:34:52.765693 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:52.765622    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df849ddbc-q4j65" event={"ID":"6d255e84-6012-48c3-8ec4-cb68205873a8","Type":"ContainerStarted","Data":"fce097826719956d1c11f7fca8fc6cd963d07fa87467c9b6e66d658626e397b7"}
Jul 16 22:34:52.765693 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:52.765639    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df849ddbc-q4j65" event={"ID":"6d255e84-6012-48c3-8ec4-cb68205873a8","Type":"ContainerStarted","Data":"6a3281cd5e0af8ca5e173b7c3f8de6433cc95e274fc86fce91d9c66ae4796daf"}
Jul 16 22:34:52.786194 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:52.786154    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df849ddbc-q4j65" podStartSLOduration=1.773339593 podStartE2EDuration="5.786141337s" totalImagesPullingTime="4.012801744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:34:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:34:52.785392534 +0000 UTC m=+66.795655197" watchObservedRunningTime="2026-07-16 22:34:52.786141337 +0000 UTC m=+66.796404001"
Jul 16 22:34:52.888141 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:52.888116    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8bst9_268b78b0-10f6-44ce-a822-a124771df729/dns-node-resolver/0.log"
Jul 16 22:34:53.239059 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:53.239000    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53ae108c-2ec1-4c24-b756-4a899fbcf19d-metrics-tls\") pod \"dns-default-nbc9n\" (UID: \"53ae108c-2ec1-4c24-b756-4a899fbcf19d\") " pod="openshift-dns/dns-default-nbc9n"
Jul 16 22:34:53.239059 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:53.239024    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 22:34:53.239566 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:34:53.239183    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ae108c-2ec1-4c24-b756-4a899fbcf19d-metrics-tls podName:53ae108c-2ec1-4c24-b756-4a899fbcf19d nodeName:}" failed. No retries permitted until 2026-07-16 22:35:25.239163872 +0000 UTC m=+99.249426517 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/53ae108c-2ec1-4c24-b756-4a899fbcf19d-metrics-tls") pod "dns-default-nbc9n" (UID: "53ae108c-2ec1-4c24-b756-4a899fbcf19d") : secret "dns-default-metrics-tls" not found
Jul 16 22:34:53.671737 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:53.671703    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:53.749912 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:53.749877    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:54.057620 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:54.057511    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:54.261812 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:54.261744    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:54.287563 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:54.287539    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-x4th8_1d026b73-92ba-465b-bc31-5843395ae3e4/node-ca/0.log"
Jul 16 22:34:54.487909 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:54.487039    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:54.689085 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:54.689051    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-btltg_93773e2d-fdc2-4715-825e-ea777c6e6916/serve-healthcheck-canary/0.log"
Jul 16 22:34:54.858248 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:54.858212    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:55.257636 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:55.257543    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:55.406268 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:55.406216    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:55.504941 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:55.504909    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:55.860887 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:55.860854    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:56.249226 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:56.249144    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:56.309030 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:56.308993    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:56.460776 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:56.460732    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:56.653780 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:56.653747    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:57.660225 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:57.660187    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:57.764007 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:57.763982    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:57.786122 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:57.786093    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7nqtn" event={"ID":"bec212bf-6a2b-4c1e-bd0e-405bfe87b5ef","Type":"ContainerStarted","Data":"d16e117e29d29c85d563d1b43c7ba614a9777ada2269b32a48b7a7f42ffa5051"}
Jul 16 22:34:57.786297 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:57.786188    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7nqtn"
Jul 16 22:34:57.787528 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:57.787504    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kxpfx" event={"ID":"c8d410fb-20ac-4175-a4b4-38d5b44636d6","Type":"ContainerStarted","Data":"9f16855ee1f999bdf65c6e0bec40d8bbbf686b8c2044b7c3020db00ac063dc61"}
Jul 16 22:34:57.803362 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:57.803323    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7nqtn" podStartSLOduration=67.374881632 podStartE2EDuration="1m11.803311356s" totalImagesPullingTime="4.428429724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:33:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:34:57.802632616 +0000 UTC m=+71.812895279" watchObservedRunningTime="2026-07-16 22:34:57.803311356 +0000 UTC m=+71.813574020"
Jul 16 22:34:57.817758 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:57.817720    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-kxpfx" podStartSLOduration=67.215004476 podStartE2EDuration="1m11.817706538s" totalImagesPullingTime="4.602702062s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:33:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:34:57.816941851 +0000 UTC m=+71.827204513" watchObservedRunningTime="2026-07-16 22:34:57.817706538 +0000 UTC m=+71.827969201"
Jul 16 22:34:57.865141 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:57.865101    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:58.037520 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:58.037438    2573 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df849ddbc-q4j65" podUID="6d255e84-6012-48c3-8ec4-cb68205873a8" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 16 22:34:58.258672 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:58.258640    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:58.450720 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:58.450689    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:58.780560 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:58.780471    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3244b5d6-318c-423d-844c-9d93d94ea9d8-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-zvtkr\" (UID: \"3244b5d6-318c-423d-844c-9d93d94ea9d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-zvtkr"
Jul 16 22:34:58.782875 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:58.782849    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3244b5d6-318c-423d-844c-9d93d94ea9d8-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-zvtkr\" (UID: \"3244b5d6-318c-423d-844c-9d93d94ea9d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-zvtkr"
Jul 16 22:34:58.816152 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:58.816117    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-p84ss\""
Jul 16 22:34:58.824293 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:58.824269    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-zvtkr"
Jul 16 22:34:58.857021 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:58.856986    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:58.881479 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:58.881408    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-metrics-certs\") pod \"router-default-6669f9958b-wkhdn\" (UID: \"4a28ac1e-8676-4819-aae2-68bf9cd4ea37\") " pod="openshift-ingress/router-default-6669f9958b-wkhdn"
Jul 16 22:34:58.881479 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:58.881478    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-service-ca-bundle\") pod \"router-default-6669f9958b-wkhdn\" (UID: \"4a28ac1e-8676-4819-aae2-68bf9cd4ea37\") " pod="openshift-ingress/router-default-6669f9958b-wkhdn"
Jul 16 22:34:58.882200 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:58.882178    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-service-ca-bundle\") pod \"router-default-6669f9958b-wkhdn\" (UID: \"4a28ac1e-8676-4819-aae2-68bf9cd4ea37\") " pod="openshift-ingress/router-default-6669f9958b-wkhdn"
Jul 16 22:34:58.884635 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:58.884614    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a28ac1e-8676-4819-aae2-68bf9cd4ea37-metrics-certs\") pod \"router-default-6669f9958b-wkhdn\" (UID: \"4a28ac1e-8676-4819-aae2-68bf9cd4ea37\") " pod="openshift-ingress/router-default-6669f9958b-wkhdn"
Jul 16 22:34:58.921781 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:58.921756    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-qjslv\""
Jul 16 22:34:58.929564 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:58.929530    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6669f9958b-wkhdn"
Jul 16 22:34:58.955741 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:58.955680    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-zvtkr"]
Jul 16 22:34:59.072872 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:59.072732    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6669f9958b-wkhdn"]
Jul 16 22:34:59.075217 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:34:59.075195    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a28ac1e_8676_4819_aae2_68bf9cd4ea37.slice/crio-3d96c66e21130ce86e7b8b12c1ddbc465135a0a3e2cb087ee22a666b18f592fe WatchSource:0}: Error finding container 3d96c66e21130ce86e7b8b12c1ddbc465135a0a3e2cb087ee22a666b18f592fe: Status 404 returned error can't find the container with id 3d96c66e21130ce86e7b8b12c1ddbc465135a0a3e2cb087ee22a666b18f592fe
Jul 16 22:34:59.252409 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:59.252380    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:59.795850 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:59.795811    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6669f9958b-wkhdn" event={"ID":"4a28ac1e-8676-4819-aae2-68bf9cd4ea37","Type":"ContainerStarted","Data":"8615a19b58b4fb92a6fa49e79c9984e178e0efd41c4e5ad97f6f2a0e6dcd27d9"}
Jul 16 22:34:59.796268 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:59.795873    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6669f9958b-wkhdn" event={"ID":"4a28ac1e-8676-4819-aae2-68bf9cd4ea37","Type":"ContainerStarted","Data":"3d96c66e21130ce86e7b8b12c1ddbc465135a0a3e2cb087ee22a666b18f592fe"}
Jul 16 22:34:59.797072 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:59.797044    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-zvtkr" event={"ID":"3244b5d6-318c-423d-844c-9d93d94ea9d8","Type":"ContainerStarted","Data":"e229a96b54e4e494154d4d72ad6a7e29eecfa3de2140aa43d18799a3dc1ea2bb"}
Jul 16 22:34:59.822745 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:59.822691    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6669f9958b-wkhdn" podStartSLOduration=17.822671919 podStartE2EDuration="17.822671919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:34:59.820680588 +0000 UTC m=+73.830943275" watchObservedRunningTime="2026-07-16 22:34:59.822671919 +0000 UTC m=+73.832934564"
Jul 16 22:34:59.891007 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:59.890976    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:59.929956 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:59.929919    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6669f9958b-wkhdn"
Jul 16 22:34:59.932386 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:34:59.932363    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6669f9958b-wkhdn"
Jul 16 22:35:00.260904 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:00.260867    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:00.361536 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:00.361495    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:00.759884 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:00.759847    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:00.808023 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:00.807936    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-zvtkr" event={"ID":"3244b5d6-318c-423d-844c-9d93d94ea9d8","Type":"ContainerStarted","Data":"7bcbc87fe45c34375b4ae9ab15ad9c7d7d7e7646b86dca133e25e30372ac2f2f"}
Jul 16 22:35:00.808475 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:00.808024    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-zvtkr" event={"ID":"3244b5d6-318c-423d-844c-9d93d94ea9d8","Type":"ContainerStarted","Data":"bf5866f37b86e941859ce6c16a29b575eb05066ce66dd175763de67b76cca081"}
Jul 16 22:35:00.808475 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:00.808173    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6669f9958b-wkhdn"
Jul 16 22:35:00.809507 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:00.809475    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6669f9958b-wkhdn"
Jul 16 22:35:00.831614 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:00.831563    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-zvtkr" podStartSLOduration=17.332174134 podStartE2EDuration="18.831546462s" totalImagesPullingTime="1.499372328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:34:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:35:00.831381255 +0000 UTC m=+74.841643919" watchObservedRunningTime="2026-07-16 22:35:00.831546462 +0000 UTC m=+74.841809126"
Jul 16 22:35:00.863107 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:00.863075    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:00.953066 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:00.953033    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:01.177089 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:01.177055    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:01.278445 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:01.278391    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:01.954600 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:01.954565    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:02.058419 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:02.058382    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:02.154421 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:02.154387    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:02.491019 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:02.490985    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:02.658224 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:02.658187    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:08.036802 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:08.036760    2573 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df849ddbc-q4j65" podUID="6d255e84-6012-48c3-8ec4-cb68205873a8" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 16 22:35:11.066229 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:11.066190    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7qrwd"]
Jul 16 22:35:11.071079 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:11.071048    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7qrwd"
Jul 16 22:35:11.082798 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:11.082773    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lklv7\""
Jul 16 22:35:11.082931 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:11.082820    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 16 22:35:11.082931 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:11.082820    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 16 22:35:11.082931 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:11.082873    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 16 22:35:11.082931 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:11.082824    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 16 22:35:11.093948 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:11.093926    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7qrwd"]
Jul 16 22:35:11.170643 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:11.170605    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/420b20df-d1c4-4c01-afc1-bee33dc7ba03-data-volume\") pod \"insights-runtime-extractor-7qrwd\" (UID: \"420b20df-d1c4-4c01-afc1-bee33dc7ba03\") " pod="openshift-insights/insights-runtime-extractor-7qrwd"
Jul 16 22:35:11.170815 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:11.170652    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/420b20df-d1c4-4c01-afc1-bee33dc7ba03-crio-socket\") pod \"insights-runtime-extractor-7qrwd\" (UID: \"420b20df-d1c4-4c01-afc1-bee33dc7ba03\") " pod="openshift-insights/insights-runtime-extractor-7qrwd"
Jul 16 22:35:11.170815 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:11.170775    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ghjz\" (UniqueName: \"kubernetes.io/projected/420b20df-d1c4-4c01-afc1-bee33dc7ba03-kube-api-access-7ghjz\") pod \"insights-runtime-extractor-7qrwd\" (UID: \"420b20df-d1c4-4c01-afc1-bee33dc7ba03\") " pod="openshift-insights/insights-runtime-extractor-7qrwd"
Jul 16 22:35:11.170888 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:11.170824    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/420b20df-d1c4-4c01-afc1-bee33dc7ba03-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7qrwd\" (UID: \"420b20df-d1c4-4c01-afc1-bee33dc7ba03\") " pod="openshift-insights/insights-runtime-extractor-7qrwd"
Jul 16 22:35:11.170888 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:11.170850    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/420b20df-d1c4-4c01-afc1-bee33dc7ba03-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7qrwd\" (UID: \"420b20df-d1c4-4c01-afc1-bee33dc7ba03\") " pod="openshift-insights/insights-runtime-extractor-7qrwd"
Jul 16 22:35:11.231542 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:11.231503    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:11.271570 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:11.271474    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/420b20df-d1c4-4c01-afc1-bee33dc7ba03-crio-socket\") pod \"insights-runtime-extractor-7qrwd\" (UID: \"420b20df-d1c4-4c01-afc1-bee33dc7ba03\") " pod="openshift-insights/insights-runtime-extractor-7qrwd"
Jul 16 22:35:11.271570 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:11.271558    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ghjz\" (UniqueName: \"kubernetes.io/projected/420b20df-d1c4-4c01-afc1-bee33dc7ba03-kube-api-access-7ghjz\") pod \"insights-runtime-extractor-7qrwd\" (UID: \"420b20df-d1c4-4c01-afc1-bee33dc7ba03\") " pod="openshift-insights/insights-runtime-extractor-7qrwd"
Jul 16 22:35:11.271755 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:11.271603    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/420b20df-d1c4-4c01-afc1-bee33dc7ba03-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7qrwd\" (UID: \"420b20df-d1c4-4c01-afc1-bee33dc7ba03\") " pod="openshift-insights/insights-runtime-extractor-7qrwd"
Jul 16 22:35:11.271755 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:11.271635    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/420b20df-d1c4-4c01-afc1-bee33dc7ba03-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7qrwd\" (UID: \"420b20df-d1c4-4c01-afc1-bee33dc7ba03\") " pod="openshift-insights/insights-runtime-extractor-7qrwd"
Jul 16 22:35:11.271755 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:11.271698    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/420b20df-d1c4-4c01-afc1-bee33dc7ba03-crio-socket\") pod \"insights-runtime-extractor-7qrwd\" (UID: \"420b20df-d1c4-4c01-afc1-bee33dc7ba03\") " pod="openshift-insights/insights-runtime-extractor-7qrwd"
Jul 16 22:35:11.271844 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:11.271753    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/420b20df-d1c4-4c01-afc1-bee33dc7ba03-data-volume\") pod \"insights-runtime-extractor-7qrwd\" (UID: \"420b20df-d1c4-4c01-afc1-bee33dc7ba03\") " pod="openshift-insights/insights-runtime-extractor-7qrwd"
Jul 16 22:35:11.272107 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:11.272092    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/420b20df-d1c4-4c01-afc1-bee33dc7ba03-data-volume\") pod \"insights-runtime-extractor-7qrwd\" (UID: \"420b20df-d1c4-4c01-afc1-bee33dc7ba03\") " pod="openshift-insights/insights-runtime-extractor-7qrwd"
Jul 16 22:35:11.272214 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:11.272196    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/420b20df-d1c4-4c01-afc1-bee33dc7ba03-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7qrwd\" (UID: \"420b20df-d1c4-4c01-afc1-bee33dc7ba03\") " pod="openshift-insights/insights-runtime-extractor-7qrwd"
Jul 16 22:35:11.274006 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:11.273988    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/420b20df-d1c4-4c01-afc1-bee33dc7ba03-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7qrwd\" (UID: \"420b20df-d1c4-4c01-afc1-bee33dc7ba03\") " pod="openshift-insights/insights-runtime-extractor-7qrwd"
Jul 16 22:35:11.284089 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:11.284061    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ghjz\" (UniqueName: \"kubernetes.io/projected/420b20df-d1c4-4c01-afc1-bee33dc7ba03-kube-api-access-7ghjz\") pod \"insights-runtime-extractor-7qrwd\" (UID: \"420b20df-d1c4-4c01-afc1-bee33dc7ba03\") " pod="openshift-insights/insights-runtime-extractor-7qrwd"
Jul 16 22:35:11.370764 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:11.370725    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:11.382144 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:11.382102    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7qrwd"
Jul 16 22:35:11.563364 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:11.563273    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7qrwd"]
Jul 16 22:35:11.567639 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:35:11.567597    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420b20df_d1c4_4c01_afc1_bee33dc7ba03.slice/crio-1f81e216c824eee4dcede92674b00def1a5b86d62c968136afda3d1fa400e8cc WatchSource:0}: Error finding container 1f81e216c824eee4dcede92674b00def1a5b86d62c968136afda3d1fa400e8cc: Status 404 returned error can't find the container with id 1f81e216c824eee4dcede92674b00def1a5b86d62c968136afda3d1fa400e8cc
Jul 16 22:35:11.617790 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:11.617752    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:11.723772 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:11.723733    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:11.844892 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:11.844853    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7qrwd" event={"ID":"420b20df-d1c4-4c01-afc1-bee33dc7ba03","Type":"ContainerStarted","Data":"1f81e216c824eee4dcede92674b00def1a5b86d62c968136afda3d1fa400e8cc"}
Jul 16 22:35:13.378358 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:13.378315    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:13.549444 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:13.549408    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:15.175804 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:15.175762    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:15.334566 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:15.334522    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:15.490613 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:15.490535    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:17.419796 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:17.419701    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:17.523545 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:17.523498    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:17.874220 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:17.874118    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7qrwd" event={"ID":"420b20df-d1c4-4c01-afc1-bee33dc7ba03","Type":"ContainerStarted","Data":"51d5b00a9120f49c9b36bd744f54a5ad497bb0e0921dae2d10f1c19354343497"}
Jul 16 22:35:18.038814 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:18.038753    2573 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df849ddbc-q4j65" podUID="6d255e84-6012-48c3-8ec4-cb68205873a8" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 16 22:35:18.039007 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:18.038855    2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df849ddbc-q4j65"
Jul 16 22:35:18.039679 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:18.039632    2573 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"7e9cf7f6e027841bc47e80794d09deb886809a5a8395ee9bcf661e7b2f5b32e1"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df849ddbc-q4j65" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 16 22:35:18.039833 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:18.039706    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df849ddbc-q4j65" podUID="6d255e84-6012-48c3-8ec4-cb68205873a8" containerName="service-proxy" containerID="cri-o://7e9cf7f6e027841bc47e80794d09deb886809a5a8395ee9bcf661e7b2f5b32e1" gracePeriod=30
Jul 16 22:35:18.558421 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:18.558329    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:18.881205 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:18.881163    2573 generic.go:356] "Generic (PLEG): container finished" podID="6d255e84-6012-48c3-8ec4-cb68205873a8" containerID="7e9cf7f6e027841bc47e80794d09deb886809a5a8395ee9bcf661e7b2f5b32e1" exitCode=2
Jul 16 22:35:18.881501 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:18.881253    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df849ddbc-q4j65" event={"ID":"6d255e84-6012-48c3-8ec4-cb68205873a8","Type":"ContainerDied","Data":"7e9cf7f6e027841bc47e80794d09deb886809a5a8395ee9bcf661e7b2f5b32e1"}
Jul 16 22:35:18.881501 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:18.881310    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df849ddbc-q4j65" event={"ID":"6d255e84-6012-48c3-8ec4-cb68205873a8","Type":"ContainerStarted","Data":"4cdea835b42c496a452df26e4cb0ccf8646fcee3e1cd3af5adbb932d13f80cef"}
Jul 16 22:35:20.891221 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:20.891182    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7qrwd" event={"ID":"420b20df-d1c4-4c01-afc1-bee33dc7ba03","Type":"ContainerStarted","Data":"a8b1b3a7439f5d3496eb0535683f7fdb3eeb5b7ffb6b04ffa0b834f9201d3245"}
Jul 16 22:35:21.372730 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:21.372686    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:21.498951 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:21.498913    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:21.640112 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:21.640010    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:21.764939 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:21.764865    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:21.792780 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:21.792563    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fr7wp"]
Jul 16 22:35:21.795326 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:21.795296    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fr7wp"
Jul 16 22:35:21.799211 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:21.798425    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xww28\""
Jul 16 22:35:21.799211 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:21.798720    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 16 22:35:21.799211 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:21.799005    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 16 22:35:21.799211 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:21.799191    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 16 22:35:21.799546 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:21.799272    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 16 22:35:21.799546 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:21.799350    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 16 22:35:21.799546 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:21.799198    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 16 22:35:21.871514 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:21.871471    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b4e74e3-c3b1-40c8-9f0b-34f84325f497-node-exporter-tls\") pod \"node-exporter-fr7wp\" (UID: \"1b4e74e3-c3b1-40c8-9f0b-34f84325f497\") " pod="openshift-monitoring/node-exporter-fr7wp"
Jul 16 22:35:21.871756 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:21.871616    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b4e74e3-c3b1-40c8-9f0b-34f84325f497-metrics-client-ca\") pod \"node-exporter-fr7wp\" (UID: \"1b4e74e3-c3b1-40c8-9f0b-34f84325f497\") " pod="openshift-monitoring/node-exporter-fr7wp"
Jul 16 22:35:21.871756 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:21.871651    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b4e74e3-c3b1-40c8-9f0b-34f84325f497-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fr7wp\" (UID: \"1b4e74e3-c3b1-40c8-9f0b-34f84325f497\") " pod="openshift-monitoring/node-exporter-fr7wp"
Jul 16 22:35:21.871756 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:21.871683    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1b4e74e3-c3b1-40c8-9f0b-34f84325f497-root\") pod \"node-exporter-fr7wp\" (UID: \"1b4e74e3-c3b1-40c8-9f0b-34f84325f497\") " pod="openshift-monitoring/node-exporter-fr7wp"
Jul 16 22:35:21.871756 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:21.871712    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1b4e74e3-c3b1-40c8-9f0b-34f84325f497-node-exporter-textfile\") pod \"node-exporter-fr7wp\" (UID: \"1b4e74e3-c3b1-40c8-9f0b-34f84325f497\") " pod="openshift-monitoring/node-exporter-fr7wp"
Jul 16 22:35:21.871974 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:21.871780    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1b4e74e3-c3b1-40c8-9f0b-34f84325f497-node-exporter-wtmp\") pod \"node-exporter-fr7wp\" (UID: \"1b4e74e3-c3b1-40c8-9f0b-34f84325f497\") " pod="openshift-monitoring/node-exporter-fr7wp"
Jul 16 22:35:21.871974 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:21.871815    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1b4e74e3-c3b1-40c8-9f0b-34f84325f497-node-exporter-accelerators-collector-config\") pod \"node-exporter-fr7wp\" (UID: \"1b4e74e3-c3b1-40c8-9f0b-34f84325f497\") " pod="openshift-monitoring/node-exporter-fr7wp"
Jul 16 22:35:21.871974 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:21.871846    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzhq7\" (UniqueName: \"kubernetes.io/projected/1b4e74e3-c3b1-40c8-9f0b-34f84325f497-kube-api-access-dzhq7\") pod \"node-exporter-fr7wp\" (UID: \"1b4e74e3-c3b1-40c8-9f0b-34f84325f497\") " pod="openshift-monitoring/node-exporter-fr7wp"
Jul 16 22:35:21.871974 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:21.871905    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b4e74e3-c3b1-40c8-9f0b-34f84325f497-sys\") pod \"node-exporter-fr7wp\" (UID: \"1b4e74e3-c3b1-40c8-9f0b-34f84325f497\") " pod="openshift-monitoring/node-exporter-fr7wp"
Jul 16 22:35:21.974411 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:21.972438    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1b4e74e3-c3b1-40c8-9f0b-34f84325f497-root\") pod \"node-exporter-fr7wp\" (UID: \"1b4e74e3-c3b1-40c8-9f0b-34f84325f497\") " pod="openshift-monitoring/node-exporter-fr7wp"
Jul 16 22:35:21.974411 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:21.972498    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1b4e74e3-c3b1-40c8-9f0b-34f84325f497-node-exporter-textfile\") pod \"node-exporter-fr7wp\" (UID: \"1b4e74e3-c3b1-40c8-9f0b-34f84325f497\") " pod="openshift-monitoring/node-exporter-fr7wp"
Jul 16 22:35:21.974411 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:21.972544    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1b4e74e3-c3b1-40c8-9f0b-34f84325f497-node-exporter-wtmp\") pod \"node-exporter-fr7wp\" (UID: \"1b4e74e3-c3b1-40c8-9f0b-34f84325f497\") " pod="openshift-monitoring/node-exporter-fr7wp"
Jul 16 22:35:21.974411 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:21.972575    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1b4e74e3-c3b1-40c8-9f0b-34f84325f497-node-exporter-accelerators-collector-config\") pod \"node-exporter-fr7wp\" (UID: \"1b4e74e3-c3b1-40c8-9f0b-34f84325f497\") " pod="openshift-monitoring/node-exporter-fr7wp"
Jul 16 22:35:21.974411 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:21.972605    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzhq7\" (UniqueName: \"kubernetes.io/projected/1b4e74e3-c3b1-40c8-9f0b-34f84325f497-kube-api-access-dzhq7\") pod \"node-exporter-fr7wp\" (UID: \"1b4e74e3-c3b1-40c8-9f0b-34f84325f497\") " pod="openshift-monitoring/node-exporter-fr7wp"
Jul 16 22:35:21.974411 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:21.972638    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b4e74e3-c3b1-40c8-9f0b-34f84325f497-sys\") pod \"node-exporter-fr7wp\" (UID: \"1b4e74e3-c3b1-40c8-9f0b-34f84325f497\") " pod="openshift-monitoring/node-exporter-fr7wp"
Jul 16 22:35:21.974411 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:21.972682    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b4e74e3-c3b1-40c8-9f0b-34f84325f497-node-exporter-tls\") pod \"node-exporter-fr7wp\" (UID: \"1b4e74e3-c3b1-40c8-9f0b-34f84325f497\") " pod="openshift-monitoring/node-exporter-fr7wp"
Jul 16 22:35:21.974411 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:21.972782    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b4e74e3-c3b1-40c8-9f0b-34f84325f497-metrics-client-ca\") pod \"node-exporter-fr7wp\" (UID: \"1b4e74e3-c3b1-40c8-9f0b-34f84325f497\") " pod="openshift-monitoring/node-exporter-fr7wp"
Jul 16 22:35:21.974411 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:21.972813    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b4e74e3-c3b1-40c8-9f0b-34f84325f497-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fr7wp\" (UID: \"1b4e74e3-c3b1-40c8-9f0b-34f84325f497\") " pod="openshift-monitoring/node-exporter-fr7wp"
Jul 16 22:35:21.974411 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:21.972983    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1b4e74e3-c3b1-40c8-9f0b-34f84325f497-node-exporter-textfile\") pod \"node-exporter-fr7wp\" (UID: \"1b4e74e3-c3b1-40c8-9f0b-34f84325f497\") " pod="openshift-monitoring/node-exporter-fr7wp"
Jul 16 22:35:21.974411 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:21.973058    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1b4e74e3-c3b1-40c8-9f0b-34f84325f497-root\") pod \"node-exporter-fr7wp\" (UID: \"1b4e74e3-c3b1-40c8-9f0b-34f84325f497\") " pod="openshift-monitoring/node-exporter-fr7wp"
Jul 16 22:35:21.974411 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:21.973506    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1b4e74e3-c3b1-40c8-9f0b-34f84325f497-node-exporter-wtmp\") pod \"node-exporter-fr7wp\" (UID: \"1b4e74e3-c3b1-40c8-9f0b-34f84325f497\") " pod="openshift-monitoring/node-exporter-fr7wp"
Jul 16 22:35:21.974411 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:21.973816    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:21.974411 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:35:21.973980    2573 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 16 22:35:21.975645 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:35:21.974046    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b4e74e3-c3b1-40c8-9f0b-34f84325f497-node-exporter-tls podName:1b4e74e3-c3b1-40c8-9f0b-34f84325f497 nodeName:}" failed. No retries permitted until 2026-07-16 22:35:22.474025469 +0000 UTC m=+96.484288116 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/1b4e74e3-c3b1-40c8-9f0b-34f84325f497-node-exporter-tls") pod "node-exporter-fr7wp" (UID: "1b4e74e3-c3b1-40c8-9f0b-34f84325f497") : secret "node-exporter-tls" not found
Jul 16 22:35:21.975645 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:21.974291    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b4e74e3-c3b1-40c8-9f0b-34f84325f497-sys\") pod \"node-exporter-fr7wp\" (UID: \"1b4e74e3-c3b1-40c8-9f0b-34f84325f497\") " pod="openshift-monitoring/node-exporter-fr7wp"
Jul 16 22:35:21.975645 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:21.974290    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1b4e74e3-c3b1-40c8-9f0b-34f84325f497-node-exporter-accelerators-collector-config\") pod \"node-exporter-fr7wp\" (UID: \"1b4e74e3-c3b1-40c8-9f0b-34f84325f497\") " pod="openshift-monitoring/node-exporter-fr7wp"
Jul 16 22:35:21.975645 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:21.974830    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b4e74e3-c3b1-40c8-9f0b-34f84325f497-metrics-client-ca\") pod \"node-exporter-fr7wp\" (UID: \"1b4e74e3-c3b1-40c8-9f0b-34f84325f497\") " pod="openshift-monitoring/node-exporter-fr7wp"
Jul 16 22:35:21.976283 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:21.976254    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b4e74e3-c3b1-40c8-9f0b-34f84325f497-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fr7wp\" (UID: \"1b4e74e3-c3b1-40c8-9f0b-34f84325f497\") " pod="openshift-monitoring/node-exporter-fr7wp"
Jul 16 22:35:21.983956 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:21.983879    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzhq7\" (UniqueName: \"kubernetes.io/projected/1b4e74e3-c3b1-40c8-9f0b-34f84325f497-kube-api-access-dzhq7\") pod \"node-exporter-fr7wp\" (UID: \"1b4e74e3-c3b1-40c8-9f0b-34f84325f497\") " pod="openshift-monitoring/node-exporter-fr7wp"
Jul 16 22:35:22.078915 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:22.078814    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:22.221862 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:22.221828    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:22.322040 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:22.321953    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:22.425075 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:22.425021    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:22.479528 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:22.479483    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b4e74e3-c3b1-40c8-9f0b-34f84325f497-node-exporter-tls\") pod \"node-exporter-fr7wp\" (UID: \"1b4e74e3-c3b1-40c8-9f0b-34f84325f497\") " pod="openshift-monitoring/node-exporter-fr7wp"
Jul 16 22:35:22.482603 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:22.482569    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b4e74e3-c3b1-40c8-9f0b-34f84325f497-node-exporter-tls\") pod \"node-exporter-fr7wp\" (UID: \"1b4e74e3-c3b1-40c8-9f0b-34f84325f497\") " pod="openshift-monitoring/node-exporter-fr7wp"
Jul 16 22:35:22.602762 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:22.602722    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:22.695816 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:22.695776    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:22.708469 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:22.708430    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fr7wp"
Jul 16 22:35:22.814815 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:22.814779    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:23.452419 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:23.452377    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:23.576723 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:23.576684    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:24.179508 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:35:24.179472    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b4e74e3_c3b1_40c8_9f0b_34f84325f497.slice/crio-dfbc4d9dcf5e88bccc07e0e44fd8ac9efa9cda723c81b395a6009dd169f88e13 WatchSource:0}: Error finding container dfbc4d9dcf5e88bccc07e0e44fd8ac9efa9cda723c81b395a6009dd169f88e13: Status 404 returned error can't find the container with id dfbc4d9dcf5e88bccc07e0e44fd8ac9efa9cda723c81b395a6009dd169f88e13
Jul 16 22:35:24.610654 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:24.610597    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:24.711953 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:24.711915    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:24.813366 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:24.813325    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:24.910607 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:24.910509    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7qrwd" event={"ID":"420b20df-d1c4-4c01-afc1-bee33dc7ba03","Type":"ContainerStarted","Data":"433c16763cfd40cf29f0a614d845bd978ed11fa8f67bb3197cdd0a98601fb576"}
Jul 16 22:35:24.912091 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:24.912049    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fr7wp" event={"ID":"1b4e74e3-c3b1-40c8-9f0b-34f84325f497","Type":"ContainerStarted","Data":"dfbc4d9dcf5e88bccc07e0e44fd8ac9efa9cda723c81b395a6009dd169f88e13"}
Jul 16 22:35:24.929748 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:24.929693    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7qrwd" podStartSLOduration=1.416444465 podStartE2EDuration="13.929669984s" totalImagesPullingTime="12.513225519s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:35:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:35:24.928093889 +0000 UTC m=+98.938356568" watchObservedRunningTime="2026-07-16 22:35:24.929669984 +0000 UTC m=+98.939932653"
Jul 16 22:35:25.257821 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:25.257711    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:25.305039 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:25.304982    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53ae108c-2ec1-4c24-b756-4a899fbcf19d-metrics-tls\") pod \"dns-default-nbc9n\" (UID: \"53ae108c-2ec1-4c24-b756-4a899fbcf19d\") " pod="openshift-dns/dns-default-nbc9n"
Jul 16 22:35:25.307868 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:25.307833    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53ae108c-2ec1-4c24-b756-4a899fbcf19d-metrics-tls\") pod \"dns-default-nbc9n\" (UID: \"53ae108c-2ec1-4c24-b756-4a899fbcf19d\") " pod="openshift-dns/dns-default-nbc9n"
Jul 16 22:35:25.313579 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:25.313549    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-57zhl\""
Jul 16 22:35:25.321493 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:25.321457    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nbc9n"
Jul 16 22:35:25.514739 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:25.514647    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nbc9n"]
Jul 16 22:35:25.518025 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:35:25.517986    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53ae108c_2ec1_4c24_b756_4a899fbcf19d.slice/crio-88cc580a989b324d24da3767b608904bee55f79f46fa5dbbb993c6604e8a9818 WatchSource:0}: Error finding container 88cc580a989b324d24da3767b608904bee55f79f46fa5dbbb993c6604e8a9818: Status 404 returned error can't find the container with id 88cc580a989b324d24da3767b608904bee55f79f46fa5dbbb993c6604e8a9818
Jul 16 22:35:25.917403 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:25.917359    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fr7wp" event={"ID":"1b4e74e3-c3b1-40c8-9f0b-34f84325f497","Type":"ContainerStarted","Data":"d36616fd157db32936753be61cdd53d2c850435df0eaba8c39a05552b9a88e63"}
Jul 16 22:35:25.918728 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:25.918699    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nbc9n" event={"ID":"53ae108c-2ec1-4c24-b756-4a899fbcf19d","Type":"ContainerStarted","Data":"88cc580a989b324d24da3767b608904bee55f79f46fa5dbbb993c6604e8a9818"}
Jul 16 22:35:26.464513 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:26.464473    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:26.665967 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:26.665928    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:26.925089 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:26.924993    2573 generic.go:356] "Generic (PLEG): container finished" podID="1b4e74e3-c3b1-40c8-9f0b-34f84325f497" containerID="d36616fd157db32936753be61cdd53d2c850435df0eaba8c39a05552b9a88e63" exitCode=0
Jul 16 22:35:26.925565 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:26.925084    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fr7wp" event={"ID":"1b4e74e3-c3b1-40c8-9f0b-34f84325f497","Type":"ContainerDied","Data":"d36616fd157db32936753be61cdd53d2c850435df0eaba8c39a05552b9a88e63"}
Jul 16 22:35:27.931392 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:27.931356    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nbc9n" event={"ID":"53ae108c-2ec1-4c24-b756-4a899fbcf19d","Type":"ContainerStarted","Data":"8beb2dd4790b13054244cdc4361dfe2394e0505a6b88ca6d3fc3b699a9c6bd63"}
Jul 16 22:35:27.931866 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:27.931426    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nbc9n" event={"ID":"53ae108c-2ec1-4c24-b756-4a899fbcf19d","Type":"ContainerStarted","Data":"2f1207dc1e8b0588114d03399d704ca702cf8d388bb5ed2b109fa7e7158e2b19"}
Jul 16 22:35:27.931866 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:27.931458    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nbc9n"
Jul 16 22:35:27.934371 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:27.934339    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fr7wp" event={"ID":"1b4e74e3-c3b1-40c8-9f0b-34f84325f497","Type":"ContainerStarted","Data":"e74275606a018ae452858af0bac4f557ae2300392764203049cb4252c356aa91"}
Jul 16 22:35:27.934532 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:27.934393    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fr7wp" event={"ID":"1b4e74e3-c3b1-40c8-9f0b-34f84325f497","Type":"ContainerStarted","Data":"1a3d545dd3018ba3b5f86540ca0e8565c87c9741b7a222782fe03c4a30656367"}
Jul 16 22:35:27.968165 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:27.968127    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:27.989558 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:27.989500    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-nbc9n" podStartSLOduration=64.85181879699999 podStartE2EDuration="1m6.989480046s" totalImagesPullingTime="2.137661249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:34:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:35:27.955051357 +0000 UTC m=+101.965314030" watchObservedRunningTime="2026-07-16 22:35:27.989480046 +0000 UTC m=+101.999742725"
Jul 16 22:35:27.989753 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:27.989686    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fr7wp" podStartSLOduration=5.302109223 podStartE2EDuration="6.989673604s" totalImagesPullingTime="1.584809793s" totalInitContainerRuntime="102.754588ms" isStatefulPod=true podCreationTimestamp="2026-07-16 22:35:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:35:27.987509845 +0000 UTC m=+101.997772506" watchObservedRunningTime="2026-07-16 22:35:27.989673604 +0000 UTC m=+101.999936269"
Jul 16 22:35:28.792695 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:28.792659    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7nqtn"
Jul 16 22:35:30.126679 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:30.126641    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:30.185170 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:30.185133    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:37.942476 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:37.942428    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nbc9n"
Jul 16 22:35:40.528531 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:40.528494    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:40.628984 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:40.628949    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:40.728087 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:40.728054    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:40.828165 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:40.828132    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:41.001585 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:41.001552    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:41.190948 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:41.190837    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:41.526756 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:41.526675    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:41.657767 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:41.657730    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:41.749295 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:41.749259    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:42.302871 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:42.302839    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:42.400080 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:42.400049    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:42.459721 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:42.459689    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:42.596206 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:42.596179    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:42.695970 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:42.695928    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:43.236722 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:43.236688    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:43.304510 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:43.304476    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:43.451198 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:43.451159    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:43.518574 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:43.518491    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:43.881694 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:43.881659    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:43.942796 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:43.942765    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:49.885670 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:49.885636    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:50.169205 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:50.169119    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:50.315450 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:50.315414    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:50.685110 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:50.685066    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:50.788154 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:50.788114    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:51.358094 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:51.358048    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:51.542680 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:51.542647    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:53.422703 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:53.422664    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:53.520939 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:53.520885    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:56.265573 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:56.265528    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b332999-321a-4e3f-99bf-19517a268e28-metrics-certs\") pod \"network-metrics-daemon-6xfmc\" (UID: \"9b332999-321a-4e3f-99bf-19517a268e28\") " pod="openshift-multus/network-metrics-daemon-6xfmc"
Jul 16 22:35:56.268103 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:56.268072    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b332999-321a-4e3f-99bf-19517a268e28-metrics-certs\") pod \"network-metrics-daemon-6xfmc\" (UID: \"9b332999-321a-4e3f-99bf-19517a268e28\") " pod="openshift-multus/network-metrics-daemon-6xfmc"
Jul 16 22:35:56.325300 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:56.325258    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9dnx4\""
Jul 16 22:35:56.332806 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:56.332773    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6xfmc"
Jul 16 22:35:56.521507 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:56.521422    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6xfmc"]
Jul 16 22:35:56.525625 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:35:56.525591    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b332999_321a_4e3f_99bf_19517a268e28.slice/crio-3faa4f04d524a9aee5a7833131278f9f91ac82530acacaef46de624cd507c86e WatchSource:0}: Error finding container 3faa4f04d524a9aee5a7833131278f9f91ac82530acacaef46de624cd507c86e: Status 404 returned error can't find the container with id 3faa4f04d524a9aee5a7833131278f9f91ac82530acacaef46de624cd507c86e
Jul 16 22:35:57.036126 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:57.036083    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6xfmc" event={"ID":"9b332999-321a-4e3f-99bf-19517a268e28","Type":"ContainerStarted","Data":"3faa4f04d524a9aee5a7833131278f9f91ac82530acacaef46de624cd507c86e"}
Jul 16 22:35:57.037471 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:57.037439    2573 generic.go:356] "Generic (PLEG): container finished" podID="6492a043-84e1-4d73-aac5-4239e445d109" containerID="78305322d55ede1cd044f5d60cc6507916dad04fe5c3492efa3f1139c80c745d" exitCode=0
Jul 16 22:35:57.037583 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:57.037511    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-zm76k" event={"ID":"6492a043-84e1-4d73-aac5-4239e445d109","Type":"ContainerDied","Data":"78305322d55ede1cd044f5d60cc6507916dad04fe5c3492efa3f1139c80c745d"}
Jul 16 22:35:57.037852 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:57.037839    2573 scope.go:117] "RemoveContainer" containerID="78305322d55ede1cd044f5d60cc6507916dad04fe5c3492efa3f1139c80c745d"
Jul 16 22:35:57.217443 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:57.217389    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:58.043492 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:58.043445    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6xfmc" event={"ID":"9b332999-321a-4e3f-99bf-19517a268e28","Type":"ContainerStarted","Data":"a19cc784cbbd27d55498d48459a5a1b81db0eea7fd2bb5f06ed23fe692a775ab"}
Jul 16 22:35:58.043944 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:58.043522    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6xfmc" event={"ID":"9b332999-321a-4e3f-99bf-19517a268e28","Type":"ContainerStarted","Data":"7c1ed5379bf77842b4480fd4b11fa8d711b0d8881b00567005a9dee962d7ea53"}
Jul 16 22:35:58.045224 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:58.045196    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-zm76k" event={"ID":"6492a043-84e1-4d73-aac5-4239e445d109","Type":"ContainerStarted","Data":"4509e440493604ba5d9686368f0434271605f1dc0bbf3c7086957a94b642fd9b"}
Jul 16 22:35:58.061017 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:58.060949    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6xfmc" podStartSLOduration=130.966885543 podStartE2EDuration="2m12.060926168s" totalImagesPullingTime="1.094040625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:33:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:35:58.06080415 +0000 UTC m=+132.071066841" watchObservedRunningTime="2026-07-16 22:35:58.060926168 +0000 UTC m=+132.071188830"
Jul 16 22:35:58.195133 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:58.195029    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:58.291806 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:58.291763    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:58.477522 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:58.477426    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:58.582758 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:58.582724    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:58.685375 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:35:58.685338    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:36:02.047980 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:36:02.047941    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:36:02.106548 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:36:02.106513    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:36:05.246989 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:36:05.246957    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fr7wp_1b4e74e3-c3b1-40c8-9f0b-34f84325f497/init-textfile/0.log"
Jul 16 22:36:05.447629 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:36:05.447591    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fr7wp_1b4e74e3-c3b1-40c8-9f0b-34f84325f497/node-exporter/0.log"
Jul 16 22:36:05.644572 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:36:05.644542    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fr7wp_1b4e74e3-c3b1-40c8-9f0b-34f84325f497/kube-rbac-proxy/0.log"
Jul 16 22:36:08.120798 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:36:08.120767    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:36:08.671586 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:36:08.671541    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:36:08.767149 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:36:08.767107    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:36:09.181793 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:36:09.181758    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:36:09.243445 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:36:09.243408    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:36:11.502071 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:36:11.502035    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:36:11.600844 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:36:11.600807    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:36:15.575038 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:36:15.574997    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:36:15.641789 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:36:15.641758    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:36:26.506810 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:36:26.506765    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:36:26.576306 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:36:26.576271    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:36:40.940955 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:36:40.940874    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:36:41.201325 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:36:41.201206    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:36:49.437410 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:36:49.437373    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:36:49.524048 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:36:49.524013    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:36:55.407891 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:36:55.407850    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:37:00.817756 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:37:00.817722    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:37:00.919045 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:37:00.919002    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:37:00.976749 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:37:00.976700    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:37:27.483643 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:37:27.483591    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:37:32.511410 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:37:32.511364    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:37:32.570281 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:37:32.570221    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:37:56.193907 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:37:56.193872    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:37:56.294480 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:37:56.294444    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:38:08.165920 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:38:08.165879    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:38:08.231576 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:38:08.231544    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:38:11.520489 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:38:11.520411    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:38:11.621569 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:38:11.621535    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:38:26.207659 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:38:26.207621    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:38:40.799572 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:38:40.799544    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:38:40.867881 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:38:40.867854    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:38:40.934422 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:38:40.934394    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:38:41.554786 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:38:41.554753    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:38:41.626567 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:38:41.626544    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:38:46.352555 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:38:46.352526    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r9jm4_506f904a-3272-4586-8528-146884e728f9/ovn-acl-logging/0.log"
Jul 16 22:38:46.354788 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:38:46.354766    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r9jm4_506f904a-3272-4586-8528-146884e728f9/ovn-acl-logging/0.log"
Jul 16 22:38:46.355543 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:38:46.355523    2573 kubelet.go:1648] "Image garbage collection succeeded"
Jul 16 22:38:46.497562 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:38:46.497536    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:38:46.599111 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:38:46.599083    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:38:47.486450 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:38:47.486409    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l7hvh"]
Jul 16 22:38:47.489346 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:38:47.489328    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l7hvh"
Jul 16 22:38:47.492384 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:38:47.492362    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 16 22:38:47.492479 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:38:47.492434    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-p4dqb\""
Jul 16 22:38:47.493153 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:38:47.493138    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 16 22:38:47.502462 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:38:47.502433    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l7hvh"]
Jul 16 22:38:47.657305 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:38:47.657275    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bm59\" (UniqueName: \"kubernetes.io/projected/083d303e-404a-4590-9b07-b44428381368-kube-api-access-4bm59\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l7hvh\" (UID: \"083d303e-404a-4590-9b07-b44428381368\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l7hvh"
Jul 16 22:38:47.657446 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:38:47.657327    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/083d303e-404a-4590-9b07-b44428381368-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l7hvh\" (UID: \"083d303e-404a-4590-9b07-b44428381368\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l7hvh"
Jul 16 22:38:47.657446 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:38:47.657415    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/083d303e-404a-4590-9b07-b44428381368-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l7hvh\" (UID: \"083d303e-404a-4590-9b07-b44428381368\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l7hvh"
Jul 16 22:38:47.758613 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:38:47.758553    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/083d303e-404a-4590-9b07-b44428381368-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l7hvh\" (UID: \"083d303e-404a-4590-9b07-b44428381368\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l7hvh"
Jul 16 22:38:47.758733 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:38:47.758609    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/083d303e-404a-4590-9b07-b44428381368-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l7hvh\" (UID: \"083d303e-404a-4590-9b07-b44428381368\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l7hvh"
Jul 16 22:38:47.758733 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:38:47.758638    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bm59\" (UniqueName: \"kubernetes.io/projected/083d303e-404a-4590-9b07-b44428381368-kube-api-access-4bm59\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l7hvh\" (UID: \"083d303e-404a-4590-9b07-b44428381368\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l7hvh"
Jul 16 22:38:47.758899 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:38:47.758881    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/083d303e-404a-4590-9b07-b44428381368-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l7hvh\" (UID: \"083d303e-404a-4590-9b07-b44428381368\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l7hvh"
Jul 16 22:38:47.758961 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:38:47.758944    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/083d303e-404a-4590-9b07-b44428381368-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l7hvh\" (UID: \"083d303e-404a-4590-9b07-b44428381368\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l7hvh"
Jul 16 22:38:47.766749 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:38:47.766730    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bm59\" (UniqueName: \"kubernetes.io/projected/083d303e-404a-4590-9b07-b44428381368-kube-api-access-4bm59\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l7hvh\" (UID: \"083d303e-404a-4590-9b07-b44428381368\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l7hvh"
Jul 16 22:38:47.797851 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:38:47.797824    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l7hvh"
Jul 16 22:38:47.916011 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:38:47.915979    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l7hvh"]
Jul 16 22:38:47.919505 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:38:47.919467    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod083d303e_404a_4590_9b07_b44428381368.slice/crio-3a7e89a573c44d6a55e2cf16b4292a8e4df164e7971190326ef5344501a88560 WatchSource:0}: Error finding container 3a7e89a573c44d6a55e2cf16b4292a8e4df164e7971190326ef5344501a88560: Status 404 returned error can't find the container with id 3a7e89a573c44d6a55e2cf16b4292a8e4df164e7971190326ef5344501a88560
Jul 16 22:38:47.921423 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:38:47.921402    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 22:38:48.628006 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:38:48.627966    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l7hvh" event={"ID":"083d303e-404a-4590-9b07-b44428381368","Type":"ContainerStarted","Data":"3a7e89a573c44d6a55e2cf16b4292a8e4df164e7971190326ef5344501a88560"}
Jul 16 22:38:49.672615 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:38:49.672576    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:38:51.242427 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:38:51.242396    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:38:51.300132 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:38:51.300106    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:38:51.357111 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:38:51.357085    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:38:54.653772 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:38:54.653744    2573 generic.go:356] "Generic (PLEG): container finished" podID="083d303e-404a-4590-9b07-b44428381368" containerID="7d459c4512d4691b89b3b0cc4a529387679991788e3bc44d7357e25f3378380e" exitCode=0
Jul 16 22:38:54.654092 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:38:54.653799    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l7hvh" event={"ID":"083d303e-404a-4590-9b07-b44428381368","Type":"ContainerDied","Data":"7d459c4512d4691b89b3b0cc4a529387679991788e3bc44d7357e25f3378380e"}
Jul 16 22:38:57.664836 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:38:57.664805    2573 generic.go:356] "Generic (PLEG): container finished" podID="083d303e-404a-4590-9b07-b44428381368" containerID="b00f88e6443aa44754e9de59b65da4482a6730d1f31bdd50a945866b8cc6cd51" exitCode=0
Jul 16 22:38:57.665166 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:38:57.664878    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l7hvh" event={"ID":"083d303e-404a-4590-9b07-b44428381368","Type":"ContainerDied","Data":"b00f88e6443aa44754e9de59b65da4482a6730d1f31bdd50a945866b8cc6cd51"}
Jul 16 22:39:04.693978 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:04.693944    2573 generic.go:356] "Generic (PLEG): container finished" podID="083d303e-404a-4590-9b07-b44428381368" containerID="7b372f442e1e49a8c6626926c423dbeb09e71b2ba70a9ad3d772bc38effc5236" exitCode=0
Jul 16 22:39:04.694296 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:04.693987    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l7hvh" event={"ID":"083d303e-404a-4590-9b07-b44428381368","Type":"ContainerDied","Data":"7b372f442e1e49a8c6626926c423dbeb09e71b2ba70a9ad3d772bc38effc5236"}
Jul 16 22:39:05.814367 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:05.814343    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l7hvh"
Jul 16 22:39:05.988471 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:05.988410    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/083d303e-404a-4590-9b07-b44428381368-util\") pod \"083d303e-404a-4590-9b07-b44428381368\" (UID: \"083d303e-404a-4590-9b07-b44428381368\") "
Jul 16 22:39:05.988471 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:05.988467    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bm59\" (UniqueName: \"kubernetes.io/projected/083d303e-404a-4590-9b07-b44428381368-kube-api-access-4bm59\") pod \"083d303e-404a-4590-9b07-b44428381368\" (UID: \"083d303e-404a-4590-9b07-b44428381368\") "
Jul 16 22:39:05.988624 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:05.988488    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/083d303e-404a-4590-9b07-b44428381368-bundle\") pod \"083d303e-404a-4590-9b07-b44428381368\" (UID: \"083d303e-404a-4590-9b07-b44428381368\") "
Jul 16 22:39:05.989107 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:05.989082    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/083d303e-404a-4590-9b07-b44428381368-bundle" (OuterVolumeSpecName: "bundle") pod "083d303e-404a-4590-9b07-b44428381368" (UID: "083d303e-404a-4590-9b07-b44428381368"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:39:05.990502 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:05.990477    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/083d303e-404a-4590-9b07-b44428381368-kube-api-access-4bm59" (OuterVolumeSpecName: "kube-api-access-4bm59") pod "083d303e-404a-4590-9b07-b44428381368" (UID: "083d303e-404a-4590-9b07-b44428381368"). InnerVolumeSpecName "kube-api-access-4bm59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:39:05.992563 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:05.992541    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/083d303e-404a-4590-9b07-b44428381368-util" (OuterVolumeSpecName: "util") pod "083d303e-404a-4590-9b07-b44428381368" (UID: "083d303e-404a-4590-9b07-b44428381368"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:39:06.089248 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:06.089222    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bm59\" (UniqueName: \"kubernetes.io/projected/083d303e-404a-4590-9b07-b44428381368-kube-api-access-4bm59\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 22:39:06.089313 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:06.089265    2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/083d303e-404a-4590-9b07-b44428381368-bundle\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 22:39:06.089313 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:06.089275    2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/083d303e-404a-4590-9b07-b44428381368-util\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 22:39:06.702371 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:06.702349    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l7hvh"
Jul 16 22:39:06.702488 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:06.702347    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l7hvh" event={"ID":"083d303e-404a-4590-9b07-b44428381368","Type":"ContainerDied","Data":"3a7e89a573c44d6a55e2cf16b4292a8e4df164e7971190326ef5344501a88560"}
Jul 16 22:39:06.702488 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:06.702471    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a7e89a573c44d6a55e2cf16b4292a8e4df164e7971190326ef5344501a88560"
Jul 16 22:39:08.525817 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:08.525784    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:39:09.939533 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:09.939498    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:39:12.131905 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:12.131868    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:39:12.192975 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:12.192945    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:39:14.564731 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:14.564700    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:39:14.687335 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:14.687306    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:39:15.293671 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:15.293637    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrznw"]
Jul 16 22:39:15.293875 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:15.293862    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="083d303e-404a-4590-9b07-b44428381368" containerName="pull"
Jul 16 22:39:15.293917 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:15.293875    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="083d303e-404a-4590-9b07-b44428381368" containerName="pull"
Jul 16 22:39:15.293917 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:15.293884    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="083d303e-404a-4590-9b07-b44428381368" containerName="extract"
Jul 16 22:39:15.293917 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:15.293890    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="083d303e-404a-4590-9b07-b44428381368" containerName="extract"
Jul 16 22:39:15.293917 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:15.293902    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="083d303e-404a-4590-9b07-b44428381368" containerName="util"
Jul 16 22:39:15.293917 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:15.293908    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="083d303e-404a-4590-9b07-b44428381368" containerName="util"
Jul 16 22:39:15.294062 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:15.293952    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="083d303e-404a-4590-9b07-b44428381368" containerName="extract"
Jul 16 22:39:15.299343 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:15.299323    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrznw"
Jul 16 22:39:15.306397 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:15.306376    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-p4dqb\""
Jul 16 22:39:15.306525 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:15.306439    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 16 22:39:15.307109 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:15.307092    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 16 22:39:15.317097 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:15.317077    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrznw"]
Jul 16 22:39:15.449393 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:15.449367    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/400740d2-1509-4725-8249-338764332943-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrznw\" (UID: \"400740d2-1509-4725-8249-338764332943\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrznw"
Jul 16 22:39:15.449488 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:15.449402    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpgcc\" (UniqueName: \"kubernetes.io/projected/400740d2-1509-4725-8249-338764332943-kube-api-access-bpgcc\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrznw\" (UID: \"400740d2-1509-4725-8249-338764332943\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrznw"
Jul 16 22:39:15.449488 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:15.449427    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/400740d2-1509-4725-8249-338764332943-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrznw\" (UID: \"400740d2-1509-4725-8249-338764332943\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrznw"
Jul 16 22:39:15.549786 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:15.549731    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/400740d2-1509-4725-8249-338764332943-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrznw\" (UID: \"400740d2-1509-4725-8249-338764332943\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrznw"
Jul 16 22:39:15.549786 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:15.549770    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpgcc\" (UniqueName: \"kubernetes.io/projected/400740d2-1509-4725-8249-338764332943-kube-api-access-bpgcc\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrznw\" (UID: \"400740d2-1509-4725-8249-338764332943\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrznw"
Jul 16 22:39:15.549911 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:15.549793    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/400740d2-1509-4725-8249-338764332943-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrznw\" (UID: \"400740d2-1509-4725-8249-338764332943\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrznw"
Jul 16 22:39:15.550109 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:15.550094    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/400740d2-1509-4725-8249-338764332943-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrznw\" (UID: \"400740d2-1509-4725-8249-338764332943\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrznw"
Jul 16 22:39:15.550143 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:15.550129    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/400740d2-1509-4725-8249-338764332943-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrznw\" (UID: \"400740d2-1509-4725-8249-338764332943\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrznw"
Jul 16 22:39:15.562281 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:15.562230    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpgcc\" (UniqueName: \"kubernetes.io/projected/400740d2-1509-4725-8249-338764332943-kube-api-access-bpgcc\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrznw\" (UID: \"400740d2-1509-4725-8249-338764332943\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrznw"
Jul 16 22:39:15.607832 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:15.607808    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrznw"
Jul 16 22:39:15.748459 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:15.748432    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrznw"]
Jul 16 22:39:15.751961 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:39:15.751934    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod400740d2_1509_4725_8249_338764332943.slice/crio-135b3ff339fac8109b0c3f8b54b4ab818217ba64e0b5def1559fa33f33765228 WatchSource:0}: Error finding container 135b3ff339fac8109b0c3f8b54b4ab818217ba64e0b5def1559fa33f33765228: Status 404 returned error can't find the container with id 135b3ff339fac8109b0c3f8b54b4ab818217ba64e0b5def1559fa33f33765228
Jul 16 22:39:15.780913 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:15.780891    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-rw4sx"]
Jul 16 22:39:15.783049 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:15.783032    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-rw4sx"
Jul 16 22:39:15.790407 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:15.790385    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 16 22:39:15.790507 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:15.790390    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-snxmg\""
Jul 16 22:39:15.790507 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:15.790396    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 16 22:39:15.803981 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:15.803930    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-rw4sx"]
Jul 16 22:39:15.952446 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:15.952418    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eb3d0f56-a8f9-4485-b0cb-40cace205daa-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-rw4sx\" (UID: \"eb3d0f56-a8f9-4485-b0cb-40cace205daa\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rw4sx"
Jul 16 22:39:15.952552 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:15.952457    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhdgz\" (UniqueName: \"kubernetes.io/projected/eb3d0f56-a8f9-4485-b0cb-40cace205daa-kube-api-access-zhdgz\") pod \"cert-manager-webhook-6486ffd997-rw4sx\" (UID: \"eb3d0f56-a8f9-4485-b0cb-40cace205daa\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rw4sx"
Jul 16 22:39:16.053544 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:16.053519    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eb3d0f56-a8f9-4485-b0cb-40cace205daa-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-rw4sx\" (UID: \"eb3d0f56-a8f9-4485-b0cb-40cace205daa\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rw4sx"
Jul 16 22:39:16.053641 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:16.053555    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhdgz\" (UniqueName: \"kubernetes.io/projected/eb3d0f56-a8f9-4485-b0cb-40cace205daa-kube-api-access-zhdgz\") pod \"cert-manager-webhook-6486ffd997-rw4sx\" (UID: \"eb3d0f56-a8f9-4485-b0cb-40cace205daa\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rw4sx"
Jul 16 22:39:16.064026 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:16.063972    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eb3d0f56-a8f9-4485-b0cb-40cace205daa-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-rw4sx\" (UID: \"eb3d0f56-a8f9-4485-b0cb-40cace205daa\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rw4sx"
Jul 16 22:39:16.064179 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:16.064163    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhdgz\" (UniqueName: \"kubernetes.io/projected/eb3d0f56-a8f9-4485-b0cb-40cace205daa-kube-api-access-zhdgz\") pod \"cert-manager-webhook-6486ffd997-rw4sx\" (UID: \"eb3d0f56-a8f9-4485-b0cb-40cace205daa\") " pod="cert-manager/cert-manager-webhook-6486ffd997-rw4sx"
Jul 16 22:39:16.120528 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:16.120509    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-rw4sx"
Jul 16 22:39:16.246265 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:16.246224    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-rw4sx"]
Jul 16 22:39:16.248989 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:39:16.248961    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb3d0f56_a8f9_4485_b0cb_40cace205daa.slice/crio-fdd8b611e2f15b5a7a4721f4f23d1015157718cdbed100e567ac764b2ef24492 WatchSource:0}: Error finding container fdd8b611e2f15b5a7a4721f4f23d1015157718cdbed100e567ac764b2ef24492: Status 404 returned error can't find the container with id fdd8b611e2f15b5a7a4721f4f23d1015157718cdbed100e567ac764b2ef24492
Jul 16 22:39:16.742744 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:16.742712    2573 generic.go:356] "Generic (PLEG): container finished" podID="400740d2-1509-4725-8249-338764332943" containerID="3860221ef8d4fbe49c4205b09bdcb3c26a030ae04729e9af6fd2aae4e9369d4e" exitCode=0
Jul 16 22:39:16.743105 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:16.742803    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrznw" event={"ID":"400740d2-1509-4725-8249-338764332943","Type":"ContainerDied","Data":"3860221ef8d4fbe49c4205b09bdcb3c26a030ae04729e9af6fd2aae4e9369d4e"}
Jul 16 22:39:16.743105 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:16.742906    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrznw" event={"ID":"400740d2-1509-4725-8249-338764332943","Type":"ContainerStarted","Data":"135b3ff339fac8109b0c3f8b54b4ab818217ba64e0b5def1559fa33f33765228"}
Jul 16 22:39:16.744025 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:16.743998    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-rw4sx" event={"ID":"eb3d0f56-a8f9-4485-b0cb-40cace205daa","Type":"ContainerStarted","Data":"fdd8b611e2f15b5a7a4721f4f23d1015157718cdbed100e567ac764b2ef24492"}
Jul 16 22:39:16.952754 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:16.952724    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:39:17.221525 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:17.221393    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:39:19.757984 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:19.757916    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-rw4sx" event={"ID":"eb3d0f56-a8f9-4485-b0cb-40cace205daa","Type":"ContainerStarted","Data":"361cb9388040ce1bff665c336ff10de719f0390f9de50a54e23664f630960b11"}
Jul 16 22:39:19.758366 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:19.757987    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-rw4sx"
Jul 16 22:39:19.759885 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:19.759853    2573 generic.go:356] "Generic (PLEG): container finished" podID="400740d2-1509-4725-8249-338764332943" containerID="57b1a40ed91239ec56a2b37c283cf1ff4601264eac56c9b12235f5a1c4d3b39b" exitCode=0
Jul 16 22:39:19.759997 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:19.759901    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrznw" event={"ID":"400740d2-1509-4725-8249-338764332943","Type":"ContainerDied","Data":"57b1a40ed91239ec56a2b37c283cf1ff4601264eac56c9b12235f5a1c4d3b39b"}
Jul 16 22:39:19.783432 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:19.783384    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-rw4sx" podStartSLOduration=1.5848376370000001 podStartE2EDuration="4.783368649s" totalImagesPullingTime="3.198531012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:39:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:39:19.78114283 +0000 UTC m=+333.791405493" watchObservedRunningTime="2026-07-16 22:39:19.783368649 +0000 UTC m=+333.793631313"
Jul 16 22:39:20.765617 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:20.765577    2573 generic.go:356] "Generic (PLEG): container finished" podID="400740d2-1509-4725-8249-338764332943" containerID="611f42aa6388f83fc0ca56c247dc7fed910429f5c55ed75628016ecbb6b67b50" exitCode=0
Jul 16 22:39:20.766084 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:20.765678    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrznw" event={"ID":"400740d2-1509-4725-8249-338764332943","Type":"ContainerDied","Data":"611f42aa6388f83fc0ca56c247dc7fed910429f5c55ed75628016ecbb6b67b50"}
Jul 16 22:39:21.886743 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:21.886717    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrznw"
Jul 16 22:39:21.998419 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:21.998394    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/400740d2-1509-4725-8249-338764332943-bundle\") pod \"400740d2-1509-4725-8249-338764332943\" (UID: \"400740d2-1509-4725-8249-338764332943\") "
Jul 16 22:39:21.998541 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:21.998427    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/400740d2-1509-4725-8249-338764332943-util\") pod \"400740d2-1509-4725-8249-338764332943\" (UID: \"400740d2-1509-4725-8249-338764332943\") "
Jul 16 22:39:21.998541 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:21.998455    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bpgcc\" (UniqueName: \"kubernetes.io/projected/400740d2-1509-4725-8249-338764332943-kube-api-access-bpgcc\") pod \"400740d2-1509-4725-8249-338764332943\" (UID: \"400740d2-1509-4725-8249-338764332943\") "
Jul 16 22:39:21.998815 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:21.998790    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/400740d2-1509-4725-8249-338764332943-bundle" (OuterVolumeSpecName: "bundle") pod "400740d2-1509-4725-8249-338764332943" (UID: "400740d2-1509-4725-8249-338764332943"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:39:22.000383 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:22.000357    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/400740d2-1509-4725-8249-338764332943-kube-api-access-bpgcc" (OuterVolumeSpecName: "kube-api-access-bpgcc") pod "400740d2-1509-4725-8249-338764332943" (UID: "400740d2-1509-4725-8249-338764332943"). InnerVolumeSpecName "kube-api-access-bpgcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:39:22.003363 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:22.003292    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/400740d2-1509-4725-8249-338764332943-util" (OuterVolumeSpecName: "util") pod "400740d2-1509-4725-8249-338764332943" (UID: "400740d2-1509-4725-8249-338764332943"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:39:22.099132 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:22.099109    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpgcc\" (UniqueName: \"kubernetes.io/projected/400740d2-1509-4725-8249-338764332943-kube-api-access-bpgcc\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 22:39:22.099132 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:22.099130    2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/400740d2-1509-4725-8249-338764332943-bundle\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 22:39:22.099281 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:22.099139    2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/400740d2-1509-4725-8249-338764332943-util\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 22:39:22.775454 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:22.775410    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrznw" event={"ID":"400740d2-1509-4725-8249-338764332943","Type":"ContainerDied","Data":"135b3ff339fac8109b0c3f8b54b4ab818217ba64e0b5def1559fa33f33765228"}
Jul 16 22:39:22.775601 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:22.775474    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="135b3ff339fac8109b0c3f8b54b4ab818217ba64e0b5def1559fa33f33765228"
Jul 16 22:39:22.775601 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:22.775475    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fmrznw"
Jul 16 22:39:24.794973 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:24.794926    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:39:25.768535 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:25.768503    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-rw4sx"
Jul 16 22:39:26.783311 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:26.783275    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:39:26.886456 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:26.886420    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:39:27.105019 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:27.104996    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:39:27.210346 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:27.210325    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:39:33.043817 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:33.043780    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:39:33.146807 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:33.146779    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:39:34.518379 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:34.518353    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc9lc4"]
Jul 16 22:39:34.518736 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:34.518582    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="400740d2-1509-4725-8249-338764332943" containerName="pull"
Jul 16 22:39:34.518736 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:34.518594    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="400740d2-1509-4725-8249-338764332943" containerName="pull"
Jul 16 22:39:34.518736 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:34.518601    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="400740d2-1509-4725-8249-338764332943" containerName="extract"
Jul 16 22:39:34.518736 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:34.518606    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="400740d2-1509-4725-8249-338764332943" containerName="extract"
Jul 16 22:39:34.518736 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:34.518618    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="400740d2-1509-4725-8249-338764332943" containerName="util"
Jul 16 22:39:34.518736 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:34.518623    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="400740d2-1509-4725-8249-338764332943" containerName="util"
Jul 16 22:39:34.518736 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:34.518663    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="400740d2-1509-4725-8249-338764332943" containerName="extract"
Jul 16 22:39:34.521835 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:34.521813    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc9lc4"
Jul 16 22:39:34.524727 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:34.524707    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 16 22:39:34.525430 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:34.525411    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 16 22:39:34.525527 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:34.525432    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-p4dqb\""
Jul 16 22:39:34.530460 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:34.530424    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc9lc4"]
Jul 16 22:39:34.678089 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:34.678060    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/322e3090-597e-42df-be25-3a3c7104bf8b-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc9lc4\" (UID: \"322e3090-597e-42df-be25-3a3c7104bf8b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc9lc4"
Jul 16 22:39:34.678222 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:34.678141    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwb4k\" (UniqueName: \"kubernetes.io/projected/322e3090-597e-42df-be25-3a3c7104bf8b-kube-api-access-rwb4k\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc9lc4\" (UID: \"322e3090-597e-42df-be25-3a3c7104bf8b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc9lc4"
Jul 16 22:39:34.678222 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:34.678181    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/322e3090-597e-42df-be25-3a3c7104bf8b-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc9lc4\" (UID: \"322e3090-597e-42df-be25-3a3c7104bf8b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc9lc4"
Jul 16 22:39:34.728351 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:34.728323    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:39:34.778785 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:34.778737    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/322e3090-597e-42df-be25-3a3c7104bf8b-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc9lc4\" (UID: \"322e3090-597e-42df-be25-3a3c7104bf8b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc9lc4"
Jul 16 22:39:34.778855 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:34.778786    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwb4k\" (UniqueName: \"kubernetes.io/projected/322e3090-597e-42df-be25-3a3c7104bf8b-kube-api-access-rwb4k\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc9lc4\" (UID: \"322e3090-597e-42df-be25-3a3c7104bf8b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc9lc4"
Jul 16 22:39:34.778855 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:34.778819    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/322e3090-597e-42df-be25-3a3c7104bf8b-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc9lc4\" (UID: \"322e3090-597e-42df-be25-3a3c7104bf8b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc9lc4"
Jul 16 22:39:34.779168 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:34.779148    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/322e3090-597e-42df-be25-3a3c7104bf8b-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc9lc4\" (UID: \"322e3090-597e-42df-be25-3a3c7104bf8b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc9lc4"
Jul 16 22:39:34.779168 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:34.779162    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/322e3090-597e-42df-be25-3a3c7104bf8b-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc9lc4\" (UID: \"322e3090-597e-42df-be25-3a3c7104bf8b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc9lc4"
Jul 16 22:39:34.787964 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:34.787939    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwb4k\" (UniqueName: \"kubernetes.io/projected/322e3090-597e-42df-be25-3a3c7104bf8b-kube-api-access-rwb4k\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc9lc4\" (UID: \"322e3090-597e-42df-be25-3a3c7104bf8b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc9lc4"
Jul 16 22:39:34.831217 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:34.831195    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc9lc4"
Jul 16 22:39:34.953071 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:34.953037    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc9lc4"]
Jul 16 22:39:34.955500 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:39:34.955471    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod322e3090_597e_42df_be25_3a3c7104bf8b.slice/crio-0b9a2b6a2cd114e7ed155c16fc825cff94e1fcdd9192966a2fc44273fb91ef5c WatchSource:0}: Error finding container 0b9a2b6a2cd114e7ed155c16fc825cff94e1fcdd9192966a2fc44273fb91ef5c: Status 404 returned error can't find the container with id 0b9a2b6a2cd114e7ed155c16fc825cff94e1fcdd9192966a2fc44273fb91ef5c
Jul 16 22:39:35.821494 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:35.821464    2573 generic.go:356] "Generic (PLEG): container finished" podID="322e3090-597e-42df-be25-3a3c7104bf8b" containerID="14ff1665cb1e460a78fa7e180292c0ef8acf4b40bb687b9213714a28c275151a" exitCode=0
Jul 16 22:39:35.821830 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:35.821563    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc9lc4" event={"ID":"322e3090-597e-42df-be25-3a3c7104bf8b","Type":"ContainerDied","Data":"14ff1665cb1e460a78fa7e180292c0ef8acf4b40bb687b9213714a28c275151a"}
Jul 16 22:39:35.821830 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:35.821630    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc9lc4" event={"ID":"322e3090-597e-42df-be25-3a3c7104bf8b","Type":"ContainerStarted","Data":"0b9a2b6a2cd114e7ed155c16fc825cff94e1fcdd9192966a2fc44273fb91ef5c"}
Jul 16 22:39:36.827086 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:36.827052    2573 generic.go:356] "Generic (PLEG): container finished" podID="322e3090-597e-42df-be25-3a3c7104bf8b" containerID="b3f86228339a27fe89d83ac64777ce5e38ec6a991f602a0d75f99ad7491b8c73" exitCode=0
Jul 16 22:39:36.827483 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:36.827150    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc9lc4" event={"ID":"322e3090-597e-42df-be25-3a3c7104bf8b","Type":"ContainerDied","Data":"b3f86228339a27fe89d83ac64777ce5e38ec6a991f602a0d75f99ad7491b8c73"}
Jul 16 22:39:37.834728 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:37.834687    2573 generic.go:356] "Generic (PLEG): container finished" podID="322e3090-597e-42df-be25-3a3c7104bf8b" containerID="d6ee8099a6daf326f68bd65d9ca03ebac46572bd1cf91bb1310f16c7cf1ba7a1" exitCode=0
Jul 16 22:39:37.835102 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:37.834777    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc9lc4" event={"ID":"322e3090-597e-42df-be25-3a3c7104bf8b","Type":"ContainerDied","Data":"d6ee8099a6daf326f68bd65d9ca03ebac46572bd1cf91bb1310f16c7cf1ba7a1"}
Jul 16 22:39:38.961612 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:38.961590    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc9lc4"
Jul 16 22:39:39.008794 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:39.008767    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/322e3090-597e-42df-be25-3a3c7104bf8b-bundle\") pod \"322e3090-597e-42df-be25-3a3c7104bf8b\" (UID: \"322e3090-597e-42df-be25-3a3c7104bf8b\") "
Jul 16 22:39:39.008901 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:39.008824    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/322e3090-597e-42df-be25-3a3c7104bf8b-util\") pod \"322e3090-597e-42df-be25-3a3c7104bf8b\" (UID: \"322e3090-597e-42df-be25-3a3c7104bf8b\") "
Jul 16 22:39:39.008901 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:39.008860    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwb4k\" (UniqueName: \"kubernetes.io/projected/322e3090-597e-42df-be25-3a3c7104bf8b-kube-api-access-rwb4k\") pod \"322e3090-597e-42df-be25-3a3c7104bf8b\" (UID: \"322e3090-597e-42df-be25-3a3c7104bf8b\") "
Jul 16 22:39:39.009469 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:39.009443    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/322e3090-597e-42df-be25-3a3c7104bf8b-bundle" (OuterVolumeSpecName: "bundle") pod "322e3090-597e-42df-be25-3a3c7104bf8b" (UID: "322e3090-597e-42df-be25-3a3c7104bf8b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:39:39.010716 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:39.010691    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/322e3090-597e-42df-be25-3a3c7104bf8b-kube-api-access-rwb4k" (OuterVolumeSpecName: "kube-api-access-rwb4k") pod "322e3090-597e-42df-be25-3a3c7104bf8b" (UID: "322e3090-597e-42df-be25-3a3c7104bf8b"). InnerVolumeSpecName "kube-api-access-rwb4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:39:39.014886 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:39.014852    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/322e3090-597e-42df-be25-3a3c7104bf8b-util" (OuterVolumeSpecName: "util") pod "322e3090-597e-42df-be25-3a3c7104bf8b" (UID: "322e3090-597e-42df-be25-3a3c7104bf8b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:39:39.109513 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:39.109463    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwb4k\" (UniqueName: \"kubernetes.io/projected/322e3090-597e-42df-be25-3a3c7104bf8b-kube-api-access-rwb4k\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 22:39:39.109513 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:39.109483    2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/322e3090-597e-42df-be25-3a3c7104bf8b-bundle\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 22:39:39.109513 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:39.109492    2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/322e3090-597e-42df-be25-3a3c7104bf8b-util\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 22:39:39.845463 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:39.845421    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc9lc4" event={"ID":"322e3090-597e-42df-be25-3a3c7104bf8b","Type":"ContainerDied","Data":"0b9a2b6a2cd114e7ed155c16fc825cff94e1fcdd9192966a2fc44273fb91ef5c"}
Jul 16 22:39:39.845463 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:39.845447    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afc9lc4"
Jul 16 22:39:39.845696 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:39.845474    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b9a2b6a2cd114e7ed155c16fc825cff94e1fcdd9192966a2fc44273fb91ef5c"
Jul 16 22:39:46.829986 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:46.829954    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:39:46.943738 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:46.943701    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:39:47.467224 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:47.467194    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:39:47.853324 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:47.853297    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:39:47.945173 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:47.945136    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:39:47.963723 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:47.963688    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-crfwm"]
Jul 16 22:39:47.963951 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:47.963938    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="322e3090-597e-42df-be25-3a3c7104bf8b" containerName="util"
Jul 16 22:39:47.963993 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:47.963952    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="322e3090-597e-42df-be25-3a3c7104bf8b" containerName="util"
Jul 16 22:39:47.963993 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:47.963967    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="322e3090-597e-42df-be25-3a3c7104bf8b" containerName="pull"
Jul 16 22:39:47.963993 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:47.963972    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="322e3090-597e-42df-be25-3a3c7104bf8b" containerName="pull"
Jul 16 22:39:47.963993 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:47.963980    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="322e3090-597e-42df-be25-3a3c7104bf8b" containerName="extract"
Jul 16 22:39:47.963993 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:47.963986    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="322e3090-597e-42df-be25-3a3c7104bf8b" containerName="extract"
Jul 16 22:39:47.964130 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:47.964037    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="322e3090-597e-42df-be25-3a3c7104bf8b" containerName="extract"
Jul 16 22:39:47.968171 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:47.968152    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-crfwm"
Jul 16 22:39:47.972145 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:47.972118    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 16 22:39:47.972575 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:47.972556    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 16 22:39:47.972965 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:47.972942    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-xs9sp\""
Jul 16 22:39:47.972965 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:47.972949    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 16 22:39:47.974629 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:47.974609    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 16 22:39:48.002983 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:48.002963    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-crfwm"]
Jul 16 22:39:48.067685 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:48.067660    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/64962d67-8646-4f77-8e5b-b2accdb71a35-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bb6b77ff4-crfwm\" (UID: \"64962d67-8646-4f77-8e5b-b2accdb71a35\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-crfwm"
Jul 16 22:39:48.067766 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:48.067692    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5szf\" (UniqueName: \"kubernetes.io/projected/64962d67-8646-4f77-8e5b-b2accdb71a35-kube-api-access-j5szf\") pod \"opendatahub-operator-controller-manager-6bb6b77ff4-crfwm\" (UID: \"64962d67-8646-4f77-8e5b-b2accdb71a35\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-crfwm"
Jul 16 22:39:48.067766 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:48.067714    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64962d67-8646-4f77-8e5b-b2accdb71a35-tmp\") pod \"opendatahub-operator-controller-manager-6bb6b77ff4-crfwm\" (UID: \"64962d67-8646-4f77-8e5b-b2accdb71a35\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-crfwm"
Jul 16 22:39:48.067838 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:48.067786    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/64962d67-8646-4f77-8e5b-b2accdb71a35-manifests\") pod \"opendatahub-operator-controller-manager-6bb6b77ff4-crfwm\" (UID: \"64962d67-8646-4f77-8e5b-b2accdb71a35\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-crfwm"
Jul 16 22:39:48.067870 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:48.067857    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/64962d67-8646-4f77-8e5b-b2accdb71a35-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bb6b77ff4-crfwm\" (UID: \"64962d67-8646-4f77-8e5b-b2accdb71a35\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-crfwm"
Jul 16 22:39:48.168371 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:48.168319    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5szf\" (UniqueName: \"kubernetes.io/projected/64962d67-8646-4f77-8e5b-b2accdb71a35-kube-api-access-j5szf\") pod \"opendatahub-operator-controller-manager-6bb6b77ff4-crfwm\" (UID: \"64962d67-8646-4f77-8e5b-b2accdb71a35\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-crfwm"
Jul 16 22:39:48.168371 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:48.168349    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64962d67-8646-4f77-8e5b-b2accdb71a35-tmp\") pod \"opendatahub-operator-controller-manager-6bb6b77ff4-crfwm\" (UID: \"64962d67-8646-4f77-8e5b-b2accdb71a35\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-crfwm"
Jul 16 22:39:48.168489 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:48.168376    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/64962d67-8646-4f77-8e5b-b2accdb71a35-manifests\") pod \"opendatahub-operator-controller-manager-6bb6b77ff4-crfwm\" (UID: \"64962d67-8646-4f77-8e5b-b2accdb71a35\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-crfwm"
Jul 16 22:39:48.168489 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:48.168417    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/64962d67-8646-4f77-8e5b-b2accdb71a35-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bb6b77ff4-crfwm\" (UID: \"64962d67-8646-4f77-8e5b-b2accdb71a35\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-crfwm"
Jul 16 22:39:48.168489 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:48.168447    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/64962d67-8646-4f77-8e5b-b2accdb71a35-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bb6b77ff4-crfwm\" (UID: \"64962d67-8646-4f77-8e5b-b2accdb71a35\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-crfwm"
Jul 16 22:39:48.168826 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:48.168805    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/64962d67-8646-4f77-8e5b-b2accdb71a35-manifests\") pod \"opendatahub-operator-controller-manager-6bb6b77ff4-crfwm\" (UID: \"64962d67-8646-4f77-8e5b-b2accdb71a35\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-crfwm"
Jul 16 22:39:48.170503 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:48.170478    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64962d67-8646-4f77-8e5b-b2accdb71a35-tmp\") pod \"opendatahub-operator-controller-manager-6bb6b77ff4-crfwm\" (UID: \"64962d67-8646-4f77-8e5b-b2accdb71a35\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-crfwm"
Jul 16 22:39:48.170832 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:48.170812    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/64962d67-8646-4f77-8e5b-b2accdb71a35-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bb6b77ff4-crfwm\" (UID: \"64962d67-8646-4f77-8e5b-b2accdb71a35\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-crfwm"
Jul 16 22:39:48.170885 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:48.170821    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/64962d67-8646-4f77-8e5b-b2accdb71a35-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bb6b77ff4-crfwm\" (UID: \"64962d67-8646-4f77-8e5b-b2accdb71a35\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-crfwm"
Jul 16 22:39:48.185376 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:48.185357    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5szf\" (UniqueName: \"kubernetes.io/projected/64962d67-8646-4f77-8e5b-b2accdb71a35-kube-api-access-j5szf\") pod \"opendatahub-operator-controller-manager-6bb6b77ff4-crfwm\" (UID: \"64962d67-8646-4f77-8e5b-b2accdb71a35\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-crfwm"
Jul 16 22:39:48.200149 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:48.200129    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:39:48.278423 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:48.278398    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-crfwm"
Jul 16 22:39:48.298903 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:48.298881    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:39:48.407390 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:48.407357    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:39:48.408198 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:48.408173    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-crfwm"]
Jul 16 22:39:48.411942 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:39:48.411919    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64962d67_8646_4f77_8e5b_b2accdb71a35.slice/crio-395121a0744400352cdc20095dd32ee6ac3c289ab5e3fc0b94cc4ea1ae560072 WatchSource:0}: Error finding container 395121a0744400352cdc20095dd32ee6ac3c289ab5e3fc0b94cc4ea1ae560072: Status 404 returned error can't find the container with id 395121a0744400352cdc20095dd32ee6ac3c289ab5e3fc0b94cc4ea1ae560072
Jul 16 22:39:48.508038 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:48.507974    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:39:48.601320 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:48.601296    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:39:48.702261 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:48.702217    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:39:48.838723 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:48.838700    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:39:48.879198 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:48.879175    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-crfwm" event={"ID":"64962d67-8646-4f77-8e5b-b2accdb71a35","Type":"ContainerStarted","Data":"395121a0744400352cdc20095dd32ee6ac3c289ab5e3fc0b94cc4ea1ae560072"}
Jul 16 22:39:49.033985 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:39:49.033951    2573 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image" image="quay.io/opendatahub/opendatahub-operator:v3.5.0-ea.2"
Jul 16 22:39:49.034078 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:39:49.034061    2573 kuberuntime_manager.go:1459] "Unhandled Error" err="init container copy-manifests start failed in pod opendatahub-operator-controller-manager-6bb6b77ff4-crfwm_opendatahub(64962d67-8646-4f77-8e5b-b2accdb71a35): ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image" logger="UnhandledError"
Jul 16 22:39:49.034126 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:39:49.034093    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"copy-manifests\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image\"" pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-crfwm" podUID="64962d67-8646-4f77-8e5b-b2accdb71a35"
Jul 16 22:39:49.884498 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:39:49.884451    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"copy-manifests\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/opendatahub/opendatahub-operator:v3.5.0-ea.2\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image\"" pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-crfwm" podUID="64962d67-8646-4f77-8e5b-b2accdb71a35"
Jul 16 22:39:55.072289 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:55.072255    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:39:55.156558 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:39:55.156529    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:05.956813 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:05.956775    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-crfwm" event={"ID":"64962d67-8646-4f77-8e5b-b2accdb71a35","Type":"ContainerStarted","Data":"aa1128f2c680c08f8c31e6099eb75a28dd62f0e4beb584d5e9a624be24cba624"}
Jul 16 22:40:06.962665 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:06.962628    2573 generic.go:356] "Generic (PLEG): container finished" podID="64962d67-8646-4f77-8e5b-b2accdb71a35" containerID="aa1128f2c680c08f8c31e6099eb75a28dd62f0e4beb584d5e9a624be24cba624" exitCode=0
Jul 16 22:40:06.963020 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:06.962703    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-crfwm" event={"ID":"64962d67-8646-4f77-8e5b-b2accdb71a35","Type":"ContainerDied","Data":"aa1128f2c680c08f8c31e6099eb75a28dd62f0e4beb584d5e9a624be24cba624"}
Jul 16 22:40:07.969568 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:07.969533    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-crfwm" event={"ID":"64962d67-8646-4f77-8e5b-b2accdb71a35","Type":"ContainerStarted","Data":"e586e8b1e568e99f86a696d7ed7f27ca17d715c1aed114d7334a83e4d32df47f"}
Jul 16 22:40:07.969960 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:07.969755    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-crfwm"
Jul 16 22:40:07.995091 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:07.995049    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-crfwm" podStartSLOduration=1.105443193 podStartE2EDuration="20.995035861s" totalImagesPullingTime="18.891755155s" totalInitContainerRuntime="997.837513ms" isStatefulPod=false podCreationTimestamp="2026-07-16 22:39:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-16 22:40:07.992576312 +0000 UTC m=+382.002838974" watchObservedRunningTime="2026-07-16 22:40:07.995035861 +0000 UTC m=+382.005298524"
Jul 16 22:40:18.976082 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:18.976048    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-crfwm"
Jul 16 22:40:20.843711 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:20.843681    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:21.122203 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:21.122133    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:21.433619 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:21.433555    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:21.570339 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:21.570307    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-b54d476c5-9kgs2"]
Jul 16 22:40:21.577769 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:21.577754    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b54d476c5-9kgs2"
Jul 16 22:40:21.590332 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:21.590315    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 16 22:40:21.606310 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:21.606291    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 16 22:40:21.667823 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:21.667799    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b54d476c5-9kgs2"]
Jul 16 22:40:21.701759 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:21.701709    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81eb7d79-86f5-4007-87df-a70530de357a-tls-certs\") pod \"kube-auth-proxy-b54d476c5-9kgs2\" (UID: \"81eb7d79-86f5-4007-87df-a70530de357a\") " pod="openshift-ingress/kube-auth-proxy-b54d476c5-9kgs2"
Jul 16 22:40:21.701759 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:21.701734    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81eb7d79-86f5-4007-87df-a70530de357a-tmp\") pod \"kube-auth-proxy-b54d476c5-9kgs2\" (UID: \"81eb7d79-86f5-4007-87df-a70530de357a\") " pod="openshift-ingress/kube-auth-proxy-b54d476c5-9kgs2"
Jul 16 22:40:21.701759 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:21.701751    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9lhp\" (UniqueName: \"kubernetes.io/projected/81eb7d79-86f5-4007-87df-a70530de357a-kube-api-access-b9lhp\") pod \"kube-auth-proxy-b54d476c5-9kgs2\" (UID: \"81eb7d79-86f5-4007-87df-a70530de357a\") " pod="openshift-ingress/kube-auth-proxy-b54d476c5-9kgs2"
Jul 16 22:40:21.713439 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:21.713418    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:21.802844 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:21.802820    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81eb7d79-86f5-4007-87df-a70530de357a-tls-certs\") pod \"kube-auth-proxy-b54d476c5-9kgs2\" (UID: \"81eb7d79-86f5-4007-87df-a70530de357a\") " pod="openshift-ingress/kube-auth-proxy-b54d476c5-9kgs2"
Jul 16 22:40:21.802925 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:21.802850    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81eb7d79-86f5-4007-87df-a70530de357a-tmp\") pod \"kube-auth-proxy-b54d476c5-9kgs2\" (UID: \"81eb7d79-86f5-4007-87df-a70530de357a\") " pod="openshift-ingress/kube-auth-proxy-b54d476c5-9kgs2"
Jul 16 22:40:21.802925 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:21.802869    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9lhp\" (UniqueName: \"kubernetes.io/projected/81eb7d79-86f5-4007-87df-a70530de357a-kube-api-access-b9lhp\") pod \"kube-auth-proxy-b54d476c5-9kgs2\" (UID: \"81eb7d79-86f5-4007-87df-a70530de357a\") " pod="openshift-ingress/kube-auth-proxy-b54d476c5-9kgs2"
Jul 16 22:40:21.805181 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:21.805153    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81eb7d79-86f5-4007-87df-a70530de357a-tmp\") pod \"kube-auth-proxy-b54d476c5-9kgs2\" (UID: \"81eb7d79-86f5-4007-87df-a70530de357a\") " pod="openshift-ingress/kube-auth-proxy-b54d476c5-9kgs2"
Jul 16 22:40:21.805303 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:21.805288    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81eb7d79-86f5-4007-87df-a70530de357a-tls-certs\") pod \"kube-auth-proxy-b54d476c5-9kgs2\" (UID: \"81eb7d79-86f5-4007-87df-a70530de357a\") " pod="openshift-ingress/kube-auth-proxy-b54d476c5-9kgs2"
Jul 16 22:40:21.814486 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:21.814466    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9lhp\" (UniqueName: \"kubernetes.io/projected/81eb7d79-86f5-4007-87df-a70530de357a-kube-api-access-b9lhp\") pod \"kube-auth-proxy-b54d476c5-9kgs2\" (UID: \"81eb7d79-86f5-4007-87df-a70530de357a\") " pod="openshift-ingress/kube-auth-proxy-b54d476c5-9kgs2"
Jul 16 22:40:21.886981 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:21.886957    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b54d476c5-9kgs2"
Jul 16 22:40:22.021979 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:22.021953    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b54d476c5-9kgs2"]
Jul 16 22:40:22.024315 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:40:22.024292    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81eb7d79_86f5_4007_87df_a70530de357a.slice/crio-21e512216d910eb1c39f018518217ec6e79acf41faa57d68aa11ce53324cd545 WatchSource:0}: Error finding container 21e512216d910eb1c39f018518217ec6e79acf41faa57d68aa11ce53324cd545: Status 404 returned error can't find the container with id 21e512216d910eb1c39f018518217ec6e79acf41faa57d68aa11ce53324cd545
Jul 16 22:40:22.039574 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:22.039547    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b54d476c5-9kgs2" event={"ID":"81eb7d79-86f5-4007-87df-a70530de357a","Type":"ContainerStarted","Data":"21e512216d910eb1c39f018518217ec6e79acf41faa57d68aa11ce53324cd545"}
Jul 16 22:40:22.049544 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:22.049524    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9bdlb"]
Jul 16 22:40:22.051798 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:22.051784    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-9bdlb"
Jul 16 22:40:22.056506 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:22.056487    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-f2ztv\""
Jul 16 22:40:22.068330 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:22.068312    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 16 22:40:22.089341 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:22.089319    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9bdlb"]
Jul 16 22:40:22.105525 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:22.105503    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/752d23b3-773c-484b-8e97-1c511173fdc8-cert\") pod \"odh-model-controller-7998c67dc4-9bdlb\" (UID: \"752d23b3-773c-484b-8e97-1c511173fdc8\") " pod="opendatahub/odh-model-controller-7998c67dc4-9bdlb"
Jul 16 22:40:22.105602 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:22.105560    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg25k\" (UniqueName: \"kubernetes.io/projected/752d23b3-773c-484b-8e97-1c511173fdc8-kube-api-access-hg25k\") pod \"odh-model-controller-7998c67dc4-9bdlb\" (UID: \"752d23b3-773c-484b-8e97-1c511173fdc8\") " pod="opendatahub/odh-model-controller-7998c67dc4-9bdlb"
Jul 16 22:40:22.134339 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:22.134308    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:22.206110 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:22.206089    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg25k\" (UniqueName: \"kubernetes.io/projected/752d23b3-773c-484b-8e97-1c511173fdc8-kube-api-access-hg25k\") pod \"odh-model-controller-7998c67dc4-9bdlb\" (UID: \"752d23b3-773c-484b-8e97-1c511173fdc8\") " pod="opendatahub/odh-model-controller-7998c67dc4-9bdlb"
Jul 16 22:40:22.206210 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:22.206129    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/752d23b3-773c-484b-8e97-1c511173fdc8-cert\") pod \"odh-model-controller-7998c67dc4-9bdlb\" (UID: \"752d23b3-773c-484b-8e97-1c511173fdc8\") " pod="opendatahub/odh-model-controller-7998c67dc4-9bdlb"
Jul 16 22:40:22.206287 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:40:22.206221    2573 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 16 22:40:22.206340 ip-10-0-140-146 kubenswrapper[2573]: E0716 22:40:22.206287    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/752d23b3-773c-484b-8e97-1c511173fdc8-cert podName:752d23b3-773c-484b-8e97-1c511173fdc8 nodeName:}" failed. No retries permitted until 2026-07-16 22:40:22.706271915 +0000 UTC m=+396.716534556 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/752d23b3-773c-484b-8e97-1c511173fdc8-cert") pod "odh-model-controller-7998c67dc4-9bdlb" (UID: "752d23b3-773c-484b-8e97-1c511173fdc8") : secret "odh-model-controller-webhook-cert" not found
Jul 16 22:40:22.227024 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:22.226971    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg25k\" (UniqueName: \"kubernetes.io/projected/752d23b3-773c-484b-8e97-1c511173fdc8-kube-api-access-hg25k\") pod \"odh-model-controller-7998c67dc4-9bdlb\" (UID: \"752d23b3-773c-484b-8e97-1c511173fdc8\") " pod="opendatahub/odh-model-controller-7998c67dc4-9bdlb"
Jul 16 22:40:22.276404 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:22.276384    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:22.489052 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:22.488990    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:22.709817 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:22.709790    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/752d23b3-773c-484b-8e97-1c511173fdc8-cert\") pod \"odh-model-controller-7998c67dc4-9bdlb\" (UID: \"752d23b3-773c-484b-8e97-1c511173fdc8\") " pod="opendatahub/odh-model-controller-7998c67dc4-9bdlb"
Jul 16 22:40:22.712152 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:22.712132    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/752d23b3-773c-484b-8e97-1c511173fdc8-cert\") pod \"odh-model-controller-7998c67dc4-9bdlb\" (UID: \"752d23b3-773c-484b-8e97-1c511173fdc8\") " pod="opendatahub/odh-model-controller-7998c67dc4-9bdlb"
Jul 16 22:40:22.714660 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:22.714636    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:22.961557 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:22.961533    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-9bdlb"
Jul 16 22:40:23.106018 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:40:23.105987    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod752d23b3_773c_484b_8e97_1c511173fdc8.slice/crio-ca26822d9be38a621ee3412da644f29473ffe4512c0dcc3e82892f736635a1db WatchSource:0}: Error finding container ca26822d9be38a621ee3412da644f29473ffe4512c0dcc3e82892f736635a1db: Status 404 returned error can't find the container with id ca26822d9be38a621ee3412da644f29473ffe4512c0dcc3e82892f736635a1db
Jul 16 22:40:23.107497 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:23.107471    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9bdlb"]
Jul 16 22:40:23.420905 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:23.420876    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:23.450113 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:23.450080    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-tnzvx"]
Jul 16 22:40:23.453291 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:23.453275    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-tnzvx"
Jul 16 22:40:23.456435 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:23.456418    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 16 22:40:23.457331 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:23.457314    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-xxskh\""
Jul 16 22:40:23.466354 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:23.466335    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-tnzvx"]
Jul 16 22:40:23.516656 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:23.516635    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjktv\" (UniqueName: \"kubernetes.io/projected/0e390942-75d4-4f6c-8e43-287304f8b838-kube-api-access-cjktv\") pod \"model-serving-api-54d89bd5fc-tnzvx\" (UID: \"0e390942-75d4-4f6c-8e43-287304f8b838\") " pod="opendatahub/model-serving-api-54d89bd5fc-tnzvx"
Jul 16 22:40:23.516765 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:23.516671    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e390942-75d4-4f6c-8e43-287304f8b838-tls-certs\") pod \"model-serving-api-54d89bd5fc-tnzvx\" (UID: \"0e390942-75d4-4f6c-8e43-287304f8b838\") " pod="opendatahub/model-serving-api-54d89bd5fc-tnzvx"
Jul 16 22:40:23.617001 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:23.616975    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjktv\" (UniqueName: \"kubernetes.io/projected/0e390942-75d4-4f6c-8e43-287304f8b838-kube-api-access-cjktv\") pod \"model-serving-api-54d89bd5fc-tnzvx\" (UID: \"0e390942-75d4-4f6c-8e43-287304f8b838\") " pod="opendatahub/model-serving-api-54d89bd5fc-tnzvx"
Jul 16 22:40:23.617120 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:23.617010    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e390942-75d4-4f6c-8e43-287304f8b838-tls-certs\") pod \"model-serving-api-54d89bd5fc-tnzvx\" (UID: \"0e390942-75d4-4f6c-8e43-287304f8b838\") " pod="opendatahub/model-serving-api-54d89bd5fc-tnzvx"
Jul 16 22:40:23.619541 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:23.619518    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e390942-75d4-4f6c-8e43-287304f8b838-tls-certs\") pod \"model-serving-api-54d89bd5fc-tnzvx\" (UID: \"0e390942-75d4-4f6c-8e43-287304f8b838\") " pod="opendatahub/model-serving-api-54d89bd5fc-tnzvx"
Jul 16 22:40:23.643272 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:23.643227    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjktv\" (UniqueName: \"kubernetes.io/projected/0e390942-75d4-4f6c-8e43-287304f8b838-kube-api-access-cjktv\") pod \"model-serving-api-54d89bd5fc-tnzvx\" (UID: \"0e390942-75d4-4f6c-8e43-287304f8b838\") " pod="opendatahub/model-serving-api-54d89bd5fc-tnzvx"
Jul 16 22:40:23.692699 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:23.692646    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:23.762759 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:23.762736    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-tnzvx"
Jul 16 22:40:23.924229 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:23.924112    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-tnzvx"]
Jul 16 22:40:24.014175 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:24.014052    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:24.051051 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:24.051019    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-9bdlb" event={"ID":"752d23b3-773c-484b-8e97-1c511173fdc8","Type":"ContainerStarted","Data":"ca26822d9be38a621ee3412da644f29473ffe4512c0dcc3e82892f736635a1db"}
Jul 16 22:40:24.114319 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:24.114286    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:25.058017 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:25.057934    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-tnzvx" event={"ID":"0e390942-75d4-4f6c-8e43-287304f8b838","Type":"ContainerStarted","Data":"237e9afab4d9acc9fff37be770b8c8c09e93c2913c7e7cb0782f4672e734dd9f"}
Jul 16 22:40:25.451770 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:25.451740    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:26.076347 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:26.076309    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b54d476c5-9kgs2" event={"ID":"81eb7d79-86f5-4007-87df-a70530de357a","Type":"ContainerStarted","Data":"fc3f264585d08ba49ebb5249ec4087f173cf70735af9cdedcb4a39a9d489a6e3"}
Jul 16 22:40:26.113104 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:26.113056    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-b54d476c5-9kgs2" podStartSLOduration=1.164412241 podStartE2EDuration="5.113036136s" totalImagesPullingTime="3.948623895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:40:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:40:26.112311741 +0000 UTC m=+400.122574431" watchObservedRunningTime="2026-07-16 22:40:26.113036136 +0000 UTC m=+400.123298799"
Jul 16 22:40:27.082356 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:27.082304    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-9bdlb" event={"ID":"752d23b3-773c-484b-8e97-1c511173fdc8","Type":"ContainerStarted","Data":"d8000a9e5b6828b7feffd8ab85c5437929da27f83dd565140e6516ad86487337"}
Jul 16 22:40:27.082759 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:27.082480    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-9bdlb"
Jul 16 22:40:27.083808 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:27.083785    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-tnzvx" event={"ID":"0e390942-75d4-4f6c-8e43-287304f8b838","Type":"ContainerStarted","Data":"40872a8622df56c3efc012357b70f03758d3f5e2f71dddd516c19cf1fdf699a1"}
Jul 16 22:40:27.102564 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:27.102496    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-9bdlb" podStartSLOduration=2.177339356 podStartE2EDuration="5.102481084s" totalImagesPullingTime="2.925141728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:40:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:40:27.101397793 +0000 UTC m=+401.111660456" watchObservedRunningTime="2026-07-16 22:40:27.102481084 +0000 UTC m=+401.112743747"
Jul 16 22:40:27.128876 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:27.128837    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-tnzvx" podStartSLOduration=2.263872582 podStartE2EDuration="4.128824325s" totalImagesPullingTime="1.864951743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:40:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:40:27.128372152 +0000 UTC m=+401.138634825" watchObservedRunningTime="2026-07-16 22:40:27.128824325 +0000 UTC m=+401.139086988"
Jul 16 22:40:27.243887 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:27.243861    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-84c74b69b8-xzh5k"]
Jul 16 22:40:27.246866 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:27.246849    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-84c74b69b8-xzh5k"
Jul 16 22:40:27.249754 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:27.249736    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 16 22:40:27.250015 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:27.249996    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-sj7rv\""
Jul 16 22:40:27.250264 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:27.250232    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 16 22:40:27.250264 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:27.250251    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 16 22:40:27.250425 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:27.250264    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 16 22:40:27.250425 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:27.250233    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 16 22:40:27.263043 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:27.263025    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-84c74b69b8-xzh5k"]
Jul 16 22:40:27.348328 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:27.348278    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c62a9d3-c22d-4f21-af46-c3f713816d18-metrics-cert\") pod \"lws-controller-manager-84c74b69b8-xzh5k\" (UID: \"6c62a9d3-c22d-4f21-af46-c3f713816d18\") " pod="openshift-lws-operator/lws-controller-manager-84c74b69b8-xzh5k"
Jul 16 22:40:27.348328 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:27.348311    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6c62a9d3-c22d-4f21-af46-c3f713816d18-manager-config\") pod \"lws-controller-manager-84c74b69b8-xzh5k\" (UID: \"6c62a9d3-c22d-4f21-af46-c3f713816d18\") " pod="openshift-lws-operator/lws-controller-manager-84c74b69b8-xzh5k"
Jul 16 22:40:27.348483 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:27.348344    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt7xr\" (UniqueName: \"kubernetes.io/projected/6c62a9d3-c22d-4f21-af46-c3f713816d18-kube-api-access-gt7xr\") pod \"lws-controller-manager-84c74b69b8-xzh5k\" (UID: \"6c62a9d3-c22d-4f21-af46-c3f713816d18\") " pod="openshift-lws-operator/lws-controller-manager-84c74b69b8-xzh5k"
Jul 16 22:40:27.348483 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:27.348399    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c62a9d3-c22d-4f21-af46-c3f713816d18-cert\") pod \"lws-controller-manager-84c74b69b8-xzh5k\" (UID: \"6c62a9d3-c22d-4f21-af46-c3f713816d18\") " pod="openshift-lws-operator/lws-controller-manager-84c74b69b8-xzh5k"
Jul 16 22:40:27.448768 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:27.448744    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt7xr\" (UniqueName: \"kubernetes.io/projected/6c62a9d3-c22d-4f21-af46-c3f713816d18-kube-api-access-gt7xr\") pod \"lws-controller-manager-84c74b69b8-xzh5k\" (UID: \"6c62a9d3-c22d-4f21-af46-c3f713816d18\") " pod="openshift-lws-operator/lws-controller-manager-84c74b69b8-xzh5k"
Jul 16 22:40:27.448871 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:27.448772    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c62a9d3-c22d-4f21-af46-c3f713816d18-cert\") pod \"lws-controller-manager-84c74b69b8-xzh5k\" (UID: \"6c62a9d3-c22d-4f21-af46-c3f713816d18\") " pod="openshift-lws-operator/lws-controller-manager-84c74b69b8-xzh5k"
Jul 16 22:40:27.448871 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:27.448809    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c62a9d3-c22d-4f21-af46-c3f713816d18-metrics-cert\") pod \"lws-controller-manager-84c74b69b8-xzh5k\" (UID: \"6c62a9d3-c22d-4f21-af46-c3f713816d18\") " pod="openshift-lws-operator/lws-controller-manager-84c74b69b8-xzh5k"
Jul 16 22:40:27.448871 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:27.448843    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6c62a9d3-c22d-4f21-af46-c3f713816d18-manager-config\") pod \"lws-controller-manager-84c74b69b8-xzh5k\" (UID: \"6c62a9d3-c22d-4f21-af46-c3f713816d18\") " pod="openshift-lws-operator/lws-controller-manager-84c74b69b8-xzh5k"
Jul 16 22:40:27.449520 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:27.449501    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6c62a9d3-c22d-4f21-af46-c3f713816d18-manager-config\") pod \"lws-controller-manager-84c74b69b8-xzh5k\" (UID: \"6c62a9d3-c22d-4f21-af46-c3f713816d18\") " pod="openshift-lws-operator/lws-controller-manager-84c74b69b8-xzh5k"
Jul 16 22:40:27.451249 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:27.451215    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c62a9d3-c22d-4f21-af46-c3f713816d18-metrics-cert\") pod \"lws-controller-manager-84c74b69b8-xzh5k\" (UID: \"6c62a9d3-c22d-4f21-af46-c3f713816d18\") " pod="openshift-lws-operator/lws-controller-manager-84c74b69b8-xzh5k"
Jul 16 22:40:27.451322 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:27.451301    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c62a9d3-c22d-4f21-af46-c3f713816d18-cert\") pod \"lws-controller-manager-84c74b69b8-xzh5k\" (UID: \"6c62a9d3-c22d-4f21-af46-c3f713816d18\") " pod="openshift-lws-operator/lws-controller-manager-84c74b69b8-xzh5k"
Jul 16 22:40:27.458552 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:27.458534    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt7xr\" (UniqueName: \"kubernetes.io/projected/6c62a9d3-c22d-4f21-af46-c3f713816d18-kube-api-access-gt7xr\") pod \"lws-controller-manager-84c74b69b8-xzh5k\" (UID: \"6c62a9d3-c22d-4f21-af46-c3f713816d18\") " pod="openshift-lws-operator/lws-controller-manager-84c74b69b8-xzh5k"
Jul 16 22:40:27.555763 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:27.555733    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-84c74b69b8-xzh5k"
Jul 16 22:40:27.745977 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:40:27.745942    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c62a9d3_c22d_4f21_af46_c3f713816d18.slice/crio-91ff8748d23e66f0b1570f6eb25d984c16ec6db8d9b2c1618dc0b984ec7edf9f WatchSource:0}: Error finding container 91ff8748d23e66f0b1570f6eb25d984c16ec6db8d9b2c1618dc0b984ec7edf9f: Status 404 returned error can't find the container with id 91ff8748d23e66f0b1570f6eb25d984c16ec6db8d9b2c1618dc0b984ec7edf9f
Jul 16 22:40:27.750334 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:27.750310    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-84c74b69b8-xzh5k"]
Jul 16 22:40:28.089175 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:28.089137    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-84c74b69b8-xzh5k" event={"ID":"6c62a9d3-c22d-4f21-af46-c3f713816d18","Type":"ContainerStarted","Data":"91ff8748d23e66f0b1570f6eb25d984c16ec6db8d9b2c1618dc0b984ec7edf9f"}
Jul 16 22:40:28.089542 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:28.089340    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-tnzvx"
Jul 16 22:40:28.270036 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:28.270010    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:28.392759 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:28.392681    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:29.549611 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:29.549535    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:29.781093 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:29.781062    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:29.912990 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:29.912965    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:30.102659 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:30.102626    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-84c74b69b8-xzh5k" event={"ID":"6c62a9d3-c22d-4f21-af46-c3f713816d18","Type":"ContainerStarted","Data":"33344a244551206c2c32f94003751c9c88441832e586f7d41a95e6a5ce835723"}
Jul 16 22:40:30.102775 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:30.102758    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-84c74b69b8-xzh5k"
Jul 16 22:40:34.093963 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:34.093922    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:34.334785 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:34.334752    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:34.516123 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:34.516057    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:34.781083 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:34.781021    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:35.519539 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:35.519504    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:35.637342 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:35.637304    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:35.803446 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:35.803375    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:38.091909 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:38.091878    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-9bdlb"
Jul 16 22:40:38.159340 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:38.159289    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-84c74b69b8-xzh5k" podStartSLOduration=9.696654007 podStartE2EDuration="11.159271737s" totalImagesPullingTime="1.46261773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:40:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:40:30.146095922 +0000 UTC m=+404.156358595" watchObservedRunningTime="2026-07-16 22:40:38.159271737 +0000 UTC m=+412.169534398"
Jul 16 22:40:39.101076 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:39.101045    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-tnzvx"
Jul 16 22:40:41.109489 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:40:41.109449    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-84c74b69b8-xzh5k"
Jul 16 22:41:22.318788 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:22.318708    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:41:22.419118 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:22.419087    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:41:22.505809 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:22.505781    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:41:37.868522 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:37.868487    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1k4f4l"]
Jul 16 22:41:37.870648 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:37.870632    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1k4f4l"
Jul 16 22:41:37.877287 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:37.877267    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 16 22:41:37.877407 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:37.877353    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-vhn47\""
Jul 16 22:41:37.878062 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:37.878048    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 16 22:41:37.892266 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:37.892230    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1k4f4l"]
Jul 16 22:41:37.930103 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:37.930083    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm7ls\" (UniqueName: \"kubernetes.io/projected/3aa4994d-8f92-42a9-9150-fbd6f9c164fb-kube-api-access-zm7ls\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1k4f4l\" (UID: \"3aa4994d-8f92-42a9-9150-fbd6f9c164fb\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1k4f4l"
Jul 16 22:41:37.930191 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:37.930117    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3aa4994d-8f92-42a9-9150-fbd6f9c164fb-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1k4f4l\" (UID: \"3aa4994d-8f92-42a9-9150-fbd6f9c164fb\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1k4f4l"
Jul 16 22:41:37.930191 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:37.930139    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3aa4994d-8f92-42a9-9150-fbd6f9c164fb-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1k4f4l\" (UID: \"3aa4994d-8f92-42a9-9150-fbd6f9c164fb\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1k4f4l"
Jul 16 22:41:38.030762 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:38.030742    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm7ls\" (UniqueName: \"kubernetes.io/projected/3aa4994d-8f92-42a9-9150-fbd6f9c164fb-kube-api-access-zm7ls\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1k4f4l\" (UID: \"3aa4994d-8f92-42a9-9150-fbd6f9c164fb\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1k4f4l"
Jul 16 22:41:38.030857 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:38.030777    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3aa4994d-8f92-42a9-9150-fbd6f9c164fb-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1k4f4l\" (UID: \"3aa4994d-8f92-42a9-9150-fbd6f9c164fb\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1k4f4l"
Jul 16 22:41:38.030857 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:38.030806    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3aa4994d-8f92-42a9-9150-fbd6f9c164fb-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1k4f4l\" (UID: \"3aa4994d-8f92-42a9-9150-fbd6f9c164fb\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1k4f4l"
Jul 16 22:41:38.031108 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:38.031090    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3aa4994d-8f92-42a9-9150-fbd6f9c164fb-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1k4f4l\" (UID: \"3aa4994d-8f92-42a9-9150-fbd6f9c164fb\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1k4f4l"
Jul 16 22:41:38.031158 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:38.031143    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3aa4994d-8f92-42a9-9150-fbd6f9c164fb-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1k4f4l\" (UID: \"3aa4994d-8f92-42a9-9150-fbd6f9c164fb\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1k4f4l"
Jul 16 22:41:38.043010 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:38.042990    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm7ls\" (UniqueName: \"kubernetes.io/projected/3aa4994d-8f92-42a9-9150-fbd6f9c164fb-kube-api-access-zm7ls\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1k4f4l\" (UID: \"3aa4994d-8f92-42a9-9150-fbd6f9c164fb\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1k4f4l"
Jul 16 22:41:38.178876 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:38.178825    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1k4f4l"
Jul 16 22:41:38.315771 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:38.315743    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1k4f4l"]
Jul 16 22:41:38.319104 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:41:38.319077    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa4994d_8f92_42a9_9150_fbd6f9c164fb.slice/crio-39f6e77e8512391d939d92eb233659c528f38a0b76cbcc943a723ca9a2c5307f WatchSource:0}: Error finding container 39f6e77e8512391d939d92eb233659c528f38a0b76cbcc943a723ca9a2c5307f: Status 404 returned error can't find the container with id 39f6e77e8512391d939d92eb233659c528f38a0b76cbcc943a723ca9a2c5307f
Jul 16 22:41:38.385470 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:38.385434    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1k4f4l" event={"ID":"3aa4994d-8f92-42a9-9150-fbd6f9c164fb","Type":"ContainerStarted","Data":"0c1430746498091d953e28b3b7031f101b73d964b1c1c15e95048b0bd95636c1"}
Jul 16 22:41:38.385572 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:38.385505    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1k4f4l" event={"ID":"3aa4994d-8f92-42a9-9150-fbd6f9c164fb","Type":"ContainerStarted","Data":"39f6e77e8512391d939d92eb233659c528f38a0b76cbcc943a723ca9a2c5307f"}
Jul 16 22:41:38.700164 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:38.700141    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qlxnc"]
Jul 16 22:41:38.702410 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:38.702395    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qlxnc"
Jul 16 22:41:38.723079 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:38.723056    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qlxnc"]
Jul 16 22:41:38.734770 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:38.734746    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c2a323d2-7e88-4e00-9c09-918cffae5ae2-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qlxnc\" (UID: \"c2a323d2-7e88-4e00-9c09-918cffae5ae2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qlxnc"
Jul 16 22:41:38.734853 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:38.734793    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c2a323d2-7e88-4e00-9c09-918cffae5ae2-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qlxnc\" (UID: \"c2a323d2-7e88-4e00-9c09-918cffae5ae2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qlxnc"
Jul 16 22:41:38.734905 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:38.734859    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srjg2\" (UniqueName: \"kubernetes.io/projected/c2a323d2-7e88-4e00-9c09-918cffae5ae2-kube-api-access-srjg2\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qlxnc\" (UID: \"c2a323d2-7e88-4e00-9c09-918cffae5ae2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qlxnc"
Jul 16 22:41:38.835657 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:38.835637    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c2a323d2-7e88-4e00-9c09-918cffae5ae2-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qlxnc\" (UID: \"c2a323d2-7e88-4e00-9c09-918cffae5ae2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qlxnc"
Jul 16 22:41:38.835739 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:38.835678    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srjg2\" (UniqueName: \"kubernetes.io/projected/c2a323d2-7e88-4e00-9c09-918cffae5ae2-kube-api-access-srjg2\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qlxnc\" (UID: \"c2a323d2-7e88-4e00-9c09-918cffae5ae2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qlxnc"
Jul 16 22:41:38.835739 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:38.835730    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c2a323d2-7e88-4e00-9c09-918cffae5ae2-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qlxnc\" (UID: \"c2a323d2-7e88-4e00-9c09-918cffae5ae2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qlxnc"
Jul 16 22:41:38.836027 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:38.836009    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c2a323d2-7e88-4e00-9c09-918cffae5ae2-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qlxnc\" (UID: \"c2a323d2-7e88-4e00-9c09-918cffae5ae2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qlxnc"
Jul 16 22:41:38.836065 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:38.836044    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c2a323d2-7e88-4e00-9c09-918cffae5ae2-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qlxnc\" (UID: \"c2a323d2-7e88-4e00-9c09-918cffae5ae2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qlxnc"
Jul 16 22:41:38.847646 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:38.847625    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srjg2\" (UniqueName: \"kubernetes.io/projected/c2a323d2-7e88-4e00-9c09-918cffae5ae2-kube-api-access-srjg2\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qlxnc\" (UID: \"c2a323d2-7e88-4e00-9c09-918cffae5ae2\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qlxnc"
Jul 16 22:41:39.011121 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:39.011067    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qlxnc"
Jul 16 22:41:39.148043 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:39.147934    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qlxnc"]
Jul 16 22:41:39.150430 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:41:39.150407    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2a323d2_7e88_4e00_9c09_918cffae5ae2.slice/crio-6987b7dae7eef2ff2ff89bbd1866cff7dffe598280534b7989c31349d14598e9 WatchSource:0}: Error finding container 6987b7dae7eef2ff2ff89bbd1866cff7dffe598280534b7989c31349d14598e9: Status 404 returned error can't find the container with id 6987b7dae7eef2ff2ff89bbd1866cff7dffe598280534b7989c31349d14598e9
Jul 16 22:41:39.391709 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:39.391679    2573 generic.go:356] "Generic (PLEG): container finished" podID="c2a323d2-7e88-4e00-9c09-918cffae5ae2" containerID="0ef89d17b698369c88ec141870a03bf9261b426eae43caae5f9cf00dfc5f9744" exitCode=0
Jul 16 22:41:39.391818 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:39.391735    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qlxnc" event={"ID":"c2a323d2-7e88-4e00-9c09-918cffae5ae2","Type":"ContainerDied","Data":"0ef89d17b698369c88ec141870a03bf9261b426eae43caae5f9cf00dfc5f9744"}
Jul 16 22:41:39.391818 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:39.391787    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qlxnc" event={"ID":"c2a323d2-7e88-4e00-9c09-918cffae5ae2","Type":"ContainerStarted","Data":"6987b7dae7eef2ff2ff89bbd1866cff7dffe598280534b7989c31349d14598e9"}
Jul 16 22:41:39.393316 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:39.393293    2573 generic.go:356] "Generic (PLEG): container finished" podID="3aa4994d-8f92-42a9-9150-fbd6f9c164fb" containerID="0c1430746498091d953e28b3b7031f101b73d964b1c1c15e95048b0bd95636c1" exitCode=0
Jul 16 22:41:39.393411 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:39.393347    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1k4f4l" event={"ID":"3aa4994d-8f92-42a9-9150-fbd6f9c164fb","Type":"ContainerDied","Data":"0c1430746498091d953e28b3b7031f101b73d964b1c1c15e95048b0bd95636c1"}
Jul 16 22:41:39.468746 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:39.468722    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hqb5q"]
Jul 16 22:41:39.470834 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:39.470818    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hqb5q"
Jul 16 22:41:39.483285 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:39.482791    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hqb5q"]
Jul 16 22:41:39.540809 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:39.540776    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83bf8282-8854-4c8b-967f-09b40c128e93-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hqb5q\" (UID: \"83bf8282-8854-4c8b-967f-09b40c128e93\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hqb5q"
Jul 16 22:41:39.540942 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:39.540831    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgrqx\" (UniqueName: \"kubernetes.io/projected/83bf8282-8854-4c8b-967f-09b40c128e93-kube-api-access-cgrqx\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hqb5q\" (UID: \"83bf8282-8854-4c8b-967f-09b40c128e93\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hqb5q"
Jul 16 22:41:39.540942 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:39.540868    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83bf8282-8854-4c8b-967f-09b40c128e93-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hqb5q\" (UID: \"83bf8282-8854-4c8b-967f-09b40c128e93\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hqb5q"
Jul 16 22:41:39.641492 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:39.641461    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83bf8282-8854-4c8b-967f-09b40c128e93-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hqb5q\" (UID: \"83bf8282-8854-4c8b-967f-09b40c128e93\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hqb5q"
Jul 16 22:41:39.641648 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:39.641503    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgrqx\" (UniqueName: \"kubernetes.io/projected/83bf8282-8854-4c8b-967f-09b40c128e93-kube-api-access-cgrqx\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hqb5q\" (UID: \"83bf8282-8854-4c8b-967f-09b40c128e93\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hqb5q"
Jul 16 22:41:39.641648 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:39.641532    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83bf8282-8854-4c8b-967f-09b40c128e93-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hqb5q\" (UID: \"83bf8282-8854-4c8b-967f-09b40c128e93\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hqb5q"
Jul 16 22:41:39.641843 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:39.641797    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83bf8282-8854-4c8b-967f-09b40c128e93-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hqb5q\" (UID: \"83bf8282-8854-4c8b-967f-09b40c128e93\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hqb5q"
Jul 16 22:41:39.641843 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:39.641828    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83bf8282-8854-4c8b-967f-09b40c128e93-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hqb5q\" (UID: \"83bf8282-8854-4c8b-967f-09b40c128e93\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hqb5q"
Jul 16 22:41:39.650081 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:39.650051    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgrqx\" (UniqueName: \"kubernetes.io/projected/83bf8282-8854-4c8b-967f-09b40c128e93-kube-api-access-cgrqx\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hqb5q\" (UID: \"83bf8282-8854-4c8b-967f-09b40c128e93\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hqb5q"
Jul 16 22:41:39.784713 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:39.784684    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hqb5q"
Jul 16 22:41:39.908760 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:39.908736    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hqb5q"]
Jul 16 22:41:39.910806 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:41:39.910781    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83bf8282_8854_4c8b_967f_09b40c128e93.slice/crio-1fd79b60a1ec5aad24e5904becce83dbb58fe87a9e71509a566c6b85990e46e7 WatchSource:0}: Error finding container 1fd79b60a1ec5aad24e5904becce83dbb58fe87a9e71509a566c6b85990e46e7: Status 404 returned error can't find the container with id 1fd79b60a1ec5aad24e5904becce83dbb58fe87a9e71509a566c6b85990e46e7
Jul 16 22:41:40.044773 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:40.044738    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwrc8"]
Jul 16 22:41:40.048066 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:40.048050    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwrc8"
Jul 16 22:41:40.057041 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:40.057019    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwrc8"]
Jul 16 22:41:40.144601 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:40.144570    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5llf6\" (UniqueName: \"kubernetes.io/projected/2c500107-02bb-469f-8b28-5eebf1ed90b8-kube-api-access-5llf6\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwrc8\" (UID: \"2c500107-02bb-469f-8b28-5eebf1ed90b8\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwrc8"
Jul 16 22:41:40.144755 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:40.144642    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c500107-02bb-469f-8b28-5eebf1ed90b8-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwrc8\" (UID: \"2c500107-02bb-469f-8b28-5eebf1ed90b8\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwrc8"
Jul 16 22:41:40.144755 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:40.144670    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c500107-02bb-469f-8b28-5eebf1ed90b8-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwrc8\" (UID: \"2c500107-02bb-469f-8b28-5eebf1ed90b8\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwrc8"
Jul 16 22:41:40.244951 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:40.244920    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5llf6\" (UniqueName: \"kubernetes.io/projected/2c500107-02bb-469f-8b28-5eebf1ed90b8-kube-api-access-5llf6\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwrc8\" (UID: \"2c500107-02bb-469f-8b28-5eebf1ed90b8\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwrc8"
Jul 16 22:41:40.245043 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:40.244966    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c500107-02bb-469f-8b28-5eebf1ed90b8-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwrc8\" (UID: \"2c500107-02bb-469f-8b28-5eebf1ed90b8\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwrc8"
Jul 16 22:41:40.245043 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:40.244985    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c500107-02bb-469f-8b28-5eebf1ed90b8-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwrc8\" (UID: \"2c500107-02bb-469f-8b28-5eebf1ed90b8\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwrc8"
Jul 16 22:41:40.245290 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:40.245270    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c500107-02bb-469f-8b28-5eebf1ed90b8-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwrc8\" (UID: \"2c500107-02bb-469f-8b28-5eebf1ed90b8\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwrc8"
Jul 16 22:41:40.245365 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:40.245350    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c500107-02bb-469f-8b28-5eebf1ed90b8-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwrc8\" (UID: \"2c500107-02bb-469f-8b28-5eebf1ed90b8\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwrc8"
Jul 16 22:41:40.254547 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:40.254522    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5llf6\" (UniqueName: \"kubernetes.io/projected/2c500107-02bb-469f-8b28-5eebf1ed90b8-kube-api-access-5llf6\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwrc8\" (UID: \"2c500107-02bb-469f-8b28-5eebf1ed90b8\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwrc8"
Jul 16 22:41:40.371655 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:40.371632    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwrc8"
Jul 16 22:41:40.400009 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:40.399983    2573 generic.go:356] "Generic (PLEG): container finished" podID="83bf8282-8854-4c8b-967f-09b40c128e93" containerID="2b7cbe7981f9472d6089b9be59d044c15d9d2035b8ad6159a2c43396aca7cc43" exitCode=0
Jul 16 22:41:40.400127 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:40.400062    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hqb5q" event={"ID":"83bf8282-8854-4c8b-967f-09b40c128e93","Type":"ContainerDied","Data":"2b7cbe7981f9472d6089b9be59d044c15d9d2035b8ad6159a2c43396aca7cc43"}
Jul 16 22:41:40.400127 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:40.400121    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hqb5q" event={"ID":"83bf8282-8854-4c8b-967f-09b40c128e93","Type":"ContainerStarted","Data":"1fd79b60a1ec5aad24e5904becce83dbb58fe87a9e71509a566c6b85990e46e7"}
Jul 16 22:41:40.402005 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:40.401980    2573 generic.go:356] "Generic (PLEG): container finished" podID="3aa4994d-8f92-42a9-9150-fbd6f9c164fb" containerID="3d92e2911b6903ff05e921292a26b0a2738e15ac373e8a27ff0a1d8915478d33" exitCode=0
Jul 16 22:41:40.402110 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:40.402064    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1k4f4l" event={"ID":"3aa4994d-8f92-42a9-9150-fbd6f9c164fb","Type":"ContainerDied","Data":"3d92e2911b6903ff05e921292a26b0a2738e15ac373e8a27ff0a1d8915478d33"}
Jul 16 22:41:40.404445 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:40.404398    2573 generic.go:356] "Generic (PLEG): container finished" podID="c2a323d2-7e88-4e00-9c09-918cffae5ae2" containerID="698e45e7c89cde244185577d7dbb7c993685456166c3a452cb3b35ff2e643307" exitCode=0
Jul 16 22:41:40.404445 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:40.404437    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qlxnc" event={"ID":"c2a323d2-7e88-4e00-9c09-918cffae5ae2","Type":"ContainerDied","Data":"698e45e7c89cde244185577d7dbb7c993685456166c3a452cb3b35ff2e643307"}
Jul 16 22:41:40.509633 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:40.509526    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwrc8"]
Jul 16 22:41:40.512189 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:41:40.512159    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c500107_02bb_469f_8b28_5eebf1ed90b8.slice/crio-0628b50fc1b458e0e2aa840b2b300bd80ad94a87ff4c31ce9f60913f8af91ef8 WatchSource:0}: Error finding container 0628b50fc1b458e0e2aa840b2b300bd80ad94a87ff4c31ce9f60913f8af91ef8: Status 404 returned error can't find the container with id 0628b50fc1b458e0e2aa840b2b300bd80ad94a87ff4c31ce9f60913f8af91ef8
Jul 16 22:41:41.411891 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:41.411827    2573 generic.go:356] "Generic (PLEG): container finished" podID="3aa4994d-8f92-42a9-9150-fbd6f9c164fb" containerID="1d7553433fea7435220c0d2fc90daeeba58dcc96a13992ce1366a0b2e0469fc6" exitCode=0
Jul 16 22:41:41.412258 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:41.411911    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1k4f4l" event={"ID":"3aa4994d-8f92-42a9-9150-fbd6f9c164fb","Type":"ContainerDied","Data":"1d7553433fea7435220c0d2fc90daeeba58dcc96a13992ce1366a0b2e0469fc6"}
Jul 16 22:41:41.413719 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:41.413699    2573 generic.go:356] "Generic (PLEG): container finished" podID="c2a323d2-7e88-4e00-9c09-918cffae5ae2" containerID="bfb810ce8f60ebf1112f39eaac7e41b94cab8a0c812211a588a99755212961b3" exitCode=0
Jul 16 22:41:41.413821 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:41.413782    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qlxnc" event={"ID":"c2a323d2-7e88-4e00-9c09-918cffae5ae2","Type":"ContainerDied","Data":"bfb810ce8f60ebf1112f39eaac7e41b94cab8a0c812211a588a99755212961b3"}
Jul 16 22:41:41.415202 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:41.415183    2573 generic.go:356] "Generic (PLEG): container finished" podID="83bf8282-8854-4c8b-967f-09b40c128e93" containerID="01d796d1ae1cda116b00c514cbd5a77edc36b1beec3c825815366d18e9db08f1" exitCode=0
Jul 16 22:41:41.415333 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:41.415265    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hqb5q" event={"ID":"83bf8282-8854-4c8b-967f-09b40c128e93","Type":"ContainerDied","Data":"01d796d1ae1cda116b00c514cbd5a77edc36b1beec3c825815366d18e9db08f1"}
Jul 16 22:41:41.416766 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:41.416670    2573 generic.go:356] "Generic (PLEG): container finished" podID="2c500107-02bb-469f-8b28-5eebf1ed90b8" containerID="ad2fa047f96d7e50e37fea673ed9fc3830be35d8fd0fc07d1e27c22e86c08236" exitCode=0
Jul 16 22:41:41.416766 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:41.416732    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwrc8" event={"ID":"2c500107-02bb-469f-8b28-5eebf1ed90b8","Type":"ContainerDied","Data":"ad2fa047f96d7e50e37fea673ed9fc3830be35d8fd0fc07d1e27c22e86c08236"}
Jul 16 22:41:41.416847 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:41.416766    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwrc8" event={"ID":"2c500107-02bb-469f-8b28-5eebf1ed90b8","Type":"ContainerStarted","Data":"0628b50fc1b458e0e2aa840b2b300bd80ad94a87ff4c31ce9f60913f8af91ef8"}
Jul 16 22:41:42.423372 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:42.423339    2573 generic.go:356] "Generic (PLEG): container finished" podID="83bf8282-8854-4c8b-967f-09b40c128e93" containerID="883ff84c905770bd049a23c8fe8ce911f15f9c2d2b5b4c0e27f157408b2fc930" exitCode=0
Jul 16 22:41:42.423717 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:42.423396    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hqb5q" event={"ID":"83bf8282-8854-4c8b-967f-09b40c128e93","Type":"ContainerDied","Data":"883ff84c905770bd049a23c8fe8ce911f15f9c2d2b5b4c0e27f157408b2fc930"}
Jul 16 22:41:42.425143 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:42.425120    2573 generic.go:356] "Generic (PLEG): container finished" podID="2c500107-02bb-469f-8b28-5eebf1ed90b8" containerID="2cd966d9ba9732ff9f2eb03d7b660463193d8063c0eba38a40b14c9eb96bf80f" exitCode=0
Jul 16 22:41:42.425248 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:42.425205    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwrc8" event={"ID":"2c500107-02bb-469f-8b28-5eebf1ed90b8","Type":"ContainerDied","Data":"2cd966d9ba9732ff9f2eb03d7b660463193d8063c0eba38a40b14c9eb96bf80f"}
Jul 16 22:41:42.558872 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:42.558847    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qlxnc"
Jul 16 22:41:42.563295 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:42.563273    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-srjg2\" (UniqueName: \"kubernetes.io/projected/c2a323d2-7e88-4e00-9c09-918cffae5ae2-kube-api-access-srjg2\") pod \"c2a323d2-7e88-4e00-9c09-918cffae5ae2\" (UID: \"c2a323d2-7e88-4e00-9c09-918cffae5ae2\") "
Jul 16 22:41:42.563407 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:42.563326    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c2a323d2-7e88-4e00-9c09-918cffae5ae2-util\") pod \"c2a323d2-7e88-4e00-9c09-918cffae5ae2\" (UID: \"c2a323d2-7e88-4e00-9c09-918cffae5ae2\") "
Jul 16 22:41:42.563407 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:42.563385    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c2a323d2-7e88-4e00-9c09-918cffae5ae2-bundle\") pod \"c2a323d2-7e88-4e00-9c09-918cffae5ae2\" (UID: \"c2a323d2-7e88-4e00-9c09-918cffae5ae2\") "
Jul 16 22:41:42.563889 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:42.563863    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2a323d2-7e88-4e00-9c09-918cffae5ae2-bundle" (OuterVolumeSpecName: "bundle") pod "c2a323d2-7e88-4e00-9c09-918cffae5ae2" (UID: "c2a323d2-7e88-4e00-9c09-918cffae5ae2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:41:42.565795 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:42.565773    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2a323d2-7e88-4e00-9c09-918cffae5ae2-kube-api-access-srjg2" (OuterVolumeSpecName: "kube-api-access-srjg2") pod "c2a323d2-7e88-4e00-9c09-918cffae5ae2" (UID: "c2a323d2-7e88-4e00-9c09-918cffae5ae2"). InnerVolumeSpecName "kube-api-access-srjg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:41:42.568602 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:42.568577    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2a323d2-7e88-4e00-9c09-918cffae5ae2-util" (OuterVolumeSpecName: "util") pod "c2a323d2-7e88-4e00-9c09-918cffae5ae2" (UID: "c2a323d2-7e88-4e00-9c09-918cffae5ae2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:41:42.623805 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:42.623786    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1k4f4l"
Jul 16 22:41:42.664506 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:42.664485    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3aa4994d-8f92-42a9-9150-fbd6f9c164fb-util\") pod \"3aa4994d-8f92-42a9-9150-fbd6f9c164fb\" (UID: \"3aa4994d-8f92-42a9-9150-fbd6f9c164fb\") "
Jul 16 22:41:42.664623 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:42.664539    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zm7ls\" (UniqueName: \"kubernetes.io/projected/3aa4994d-8f92-42a9-9150-fbd6f9c164fb-kube-api-access-zm7ls\") pod \"3aa4994d-8f92-42a9-9150-fbd6f9c164fb\" (UID: \"3aa4994d-8f92-42a9-9150-fbd6f9c164fb\") "
Jul 16 22:41:42.664623 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:42.664558    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3aa4994d-8f92-42a9-9150-fbd6f9c164fb-bundle\") pod \"3aa4994d-8f92-42a9-9150-fbd6f9c164fb\" (UID: \"3aa4994d-8f92-42a9-9150-fbd6f9c164fb\") "
Jul 16 22:41:42.664735 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:42.664709    2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c2a323d2-7e88-4e00-9c09-918cffae5ae2-bundle\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 22:41:42.664735 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:42.664727    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srjg2\" (UniqueName: \"kubernetes.io/projected/c2a323d2-7e88-4e00-9c09-918cffae5ae2-kube-api-access-srjg2\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 22:41:42.664735 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:42.664742    2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c2a323d2-7e88-4e00-9c09-918cffae5ae2-util\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 22:41:42.665031 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:42.665007    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa4994d-8f92-42a9-9150-fbd6f9c164fb-bundle" (OuterVolumeSpecName: "bundle") pod "3aa4994d-8f92-42a9-9150-fbd6f9c164fb" (UID: "3aa4994d-8f92-42a9-9150-fbd6f9c164fb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:41:42.666413 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:42.666392    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aa4994d-8f92-42a9-9150-fbd6f9c164fb-kube-api-access-zm7ls" (OuterVolumeSpecName: "kube-api-access-zm7ls") pod "3aa4994d-8f92-42a9-9150-fbd6f9c164fb" (UID: "3aa4994d-8f92-42a9-9150-fbd6f9c164fb"). InnerVolumeSpecName "kube-api-access-zm7ls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:41:42.669783 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:42.669746    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa4994d-8f92-42a9-9150-fbd6f9c164fb-util" (OuterVolumeSpecName: "util") pod "3aa4994d-8f92-42a9-9150-fbd6f9c164fb" (UID: "3aa4994d-8f92-42a9-9150-fbd6f9c164fb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:41:42.765686 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:42.765667    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm7ls\" (UniqueName: \"kubernetes.io/projected/3aa4994d-8f92-42a9-9150-fbd6f9c164fb-kube-api-access-zm7ls\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 22:41:42.765686 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:42.765687    2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3aa4994d-8f92-42a9-9150-fbd6f9c164fb-bundle\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 22:41:42.765804 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:42.765696    2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3aa4994d-8f92-42a9-9150-fbd6f9c164fb-util\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 22:41:43.431531 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:43.431503    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1k4f4l"
Jul 16 22:41:43.431531 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:43.431524    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1k4f4l" event={"ID":"3aa4994d-8f92-42a9-9150-fbd6f9c164fb","Type":"ContainerDied","Data":"39f6e77e8512391d939d92eb233659c528f38a0b76cbcc943a723ca9a2c5307f"}
Jul 16 22:41:43.432027 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:43.431570    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39f6e77e8512391d939d92eb233659c528f38a0b76cbcc943a723ca9a2c5307f"
Jul 16 22:41:43.433295 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:43.433272    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qlxnc" event={"ID":"c2a323d2-7e88-4e00-9c09-918cffae5ae2","Type":"ContainerDied","Data":"6987b7dae7eef2ff2ff89bbd1866cff7dffe598280534b7989c31349d14598e9"}
Jul 16 22:41:43.433417 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:43.433299    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qlxnc"
Jul 16 22:41:43.433417 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:43.433314    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6987b7dae7eef2ff2ff89bbd1866cff7dffe598280534b7989c31349d14598e9"
Jul 16 22:41:43.435538 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:43.435511    2573 generic.go:356] "Generic (PLEG): container finished" podID="2c500107-02bb-469f-8b28-5eebf1ed90b8" containerID="a04429a27c5d8f0990ec01508bd22b80fa3d42f290e452ce9d6db7c035650915" exitCode=0
Jul 16 22:41:43.435649 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:43.435589    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwrc8" event={"ID":"2c500107-02bb-469f-8b28-5eebf1ed90b8","Type":"ContainerDied","Data":"a04429a27c5d8f0990ec01508bd22b80fa3d42f290e452ce9d6db7c035650915"}
Jul 16 22:41:43.556349 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:43.556329    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hqb5q"
Jul 16 22:41:43.571039 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:43.571018    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83bf8282-8854-4c8b-967f-09b40c128e93-bundle\") pod \"83bf8282-8854-4c8b-967f-09b40c128e93\" (UID: \"83bf8282-8854-4c8b-967f-09b40c128e93\") "
Jul 16 22:41:43.571147 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:43.571067    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgrqx\" (UniqueName: \"kubernetes.io/projected/83bf8282-8854-4c8b-967f-09b40c128e93-kube-api-access-cgrqx\") pod \"83bf8282-8854-4c8b-967f-09b40c128e93\" (UID: \"83bf8282-8854-4c8b-967f-09b40c128e93\") "
Jul 16 22:41:43.571147 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:43.571104    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83bf8282-8854-4c8b-967f-09b40c128e93-util\") pod \"83bf8282-8854-4c8b-967f-09b40c128e93\" (UID: \"83bf8282-8854-4c8b-967f-09b40c128e93\") "
Jul 16 22:41:43.571634 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:43.571612    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83bf8282-8854-4c8b-967f-09b40c128e93-bundle" (OuterVolumeSpecName: "bundle") pod "83bf8282-8854-4c8b-967f-09b40c128e93" (UID: "83bf8282-8854-4c8b-967f-09b40c128e93"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:41:43.572977 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:43.572951    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83bf8282-8854-4c8b-967f-09b40c128e93-kube-api-access-cgrqx" (OuterVolumeSpecName: "kube-api-access-cgrqx") pod "83bf8282-8854-4c8b-967f-09b40c128e93" (UID: "83bf8282-8854-4c8b-967f-09b40c128e93"). InnerVolumeSpecName "kube-api-access-cgrqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:41:43.576487 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:43.576453    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83bf8282-8854-4c8b-967f-09b40c128e93-util" (OuterVolumeSpecName: "util") pod "83bf8282-8854-4c8b-967f-09b40c128e93" (UID: "83bf8282-8854-4c8b-967f-09b40c128e93"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:41:43.672330 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:43.672298    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgrqx\" (UniqueName: \"kubernetes.io/projected/83bf8282-8854-4c8b-967f-09b40c128e93-kube-api-access-cgrqx\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 22:41:43.672330 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:43.672323    2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83bf8282-8854-4c8b-967f-09b40c128e93-util\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 22:41:43.672330 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:43.672335    2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83bf8282-8854-4c8b-967f-09b40c128e93-bundle\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 22:41:44.441948 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:44.441917    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hqb5q"
Jul 16 22:41:44.442330 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:44.441942    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hqb5q" event={"ID":"83bf8282-8854-4c8b-967f-09b40c128e93","Type":"ContainerDied","Data":"1fd79b60a1ec5aad24e5904becce83dbb58fe87a9e71509a566c6b85990e46e7"}
Jul 16 22:41:44.442330 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:44.441987    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fd79b60a1ec5aad24e5904becce83dbb58fe87a9e71509a566c6b85990e46e7"
Jul 16 22:41:44.570564 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:44.570533    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwrc8"
Jul 16 22:41:44.579627 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:44.579602    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c500107-02bb-469f-8b28-5eebf1ed90b8-bundle\") pod \"2c500107-02bb-469f-8b28-5eebf1ed90b8\" (UID: \"2c500107-02bb-469f-8b28-5eebf1ed90b8\") "
Jul 16 22:41:44.579762 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:44.579665    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c500107-02bb-469f-8b28-5eebf1ed90b8-util\") pod \"2c500107-02bb-469f-8b28-5eebf1ed90b8\" (UID: \"2c500107-02bb-469f-8b28-5eebf1ed90b8\") "
Jul 16 22:41:44.579762 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:44.579688    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5llf6\" (UniqueName: \"kubernetes.io/projected/2c500107-02bb-469f-8b28-5eebf1ed90b8-kube-api-access-5llf6\") pod \"2c500107-02bb-469f-8b28-5eebf1ed90b8\" (UID: \"2c500107-02bb-469f-8b28-5eebf1ed90b8\") "
Jul 16 22:41:44.580159 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:44.580057    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c500107-02bb-469f-8b28-5eebf1ed90b8-bundle" (OuterVolumeSpecName: "bundle") pod "2c500107-02bb-469f-8b28-5eebf1ed90b8" (UID: "2c500107-02bb-469f-8b28-5eebf1ed90b8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:41:44.582058 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:44.582021    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c500107-02bb-469f-8b28-5eebf1ed90b8-kube-api-access-5llf6" (OuterVolumeSpecName: "kube-api-access-5llf6") pod "2c500107-02bb-469f-8b28-5eebf1ed90b8" (UID: "2c500107-02bb-469f-8b28-5eebf1ed90b8"). InnerVolumeSpecName "kube-api-access-5llf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:41:44.585789 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:44.585755    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c500107-02bb-469f-8b28-5eebf1ed90b8-util" (OuterVolumeSpecName: "util") pod "2c500107-02bb-469f-8b28-5eebf1ed90b8" (UID: "2c500107-02bb-469f-8b28-5eebf1ed90b8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:41:44.680437 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:44.680415    2573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c500107-02bb-469f-8b28-5eebf1ed90b8-util\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 22:41:44.680437 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:44.680437    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5llf6\" (UniqueName: \"kubernetes.io/projected/2c500107-02bb-469f-8b28-5eebf1ed90b8-kube-api-access-5llf6\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 22:41:44.680574 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:44.680447    2573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c500107-02bb-469f-8b28-5eebf1ed90b8-bundle\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 22:41:45.447569 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:45.447533    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwrc8" event={"ID":"2c500107-02bb-469f-8b28-5eebf1ed90b8","Type":"ContainerDied","Data":"0628b50fc1b458e0e2aa840b2b300bd80ad94a87ff4c31ce9f60913f8af91ef8"}
Jul 16 22:41:45.447569 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:45.447551    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwrc8"
Jul 16 22:41:45.447940 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:45.447577    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0628b50fc1b458e0e2aa840b2b300bd80ad94a87ff4c31ce9f60913f8af91ef8"
Jul 16 22:41:46.481013 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:46.480982    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:41:46.584834 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:46.584795    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:41:50.450682 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:50.450642    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:41:50.520285 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:50.520254    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:41:52.214257 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:52.214212    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:41:55.706306 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:55.706258    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:41:57.233408 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:57.233362    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:41:57.623166 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:57.623108    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:41:58.659271 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:58.659213    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:41:58.762014 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:41:58.761988    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:42:00.213715 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:42:00.213677    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:42:01.332548 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:42:01.332512    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:42:01.442538 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:42:01.442506    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:42:01.537719 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:42:01.537687    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:42:02.621720 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:42:02.621692    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:42:03.857942 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:42:03.857908    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:42:06.482856 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:42:06.482822    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:42:06.712193 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:42:06.712160    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:42:07.249596 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:42:07.249562    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:42:07.550729 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:42:07.550640    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:42:16.510668 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:42:16.510632    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:42:16.866079 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:42:16.866033    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:42:18.955571 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:42:18.955535    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:42:19.888533 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:42:19.888488    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:42:20.293099 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:42:20.293018    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:42:22.328260 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:42:22.328206    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:42:22.787653 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:42:22.787577    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:42:28.268454 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:42:28.268421    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:42:28.467426 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:42:28.467398    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:42:36.715754 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:42:36.715718    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:42:36.869254 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:42:36.869210    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:42:38.939148 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:42:38.939079    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:42:39.712922 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:42:39.712894    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:42:39.838746 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:42:39.838714    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:42:40.538099 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:42:40.538072    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:42:40.651410 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:42:40.651374    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:42:42.299470 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:42:42.299433    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:42:42.511642 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:42:42.511612    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:42:57.938676 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:42:57.938643    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:42:58.538482 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:42:58.538450    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:43:00.202772 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.202736    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:43:00.380475 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.380444    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:43:00.459106 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.459041    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:43:00.549598 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.549571    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:43:00.549861 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.549848    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83bf8282-8854-4c8b-967f-09b40c128e93" containerName="extract"
Jul 16 22:43:00.549903 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.549863    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="83bf8282-8854-4c8b-967f-09b40c128e93" containerName="extract"
Jul 16 22:43:00.549903 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.549872    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3aa4994d-8f92-42a9-9150-fbd6f9c164fb" containerName="pull"
Jul 16 22:43:00.549903 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.549882    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3aa4994d-8f92-42a9-9150-fbd6f9c164fb" containerName="pull"
Jul 16 22:43:00.549903 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.549893    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c500107-02bb-469f-8b28-5eebf1ed90b8" containerName="util"
Jul 16 22:43:00.549903 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.549899    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c500107-02bb-469f-8b28-5eebf1ed90b8" containerName="util"
Jul 16 22:43:00.550107 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.549911    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2a323d2-7e88-4e00-9c09-918cffae5ae2" containerName="extract"
Jul 16 22:43:00.550107 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.549916    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2a323d2-7e88-4e00-9c09-918cffae5ae2" containerName="extract"
Jul 16 22:43:00.550107 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.549923    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2a323d2-7e88-4e00-9c09-918cffae5ae2" containerName="util"
Jul 16 22:43:00.550107 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.549929    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2a323d2-7e88-4e00-9c09-918cffae5ae2" containerName="util"
Jul 16 22:43:00.550107 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.549936    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2a323d2-7e88-4e00-9c09-918cffae5ae2" containerName="pull"
Jul 16 22:43:00.550107 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.549941    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2a323d2-7e88-4e00-9c09-918cffae5ae2" containerName="pull"
Jul 16 22:43:00.550107 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.549951    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c500107-02bb-469f-8b28-5eebf1ed90b8" containerName="pull"
Jul 16 22:43:00.550107 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.549958    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c500107-02bb-469f-8b28-5eebf1ed90b8" containerName="pull"
Jul 16 22:43:00.550107 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.549966    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c500107-02bb-469f-8b28-5eebf1ed90b8" containerName="extract"
Jul 16 22:43:00.550107 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.549973    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c500107-02bb-469f-8b28-5eebf1ed90b8" containerName="extract"
Jul 16 22:43:00.550107 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.549980    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3aa4994d-8f92-42a9-9150-fbd6f9c164fb" containerName="extract"
Jul 16 22:43:00.550107 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.549985    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3aa4994d-8f92-42a9-9150-fbd6f9c164fb" containerName="extract"
Jul 16 22:43:00.550107 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.549993    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3aa4994d-8f92-42a9-9150-fbd6f9c164fb" containerName="util"
Jul 16 22:43:00.550107 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.549997    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3aa4994d-8f92-42a9-9150-fbd6f9c164fb" containerName="util"
Jul 16 22:43:00.550107 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.550002    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83bf8282-8854-4c8b-967f-09b40c128e93" containerName="util"
Jul 16 22:43:00.550107 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.550006    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="83bf8282-8854-4c8b-967f-09b40c128e93" containerName="util"
Jul 16 22:43:00.550107 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.550011    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83bf8282-8854-4c8b-967f-09b40c128e93" containerName="pull"
Jul 16 22:43:00.550107 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.550015    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="83bf8282-8854-4c8b-967f-09b40c128e93" containerName="pull"
Jul 16 22:43:00.550107 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.550064    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2a323d2-7e88-4e00-9c09-918cffae5ae2" containerName="extract"
Jul 16 22:43:00.550107 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.550072    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3aa4994d-8f92-42a9-9150-fbd6f9c164fb" containerName="extract"
Jul 16 22:43:00.550107 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.550078    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c500107-02bb-469f-8b28-5eebf1ed90b8" containerName="extract"
Jul 16 22:43:00.550107 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.550084    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="83bf8282-8854-4c8b-967f-09b40c128e93" containerName="extract"
Jul 16 22:43:00.553028 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.553010    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-48ptc"
Jul 16 22:43:00.555837 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.555820    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 16 22:43:00.556679 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.556657    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 16 22:43:00.556775 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.556679    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 16 22:43:00.556775 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.556684    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-vhn47\""
Jul 16 22:43:00.564232 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.564209    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:43:00.604785 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.604760    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:43:00.697224 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.697197    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxzvc\" (UniqueName: \"kubernetes.io/projected/5380f474-4078-4ad2-a64f-15523278be37-kube-api-access-nxzvc\") pod \"limitador-limitador-54d6d988d5-48ptc\" (UID: \"5380f474-4078-4ad2-a64f-15523278be37\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-48ptc"
Jul 16 22:43:00.697332 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.697232    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5380f474-4078-4ad2-a64f-15523278be37-config-file\") pod \"limitador-limitador-54d6d988d5-48ptc\" (UID: \"5380f474-4078-4ad2-a64f-15523278be37\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-48ptc"
Jul 16 22:43:00.797685 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.797631    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxzvc\" (UniqueName: \"kubernetes.io/projected/5380f474-4078-4ad2-a64f-15523278be37-kube-api-access-nxzvc\") pod \"limitador-limitador-54d6d988d5-48ptc\" (UID: \"5380f474-4078-4ad2-a64f-15523278be37\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-48ptc"
Jul 16 22:43:00.797685 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.797662    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5380f474-4078-4ad2-a64f-15523278be37-config-file\") pod \"limitador-limitador-54d6d988d5-48ptc\" (UID: \"5380f474-4078-4ad2-a64f-15523278be37\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-48ptc"
Jul 16 22:43:00.798201 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.798185    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5380f474-4078-4ad2-a64f-15523278be37-config-file\") pod \"limitador-limitador-54d6d988d5-48ptc\" (UID: \"5380f474-4078-4ad2-a64f-15523278be37\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-48ptc"
Jul 16 22:43:00.807356 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.807334    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxzvc\" (UniqueName: \"kubernetes.io/projected/5380f474-4078-4ad2-a64f-15523278be37-kube-api-access-nxzvc\") pod \"limitador-limitador-54d6d988d5-48ptc\" (UID: \"5380f474-4078-4ad2-a64f-15523278be37\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-48ptc"
Jul 16 22:43:00.862574 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.862553    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-48ptc"
Jul 16 22:43:00.986457 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:00.986440    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:43:00.988659 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:43:00.988632    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5380f474_4078_4ad2_a64f_15523278be37.slice/crio-e2e894144a3327af5ecce08b7ebfb97e14d24149941fae0ef20b0a5c70a6e142 WatchSource:0}: Error finding container e2e894144a3327af5ecce08b7ebfb97e14d24149941fae0ef20b0a5c70a6e142: Status 404 returned error can't find the container with id e2e894144a3327af5ecce08b7ebfb97e14d24149941fae0ef20b0a5c70a6e142
Jul 16 22:43:01.059052 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:01.058994    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:43:01.156783 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:01.156761    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:43:01.779335 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:01.779252    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-48ptc" event={"ID":"5380f474-4078-4ad2-a64f-15523278be37","Type":"ContainerStarted","Data":"e2e894144a3327af5ecce08b7ebfb97e14d24149941fae0ef20b0a5c70a6e142"}
Jul 16 22:43:03.789015 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:03.788950    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-48ptc" event={"ID":"5380f474-4078-4ad2-a64f-15523278be37","Type":"ContainerStarted","Data":"dbcda4b3795addda362957e69fe70918dfde126a5dc0a9f9c528da9002983685"}
Jul 16 22:43:03.789425 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:03.789085    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-48ptc"
Jul 16 22:43:03.807747 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:03.807704    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-48ptc" podStartSLOduration=1.264879692 podStartE2EDuration="3.807691608s" totalImagesPullingTime="2.542811916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:43:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:43:03.805719307 +0000 UTC m=+557.815981999" watchObservedRunningTime="2026-07-16 22:43:03.807691608 +0000 UTC m=+557.817954270"
Jul 16 22:43:14.793989 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:14.793964    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-48ptc"
Jul 16 22:43:16.347143 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:16.347110    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:43:16.502700 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:16.502666    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:43:16.789066 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:16.788998    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:43:28.920690 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:28.920658    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:43:29.116033 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:29.115993    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:43:35.177337 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:35.177306    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:43:35.273991 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:35.273961    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:43:35.416611 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:35.416585    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:43:39.001678 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:39.001646    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:43:39.115959 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:39.115927    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:43:46.384359 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:46.384331    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r9jm4_506f904a-3272-4586-8528-146884e728f9/ovn-acl-logging/0.log"
Jul 16 22:43:46.384950 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:43:46.384831    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r9jm4_506f904a-3272-4586-8528-146884e728f9/ovn-acl-logging/0.log"
Jul 16 22:44:12.222710 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:44:12.222615    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:44:13.804967 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:44:13.804931    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:44:13.887125 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:44:13.887082    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:44:24.580019 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:44:24.579981    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:44:24.742380 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:44:24.742349    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:44:43.096295 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:44:43.096260    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:44:43.601271 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:44:43.601222    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:44:48.865332 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:44:48.865304    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:44:49.554940 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:44:49.554905    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:45:10.082253 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:10.082200    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:45:11.333123 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:11.333092    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:45:23.976659 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:23.976628    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:45:24.049536 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:24.049503    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:45:30.923084 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:30.923051    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:45:32.557418 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:32.557377    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:45:35.773606 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:35.773558    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:45:38.367535 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:38.367499    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs"]
Jul 16 22:45:38.370599 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:38.370581    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs"
Jul 16 22:45:38.374757 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:38.374733    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 16 22:45:38.374911 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:38.374735    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 16 22:45:38.374911 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:38.374735    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dgvpk\""
Jul 16 22:45:38.374911 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:38.374776    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 16 22:45:38.397483 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:38.397456    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs"]
Jul 16 22:45:38.440185 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:38.440149    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:45:38.465801 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:38.465780    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeec56ff-611a-40e0-bda4-006ef5149278-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs\" (UID: \"eeec56ff-611a-40e0-bda4-006ef5149278\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs"
Jul 16 22:45:38.465920 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:38.465827    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeec56ff-611a-40e0-bda4-006ef5149278-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs\" (UID: \"eeec56ff-611a-40e0-bda4-006ef5149278\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs"
Jul 16 22:45:38.465920 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:38.465848    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7lh7\" (UniqueName: \"kubernetes.io/projected/eeec56ff-611a-40e0-bda4-006ef5149278-kube-api-access-c7lh7\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs\" (UID: \"eeec56ff-611a-40e0-bda4-006ef5149278\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs"
Jul 16 22:45:38.465920 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:38.465865    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeec56ff-611a-40e0-bda4-006ef5149278-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs\" (UID: \"eeec56ff-611a-40e0-bda4-006ef5149278\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs"
Jul 16 22:45:38.466077 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:38.465957    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeec56ff-611a-40e0-bda4-006ef5149278-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs\" (UID: \"eeec56ff-611a-40e0-bda4-006ef5149278\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs"
Jul 16 22:45:38.466077 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:38.465995    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeec56ff-611a-40e0-bda4-006ef5149278-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs\" (UID: \"eeec56ff-611a-40e0-bda4-006ef5149278\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs"
Jul 16 22:45:38.560576 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:38.560556    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:45:38.567084 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:38.567064    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeec56ff-611a-40e0-bda4-006ef5149278-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs\" (UID: \"eeec56ff-611a-40e0-bda4-006ef5149278\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs"
Jul 16 22:45:38.567154 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:38.567090    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeec56ff-611a-40e0-bda4-006ef5149278-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs\" (UID: \"eeec56ff-611a-40e0-bda4-006ef5149278\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs"
Jul 16 22:45:38.567154 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:38.567109    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeec56ff-611a-40e0-bda4-006ef5149278-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs\" (UID: \"eeec56ff-611a-40e0-bda4-006ef5149278\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs"
Jul 16 22:45:38.567154 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:38.567143    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeec56ff-611a-40e0-bda4-006ef5149278-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs\" (UID: \"eeec56ff-611a-40e0-bda4-006ef5149278\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs"
Jul 16 22:45:38.567329 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:38.567266    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7lh7\" (UniqueName: \"kubernetes.io/projected/eeec56ff-611a-40e0-bda4-006ef5149278-kube-api-access-c7lh7\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs\" (UID: \"eeec56ff-611a-40e0-bda4-006ef5149278\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs"
Jul 16 22:45:38.567329 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:38.567294    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeec56ff-611a-40e0-bda4-006ef5149278-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs\" (UID: \"eeec56ff-611a-40e0-bda4-006ef5149278\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs"
Jul 16 22:45:38.567509 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:38.567488    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeec56ff-611a-40e0-bda4-006ef5149278-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs\" (UID: \"eeec56ff-611a-40e0-bda4-006ef5149278\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs"
Jul 16 22:45:38.567592 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:38.567522    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeec56ff-611a-40e0-bda4-006ef5149278-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs\" (UID: \"eeec56ff-611a-40e0-bda4-006ef5149278\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs"
Jul 16 22:45:38.567631 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:38.567596    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeec56ff-611a-40e0-bda4-006ef5149278-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs\" (UID: \"eeec56ff-611a-40e0-bda4-006ef5149278\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs"
Jul 16 22:45:38.569271 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:38.569228    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeec56ff-611a-40e0-bda4-006ef5149278-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs\" (UID: \"eeec56ff-611a-40e0-bda4-006ef5149278\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs"
Jul 16 22:45:38.569532 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:38.569516    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeec56ff-611a-40e0-bda4-006ef5149278-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs\" (UID: \"eeec56ff-611a-40e0-bda4-006ef5149278\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs"
Jul 16 22:45:38.578620 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:38.578600    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7lh7\" (UniqueName: \"kubernetes.io/projected/eeec56ff-611a-40e0-bda4-006ef5149278-kube-api-access-c7lh7\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs\" (UID: \"eeec56ff-611a-40e0-bda4-006ef5149278\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs"
Jul 16 22:45:38.679275 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:38.679181    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs"
Jul 16 22:45:38.831875 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:38.831847    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs"]
Jul 16 22:45:38.835567 ip-10-0-140-146 kubenswrapper[2573]: W0716 22:45:38.835539    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeeec56ff_611a_40e0_bda4_006ef5149278.slice/crio-affbd41277c6f73471100731ae8b54229f6263e9336c87ba8910296eba41c210 WatchSource:0}: Error finding container affbd41277c6f73471100731ae8b54229f6263e9336c87ba8910296eba41c210: Status 404 returned error can't find the container with id affbd41277c6f73471100731ae8b54229f6263e9336c87ba8910296eba41c210
Jul 16 22:45:38.836728 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:38.836713    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 22:45:39.469006 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:39.468971    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs" event={"ID":"eeec56ff-611a-40e0-bda4-006ef5149278","Type":"ContainerStarted","Data":"affbd41277c6f73471100731ae8b54229f6263e9336c87ba8910296eba41c210"}
Jul 16 22:45:48.352001 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:48.351960    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:45:49.527723 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:49.527641    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs" event={"ID":"eeec56ff-611a-40e0-bda4-006ef5149278","Type":"ContainerStarted","Data":"cafe89508f27c72418bc917e70b8a6bacbff9a2786d60c9022c38d2db4141460"}
Jul 16 22:45:49.554675 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:49.554619    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs" podStartSLOduration=1.131564945 podStartE2EDuration="11.554600283s" totalImagesPullingTime="10.423035338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:45:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:45:49.548718365 +0000 UTC m=+723.558981068" watchObservedRunningTime="2026-07-16 22:45:49.554600283 +0000 UTC m=+723.564862946"
Jul 16 22:45:58.679838 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:58.679801    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs"
Jul 16 22:45:58.680361 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:58.679852    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs"
Jul 16 22:45:58.691807 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:58.691784    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs"
Jul 16 22:45:59.585492 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:45:59.585466    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs"
Jul 16 22:46:25.749643 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:46:25.749607    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:46:25.849411 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:46:25.849322    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:46:39.493582 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:46:39.493550    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:46:48.154106 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:46:48.154067    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:46:48.245147 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:46:48.245112    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:46:49.395555 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:46:49.395524    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:46:49.528784 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:46:49.528750    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:47:16.147994 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:47:16.147907    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:47:19.775626 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:47:19.775592    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:47:19.876517 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:47:19.876490    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:47:26.442609 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:47:26.442570    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:47:29.494919 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:47:29.494887    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:47:38.156542 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:47:38.156505    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:48:16.022450 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:48:16.022410    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:48:46.412789 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:48:46.412702    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r9jm4_506f904a-3272-4586-8528-146884e728f9/ovn-acl-logging/0.log"
Jul 16 22:48:46.414340 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:48:46.414311    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r9jm4_506f904a-3272-4586-8528-146884e728f9/ovn-acl-logging/0.log"
Jul 16 22:48:52.882764 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:48:52.882727    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:48:54.043765 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:48:54.043735    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:49:17.842062 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:49:17.842021    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:49:32.685016 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:49:32.684982    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:49:34.554838 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:49:34.554808    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:49:35.838220 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:49:35.838188    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:49:40.721314 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:49:40.721279    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:49:40.785379 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:49:40.785348    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:49:40.866503 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:49:40.866462    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:49:41.076656 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:49:41.076627    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:49:46.693846 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:49:46.693813    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:49:46.767029 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:49:46.767002    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:49:48.740280 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:49:48.740251    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:49:58.438587 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:49:58.438551    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:50:07.535813 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:50:07.535774    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:50:14.833154 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:50:14.833084    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:50:15.341094 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:50:15.341052    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:50:24.831939 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:50:24.831895    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:50:31.437464 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:50:31.437432    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:50:43.244720 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:50:43.244682    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:50:53.245808 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:50:53.245768    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:51:05.050266 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:51:05.050096    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:51:15.031360 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:51:15.031327    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:51:37.349326 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:51:37.349287    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:51:40.344620 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:51:40.344549    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:51:50.070726 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:51:50.070685    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:52:12.436186 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:52:12.436152    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:52:23.033377 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:52:23.033343    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:52:40.546490 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:52:40.546452    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:52:50.352481 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:52:50.352435    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:53:05.842134 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:53:05.842099    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:53:46.447087 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:53:46.447005    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r9jm4_506f904a-3272-4586-8528-146884e728f9/ovn-acl-logging/0.log"
Jul 16 22:53:46.448776 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:53:46.448755    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r9jm4_506f904a-3272-4586-8528-146884e728f9/ovn-acl-logging/0.log"
Jul 16 22:55:07.738895 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:55:07.738857    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:55:17.351833 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:55:17.351796    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:55:31.051734 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:55:31.051695    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:55:42.932787 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:55:42.932748    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:55:52.835874 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:55:52.835828    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:56:04.938254 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:56:04.938213    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:56:13.191607 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:56:13.191530    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:56:24.341171 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:56:24.341125    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:56:34.040801 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:56:34.040762    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:56:44.338288 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:56:44.338250    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:56:57.067949 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:56:57.067908    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:57:18.943230 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:57:18.943175    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:57:28.754563 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:57:28.754521    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:57:42.944564 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:57:42.944476    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:57:54.831772 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:57:54.831732    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:58:04.141043 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:58:04.141004    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:58:10.339693 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:58:10.339648    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:58:12.265926 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:58:12.265882    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:58:12.371994 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:58:12.371962    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:58:20.335943 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:58:20.335898    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:58:29.534739 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:58:29.534698    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:58:37.673803 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:58:37.673751    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:58:39.475178 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:58:39.475137    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 22:58:46.480653 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:58:46.480617    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r9jm4_506f904a-3272-4586-8528-146884e728f9/ovn-acl-logging/0.log"
Jul 16 22:58:46.482753 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:58:46.482730    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r9jm4_506f904a-3272-4586-8528-146884e728f9/ovn-acl-logging/0.log"
Jul 16 22:58:47.361662 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:58:47.361624    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:58:56.734593 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:58:56.734552    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:59:10.134217 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:59:10.134140    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:59:19.755249 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:59:19.755194    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:59:29.832900 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:59:29.832857    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:59:40.133786 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:59:40.133747    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 22:59:52.837489 ip-10-0-140-146 kubenswrapper[2573]: I0716 22:59:52.837456    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:00:02.039619 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:00:02.039584    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:00:18.587075 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:00:18.587040    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:00:21.048382 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:00:21.048345    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:00:29.335254 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:00:29.335207    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:00:33.633308 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:00:33.633268    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:00:43.745169 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:00:43.745084    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:00:45.885615 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:00:45.885570    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:00:53.280155 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:00:53.280124    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:00:57.433192 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:00:57.433152    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:01:01.517411 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:01:01.517376    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:01:06.575175 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:01:06.575139    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:01:09.131793 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:01:09.131757    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:01:09.467567 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:01:09.467495    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:01:09.567454 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:01:09.567425    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:01:09.933879 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:01:09.933846    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:01:15.708064 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:01:15.708029    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:01:15.808615 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:01:15.808589    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:01:15.923313 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:01:15.923286    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:01:15.993438 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:01:15.993385    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:01:17.260156 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:01:17.260122    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:01:17.553770 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:01:17.553700    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:01:17.785480 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:01:17.785452    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:01:18.533455 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:01:18.533420    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:01:18.729419 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:01:18.729392    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:01:19.874999 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:01:19.874967    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:01:24.486882 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:01:24.486850    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:03:17.911545 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:03:17.911513    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:03:19.461389 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:03:19.461355    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:03:46.514229 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:03:46.514195    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r9jm4_506f904a-3272-4586-8528-146884e728f9/ovn-acl-logging/0.log"
Jul 16 23:03:46.519990 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:03:46.519970    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r9jm4_506f904a-3272-4586-8528-146884e728f9/ovn-acl-logging/0.log"
Jul 16 23:04:15.799708 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:04:15.799676    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:04:15.919711 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:04:15.919685    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:04:22.307580 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:04:22.307545    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:04:22.403454 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:04:22.403428    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:04:24.963683 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:04:24.963649    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:04:28.262098 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:04:28.262058    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:04:28.776589 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:04:28.776555    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:04:34.856055 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:04:34.856016    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:04:34.956528 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:04:34.956480    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:04:35.113285 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:04:35.113175    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:04:35.231567 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:04:35.231535    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:04:35.371256 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:04:35.371167    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:04:36.607909 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:04:36.607874    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:04:36.909172 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:04:36.909091    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:04:37.248039 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:04:37.247957    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:04:37.589470 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:04:37.589443    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:04:37.707454 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:04:37.707424    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:04:39.161191 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:04:39.161159    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:04:39.500130 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:04:39.500097    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:04:39.623818 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:04:39.623792    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:04:43.848581 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:04:43.848548    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:05:31.177777 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:31.177743    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-bpkkv"]
Jul 16 23:05:31.180740 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:31.180725    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-bpkkv"
Jul 16 23:05:31.184851 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:31.184828    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:05:31.184958 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:31.184852    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 23:05:31.191613 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:31.191592    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-bpkkv"]
Jul 16 23:05:31.302703 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:31.302679    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9467285c-4b5b-4abd-9532-090ce0d8c853-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-bpkkv\" (UID: \"9467285c-4b5b-4abd-9532-090ce0d8c853\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-bpkkv"
Jul 16 23:05:31.403591 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:31.403555    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9467285c-4b5b-4abd-9532-090ce0d8c853-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-bpkkv\" (UID: \"9467285c-4b5b-4abd-9532-090ce0d8c853\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-bpkkv"
Jul 16 23:05:31.404222 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:31.404202    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9467285c-4b5b-4abd-9532-090ce0d8c853-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-bpkkv\" (UID: \"9467285c-4b5b-4abd-9532-090ce0d8c853\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-bpkkv"
Jul 16 23:05:31.490347 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:31.490256    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-bpkkv"
Jul 16 23:05:31.618391 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:31.618360    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-bpkkv"]
Jul 16 23:05:31.620317 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:05:31.620283    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9467285c_4b5b_4abd_9532_090ce0d8c853.slice/crio-5f127744508f552e691d47d889a93f15897378b7e0a6c731620b81b758181171 WatchSource:0}: Error finding container 5f127744508f552e691d47d889a93f15897378b7e0a6c731620b81b758181171: Status 404 returned error can't find the container with id 5f127744508f552e691d47d889a93f15897378b7e0a6c731620b81b758181171
Jul 16 23:05:31.622128 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:31.622111    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 23:05:32.000513 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:32.000476    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-bpkkv" event={"ID":"9467285c-4b5b-4abd-9532-090ce0d8c853","Type":"ContainerStarted","Data":"5f127744508f552e691d47d889a93f15897378b7e0a6c731620b81b758181171"}
Jul 16 23:05:36.021176 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:36.021139    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-bpkkv" event={"ID":"9467285c-4b5b-4abd-9532-090ce0d8c853","Type":"ContainerStarted","Data":"dabba09c74080486fa07dc1c1f6f6d4ec3e6b84a8bfb16b58197fa3aafefed02"}
Jul 16 23:05:36.038530 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:36.038482    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-bpkkv" podStartSLOduration=1.168957797 podStartE2EDuration="5.038468593s" totalImagesPullingTime="3.869510796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:05:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:05:36.037490496 +0000 UTC m=+1910.047753160" watchObservedRunningTime="2026-07-16 23:05:36.038468593 +0000 UTC m=+1910.048731257"
Jul 16 23:05:37.027009 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:37.026965    2573 generic.go:356] "Generic (PLEG): container finished" podID="9467285c-4b5b-4abd-9532-090ce0d8c853" containerID="dabba09c74080486fa07dc1c1f6f6d4ec3e6b84a8bfb16b58197fa3aafefed02" exitCode=7
Jul 16 23:05:37.027504 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:37.027062    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-bpkkv" event={"ID":"9467285c-4b5b-4abd-9532-090ce0d8c853","Type":"ContainerDied","Data":"dabba09c74080486fa07dc1c1f6f6d4ec3e6b84a8bfb16b58197fa3aafefed02"}
Jul 16 23:05:37.027772 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:37.027753    2573 scope.go:117] "RemoveContainer" containerID="dabba09c74080486fa07dc1c1f6f6d4ec3e6b84a8bfb16b58197fa3aafefed02"
Jul 16 23:05:38.033350 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:38.033265    2573 generic.go:356] "Generic (PLEG): container finished" podID="9467285c-4b5b-4abd-9532-090ce0d8c853" containerID="f52182f0b7601069d7aae8d3fdb47e53dd98f3a937e65e741c82691cb331767c" exitCode=7
Jul 16 23:05:38.033725 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:38.033343    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-bpkkv" event={"ID":"9467285c-4b5b-4abd-9532-090ce0d8c853","Type":"ContainerDied","Data":"f52182f0b7601069d7aae8d3fdb47e53dd98f3a937e65e741c82691cb331767c"}
Jul 16 23:05:38.033725 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:38.033397    2573 scope.go:117] "RemoveContainer" containerID="dabba09c74080486fa07dc1c1f6f6d4ec3e6b84a8bfb16b58197fa3aafefed02"
Jul 16 23:05:38.033725 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:38.033601    2573 scope.go:117] "RemoveContainer" containerID="f52182f0b7601069d7aae8d3fdb47e53dd98f3a937e65e741c82691cb331767c"
Jul 16 23:05:38.033838 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:05:38.033803    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-bpkkv_odh-ai-gateway-infra(9467285c-4b5b-4abd-9532-090ce0d8c853)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-bpkkv" podUID="9467285c-4b5b-4abd-9532-090ce0d8c853"
Jul 16 23:05:39.039686 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:39.039654    2573 scope.go:117] "RemoveContainer" containerID="f52182f0b7601069d7aae8d3fdb47e53dd98f3a937e65e741c82691cb331767c"
Jul 16 23:05:39.040164 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:05:39.039865    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-bpkkv_odh-ai-gateway-infra(9467285c-4b5b-4abd-9532-090ce0d8c853)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-bpkkv" podUID="9467285c-4b5b-4abd-9532-090ce0d8c853"
Jul 16 23:05:41.267918 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:41.267885    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:05:41.438070 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:41.438036    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:05:44.238957 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:44.238917    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:05:46.888584 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:46.888533    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:05:47.827654 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:47.827620    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:05:48.288418 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:48.288360    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:05:50.513777 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:50.513745    2573 scope.go:117] "RemoveContainer" containerID="f52182f0b7601069d7aae8d3fdb47e53dd98f3a937e65e741c82691cb331767c"
Jul 16 23:05:51.099536 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:51.099498    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-bpkkv" event={"ID":"9467285c-4b5b-4abd-9532-090ce0d8c853","Type":"ContainerStarted","Data":"4660207f3535b1e199f948657ac01a9b9edaea4f3ae0fda71cb932e07641c04b"}
Jul 16 23:05:52.105824 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:52.105787    2573 generic.go:356] "Generic (PLEG): container finished" podID="9467285c-4b5b-4abd-9532-090ce0d8c853" containerID="4660207f3535b1e199f948657ac01a9b9edaea4f3ae0fda71cb932e07641c04b" exitCode=6
Jul 16 23:05:52.106360 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:52.105858    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-bpkkv" event={"ID":"9467285c-4b5b-4abd-9532-090ce0d8c853","Type":"ContainerDied","Data":"4660207f3535b1e199f948657ac01a9b9edaea4f3ae0fda71cb932e07641c04b"}
Jul 16 23:05:52.106360 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:52.105922    2573 scope.go:117] "RemoveContainer" containerID="f52182f0b7601069d7aae8d3fdb47e53dd98f3a937e65e741c82691cb331767c"
Jul 16 23:05:52.106360 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:52.106148    2573 scope.go:117] "RemoveContainer" containerID="4660207f3535b1e199f948657ac01a9b9edaea4f3ae0fda71cb932e07641c04b"
Jul 16 23:05:52.106484 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:05:52.106365    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-bpkkv_odh-ai-gateway-infra(9467285c-4b5b-4abd-9532-090ce0d8c853)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-bpkkv" podUID="9467285c-4b5b-4abd-9532-090ce0d8c853"
Jul 16 23:05:52.135658 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:52.135628    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-bpkkv"]
Jul 16 23:05:53.241929 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:53.241901    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-bpkkv"
Jul 16 23:05:53.379177 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:53.379089    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9467285c-4b5b-4abd-9532-090ce0d8c853-maas-api-service-ca\") pod \"9467285c-4b5b-4abd-9532-090ce0d8c853\" (UID: \"9467285c-4b5b-4abd-9532-090ce0d8c853\") "
Jul 16 23:05:53.379503 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:53.379479    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9467285c-4b5b-4abd-9532-090ce0d8c853-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9467285c-4b5b-4abd-9532-090ce0d8c853" (UID: "9467285c-4b5b-4abd-9532-090ce0d8c853"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:05:53.480565 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:53.480528    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9467285c-4b5b-4abd-9532-090ce0d8c853-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:05:53.723701 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:53.723612    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:05:53.824703 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:53.824668    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:05:53.970403 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:53.970373    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:05:54.071517 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:54.071424    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:05:54.118952 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:54.118918    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-bpkkv" event={"ID":"9467285c-4b5b-4abd-9532-090ce0d8c853","Type":"ContainerDied","Data":"5f127744508f552e691d47d889a93f15897378b7e0a6c731620b81b758181171"}
Jul 16 23:05:54.118952 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:54.118944    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-bpkkv"
Jul 16 23:05:54.119188 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:54.118981    2573 scope.go:117] "RemoveContainer" containerID="4660207f3535b1e199f948657ac01a9b9edaea4f3ae0fda71cb932e07641c04b"
Jul 16 23:05:54.147229 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:54.147187    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-bpkkv"]
Jul 16 23:05:54.157064 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:54.157040    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-bpkkv"]
Jul 16 23:05:54.518030 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:54.517998    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9467285c-4b5b-4abd-9532-090ce0d8c853" path="/var/lib/kubelet/pods/9467285c-4b5b-4abd-9532-090ce0d8c853/volumes"
Jul 16 23:05:55.161549 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:55.161519    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:05:55.254579 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:55.254547    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:05:55.443455 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:55.443362    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:05:55.638283 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:55.638206    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:05:55.962752 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:55.962695    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:05:56.063872 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:56.063845    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:05:57.089618 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:57.089584    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:05:58.158356 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:05:58.158324    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:06:00.180940 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:00.180908    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-g2kcd"]
Jul 16 23:06:00.181329 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:00.181194    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9467285c-4b5b-4abd-9532-090ce0d8c853" containerName="revoke-keys"
Jul 16 23:06:00.181329 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:00.181205    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9467285c-4b5b-4abd-9532-090ce0d8c853" containerName="revoke-keys"
Jul 16 23:06:00.181329 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:00.181215    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9467285c-4b5b-4abd-9532-090ce0d8c853" containerName="revoke-keys"
Jul 16 23:06:00.181329 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:00.181220    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9467285c-4b5b-4abd-9532-090ce0d8c853" containerName="revoke-keys"
Jul 16 23:06:00.181329 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:00.181289    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9467285c-4b5b-4abd-9532-090ce0d8c853" containerName="revoke-keys"
Jul 16 23:06:00.181329 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:00.181297    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9467285c-4b5b-4abd-9532-090ce0d8c853" containerName="revoke-keys"
Jul 16 23:06:00.181329 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:00.181303    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9467285c-4b5b-4abd-9532-090ce0d8c853" containerName="revoke-keys"
Jul 16 23:06:00.184067 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:00.184049    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-g2kcd"
Jul 16 23:06:00.186647 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:00.186624    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:06:00.186939 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:00.186924    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 23:06:00.192803 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:00.192777    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-g2kcd"]
Jul 16 23:06:00.332446 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:00.332401    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bdef9b8-0194-43ab-bfd9-afd757d92fae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-g2kcd\" (UID: \"0bdef9b8-0194-43ab-bfd9-afd757d92fae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-g2kcd"
Jul 16 23:06:00.433866 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:00.433774    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bdef9b8-0194-43ab-bfd9-afd757d92fae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-g2kcd\" (UID: \"0bdef9b8-0194-43ab-bfd9-afd757d92fae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-g2kcd"
Jul 16 23:06:00.434431 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:00.434409    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bdef9b8-0194-43ab-bfd9-afd757d92fae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-g2kcd\" (UID: \"0bdef9b8-0194-43ab-bfd9-afd757d92fae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-g2kcd"
Jul 16 23:06:00.494814 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:00.494778    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-g2kcd"
Jul 16 23:06:00.632405 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:00.632368    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-g2kcd"]
Jul 16 23:06:00.634668 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:06:00.634638    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bdef9b8_0194_43ab_bfd9_afd757d92fae.slice/crio-56e72f9581273e74ef1beac6d6b9fe7a1faf84f6e1cbbea8c63c4e18fb934ee2 WatchSource:0}: Error finding container 56e72f9581273e74ef1beac6d6b9fe7a1faf84f6e1cbbea8c63c4e18fb934ee2: Status 404 returned error can't find the container with id 56e72f9581273e74ef1beac6d6b9fe7a1faf84f6e1cbbea8c63c4e18fb934ee2
Jul 16 23:06:01.158900 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:01.158860    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-g2kcd" event={"ID":"0bdef9b8-0194-43ab-bfd9-afd757d92fae","Type":"ContainerStarted","Data":"d62d17594a6707a7337067091ad3bc8dac64465c41bd31e65b45a2307270247b"}
Jul 16 23:06:01.159064 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:01.158922    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-g2kcd" event={"ID":"0bdef9b8-0194-43ab-bfd9-afd757d92fae","Type":"ContainerStarted","Data":"56e72f9581273e74ef1beac6d6b9fe7a1faf84f6e1cbbea8c63c4e18fb934ee2"}
Jul 16 23:06:01.179371 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:01.179321    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-g2kcd" podStartSLOduration=1.17930471 podStartE2EDuration="1.17930471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:06:01.177000787 +0000 UTC m=+1935.187263452" watchObservedRunningTime="2026-07-16 23:06:01.17930471 +0000 UTC m=+1935.189567453"
Jul 16 23:06:02.165196 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:02.165160    2573 generic.go:356] "Generic (PLEG): container finished" podID="0bdef9b8-0194-43ab-bfd9-afd757d92fae" containerID="d62d17594a6707a7337067091ad3bc8dac64465c41bd31e65b45a2307270247b" exitCode=7
Jul 16 23:06:02.165686 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:02.165265    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-g2kcd" event={"ID":"0bdef9b8-0194-43ab-bfd9-afd757d92fae","Type":"ContainerDied","Data":"d62d17594a6707a7337067091ad3bc8dac64465c41bd31e65b45a2307270247b"}
Jul 16 23:06:02.165686 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:02.165464    2573 scope.go:117] "RemoveContainer" containerID="d62d17594a6707a7337067091ad3bc8dac64465c41bd31e65b45a2307270247b"
Jul 16 23:06:03.172163 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:03.172131    2573 generic.go:356] "Generic (PLEG): container finished" podID="0bdef9b8-0194-43ab-bfd9-afd757d92fae" containerID="7a07db93cb14369724e27e00ecc76696c976918f06e7c99c8bd5d1c009b03b67" exitCode=7
Jul 16 23:06:03.172588 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:03.172220    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-g2kcd" event={"ID":"0bdef9b8-0194-43ab-bfd9-afd757d92fae","Type":"ContainerDied","Data":"7a07db93cb14369724e27e00ecc76696c976918f06e7c99c8bd5d1c009b03b67"}
Jul 16 23:06:03.172588 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:03.172296    2573 scope.go:117] "RemoveContainer" containerID="d62d17594a6707a7337067091ad3bc8dac64465c41bd31e65b45a2307270247b"
Jul 16 23:06:03.172588 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:03.172400    2573 scope.go:117] "RemoveContainer" containerID="7a07db93cb14369724e27e00ecc76696c976918f06e7c99c8bd5d1c009b03b67"
Jul 16 23:06:03.172719 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:06:03.172613    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-g2kcd_odh-ai-gateway-infra(0bdef9b8-0194-43ab-bfd9-afd757d92fae)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-g2kcd" podUID="0bdef9b8-0194-43ab-bfd9-afd757d92fae"
Jul 16 23:06:04.178734 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:04.178710    2573 scope.go:117] "RemoveContainer" containerID="7a07db93cb14369724e27e00ecc76696c976918f06e7c99c8bd5d1c009b03b67"
Jul 16 23:06:04.179100 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:06:04.178870    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-g2kcd_odh-ai-gateway-infra(0bdef9b8-0194-43ab-bfd9-afd757d92fae)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-g2kcd" podUID="0bdef9b8-0194-43ab-bfd9-afd757d92fae"
Jul 16 23:06:15.513777 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:15.513745    2573 scope.go:117] "RemoveContainer" containerID="7a07db93cb14369724e27e00ecc76696c976918f06e7c99c8bd5d1c009b03b67"
Jul 16 23:06:16.239049 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:16.239012    2573 generic.go:356] "Generic (PLEG): container finished" podID="0bdef9b8-0194-43ab-bfd9-afd757d92fae" containerID="92006addc88111f98019252e0bd54583343ffe06df7a39529702909520989e2e" exitCode=7
Jul 16 23:06:16.239223 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:16.239086    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-g2kcd" event={"ID":"0bdef9b8-0194-43ab-bfd9-afd757d92fae","Type":"ContainerDied","Data":"92006addc88111f98019252e0bd54583343ffe06df7a39529702909520989e2e"}
Jul 16 23:06:16.239223 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:16.239150    2573 scope.go:117] "RemoveContainer" containerID="7a07db93cb14369724e27e00ecc76696c976918f06e7c99c8bd5d1c009b03b67"
Jul 16 23:06:16.239471 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:16.239453    2573 scope.go:117] "RemoveContainer" containerID="92006addc88111f98019252e0bd54583343ffe06df7a39529702909520989e2e"
Jul 16 23:06:16.239667 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:06:16.239648    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-g2kcd_odh-ai-gateway-infra(0bdef9b8-0194-43ab-bfd9-afd757d92fae)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-g2kcd" podUID="0bdef9b8-0194-43ab-bfd9-afd757d92fae"
Jul 16 23:06:17.267178 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:17.267143    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-g2kcd"]
Jul 16 23:06:17.394486 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:17.394464    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-g2kcd"
Jul 16 23:06:17.451458 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:17.451413    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bdef9b8-0194-43ab-bfd9-afd757d92fae-maas-api-service-ca\") pod \"0bdef9b8-0194-43ab-bfd9-afd757d92fae\" (UID: \"0bdef9b8-0194-43ab-bfd9-afd757d92fae\") "
Jul 16 23:06:17.451812 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:17.451786    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bdef9b8-0194-43ab-bfd9-afd757d92fae-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0bdef9b8-0194-43ab-bfd9-afd757d92fae" (UID: "0bdef9b8-0194-43ab-bfd9-afd757d92fae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:06:17.552916 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:17.552818    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bdef9b8-0194-43ab-bfd9-afd757d92fae-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:06:18.251411 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:18.251373    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-g2kcd" event={"ID":"0bdef9b8-0194-43ab-bfd9-afd757d92fae","Type":"ContainerDied","Data":"56e72f9581273e74ef1beac6d6b9fe7a1faf84f6e1cbbea8c63c4e18fb934ee2"}
Jul 16 23:06:18.251596 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:18.251437    2573 scope.go:117] "RemoveContainer" containerID="92006addc88111f98019252e0bd54583343ffe06df7a39529702909520989e2e"
Jul 16 23:06:18.251596 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:18.251384    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-g2kcd"
Jul 16 23:06:18.276905 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:18.276884    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-g2kcd"]
Jul 16 23:06:18.278843 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:18.278822    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-g2kcd"]
Jul 16 23:06:18.518838 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:18.518762    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bdef9b8-0194-43ab-bfd9-afd757d92fae" path="/var/lib/kubelet/pods/0bdef9b8-0194-43ab-bfd9-afd757d92fae/volumes"
Jul 16 23:06:31.234510 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:31.234475    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-zs4zx"]
Jul 16 23:06:31.234896 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:31.234768    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bdef9b8-0194-43ab-bfd9-afd757d92fae" containerName="revoke-keys"
Jul 16 23:06:31.234896 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:31.234779    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bdef9b8-0194-43ab-bfd9-afd757d92fae" containerName="revoke-keys"
Jul 16 23:06:31.234896 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:31.234788    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9467285c-4b5b-4abd-9532-090ce0d8c853" containerName="revoke-keys"
Jul 16 23:06:31.234896 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:31.234793    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9467285c-4b5b-4abd-9532-090ce0d8c853" containerName="revoke-keys"
Jul 16 23:06:31.234896 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:31.234800    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bdef9b8-0194-43ab-bfd9-afd757d92fae" containerName="revoke-keys"
Jul 16 23:06:31.234896 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:31.234819    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bdef9b8-0194-43ab-bfd9-afd757d92fae" containerName="revoke-keys"
Jul 16 23:06:31.234896 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:31.234831    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bdef9b8-0194-43ab-bfd9-afd757d92fae" containerName="revoke-keys"
Jul 16 23:06:31.234896 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:31.234836    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bdef9b8-0194-43ab-bfd9-afd757d92fae" containerName="revoke-keys"
Jul 16 23:06:31.234896 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:31.234878    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bdef9b8-0194-43ab-bfd9-afd757d92fae" containerName="revoke-keys"
Jul 16 23:06:31.234896 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:31.234886    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bdef9b8-0194-43ab-bfd9-afd757d92fae" containerName="revoke-keys"
Jul 16 23:06:31.237750 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:31.237735    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-zs4zx"
Jul 16 23:06:31.240301 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:31.240269    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:06:31.240459 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:31.240419    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 23:06:31.246386 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:31.246360    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-zs4zx"]
Jul 16 23:06:31.368495 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:31.368461    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fce6821d-a2f8-492c-b5db-752c29601e5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-zs4zx\" (UID: \"fce6821d-a2f8-492c-b5db-752c29601e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-zs4zx"
Jul 16 23:06:31.469013 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:31.468970    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fce6821d-a2f8-492c-b5db-752c29601e5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-zs4zx\" (UID: \"fce6821d-a2f8-492c-b5db-752c29601e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-zs4zx"
Jul 16 23:06:31.469660 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:31.469632    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fce6821d-a2f8-492c-b5db-752c29601e5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-zs4zx\" (UID: \"fce6821d-a2f8-492c-b5db-752c29601e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-zs4zx"
Jul 16 23:06:31.548804 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:31.548722    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-zs4zx"
Jul 16 23:06:31.678011 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:31.677988    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-zs4zx"]
Jul 16 23:06:31.680493 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:06:31.680458    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfce6821d_a2f8_492c_b5db_752c29601e5e.slice/crio-844f96ae0bba36eecc32ace32624f20df7bbf75c2b87f7bc3a564056af368e7d WatchSource:0}: Error finding container 844f96ae0bba36eecc32ace32624f20df7bbf75c2b87f7bc3a564056af368e7d: Status 404 returned error can't find the container with id 844f96ae0bba36eecc32ace32624f20df7bbf75c2b87f7bc3a564056af368e7d
Jul 16 23:06:32.321326 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:32.321297    2573 generic.go:356] "Generic (PLEG): container finished" podID="fce6821d-a2f8-492c-b5db-752c29601e5e" containerID="2b6d1acdc11d094f1f29e7469e844f8d1ad333060ea312bd0e0311d713b258d0" exitCode=6
Jul 16 23:06:32.321660 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:32.321331    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-zs4zx" event={"ID":"fce6821d-a2f8-492c-b5db-752c29601e5e","Type":"ContainerDied","Data":"2b6d1acdc11d094f1f29e7469e844f8d1ad333060ea312bd0e0311d713b258d0"}
Jul 16 23:06:32.321660 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:32.321389    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-zs4zx" event={"ID":"fce6821d-a2f8-492c-b5db-752c29601e5e","Type":"ContainerStarted","Data":"844f96ae0bba36eecc32ace32624f20df7bbf75c2b87f7bc3a564056af368e7d"}
Jul 16 23:06:32.321660 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:32.321530    2573 scope.go:117] "RemoveContainer" containerID="2b6d1acdc11d094f1f29e7469e844f8d1ad333060ea312bd0e0311d713b258d0"
Jul 16 23:06:33.328388 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:33.328344    2573 generic.go:356] "Generic (PLEG): container finished" podID="fce6821d-a2f8-492c-b5db-752c29601e5e" containerID="dfd8ee0713a3c8918055a46e3a0c46c01ea22da714352bc43ef5a849ddfec3ef" exitCode=6
Jul 16 23:06:33.328844 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:33.328430    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-zs4zx" event={"ID":"fce6821d-a2f8-492c-b5db-752c29601e5e","Type":"ContainerDied","Data":"dfd8ee0713a3c8918055a46e3a0c46c01ea22da714352bc43ef5a849ddfec3ef"}
Jul 16 23:06:33.328844 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:33.328516    2573 scope.go:117] "RemoveContainer" containerID="2b6d1acdc11d094f1f29e7469e844f8d1ad333060ea312bd0e0311d713b258d0"
Jul 16 23:06:33.328844 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:33.328729    2573 scope.go:117] "RemoveContainer" containerID="dfd8ee0713a3c8918055a46e3a0c46c01ea22da714352bc43ef5a849ddfec3ef"
Jul 16 23:06:33.328972 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:06:33.328936    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-zs4zx_odh-ai-gateway-infra(fce6821d-a2f8-492c-b5db-752c29601e5e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-zs4zx" podUID="fce6821d-a2f8-492c-b5db-752c29601e5e"
Jul 16 23:06:34.334691 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:34.334663    2573 scope.go:117] "RemoveContainer" containerID="dfd8ee0713a3c8918055a46e3a0c46c01ea22da714352bc43ef5a849ddfec3ef"
Jul 16 23:06:34.335090 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:06:34.334827    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-zs4zx_odh-ai-gateway-infra(fce6821d-a2f8-492c-b5db-752c29601e5e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-zs4zx" podUID="fce6821d-a2f8-492c-b5db-752c29601e5e"
Jul 16 23:06:42.440665 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:42.440581    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6vh68"]
Jul 16 23:06:42.441024 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:42.440923    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bdef9b8-0194-43ab-bfd9-afd757d92fae" containerName="revoke-keys"
Jul 16 23:06:42.445047 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:42.445029    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6vh68"
Jul 16 23:06:42.460494 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:42.460466    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6vh68"]
Jul 16 23:06:42.549656 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:42.549627    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d09ec74d-77ce-42d7-9b74-882fdf7fc002-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-6vh68\" (UID: \"d09ec74d-77ce-42d7-9b74-882fdf7fc002\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6vh68"
Jul 16 23:06:42.650354 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:42.650330    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d09ec74d-77ce-42d7-9b74-882fdf7fc002-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-6vh68\" (UID: \"d09ec74d-77ce-42d7-9b74-882fdf7fc002\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6vh68"
Jul 16 23:06:42.650890 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:42.650872    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d09ec74d-77ce-42d7-9b74-882fdf7fc002-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-6vh68\" (UID: \"d09ec74d-77ce-42d7-9b74-882fdf7fc002\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6vh68"
Jul 16 23:06:42.757182 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:42.757129    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6vh68"
Jul 16 23:06:42.887750 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:42.887723    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6vh68"]
Jul 16 23:06:42.889559 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:06:42.889531    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd09ec74d_77ce_42d7_9b74_882fdf7fc002.slice/crio-3486f6ac1ca659f0cd995b260ac247ea9b081cf3238b30f901bdf93ec932e215 WatchSource:0}: Error finding container 3486f6ac1ca659f0cd995b260ac247ea9b081cf3238b30f901bdf93ec932e215: Status 404 returned error can't find the container with id 3486f6ac1ca659f0cd995b260ac247ea9b081cf3238b30f901bdf93ec932e215
Jul 16 23:06:43.380227 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:43.380131    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6vh68" event={"ID":"d09ec74d-77ce-42d7-9b74-882fdf7fc002","Type":"ContainerStarted","Data":"bfddeb12c563338a3288c38b673594efee97c2ab8b418eca8a1f05aebcfb25c6"}
Jul 16 23:06:43.380227 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:43.380195    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6vh68" event={"ID":"d09ec74d-77ce-42d7-9b74-882fdf7fc002","Type":"ContainerStarted","Data":"3486f6ac1ca659f0cd995b260ac247ea9b081cf3238b30f901bdf93ec932e215"}
Jul 16 23:06:43.397646 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:43.397461    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6vh68" podStartSLOduration=1.397442713 podStartE2EDuration="1.397442713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:06:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:06:43.397207678 +0000 UTC m=+1977.407470344" watchObservedRunningTime="2026-07-16 23:06:43.397442713 +0000 UTC m=+1977.407705377"
Jul 16 23:06:44.386885 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:44.386849    2573 generic.go:356] "Generic (PLEG): container finished" podID="d09ec74d-77ce-42d7-9b74-882fdf7fc002" containerID="bfddeb12c563338a3288c38b673594efee97c2ab8b418eca8a1f05aebcfb25c6" exitCode=6
Jul 16 23:06:44.387377 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:44.386915    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6vh68" event={"ID":"d09ec74d-77ce-42d7-9b74-882fdf7fc002","Type":"ContainerDied","Data":"bfddeb12c563338a3288c38b673594efee97c2ab8b418eca8a1f05aebcfb25c6"}
Jul 16 23:06:44.387377 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:44.387137    2573 scope.go:117] "RemoveContainer" containerID="bfddeb12c563338a3288c38b673594efee97c2ab8b418eca8a1f05aebcfb25c6"
Jul 16 23:06:45.395002 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:45.394962    2573 generic.go:356] "Generic (PLEG): container finished" podID="d09ec74d-77ce-42d7-9b74-882fdf7fc002" containerID="1773e2c66dd9c620158afa993eb0c108033d40adfd6e36258759595f8b0b454c" exitCode=6
Jul 16 23:06:45.395395 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:45.395023    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6vh68" event={"ID":"d09ec74d-77ce-42d7-9b74-882fdf7fc002","Type":"ContainerDied","Data":"1773e2c66dd9c620158afa993eb0c108033d40adfd6e36258759595f8b0b454c"}
Jul 16 23:06:45.395395 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:45.395081    2573 scope.go:117] "RemoveContainer" containerID="bfddeb12c563338a3288c38b673594efee97c2ab8b418eca8a1f05aebcfb25c6"
Jul 16 23:06:45.395395 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:45.395265    2573 scope.go:117] "RemoveContainer" containerID="1773e2c66dd9c620158afa993eb0c108033d40adfd6e36258759595f8b0b454c"
Jul 16 23:06:45.395519 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:06:45.395436    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-6vh68_odh-ai-gateway-infra(d09ec74d-77ce-42d7-9b74-882fdf7fc002)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6vh68" podUID="d09ec74d-77ce-42d7-9b74-882fdf7fc002"
Jul 16 23:06:46.400759 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:46.400734    2573 scope.go:117] "RemoveContainer" containerID="1773e2c66dd9c620158afa993eb0c108033d40adfd6e36258759595f8b0b454c"
Jul 16 23:06:46.401093 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:06:46.400897    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-6vh68_odh-ai-gateway-infra(d09ec74d-77ce-42d7-9b74-882fdf7fc002)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6vh68" podUID="d09ec74d-77ce-42d7-9b74-882fdf7fc002"
Jul 16 23:06:49.513869 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:49.513832    2573 scope.go:117] "RemoveContainer" containerID="dfd8ee0713a3c8918055a46e3a0c46c01ea22da714352bc43ef5a849ddfec3ef"
Jul 16 23:06:50.241462 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:50.241432    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hn8mf"]
Jul 16 23:06:50.244537 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:50.244522    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hn8mf"
Jul 16 23:06:50.258190 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:50.258166    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hn8mf"]
Jul 16 23:06:50.304044 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:50.304016    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa4578f9-b78f-45f0-a63d-4b6600a539bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-hn8mf\" (UID: \"fa4578f9-b78f-45f0-a63d-4b6600a539bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hn8mf"
Jul 16 23:06:50.404497 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:50.404437    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa4578f9-b78f-45f0-a63d-4b6600a539bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-hn8mf\" (UID: \"fa4578f9-b78f-45f0-a63d-4b6600a539bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hn8mf"
Jul 16 23:06:50.405061 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:50.405045    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa4578f9-b78f-45f0-a63d-4b6600a539bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-hn8mf\" (UID: \"fa4578f9-b78f-45f0-a63d-4b6600a539bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hn8mf"
Jul 16 23:06:50.424061 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:50.424036    2573 generic.go:356] "Generic (PLEG): container finished" podID="fce6821d-a2f8-492c-b5db-752c29601e5e" containerID="1ee0f20cf0b47f6ff70b37fc83a43973b0eff939373f351b9aab807a631e473e" exitCode=6
Jul 16 23:06:50.424162 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:50.424109    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-zs4zx" event={"ID":"fce6821d-a2f8-492c-b5db-752c29601e5e","Type":"ContainerDied","Data":"1ee0f20cf0b47f6ff70b37fc83a43973b0eff939373f351b9aab807a631e473e"}
Jul 16 23:06:50.424201 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:50.424162    2573 scope.go:117] "RemoveContainer" containerID="dfd8ee0713a3c8918055a46e3a0c46c01ea22da714352bc43ef5a849ddfec3ef"
Jul 16 23:06:50.424425 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:50.424410    2573 scope.go:117] "RemoveContainer" containerID="1ee0f20cf0b47f6ff70b37fc83a43973b0eff939373f351b9aab807a631e473e"
Jul 16 23:06:50.424627 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:06:50.424604    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-zs4zx_odh-ai-gateway-infra(fce6821d-a2f8-492c-b5db-752c29601e5e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-zs4zx" podUID="fce6821d-a2f8-492c-b5db-752c29601e5e"
Jul 16 23:06:50.553900 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:50.553878    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hn8mf"
Jul 16 23:06:50.687102 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:06:50.687076    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa4578f9_b78f_45f0_a63d_4b6600a539bb.slice/crio-563d8390c71e40de7394405d9c007b345f09b172bcd310afabaf494b823d30f6 WatchSource:0}: Error finding container 563d8390c71e40de7394405d9c007b345f09b172bcd310afabaf494b823d30f6: Status 404 returned error can't find the container with id 563d8390c71e40de7394405d9c007b345f09b172bcd310afabaf494b823d30f6
Jul 16 23:06:50.690122 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:50.690100    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hn8mf"]
Jul 16 23:06:51.430193 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:51.430154    2573 generic.go:356] "Generic (PLEG): container finished" podID="fa4578f9-b78f-45f0-a63d-4b6600a539bb" containerID="e394704875be92afc9b2412a5127ac359eb718321f4a54900b2b7f3ce5f4177c" exitCode=7
Jul 16 23:06:51.430350 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:51.430252    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hn8mf" event={"ID":"fa4578f9-b78f-45f0-a63d-4b6600a539bb","Type":"ContainerDied","Data":"e394704875be92afc9b2412a5127ac359eb718321f4a54900b2b7f3ce5f4177c"}
Jul 16 23:06:51.430350 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:51.430307    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hn8mf" event={"ID":"fa4578f9-b78f-45f0-a63d-4b6600a539bb","Type":"ContainerStarted","Data":"563d8390c71e40de7394405d9c007b345f09b172bcd310afabaf494b823d30f6"}
Jul 16 23:06:51.430442 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:51.430406    2573 scope.go:117] "RemoveContainer" containerID="e394704875be92afc9b2412a5127ac359eb718321f4a54900b2b7f3ce5f4177c"
Jul 16 23:06:51.460269 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:51.460245    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-zs4zx"]
Jul 16 23:06:51.599832 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:51.599810    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-zs4zx"
Jul 16 23:06:51.712736 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:51.712711    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fce6821d-a2f8-492c-b5db-752c29601e5e-maas-api-service-ca\") pod \"fce6821d-a2f8-492c-b5db-752c29601e5e\" (UID: \"fce6821d-a2f8-492c-b5db-752c29601e5e\") "
Jul 16 23:06:51.713006 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:51.712986    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fce6821d-a2f8-492c-b5db-752c29601e5e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fce6821d-a2f8-492c-b5db-752c29601e5e" (UID: "fce6821d-a2f8-492c-b5db-752c29601e5e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:06:51.813759 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:51.813739    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fce6821d-a2f8-492c-b5db-752c29601e5e-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:06:52.438778 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:52.438747    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-zs4zx"
Jul 16 23:06:52.438961 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:52.438775    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-zs4zx" event={"ID":"fce6821d-a2f8-492c-b5db-752c29601e5e","Type":"ContainerDied","Data":"844f96ae0bba36eecc32ace32624f20df7bbf75c2b87f7bc3a564056af368e7d"}
Jul 16 23:06:52.438961 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:52.438841    2573 scope.go:117] "RemoveContainer" containerID="1ee0f20cf0b47f6ff70b37fc83a43973b0eff939373f351b9aab807a631e473e"
Jul 16 23:06:52.441040 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:52.441007    2573 generic.go:356] "Generic (PLEG): container finished" podID="fa4578f9-b78f-45f0-a63d-4b6600a539bb" containerID="881d405ecbf6eb2b3f68359a71b7e11a4b15e5e4c0ff8d6805b8f70628b6aafb" exitCode=7
Jul 16 23:06:52.441176 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:52.441092    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hn8mf" event={"ID":"fa4578f9-b78f-45f0-a63d-4b6600a539bb","Type":"ContainerDied","Data":"881d405ecbf6eb2b3f68359a71b7e11a4b15e5e4c0ff8d6805b8f70628b6aafb"}
Jul 16 23:06:52.441370 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:52.441354    2573 scope.go:117] "RemoveContainer" containerID="881d405ecbf6eb2b3f68359a71b7e11a4b15e5e4c0ff8d6805b8f70628b6aafb"
Jul 16 23:06:52.441583 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:06:52.441553    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-hn8mf_odh-ai-gateway-infra(fa4578f9-b78f-45f0-a63d-4b6600a539bb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hn8mf" podUID="fa4578f9-b78f-45f0-a63d-4b6600a539bb"
Jul 16 23:06:52.448438 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:52.448416    2573 scope.go:117] "RemoveContainer" containerID="e394704875be92afc9b2412a5127ac359eb718321f4a54900b2b7f3ce5f4177c"
Jul 16 23:06:52.487393 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:52.487370    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-zs4zx"]
Jul 16 23:06:52.489035 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:52.489015    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-zs4zx"]
Jul 16 23:06:52.518123 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:52.518103    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fce6821d-a2f8-492c-b5db-752c29601e5e" path="/var/lib/kubelet/pods/fce6821d-a2f8-492c-b5db-752c29601e5e/volumes"
Jul 16 23:06:53.448371 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:53.448343    2573 scope.go:117] "RemoveContainer" containerID="881d405ecbf6eb2b3f68359a71b7e11a4b15e5e4c0ff8d6805b8f70628b6aafb"
Jul 16 23:06:53.448819 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:06:53.448533    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-hn8mf_odh-ai-gateway-infra(fa4578f9-b78f-45f0-a63d-4b6600a539bb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hn8mf" podUID="fa4578f9-b78f-45f0-a63d-4b6600a539bb"
Jul 16 23:06:59.513519 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:06:59.513485    2573 scope.go:117] "RemoveContainer" containerID="1773e2c66dd9c620158afa993eb0c108033d40adfd6e36258759595f8b0b454c"
Jul 16 23:07:00.491222 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:00.491150    2573 generic.go:356] "Generic (PLEG): container finished" podID="d09ec74d-77ce-42d7-9b74-882fdf7fc002" containerID="93b84a4ac0f0d6ea2c17c20d8985cd526bd88399516aa9605be89c8112299285" exitCode=6
Jul 16 23:07:00.491222 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:00.491213    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6vh68" event={"ID":"d09ec74d-77ce-42d7-9b74-882fdf7fc002","Type":"ContainerDied","Data":"93b84a4ac0f0d6ea2c17c20d8985cd526bd88399516aa9605be89c8112299285"}
Jul 16 23:07:00.491397 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:00.491287    2573 scope.go:117] "RemoveContainer" containerID="1773e2c66dd9c620158afa993eb0c108033d40adfd6e36258759595f8b0b454c"
Jul 16 23:07:00.491591 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:00.491573    2573 scope.go:117] "RemoveContainer" containerID="93b84a4ac0f0d6ea2c17c20d8985cd526bd88399516aa9605be89c8112299285"
Jul 16 23:07:00.491798 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:07:00.491774    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-6vh68_odh-ai-gateway-infra(d09ec74d-77ce-42d7-9b74-882fdf7fc002)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6vh68" podUID="d09ec74d-77ce-42d7-9b74-882fdf7fc002"
Jul 16 23:07:01.529173 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:01.529140    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6vh68"]
Jul 16 23:07:01.656985 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:01.656960    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6vh68"
Jul 16 23:07:01.782410 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:01.782329    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d09ec74d-77ce-42d7-9b74-882fdf7fc002-maas-api-service-ca\") pod \"d09ec74d-77ce-42d7-9b74-882fdf7fc002\" (UID: \"d09ec74d-77ce-42d7-9b74-882fdf7fc002\") "
Jul 16 23:07:01.782642 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:01.782621    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d09ec74d-77ce-42d7-9b74-882fdf7fc002-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d09ec74d-77ce-42d7-9b74-882fdf7fc002" (UID: "d09ec74d-77ce-42d7-9b74-882fdf7fc002"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:07:01.882970 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:01.882932    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d09ec74d-77ce-42d7-9b74-882fdf7fc002-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:07:02.502863 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:02.502828    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6vh68" event={"ID":"d09ec74d-77ce-42d7-9b74-882fdf7fc002","Type":"ContainerDied","Data":"3486f6ac1ca659f0cd995b260ac247ea9b081cf3238b30f901bdf93ec932e215"}
Jul 16 23:07:02.502863 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:02.502860    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6vh68"
Jul 16 23:07:02.503093 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:02.502892    2573 scope.go:117] "RemoveContainer" containerID="93b84a4ac0f0d6ea2c17c20d8985cd526bd88399516aa9605be89c8112299285"
Jul 16 23:07:02.525113 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:02.525085    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6vh68"]
Jul 16 23:07:02.529436 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:02.529413    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6vh68"]
Jul 16 23:07:03.513868 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:03.513837    2573 scope.go:117] "RemoveContainer" containerID="881d405ecbf6eb2b3f68359a71b7e11a4b15e5e4c0ff8d6805b8f70628b6aafb"
Jul 16 23:07:04.519101 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:04.519067    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d09ec74d-77ce-42d7-9b74-882fdf7fc002" path="/var/lib/kubelet/pods/d09ec74d-77ce-42d7-9b74-882fdf7fc002/volumes"
Jul 16 23:07:04.522295 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:04.522272    2573 generic.go:356] "Generic (PLEG): container finished" podID="fa4578f9-b78f-45f0-a63d-4b6600a539bb" containerID="3295306c53b99724808abfac6152172358d7fad2c6f21fd7ffc4b0d2b8b2782d" exitCode=7
Jul 16 23:07:04.522417 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:04.522353    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hn8mf" event={"ID":"fa4578f9-b78f-45f0-a63d-4b6600a539bb","Type":"ContainerDied","Data":"3295306c53b99724808abfac6152172358d7fad2c6f21fd7ffc4b0d2b8b2782d"}
Jul 16 23:07:04.522498 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:04.522432    2573 scope.go:117] "RemoveContainer" containerID="881d405ecbf6eb2b3f68359a71b7e11a4b15e5e4c0ff8d6805b8f70628b6aafb"
Jul 16 23:07:04.522772 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:04.522747    2573 scope.go:117] "RemoveContainer" containerID="3295306c53b99724808abfac6152172358d7fad2c6f21fd7ffc4b0d2b8b2782d"
Jul 16 23:07:04.522989 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:07:04.522964    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-hn8mf_odh-ai-gateway-infra(fa4578f9-b78f-45f0-a63d-4b6600a539bb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hn8mf" podUID="fa4578f9-b78f-45f0-a63d-4b6600a539bb"
Jul 16 23:07:05.547379 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:05.547347    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hn8mf"]
Jul 16 23:07:05.677562 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:05.677537    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hn8mf"
Jul 16 23:07:05.809771 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:05.809711    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa4578f9-b78f-45f0-a63d-4b6600a539bb-maas-api-service-ca\") pod \"fa4578f9-b78f-45f0-a63d-4b6600a539bb\" (UID: \"fa4578f9-b78f-45f0-a63d-4b6600a539bb\") "
Jul 16 23:07:05.810063 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:05.810039    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa4578f9-b78f-45f0-a63d-4b6600a539bb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fa4578f9-b78f-45f0-a63d-4b6600a539bb" (UID: "fa4578f9-b78f-45f0-a63d-4b6600a539bb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:07:05.910189 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:05.910161    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa4578f9-b78f-45f0-a63d-4b6600a539bb-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:07:06.533606 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:06.533580    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hn8mf" event={"ID":"fa4578f9-b78f-45f0-a63d-4b6600a539bb","Type":"ContainerDied","Data":"563d8390c71e40de7394405d9c007b345f09b172bcd310afabaf494b823d30f6"}
Jul 16 23:07:06.533606 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:06.533603    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hn8mf"
Jul 16 23:07:06.533786 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:06.533642    2573 scope.go:117] "RemoveContainer" containerID="3295306c53b99724808abfac6152172358d7fad2c6f21fd7ffc4b0d2b8b2782d"
Jul 16 23:07:06.568555 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:06.568530    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hn8mf"]
Jul 16 23:07:06.585570 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:06.585546    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hn8mf"]
Jul 16 23:07:08.525316 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:08.525269    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa4578f9-b78f-45f0-a63d-4b6600a539bb" path="/var/lib/kubelet/pods/fa4578f9-b78f-45f0-a63d-4b6600a539bb/volumes"
Jul 16 23:07:31.271526 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:31.271489    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mk2hn"]
Jul 16 23:07:31.271935 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:31.271794    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fce6821d-a2f8-492c-b5db-752c29601e5e" containerName="revoke-keys"
Jul 16 23:07:31.271935 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:31.271806    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fce6821d-a2f8-492c-b5db-752c29601e5e" containerName="revoke-keys"
Jul 16 23:07:31.271935 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:31.271816    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa4578f9-b78f-45f0-a63d-4b6600a539bb" containerName="revoke-keys"
Jul 16 23:07:31.271935 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:31.271822    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa4578f9-b78f-45f0-a63d-4b6600a539bb" containerName="revoke-keys"
Jul 16 23:07:31.271935 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:31.271839    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d09ec74d-77ce-42d7-9b74-882fdf7fc002" containerName="revoke-keys"
Jul 16 23:07:31.271935 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:31.271844    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d09ec74d-77ce-42d7-9b74-882fdf7fc002" containerName="revoke-keys"
Jul 16 23:07:31.271935 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:31.271851    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fce6821d-a2f8-492c-b5db-752c29601e5e" containerName="revoke-keys"
Jul 16 23:07:31.271935 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:31.271856    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fce6821d-a2f8-492c-b5db-752c29601e5e" containerName="revoke-keys"
Jul 16 23:07:31.271935 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:31.271869    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d09ec74d-77ce-42d7-9b74-882fdf7fc002" containerName="revoke-keys"
Jul 16 23:07:31.271935 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:31.271874    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d09ec74d-77ce-42d7-9b74-882fdf7fc002" containerName="revoke-keys"
Jul 16 23:07:31.271935 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:31.271879    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa4578f9-b78f-45f0-a63d-4b6600a539bb" containerName="revoke-keys"
Jul 16 23:07:31.271935 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:31.271885    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa4578f9-b78f-45f0-a63d-4b6600a539bb" containerName="revoke-keys"
Jul 16 23:07:31.271935 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:31.271930    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d09ec74d-77ce-42d7-9b74-882fdf7fc002" containerName="revoke-keys"
Jul 16 23:07:31.271935 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:31.271937    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa4578f9-b78f-45f0-a63d-4b6600a539bb" containerName="revoke-keys"
Jul 16 23:07:31.272380 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:31.271946    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fce6821d-a2f8-492c-b5db-752c29601e5e" containerName="revoke-keys"
Jul 16 23:07:31.272380 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:31.271951    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d09ec74d-77ce-42d7-9b74-882fdf7fc002" containerName="revoke-keys"
Jul 16 23:07:31.272380 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:31.271957    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa4578f9-b78f-45f0-a63d-4b6600a539bb" containerName="revoke-keys"
Jul 16 23:07:31.272380 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:31.271962    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d09ec74d-77ce-42d7-9b74-882fdf7fc002" containerName="revoke-keys"
Jul 16 23:07:31.272380 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:31.271967    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fce6821d-a2f8-492c-b5db-752c29601e5e" containerName="revoke-keys"
Jul 16 23:07:31.274927 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:31.274909    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mk2hn"
Jul 16 23:07:31.277431 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:31.277409    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 23:07:31.277561 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:31.277449    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:07:31.283763 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:31.283739    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mk2hn"]
Jul 16 23:07:31.385035 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:31.385007    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8413502d-a989-496c-85ca-3225ef45fe8c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-mk2hn\" (UID: \"8413502d-a989-496c-85ca-3225ef45fe8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mk2hn"
Jul 16 23:07:31.485926 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:31.485898    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8413502d-a989-496c-85ca-3225ef45fe8c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-mk2hn\" (UID: \"8413502d-a989-496c-85ca-3225ef45fe8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mk2hn"
Jul 16 23:07:31.486509 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:31.486489    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8413502d-a989-496c-85ca-3225ef45fe8c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-mk2hn\" (UID: \"8413502d-a989-496c-85ca-3225ef45fe8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mk2hn"
Jul 16 23:07:31.586311 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:31.586288    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mk2hn"
Jul 16 23:07:31.709853 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:31.709830    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mk2hn"]
Jul 16 23:07:31.711569 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:07:31.711540    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8413502d_a989_496c_85ca_3225ef45fe8c.slice/crio-e21c58ec0ec5915f119170a24c67c8037d0d2db5b4ee7e7bd4fcdf19801e7f52 WatchSource:0}: Error finding container e21c58ec0ec5915f119170a24c67c8037d0d2db5b4ee7e7bd4fcdf19801e7f52: Status 404 returned error can't find the container with id e21c58ec0ec5915f119170a24c67c8037d0d2db5b4ee7e7bd4fcdf19801e7f52
Jul 16 23:07:32.661662 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:32.661629    2573 generic.go:356] "Generic (PLEG): container finished" podID="8413502d-a989-496c-85ca-3225ef45fe8c" containerID="094dbd3905b63cb88aa34f2b5f94ddf183fe4bffe7e32401a50c8c36cd59faed" exitCode=6
Jul 16 23:07:32.661987 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:32.661688    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mk2hn" event={"ID":"8413502d-a989-496c-85ca-3225ef45fe8c","Type":"ContainerDied","Data":"094dbd3905b63cb88aa34f2b5f94ddf183fe4bffe7e32401a50c8c36cd59faed"}
Jul 16 23:07:32.661987 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:32.661733    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mk2hn" event={"ID":"8413502d-a989-496c-85ca-3225ef45fe8c","Type":"ContainerStarted","Data":"e21c58ec0ec5915f119170a24c67c8037d0d2db5b4ee7e7bd4fcdf19801e7f52"}
Jul 16 23:07:32.661987 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:32.661897    2573 scope.go:117] "RemoveContainer" containerID="094dbd3905b63cb88aa34f2b5f94ddf183fe4bffe7e32401a50c8c36cd59faed"
Jul 16 23:07:33.668995 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:33.668959    2573 generic.go:356] "Generic (PLEG): container finished" podID="8413502d-a989-496c-85ca-3225ef45fe8c" containerID="20283794097f4f3db2f7543eeed5e0369b80e9df11c4aa45ae91a22a241a20e1" exitCode=6
Jul 16 23:07:33.669419 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:33.669007    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mk2hn" event={"ID":"8413502d-a989-496c-85ca-3225ef45fe8c","Type":"ContainerDied","Data":"20283794097f4f3db2f7543eeed5e0369b80e9df11c4aa45ae91a22a241a20e1"}
Jul 16 23:07:33.669419 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:33.669066    2573 scope.go:117] "RemoveContainer" containerID="094dbd3905b63cb88aa34f2b5f94ddf183fe4bffe7e32401a50c8c36cd59faed"
Jul 16 23:07:33.669419 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:33.669232    2573 scope.go:117] "RemoveContainer" containerID="20283794097f4f3db2f7543eeed5e0369b80e9df11c4aa45ae91a22a241a20e1"
Jul 16 23:07:33.669655 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:07:33.669463    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-mk2hn_odh-ai-gateway-infra(8413502d-a989-496c-85ca-3225ef45fe8c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mk2hn" podUID="8413502d-a989-496c-85ca-3225ef45fe8c"
Jul 16 23:07:34.675765 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:34.675735    2573 scope.go:117] "RemoveContainer" containerID="20283794097f4f3db2f7543eeed5e0369b80e9df11c4aa45ae91a22a241a20e1"
Jul 16 23:07:34.676179 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:07:34.675952    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-mk2hn_odh-ai-gateway-infra(8413502d-a989-496c-85ca-3225ef45fe8c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mk2hn" podUID="8413502d-a989-496c-85ca-3225ef45fe8c"
Jul 16 23:07:40.266367 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:40.266332    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8ggwp"]
Jul 16 23:07:40.266822 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:40.266731    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d09ec74d-77ce-42d7-9b74-882fdf7fc002" containerName="revoke-keys"
Jul 16 23:07:40.266822 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:40.266749    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d09ec74d-77ce-42d7-9b74-882fdf7fc002" containerName="revoke-keys"
Jul 16 23:07:40.266822 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:40.266776    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fce6821d-a2f8-492c-b5db-752c29601e5e" containerName="revoke-keys"
Jul 16 23:07:40.266822 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:40.266784    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fce6821d-a2f8-492c-b5db-752c29601e5e" containerName="revoke-keys"
Jul 16 23:07:40.266822 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:40.266796    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa4578f9-b78f-45f0-a63d-4b6600a539bb" containerName="revoke-keys"
Jul 16 23:07:40.266822 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:40.266803    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa4578f9-b78f-45f0-a63d-4b6600a539bb" containerName="revoke-keys"
Jul 16 23:07:40.267123 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:40.266887    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa4578f9-b78f-45f0-a63d-4b6600a539bb" containerName="revoke-keys"
Jul 16 23:07:40.267123 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:40.266898    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fce6821d-a2f8-492c-b5db-752c29601e5e" containerName="revoke-keys"
Jul 16 23:07:40.271062 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:40.271041    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8ggwp"
Jul 16 23:07:40.276897 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:40.276867    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8ggwp"]
Jul 16 23:07:40.448083 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:40.448056    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec6bda4b-feb4-4376-98cf-ee87517043bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-8ggwp\" (UID: \"ec6bda4b-feb4-4376-98cf-ee87517043bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8ggwp"
Jul 16 23:07:40.548910 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:40.548854    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec6bda4b-feb4-4376-98cf-ee87517043bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-8ggwp\" (UID: \"ec6bda4b-feb4-4376-98cf-ee87517043bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8ggwp"
Jul 16 23:07:40.549450 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:40.549433    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec6bda4b-feb4-4376-98cf-ee87517043bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-8ggwp\" (UID: \"ec6bda4b-feb4-4376-98cf-ee87517043bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8ggwp"
Jul 16 23:07:40.584147 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:40.584121    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8ggwp"
Jul 16 23:07:40.707583 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:40.707546    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8ggwp"]
Jul 16 23:07:40.716764 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:07:40.716731    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec6bda4b_feb4_4376_98cf_ee87517043bb.slice/crio-286bf7610166db7dc1daee369efc3b6e251425df81018bf62ea3b675012c22cd WatchSource:0}: Error finding container 286bf7610166db7dc1daee369efc3b6e251425df81018bf62ea3b675012c22cd: Status 404 returned error can't find the container with id 286bf7610166db7dc1daee369efc3b6e251425df81018bf62ea3b675012c22cd
Jul 16 23:07:41.723000 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:41.722964    2573 generic.go:356] "Generic (PLEG): container finished" podID="ec6bda4b-feb4-4376-98cf-ee87517043bb" containerID="d9e9bc710ba4f070e263d2447cdeeadf58d6e46c10f24f7646bc597c399000ba" exitCode=7
Jul 16 23:07:41.723460 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:41.723050    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8ggwp" event={"ID":"ec6bda4b-feb4-4376-98cf-ee87517043bb","Type":"ContainerDied","Data":"d9e9bc710ba4f070e263d2447cdeeadf58d6e46c10f24f7646bc597c399000ba"}
Jul 16 23:07:41.723460 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:41.723103    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8ggwp" event={"ID":"ec6bda4b-feb4-4376-98cf-ee87517043bb","Type":"ContainerStarted","Data":"286bf7610166db7dc1daee369efc3b6e251425df81018bf62ea3b675012c22cd"}
Jul 16 23:07:41.723460 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:41.723156    2573 scope.go:117] "RemoveContainer" containerID="d9e9bc710ba4f070e263d2447cdeeadf58d6e46c10f24f7646bc597c399000ba"
Jul 16 23:07:42.477518 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:42.477483    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-n9x69"]
Jul 16 23:07:42.480723 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:42.480702    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-n9x69"
Jul 16 23:07:42.487690 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:42.487666    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-n9x69"]
Jul 16 23:07:42.563863 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:42.563811    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e26809a-ae4b-41be-8449-45b067e1446e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-n9x69\" (UID: \"6e26809a-ae4b-41be-8449-45b067e1446e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-n9x69"
Jul 16 23:07:42.664994 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:42.664969    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e26809a-ae4b-41be-8449-45b067e1446e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-n9x69\" (UID: \"6e26809a-ae4b-41be-8449-45b067e1446e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-n9x69"
Jul 16 23:07:42.665572 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:42.665553    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e26809a-ae4b-41be-8449-45b067e1446e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-n9x69\" (UID: \"6e26809a-ae4b-41be-8449-45b067e1446e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-n9x69"
Jul 16 23:07:42.729182 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:42.729147    2573 generic.go:356] "Generic (PLEG): container finished" podID="ec6bda4b-feb4-4376-98cf-ee87517043bb" containerID="7dea5b46b006b308e57211bab628d87d7239dd85709ecdf6e1111032de6fd4b7" exitCode=7
Jul 16 23:07:42.729498 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:42.729250    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8ggwp" event={"ID":"ec6bda4b-feb4-4376-98cf-ee87517043bb","Type":"ContainerDied","Data":"7dea5b46b006b308e57211bab628d87d7239dd85709ecdf6e1111032de6fd4b7"}
Jul 16 23:07:42.729498 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:42.729303    2573 scope.go:117] "RemoveContainer" containerID="d9e9bc710ba4f070e263d2447cdeeadf58d6e46c10f24f7646bc597c399000ba"
Jul 16 23:07:42.729498 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:42.729420    2573 scope.go:117] "RemoveContainer" containerID="7dea5b46b006b308e57211bab628d87d7239dd85709ecdf6e1111032de6fd4b7"
Jul 16 23:07:42.729627 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:07:42.729607    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-8ggwp_odh-ai-gateway-infra(ec6bda4b-feb4-4376-98cf-ee87517043bb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8ggwp" podUID="ec6bda4b-feb4-4376-98cf-ee87517043bb"
Jul 16 23:07:42.791191 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:42.791169    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-n9x69"
Jul 16 23:07:42.916059 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:42.916035    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-n9x69"]
Jul 16 23:07:42.918206 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:07:42.918170    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e26809a_ae4b_41be_8449_45b067e1446e.slice/crio-e23a1f9388c198667a59a250d5326c812220388ed20ea236291ecbfa7278e9ae WatchSource:0}: Error finding container e23a1f9388c198667a59a250d5326c812220388ed20ea236291ecbfa7278e9ae: Status 404 returned error can't find the container with id e23a1f9388c198667a59a250d5326c812220388ed20ea236291ecbfa7278e9ae
Jul 16 23:07:43.735743 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:43.735675    2573 generic.go:356] "Generic (PLEG): container finished" podID="6e26809a-ae4b-41be-8449-45b067e1446e" containerID="bdedfdd188ce88c7e5e72bcd0bfb386cfd0249b467485a74519aa6e56568278d" exitCode=6
Jul 16 23:07:43.736107 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:43.735763    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-n9x69" event={"ID":"6e26809a-ae4b-41be-8449-45b067e1446e","Type":"ContainerDied","Data":"bdedfdd188ce88c7e5e72bcd0bfb386cfd0249b467485a74519aa6e56568278d"}
Jul 16 23:07:43.736107 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:43.735831    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-n9x69" event={"ID":"6e26809a-ae4b-41be-8449-45b067e1446e","Type":"ContainerStarted","Data":"e23a1f9388c198667a59a250d5326c812220388ed20ea236291ecbfa7278e9ae"}
Jul 16 23:07:43.736107 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:43.735899    2573 scope.go:117] "RemoveContainer" containerID="bdedfdd188ce88c7e5e72bcd0bfb386cfd0249b467485a74519aa6e56568278d"
Jul 16 23:07:43.737567 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:43.737548    2573 scope.go:117] "RemoveContainer" containerID="7dea5b46b006b308e57211bab628d87d7239dd85709ecdf6e1111032de6fd4b7"
Jul 16 23:07:43.737768 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:07:43.737749    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-8ggwp_odh-ai-gateway-infra(ec6bda4b-feb4-4376-98cf-ee87517043bb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8ggwp" podUID="ec6bda4b-feb4-4376-98cf-ee87517043bb"
Jul 16 23:07:44.744136 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:44.744108    2573 generic.go:356] "Generic (PLEG): container finished" podID="6e26809a-ae4b-41be-8449-45b067e1446e" containerID="0987caf27f5e9d01b7152d10e9102705d31ce59fc11a3bd7362fa0a50cfb7eee" exitCode=6
Jul 16 23:07:44.744514 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:44.744195    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-n9x69" event={"ID":"6e26809a-ae4b-41be-8449-45b067e1446e","Type":"ContainerDied","Data":"0987caf27f5e9d01b7152d10e9102705d31ce59fc11a3bd7362fa0a50cfb7eee"}
Jul 16 23:07:44.744514 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:44.744272    2573 scope.go:117] "RemoveContainer" containerID="bdedfdd188ce88c7e5e72bcd0bfb386cfd0249b467485a74519aa6e56568278d"
Jul 16 23:07:44.744514 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:44.744412    2573 scope.go:117] "RemoveContainer" containerID="0987caf27f5e9d01b7152d10e9102705d31ce59fc11a3bd7362fa0a50cfb7eee"
Jul 16 23:07:44.744638 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:07:44.744600    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-n9x69_odh-ai-gateway-infra(6e26809a-ae4b-41be-8449-45b067e1446e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-n9x69" podUID="6e26809a-ae4b-41be-8449-45b067e1446e"
Jul 16 23:07:45.751533 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:45.751500    2573 scope.go:117] "RemoveContainer" containerID="0987caf27f5e9d01b7152d10e9102705d31ce59fc11a3bd7362fa0a50cfb7eee"
Jul 16 23:07:45.751927 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:07:45.751662    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-n9x69_odh-ai-gateway-infra(6e26809a-ae4b-41be-8449-45b067e1446e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-n9x69" podUID="6e26809a-ae4b-41be-8449-45b067e1446e"
Jul 16 23:07:46.516586 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:46.516553    2573 scope.go:117] "RemoveContainer" containerID="20283794097f4f3db2f7543eeed5e0369b80e9df11c4aa45ae91a22a241a20e1"
Jul 16 23:07:46.758694 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:46.758667    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mk2hn" event={"ID":"8413502d-a989-496c-85ca-3225ef45fe8c","Type":"ContainerStarted","Data":"f3f0817327c74a09cdda2733b3974e31f86acd35c8f630b07a3ae15d2440775b"}
Jul 16 23:07:46.774229 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:46.774151    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mk2hn" podStartSLOduration=15.77413649 podStartE2EDuration="15.77413649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:07:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:07:46.773178082 +0000 UTC m=+2040.783440772" watchObservedRunningTime="2026-07-16 23:07:46.77413649 +0000 UTC m=+2040.784399153"
Jul 16 23:07:47.766512 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:47.766476    2573 generic.go:356] "Generic (PLEG): container finished" podID="8413502d-a989-496c-85ca-3225ef45fe8c" containerID="f3f0817327c74a09cdda2733b3974e31f86acd35c8f630b07a3ae15d2440775b" exitCode=6
Jul 16 23:07:47.766943 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:47.766559    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mk2hn" event={"ID":"8413502d-a989-496c-85ca-3225ef45fe8c","Type":"ContainerDied","Data":"f3f0817327c74a09cdda2733b3974e31f86acd35c8f630b07a3ae15d2440775b"}
Jul 16 23:07:47.766943 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:47.766634    2573 scope.go:117] "RemoveContainer" containerID="20283794097f4f3db2f7543eeed5e0369b80e9df11c4aa45ae91a22a241a20e1"
Jul 16 23:07:47.766943 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:47.766929    2573 scope.go:117] "RemoveContainer" containerID="f3f0817327c74a09cdda2733b3974e31f86acd35c8f630b07a3ae15d2440775b"
Jul 16 23:07:47.767158 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:07:47.767136    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-mk2hn_odh-ai-gateway-infra(8413502d-a989-496c-85ca-3225ef45fe8c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mk2hn" podUID="8413502d-a989-496c-85ca-3225ef45fe8c"
Jul 16 23:07:47.783336 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:47.783308    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mk2hn"]
Jul 16 23:07:48.896163 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:48.896142    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mk2hn"
Jul 16 23:07:48.900813 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:48.900785    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8413502d-a989-496c-85ca-3225ef45fe8c-maas-api-service-ca\") pod \"8413502d-a989-496c-85ca-3225ef45fe8c\" (UID: \"8413502d-a989-496c-85ca-3225ef45fe8c\") "
Jul 16 23:07:48.901078 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:48.901057    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8413502d-a989-496c-85ca-3225ef45fe8c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8413502d-a989-496c-85ca-3225ef45fe8c" (UID: "8413502d-a989-496c-85ca-3225ef45fe8c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:07:49.001558 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:49.001533    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8413502d-a989-496c-85ca-3225ef45fe8c-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:07:49.223220 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:49.223183    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:07:49.313927 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:49.313897    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:07:49.780165 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:49.780144    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mk2hn"
Jul 16 23:07:49.780333 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:49.780144    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mk2hn" event={"ID":"8413502d-a989-496c-85ca-3225ef45fe8c","Type":"ContainerDied","Data":"e21c58ec0ec5915f119170a24c67c8037d0d2db5b4ee7e7bd4fcdf19801e7f52"}
Jul 16 23:07:49.780333 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:49.780304    2573 scope.go:117] "RemoveContainer" containerID="f3f0817327c74a09cdda2733b3974e31f86acd35c8f630b07a3ae15d2440775b"
Jul 16 23:07:49.802923 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:49.802898    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mk2hn"]
Jul 16 23:07:49.808688 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:49.808669    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mk2hn"]
Jul 16 23:07:50.518645 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:50.518609    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8413502d-a989-496c-85ca-3225ef45fe8c" path="/var/lib/kubelet/pods/8413502d-a989-496c-85ca-3225ef45fe8c/volumes"
Jul 16 23:07:53.531249 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:53.531211    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:07:57.513625 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:57.513592    2573 scope.go:117] "RemoveContainer" containerID="7dea5b46b006b308e57211bab628d87d7239dd85709ecdf6e1111032de6fd4b7"
Jul 16 23:07:57.823888 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:57.823858    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8ggwp" event={"ID":"ec6bda4b-feb4-4376-98cf-ee87517043bb","Type":"ContainerStarted","Data":"1dc5b7e1c4b3da2d3f5934a8a685fc9c9b53cd8b3ce313adb69af84d1b9eb51d"}
Jul 16 23:07:57.840300 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:57.840259    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8ggwp" podStartSLOduration=17.840246454 podStartE2EDuration="17.840246454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:07:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:07:57.837700649 +0000 UTC m=+2051.847963334" watchObservedRunningTime="2026-07-16 23:07:57.840246454 +0000 UTC m=+2051.850509123"
Jul 16 23:07:58.830444 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:58.830390    2573 generic.go:356] "Generic (PLEG): container finished" podID="ec6bda4b-feb4-4376-98cf-ee87517043bb" containerID="1dc5b7e1c4b3da2d3f5934a8a685fc9c9b53cd8b3ce313adb69af84d1b9eb51d" exitCode=7
Jul 16 23:07:58.830879 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:58.830479    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8ggwp" event={"ID":"ec6bda4b-feb4-4376-98cf-ee87517043bb","Type":"ContainerDied","Data":"1dc5b7e1c4b3da2d3f5934a8a685fc9c9b53cd8b3ce313adb69af84d1b9eb51d"}
Jul 16 23:07:58.830879 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:58.830565    2573 scope.go:117] "RemoveContainer" containerID="7dea5b46b006b308e57211bab628d87d7239dd85709ecdf6e1111032de6fd4b7"
Jul 16 23:07:58.830879 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:58.830821    2573 scope.go:117] "RemoveContainer" containerID="1dc5b7e1c4b3da2d3f5934a8a685fc9c9b53cd8b3ce313adb69af84d1b9eb51d"
Jul 16 23:07:58.831055 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:07:58.831033    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-8ggwp_odh-ai-gateway-infra(ec6bda4b-feb4-4376-98cf-ee87517043bb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8ggwp" podUID="ec6bda4b-feb4-4376-98cf-ee87517043bb"
Jul 16 23:07:58.847966 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:58.847938    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8ggwp"]
Jul 16 23:07:59.971279 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:07:59.971256    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8ggwp"
Jul 16 23:08:00.077927 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:00.077901    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec6bda4b-feb4-4376-98cf-ee87517043bb-maas-api-service-ca\") pod \"ec6bda4b-feb4-4376-98cf-ee87517043bb\" (UID: \"ec6bda4b-feb4-4376-98cf-ee87517043bb\") "
Jul 16 23:08:00.078261 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:00.078214    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec6bda4b-feb4-4376-98cf-ee87517043bb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ec6bda4b-feb4-4376-98cf-ee87517043bb" (UID: "ec6bda4b-feb4-4376-98cf-ee87517043bb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:08:00.179323 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:00.179258    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec6bda4b-feb4-4376-98cf-ee87517043bb-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:08:00.514016 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:00.513955    2573 scope.go:117] "RemoveContainer" containerID="0987caf27f5e9d01b7152d10e9102705d31ce59fc11a3bd7362fa0a50cfb7eee"
Jul 16 23:08:00.842414 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:00.842382    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8ggwp"
Jul 16 23:08:00.842525 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:00.842432    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8ggwp" event={"ID":"ec6bda4b-feb4-4376-98cf-ee87517043bb","Type":"ContainerDied","Data":"286bf7610166db7dc1daee369efc3b6e251425df81018bf62ea3b675012c22cd"}
Jul 16 23:08:00.842525 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:00.842488    2573 scope.go:117] "RemoveContainer" containerID="1dc5b7e1c4b3da2d3f5934a8a685fc9c9b53cd8b3ce313adb69af84d1b9eb51d"
Jul 16 23:08:00.844634 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:00.844611    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-n9x69" event={"ID":"6e26809a-ae4b-41be-8449-45b067e1446e","Type":"ContainerStarted","Data":"5a3129679de3f593f4a53e2eeb3e162953d77b61e6bd27b78e1afc98e33bd5a8"}
Jul 16 23:08:00.857472 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:00.857448    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8ggwp"]
Jul 16 23:08:00.863014 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:00.862995    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8ggwp"]
Jul 16 23:08:01.852855 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:01.852822    2573 generic.go:356] "Generic (PLEG): container finished" podID="6e26809a-ae4b-41be-8449-45b067e1446e" containerID="5a3129679de3f593f4a53e2eeb3e162953d77b61e6bd27b78e1afc98e33bd5a8" exitCode=6
Jul 16 23:08:01.853371 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:01.852894    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-n9x69" event={"ID":"6e26809a-ae4b-41be-8449-45b067e1446e","Type":"ContainerDied","Data":"5a3129679de3f593f4a53e2eeb3e162953d77b61e6bd27b78e1afc98e33bd5a8"}
Jul 16 23:08:01.853371 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:01.852950    2573 scope.go:117] "RemoveContainer" containerID="0987caf27f5e9d01b7152d10e9102705d31ce59fc11a3bd7362fa0a50cfb7eee"
Jul 16 23:08:01.853371 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:01.853169    2573 scope.go:117] "RemoveContainer" containerID="5a3129679de3f593f4a53e2eeb3e162953d77b61e6bd27b78e1afc98e33bd5a8"
Jul 16 23:08:01.853371 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:08:01.853353    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-n9x69_odh-ai-gateway-infra(6e26809a-ae4b-41be-8449-45b067e1446e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-n9x69" podUID="6e26809a-ae4b-41be-8449-45b067e1446e"
Jul 16 23:08:01.884992 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:01.884962    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-n9x69"]
Jul 16 23:08:02.518926 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:02.518891    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec6bda4b-feb4-4376-98cf-ee87517043bb" path="/var/lib/kubelet/pods/ec6bda4b-feb4-4376-98cf-ee87517043bb/volumes"
Jul 16 23:08:02.990196 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:02.990177    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-n9x69"
Jul 16 23:08:03.100959 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:03.100938    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e26809a-ae4b-41be-8449-45b067e1446e-maas-api-service-ca\") pod \"6e26809a-ae4b-41be-8449-45b067e1446e\" (UID: \"6e26809a-ae4b-41be-8449-45b067e1446e\") "
Jul 16 23:08:03.101290 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:03.101268    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e26809a-ae4b-41be-8449-45b067e1446e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6e26809a-ae4b-41be-8449-45b067e1446e" (UID: "6e26809a-ae4b-41be-8449-45b067e1446e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:08:03.201795 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:03.201745    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e26809a-ae4b-41be-8449-45b067e1446e-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:08:03.866005 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:03.865970    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-n9x69" event={"ID":"6e26809a-ae4b-41be-8449-45b067e1446e","Type":"ContainerDied","Data":"e23a1f9388c198667a59a250d5326c812220388ed20ea236291ecbfa7278e9ae"}
Jul 16 23:08:03.866005 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:03.865998    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-n9x69"
Jul 16 23:08:03.866256 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:03.866038    2573 scope.go:117] "RemoveContainer" containerID="5a3129679de3f593f4a53e2eeb3e162953d77b61e6bd27b78e1afc98e33bd5a8"
Jul 16 23:08:03.887263 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:03.887226    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-n9x69"]
Jul 16 23:08:03.888904 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:03.888883    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-n9x69"]
Jul 16 23:08:04.518641 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:04.518608    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e26809a-ae4b-41be-8449-45b067e1446e" path="/var/lib/kubelet/pods/6e26809a-ae4b-41be-8449-45b067e1446e/volumes"
Jul 16 23:08:21.309083 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:21.308997    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lhrlv"]
Jul 16 23:08:21.309537 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:21.309338    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec6bda4b-feb4-4376-98cf-ee87517043bb" containerName="revoke-keys"
Jul 16 23:08:21.309537 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:21.309352    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec6bda4b-feb4-4376-98cf-ee87517043bb" containerName="revoke-keys"
Jul 16 23:08:21.309537 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:21.309360    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e26809a-ae4b-41be-8449-45b067e1446e" containerName="revoke-keys"
Jul 16 23:08:21.309537 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:21.309365    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e26809a-ae4b-41be-8449-45b067e1446e" containerName="revoke-keys"
Jul 16 23:08:21.309537 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:21.309371    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8413502d-a989-496c-85ca-3225ef45fe8c" containerName="revoke-keys"
Jul 16 23:08:21.309537 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:21.309378    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8413502d-a989-496c-85ca-3225ef45fe8c" containerName="revoke-keys"
Jul 16 23:08:21.309537 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:21.309388    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8413502d-a989-496c-85ca-3225ef45fe8c" containerName="revoke-keys"
Jul 16 23:08:21.309537 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:21.309393    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8413502d-a989-496c-85ca-3225ef45fe8c" containerName="revoke-keys"
Jul 16 23:08:21.309537 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:21.309402    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec6bda4b-feb4-4376-98cf-ee87517043bb" containerName="revoke-keys"
Jul 16 23:08:21.309537 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:21.309407    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec6bda4b-feb4-4376-98cf-ee87517043bb" containerName="revoke-keys"
Jul 16 23:08:21.309537 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:21.309418    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e26809a-ae4b-41be-8449-45b067e1446e" containerName="revoke-keys"
Jul 16 23:08:21.309537 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:21.309423    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e26809a-ae4b-41be-8449-45b067e1446e" containerName="revoke-keys"
Jul 16 23:08:21.309537 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:21.309476    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec6bda4b-feb4-4376-98cf-ee87517043bb" containerName="revoke-keys"
Jul 16 23:08:21.309537 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:21.309485    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec6bda4b-feb4-4376-98cf-ee87517043bb" containerName="revoke-keys"
Jul 16 23:08:21.309537 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:21.309491    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8413502d-a989-496c-85ca-3225ef45fe8c" containerName="revoke-keys"
Jul 16 23:08:21.309537 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:21.309496    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e26809a-ae4b-41be-8449-45b067e1446e" containerName="revoke-keys"
Jul 16 23:08:21.309537 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:21.309501    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8413502d-a989-496c-85ca-3225ef45fe8c" containerName="revoke-keys"
Jul 16 23:08:21.309537 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:21.309506    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e26809a-ae4b-41be-8449-45b067e1446e" containerName="revoke-keys"
Jul 16 23:08:21.309537 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:21.309512    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e26809a-ae4b-41be-8449-45b067e1446e" containerName="revoke-keys"
Jul 16 23:08:21.312275 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:21.312230    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lhrlv"
Jul 16 23:08:21.314514 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:21.314493    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 23:08:21.314630 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:21.314521    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:08:21.318614 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:21.318586    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lhrlv"]
Jul 16 23:08:21.441770 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:21.441739    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f47c6ce4-7ca3-47d7-bdd2-54ce635c6487-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-lhrlv\" (UID: \"f47c6ce4-7ca3-47d7-bdd2-54ce635c6487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lhrlv"
Jul 16 23:08:21.542962 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:21.542926    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f47c6ce4-7ca3-47d7-bdd2-54ce635c6487-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-lhrlv\" (UID: \"f47c6ce4-7ca3-47d7-bdd2-54ce635c6487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lhrlv"
Jul 16 23:08:21.543548 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:21.543530    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f47c6ce4-7ca3-47d7-bdd2-54ce635c6487-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-lhrlv\" (UID: \"f47c6ce4-7ca3-47d7-bdd2-54ce635c6487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lhrlv"
Jul 16 23:08:21.623396 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:21.623357    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lhrlv"
Jul 16 23:08:21.745989 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:21.745960    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lhrlv"]
Jul 16 23:08:21.748844 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:08:21.748816    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf47c6ce4_7ca3_47d7_bdd2_54ce635c6487.slice/crio-0054ca6c48d1e493476ffb38ccd32f44083becb789754226e0bfe5df8ead0310 WatchSource:0}: Error finding container 0054ca6c48d1e493476ffb38ccd32f44083becb789754226e0bfe5df8ead0310: Status 404 returned error can't find the container with id 0054ca6c48d1e493476ffb38ccd32f44083becb789754226e0bfe5df8ead0310
Jul 16 23:08:21.952369 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:21.952335    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lhrlv" event={"ID":"f47c6ce4-7ca3-47d7-bdd2-54ce635c6487","Type":"ContainerStarted","Data":"0054ca6c48d1e493476ffb38ccd32f44083becb789754226e0bfe5df8ead0310"}
Jul 16 23:08:22.958826 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:22.958794    2573 generic.go:356] "Generic (PLEG): container finished" podID="f47c6ce4-7ca3-47d7-bdd2-54ce635c6487" containerID="7417cac1dd30ede983c7a6dedea4e1b11d4d78c78122cc3b1f62d80745be6eb1" exitCode=6
Jul 16 23:08:22.959263 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:22.958876    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lhrlv" event={"ID":"f47c6ce4-7ca3-47d7-bdd2-54ce635c6487","Type":"ContainerDied","Data":"7417cac1dd30ede983c7a6dedea4e1b11d4d78c78122cc3b1f62d80745be6eb1"}
Jul 16 23:08:22.959263 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:22.959044    2573 scope.go:117] "RemoveContainer" containerID="7417cac1dd30ede983c7a6dedea4e1b11d4d78c78122cc3b1f62d80745be6eb1"
Jul 16 23:08:23.966014 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:23.965922    2573 generic.go:356] "Generic (PLEG): container finished" podID="f47c6ce4-7ca3-47d7-bdd2-54ce635c6487" containerID="c4795404058a983b0c98f9f12561d0542cac356ebd498588ef1656da1a08f7dd" exitCode=6
Jul 16 23:08:23.966380 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:23.966011    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lhrlv" event={"ID":"f47c6ce4-7ca3-47d7-bdd2-54ce635c6487","Type":"ContainerDied","Data":"c4795404058a983b0c98f9f12561d0542cac356ebd498588ef1656da1a08f7dd"}
Jul 16 23:08:23.966380 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:23.966084    2573 scope.go:117] "RemoveContainer" containerID="7417cac1dd30ede983c7a6dedea4e1b11d4d78c78122cc3b1f62d80745be6eb1"
Jul 16 23:08:23.966380 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:23.966331    2573 scope.go:117] "RemoveContainer" containerID="c4795404058a983b0c98f9f12561d0542cac356ebd498588ef1656da1a08f7dd"
Jul 16 23:08:23.966569 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:08:23.966504    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-lhrlv_odh-ai-gateway-infra(f47c6ce4-7ca3-47d7-bdd2-54ce635c6487)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lhrlv" podUID="f47c6ce4-7ca3-47d7-bdd2-54ce635c6487"
Jul 16 23:08:24.971876 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:24.971850    2573 scope.go:117] "RemoveContainer" containerID="c4795404058a983b0c98f9f12561d0542cac356ebd498588ef1656da1a08f7dd"
Jul 16 23:08:24.972263 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:08:24.972008    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-lhrlv_odh-ai-gateway-infra(f47c6ce4-7ca3-47d7-bdd2-54ce635c6487)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lhrlv" podUID="f47c6ce4-7ca3-47d7-bdd2-54ce635c6487"
Jul 16 23:08:39.513526 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:39.513474    2573 scope.go:117] "RemoveContainer" containerID="c4795404058a983b0c98f9f12561d0542cac356ebd498588ef1656da1a08f7dd"
Jul 16 23:08:40.045374 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:40.045341    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lhrlv" event={"ID":"f47c6ce4-7ca3-47d7-bdd2-54ce635c6487","Type":"ContainerStarted","Data":"0a2de14e70b914dd99eed8e09dc6bb7e667db6d65863015779542646133ada7c"}
Jul 16 23:08:40.060691 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:40.060645    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lhrlv" podStartSLOduration=19.060632222 podStartE2EDuration="19.060632222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:08:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:08:40.058755532 +0000 UTC m=+2094.069018238" watchObservedRunningTime="2026-07-16 23:08:40.060632222 +0000 UTC m=+2094.070894886"
Jul 16 23:08:40.305320 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:40.305256    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-v49p5"]
Jul 16 23:08:40.305559 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:40.305546    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e26809a-ae4b-41be-8449-45b067e1446e" containerName="revoke-keys"
Jul 16 23:08:40.305598 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:40.305561    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e26809a-ae4b-41be-8449-45b067e1446e" containerName="revoke-keys"
Jul 16 23:08:40.305598 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:40.305573    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8413502d-a989-496c-85ca-3225ef45fe8c" containerName="revoke-keys"
Jul 16 23:08:40.305598 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:40.305578    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8413502d-a989-496c-85ca-3225ef45fe8c" containerName="revoke-keys"
Jul 16 23:08:40.305598 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:40.305587    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec6bda4b-feb4-4376-98cf-ee87517043bb" containerName="revoke-keys"
Jul 16 23:08:40.305598 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:40.305593    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec6bda4b-feb4-4376-98cf-ee87517043bb" containerName="revoke-keys"
Jul 16 23:08:40.305748 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:40.305642    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec6bda4b-feb4-4376-98cf-ee87517043bb" containerName="revoke-keys"
Jul 16 23:08:40.305748 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:40.305651    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8413502d-a989-496c-85ca-3225ef45fe8c" containerName="revoke-keys"
Jul 16 23:08:40.308406 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:40.308390    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-v49p5"
Jul 16 23:08:40.315647 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:40.315624    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-v49p5"]
Jul 16 23:08:40.385660 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:40.385635    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/970bcbff-b089-45a2-b77a-7959c1f89169-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-v49p5\" (UID: \"970bcbff-b089-45a2-b77a-7959c1f89169\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-v49p5"
Jul 16 23:08:40.486708 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:40.486681    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/970bcbff-b089-45a2-b77a-7959c1f89169-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-v49p5\" (UID: \"970bcbff-b089-45a2-b77a-7959c1f89169\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-v49p5"
Jul 16 23:08:40.487231 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:40.487214    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/970bcbff-b089-45a2-b77a-7959c1f89169-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-v49p5\" (UID: \"970bcbff-b089-45a2-b77a-7959c1f89169\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-v49p5"
Jul 16 23:08:40.619100 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:40.619080    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-v49p5"
Jul 16 23:08:40.745252 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:40.745212    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-v49p5"]
Jul 16 23:08:40.747247 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:08:40.747207    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod970bcbff_b089_45a2_b77a_7959c1f89169.slice/crio-687b67993bf718e39409f06ad951125723944c3a04df5bf06b0dfff6ef03f190 WatchSource:0}: Error finding container 687b67993bf718e39409f06ad951125723944c3a04df5bf06b0dfff6ef03f190: Status 404 returned error can't find the container with id 687b67993bf718e39409f06ad951125723944c3a04df5bf06b0dfff6ef03f190
Jul 16 23:08:41.051121 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:41.051093    2573 generic.go:356] "Generic (PLEG): container finished" podID="f47c6ce4-7ca3-47d7-bdd2-54ce635c6487" containerID="0a2de14e70b914dd99eed8e09dc6bb7e667db6d65863015779542646133ada7c" exitCode=6
Jul 16 23:08:41.051228 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:41.051160    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lhrlv" event={"ID":"f47c6ce4-7ca3-47d7-bdd2-54ce635c6487","Type":"ContainerDied","Data":"0a2de14e70b914dd99eed8e09dc6bb7e667db6d65863015779542646133ada7c"}
Jul 16 23:08:41.051228 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:41.051222    2573 scope.go:117] "RemoveContainer" containerID="c4795404058a983b0c98f9f12561d0542cac356ebd498588ef1656da1a08f7dd"
Jul 16 23:08:41.051513 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:41.051497    2573 scope.go:117] "RemoveContainer" containerID="0a2de14e70b914dd99eed8e09dc6bb7e667db6d65863015779542646133ada7c"
Jul 16 23:08:41.051743 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:08:41.051713    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-lhrlv_odh-ai-gateway-infra(f47c6ce4-7ca3-47d7-bdd2-54ce635c6487)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lhrlv" podUID="f47c6ce4-7ca3-47d7-bdd2-54ce635c6487"
Jul 16 23:08:41.052721 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:41.052701    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-v49p5" event={"ID":"970bcbff-b089-45a2-b77a-7959c1f89169","Type":"ContainerStarted","Data":"a98d3b5b89759d91b1993d073f0de68b4964a6d08dcffb03f34d6456cf0c051e"}
Jul 16 23:08:41.052803 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:41.052742    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-v49p5" event={"ID":"970bcbff-b089-45a2-b77a-7959c1f89169","Type":"ContainerStarted","Data":"687b67993bf718e39409f06ad951125723944c3a04df5bf06b0dfff6ef03f190"}
Jul 16 23:08:41.071393 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:41.071372    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lhrlv"]
Jul 16 23:08:41.082844 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:41.082809    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-v49p5" podStartSLOduration=1.082795837 podStartE2EDuration="1.082795837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:08:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:08:41.081151074 +0000 UTC m=+2095.091413772" watchObservedRunningTime="2026-07-16 23:08:41.082795837 +0000 UTC m=+2095.093058501"
Jul 16 23:08:42.061641 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:42.061604    2573 generic.go:356] "Generic (PLEG): container finished" podID="970bcbff-b089-45a2-b77a-7959c1f89169" containerID="a98d3b5b89759d91b1993d073f0de68b4964a6d08dcffb03f34d6456cf0c051e" exitCode=7
Jul 16 23:08:42.061974 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:42.061654    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-v49p5" event={"ID":"970bcbff-b089-45a2-b77a-7959c1f89169","Type":"ContainerDied","Data":"a98d3b5b89759d91b1993d073f0de68b4964a6d08dcffb03f34d6456cf0c051e"}
Jul 16 23:08:42.061974 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:42.061823    2573 scope.go:117] "RemoveContainer" containerID="a98d3b5b89759d91b1993d073f0de68b4964a6d08dcffb03f34d6456cf0c051e"
Jul 16 23:08:42.185750 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:42.185729    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lhrlv"
Jul 16 23:08:42.300344 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:42.300325    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f47c6ce4-7ca3-47d7-bdd2-54ce635c6487-maas-api-service-ca\") pod \"f47c6ce4-7ca3-47d7-bdd2-54ce635c6487\" (UID: \"f47c6ce4-7ca3-47d7-bdd2-54ce635c6487\") "
Jul 16 23:08:42.300658 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:42.300635    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f47c6ce4-7ca3-47d7-bdd2-54ce635c6487-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f47c6ce4-7ca3-47d7-bdd2-54ce635c6487" (UID: "f47c6ce4-7ca3-47d7-bdd2-54ce635c6487"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:08:42.401614 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:42.401590    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f47c6ce4-7ca3-47d7-bdd2-54ce635c6487-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:08:42.519324 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:42.519301    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-4tzbz"]
Jul 16 23:08:42.519592 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:42.519576    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f47c6ce4-7ca3-47d7-bdd2-54ce635c6487" containerName="revoke-keys"
Jul 16 23:08:42.519592 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:42.519592    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f47c6ce4-7ca3-47d7-bdd2-54ce635c6487" containerName="revoke-keys"
Jul 16 23:08:42.519724 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:42.519610    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f47c6ce4-7ca3-47d7-bdd2-54ce635c6487" containerName="revoke-keys"
Jul 16 23:08:42.519724 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:42.519616    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f47c6ce4-7ca3-47d7-bdd2-54ce635c6487" containerName="revoke-keys"
Jul 16 23:08:42.519724 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:42.519664    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f47c6ce4-7ca3-47d7-bdd2-54ce635c6487" containerName="revoke-keys"
Jul 16 23:08:42.519724 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:42.519671    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f47c6ce4-7ca3-47d7-bdd2-54ce635c6487" containerName="revoke-keys"
Jul 16 23:08:42.519724 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:42.519680    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f47c6ce4-7ca3-47d7-bdd2-54ce635c6487" containerName="revoke-keys"
Jul 16 23:08:42.522456 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:42.522437    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-4tzbz"
Jul 16 23:08:42.530078 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:42.530058    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-4tzbz"]
Jul 16 23:08:42.603599 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:42.603568    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad391f4f-baed-4d47-a247-fd9c6d8b76e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-4tzbz\" (UID: \"ad391f4f-baed-4d47-a247-fd9c6d8b76e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-4tzbz"
Jul 16 23:08:42.704108 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:42.704083    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad391f4f-baed-4d47-a247-fd9c6d8b76e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-4tzbz\" (UID: \"ad391f4f-baed-4d47-a247-fd9c6d8b76e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-4tzbz"
Jul 16 23:08:42.704712 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:42.704695    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad391f4f-baed-4d47-a247-fd9c6d8b76e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-4tzbz\" (UID: \"ad391f4f-baed-4d47-a247-fd9c6d8b76e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-4tzbz"
Jul 16 23:08:42.835099 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:42.835066    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-4tzbz"
Jul 16 23:08:42.958710 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:42.958685    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-4tzbz"]
Jul 16 23:08:42.961074 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:08:42.961054    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad391f4f_baed_4d47_a247_fd9c6d8b76e0.slice/crio-944e2b22ee220c9e98a8461be658afebfc7d0177c211b8b6701de74fb732c666 WatchSource:0}: Error finding container 944e2b22ee220c9e98a8461be658afebfc7d0177c211b8b6701de74fb732c666: Status 404 returned error can't find the container with id 944e2b22ee220c9e98a8461be658afebfc7d0177c211b8b6701de74fb732c666
Jul 16 23:08:43.067777 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:43.067748    2573 generic.go:356] "Generic (PLEG): container finished" podID="970bcbff-b089-45a2-b77a-7959c1f89169" containerID="17a8653890029709f2db4121c6f37091277a7f879479a8402e43a04a8a58c54e" exitCode=7
Jul 16 23:08:43.068117 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:43.067807    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-v49p5" event={"ID":"970bcbff-b089-45a2-b77a-7959c1f89169","Type":"ContainerDied","Data":"17a8653890029709f2db4121c6f37091277a7f879479a8402e43a04a8a58c54e"}
Jul 16 23:08:43.068117 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:43.067874    2573 scope.go:117] "RemoveContainer" containerID="a98d3b5b89759d91b1993d073f0de68b4964a6d08dcffb03f34d6456cf0c051e"
Jul 16 23:08:43.068117 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:43.068052    2573 scope.go:117] "RemoveContainer" containerID="17a8653890029709f2db4121c6f37091277a7f879479a8402e43a04a8a58c54e"
Jul 16 23:08:43.068292 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:08:43.068270    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-v49p5_odh-ai-gateway-infra(970bcbff-b089-45a2-b77a-7959c1f89169)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-v49p5" podUID="970bcbff-b089-45a2-b77a-7959c1f89169"
Jul 16 23:08:43.069261 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:43.069226    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lhrlv"
Jul 16 23:08:43.069316 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:43.069261    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lhrlv" event={"ID":"f47c6ce4-7ca3-47d7-bdd2-54ce635c6487","Type":"ContainerDied","Data":"0054ca6c48d1e493476ffb38ccd32f44083becb789754226e0bfe5df8ead0310"}
Jul 16 23:08:43.070472 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:43.070446    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-4tzbz" event={"ID":"ad391f4f-baed-4d47-a247-fd9c6d8b76e0","Type":"ContainerStarted","Data":"944e2b22ee220c9e98a8461be658afebfc7d0177c211b8b6701de74fb732c666"}
Jul 16 23:08:43.084029 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:43.084011    2573 scope.go:117] "RemoveContainer" containerID="0a2de14e70b914dd99eed8e09dc6bb7e667db6d65863015779542646133ada7c"
Jul 16 23:08:43.100353 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:43.100334    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lhrlv"]
Jul 16 23:08:43.101947 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:43.101929    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lhrlv"]
Jul 16 23:08:44.081417 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:44.081381    2573 generic.go:356] "Generic (PLEG): container finished" podID="ad391f4f-baed-4d47-a247-fd9c6d8b76e0" containerID="dba864e54e543aa6d5873331c7327cf866ce7d71aa3ffdcc5ae2a48217175cd4" exitCode=6
Jul 16 23:08:44.081851 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:44.081472    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-4tzbz" event={"ID":"ad391f4f-baed-4d47-a247-fd9c6d8b76e0","Type":"ContainerDied","Data":"dba864e54e543aa6d5873331c7327cf866ce7d71aa3ffdcc5ae2a48217175cd4"}
Jul 16 23:08:44.081851 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:44.081624    2573 scope.go:117] "RemoveContainer" containerID="dba864e54e543aa6d5873331c7327cf866ce7d71aa3ffdcc5ae2a48217175cd4"
Jul 16 23:08:44.083469 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:44.083450    2573 scope.go:117] "RemoveContainer" containerID="17a8653890029709f2db4121c6f37091277a7f879479a8402e43a04a8a58c54e"
Jul 16 23:08:44.083654 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:08:44.083633    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-v49p5_odh-ai-gateway-infra(970bcbff-b089-45a2-b77a-7959c1f89169)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-v49p5" podUID="970bcbff-b089-45a2-b77a-7959c1f89169"
Jul 16 23:08:44.518889 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:44.518855    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f47c6ce4-7ca3-47d7-bdd2-54ce635c6487" path="/var/lib/kubelet/pods/f47c6ce4-7ca3-47d7-bdd2-54ce635c6487/volumes"
Jul 16 23:08:45.090875 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:45.090832    2573 generic.go:356] "Generic (PLEG): container finished" podID="ad391f4f-baed-4d47-a247-fd9c6d8b76e0" containerID="650b8764121a10b6bafde136325cf80ac48ac39c6f6abccf7348235f74572d7e" exitCode=6
Jul 16 23:08:45.091278 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:45.090922    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-4tzbz" event={"ID":"ad391f4f-baed-4d47-a247-fd9c6d8b76e0","Type":"ContainerDied","Data":"650b8764121a10b6bafde136325cf80ac48ac39c6f6abccf7348235f74572d7e"}
Jul 16 23:08:45.091278 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:45.090994    2573 scope.go:117] "RemoveContainer" containerID="dba864e54e543aa6d5873331c7327cf866ce7d71aa3ffdcc5ae2a48217175cd4"
Jul 16 23:08:45.091278 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:45.091272    2573 scope.go:117] "RemoveContainer" containerID="650b8764121a10b6bafde136325cf80ac48ac39c6f6abccf7348235f74572d7e"
Jul 16 23:08:45.091503 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:08:45.091481    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-4tzbz_odh-ai-gateway-infra(ad391f4f-baed-4d47-a247-fd9c6d8b76e0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-4tzbz" podUID="ad391f4f-baed-4d47-a247-fd9c6d8b76e0"
Jul 16 23:08:46.097974 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:46.097949    2573 scope.go:117] "RemoveContainer" containerID="650b8764121a10b6bafde136325cf80ac48ac39c6f6abccf7348235f74572d7e"
Jul 16 23:08:46.098483 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:08:46.098167    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-4tzbz_odh-ai-gateway-infra(ad391f4f-baed-4d47-a247-fd9c6d8b76e0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-4tzbz" podUID="ad391f4f-baed-4d47-a247-fd9c6d8b76e0"
Jul 16 23:08:46.544894 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:46.544822    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r9jm4_506f904a-3272-4586-8528-146884e728f9/ovn-acl-logging/0.log"
Jul 16 23:08:46.550776 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:46.550756    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r9jm4_506f904a-3272-4586-8528-146884e728f9/ovn-acl-logging/0.log"
Jul 16 23:08:57.513372 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:57.513337    2573 scope.go:117] "RemoveContainer" containerID="17a8653890029709f2db4121c6f37091277a7f879479a8402e43a04a8a58c54e"
Jul 16 23:08:58.156347 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:58.156316    2573 generic.go:356] "Generic (PLEG): container finished" podID="970bcbff-b089-45a2-b77a-7959c1f89169" containerID="d1de3d75f99f30082d5d4d17ba7f641fa4669b7284ad15cab5677da4aba47669" exitCode=7
Jul 16 23:08:58.156465 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:58.156394    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-v49p5" event={"ID":"970bcbff-b089-45a2-b77a-7959c1f89169","Type":"ContainerDied","Data":"d1de3d75f99f30082d5d4d17ba7f641fa4669b7284ad15cab5677da4aba47669"}
Jul 16 23:08:58.156465 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:58.156454    2573 scope.go:117] "RemoveContainer" containerID="17a8653890029709f2db4121c6f37091277a7f879479a8402e43a04a8a58c54e"
Jul 16 23:08:58.156754 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:58.156734    2573 scope.go:117] "RemoveContainer" containerID="d1de3d75f99f30082d5d4d17ba7f641fa4669b7284ad15cab5677da4aba47669"
Jul 16 23:08:58.156948 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:08:58.156926    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-v49p5_odh-ai-gateway-infra(970bcbff-b089-45a2-b77a-7959c1f89169)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-v49p5" podUID="970bcbff-b089-45a2-b77a-7959c1f89169"
Jul 16 23:08:59.181987 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:59.181955    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-v49p5"]
Jul 16 23:08:59.307710 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:59.307685    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-v49p5"
Jul 16 23:08:59.425847 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:59.425820    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/970bcbff-b089-45a2-b77a-7959c1f89169-maas-api-service-ca\") pod \"970bcbff-b089-45a2-b77a-7959c1f89169\" (UID: \"970bcbff-b089-45a2-b77a-7959c1f89169\") "
Jul 16 23:08:59.426156 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:59.426133    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/970bcbff-b089-45a2-b77a-7959c1f89169-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "970bcbff-b089-45a2-b77a-7959c1f89169" (UID: "970bcbff-b089-45a2-b77a-7959c1f89169"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:08:59.527015 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:08:59.526959    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/970bcbff-b089-45a2-b77a-7959c1f89169-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:09:00.168697 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:00.168670    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-v49p5"
Jul 16 23:09:00.168697 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:00.168681    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-v49p5" event={"ID":"970bcbff-b089-45a2-b77a-7959c1f89169","Type":"ContainerDied","Data":"687b67993bf718e39409f06ad951125723944c3a04df5bf06b0dfff6ef03f190"}
Jul 16 23:09:00.168888 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:00.168746    2573 scope.go:117] "RemoveContainer" containerID="d1de3d75f99f30082d5d4d17ba7f641fa4669b7284ad15cab5677da4aba47669"
Jul 16 23:09:00.188885 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:00.188857    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-v49p5"]
Jul 16 23:09:00.196492 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:00.196471    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-v49p5"]
Jul 16 23:09:00.518733 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:00.518657    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="970bcbff-b089-45a2-b77a-7959c1f89169" path="/var/lib/kubelet/pods/970bcbff-b089-45a2-b77a-7959c1f89169/volumes"
Jul 16 23:09:01.513665 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:01.513637    2573 scope.go:117] "RemoveContainer" containerID="650b8764121a10b6bafde136325cf80ac48ac39c6f6abccf7348235f74572d7e"
Jul 16 23:09:02.180341 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:02.180316    2573 generic.go:356] "Generic (PLEG): container finished" podID="ad391f4f-baed-4d47-a247-fd9c6d8b76e0" containerID="d8e5f4065e4e3af32145471916987f92e18760804ed82cbcc55bbf4c980782d1" exitCode=6
Jul 16 23:09:02.180468 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:02.180398    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-4tzbz" event={"ID":"ad391f4f-baed-4d47-a247-fd9c6d8b76e0","Type":"ContainerDied","Data":"d8e5f4065e4e3af32145471916987f92e18760804ed82cbcc55bbf4c980782d1"}
Jul 16 23:09:02.180468 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:02.180463    2573 scope.go:117] "RemoveContainer" containerID="650b8764121a10b6bafde136325cf80ac48ac39c6f6abccf7348235f74572d7e"
Jul 16 23:09:02.180731 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:02.180712    2573 scope.go:117] "RemoveContainer" containerID="d8e5f4065e4e3af32145471916987f92e18760804ed82cbcc55bbf4c980782d1"
Jul 16 23:09:02.180915 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:09:02.180894    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-4tzbz_odh-ai-gateway-infra(ad391f4f-baed-4d47-a247-fd9c6d8b76e0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-4tzbz" podUID="ad391f4f-baed-4d47-a247-fd9c6d8b76e0"
Jul 16 23:09:03.207430 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:03.207392    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-4tzbz"]
Jul 16 23:09:03.336764 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:03.336743    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-4tzbz"
Jul 16 23:09:03.453149 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:03.453121    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad391f4f-baed-4d47-a247-fd9c6d8b76e0-maas-api-service-ca\") pod \"ad391f4f-baed-4d47-a247-fd9c6d8b76e0\" (UID: \"ad391f4f-baed-4d47-a247-fd9c6d8b76e0\") "
Jul 16 23:09:03.453506 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:03.453479    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad391f4f-baed-4d47-a247-fd9c6d8b76e0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ad391f4f-baed-4d47-a247-fd9c6d8b76e0" (UID: "ad391f4f-baed-4d47-a247-fd9c6d8b76e0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:09:03.554420 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:03.554350    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad391f4f-baed-4d47-a247-fd9c6d8b76e0-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:09:04.194483 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:04.194447    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-4tzbz"
Jul 16 23:09:04.194483 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:04.194476    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-4tzbz" event={"ID":"ad391f4f-baed-4d47-a247-fd9c6d8b76e0","Type":"ContainerDied","Data":"944e2b22ee220c9e98a8461be658afebfc7d0177c211b8b6701de74fb732c666"}
Jul 16 23:09:04.194711 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:04.194550    2573 scope.go:117] "RemoveContainer" containerID="d8e5f4065e4e3af32145471916987f92e18760804ed82cbcc55bbf4c980782d1"
Jul 16 23:09:04.217586 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:04.217557    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-4tzbz"]
Jul 16 23:09:04.229889 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:04.227987    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-4tzbz"]
Jul 16 23:09:04.518835 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:04.518766    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad391f4f-baed-4d47-a247-fd9c6d8b76e0" path="/var/lib/kubelet/pods/ad391f4f-baed-4d47-a247-fd9c6d8b76e0/volumes"
Jul 16 23:09:16.575149 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:16.575108    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:09:17.610524 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:17.610487    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:09:21.279500 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:21.279453    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:09:21.349572 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:21.349540    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:09:21.357250 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:21.357214    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-pnjph"]
Jul 16 23:09:21.357505 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:21.357492    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f47c6ce4-7ca3-47d7-bdd2-54ce635c6487" containerName="revoke-keys"
Jul 16 23:09:21.357548 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:21.357506    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f47c6ce4-7ca3-47d7-bdd2-54ce635c6487" containerName="revoke-keys"
Jul 16 23:09:21.357548 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:21.357518    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="970bcbff-b089-45a2-b77a-7959c1f89169" containerName="revoke-keys"
Jul 16 23:09:21.357548 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:21.357524    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="970bcbff-b089-45a2-b77a-7959c1f89169" containerName="revoke-keys"
Jul 16 23:09:21.357548 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:21.357535    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="970bcbff-b089-45a2-b77a-7959c1f89169" containerName="revoke-keys"
Jul 16 23:09:21.357548 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:21.357541    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="970bcbff-b089-45a2-b77a-7959c1f89169" containerName="revoke-keys"
Jul 16 23:09:21.357548 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:21.357549    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad391f4f-baed-4d47-a247-fd9c6d8b76e0" containerName="revoke-keys"
Jul 16 23:09:21.357727 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:21.357556    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad391f4f-baed-4d47-a247-fd9c6d8b76e0" containerName="revoke-keys"
Jul 16 23:09:21.357727 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:21.357568    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad391f4f-baed-4d47-a247-fd9c6d8b76e0" containerName="revoke-keys"
Jul 16 23:09:21.357727 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:21.357573    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad391f4f-baed-4d47-a247-fd9c6d8b76e0" containerName="revoke-keys"
Jul 16 23:09:21.357727 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:21.357579    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad391f4f-baed-4d47-a247-fd9c6d8b76e0" containerName="revoke-keys"
Jul 16 23:09:21.357727 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:21.357584    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad391f4f-baed-4d47-a247-fd9c6d8b76e0" containerName="revoke-keys"
Jul 16 23:09:21.357727 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:21.357632    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad391f4f-baed-4d47-a247-fd9c6d8b76e0" containerName="revoke-keys"
Jul 16 23:09:21.357727 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:21.357639    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad391f4f-baed-4d47-a247-fd9c6d8b76e0" containerName="revoke-keys"
Jul 16 23:09:21.357727 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:21.357644    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="970bcbff-b089-45a2-b77a-7959c1f89169" containerName="revoke-keys"
Jul 16 23:09:21.357727 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:21.357651    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="970bcbff-b089-45a2-b77a-7959c1f89169" containerName="revoke-keys"
Jul 16 23:09:21.357727 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:21.357658    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="970bcbff-b089-45a2-b77a-7959c1f89169" containerName="revoke-keys"
Jul 16 23:09:21.360389 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:21.360374    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-pnjph"
Jul 16 23:09:21.362661 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:21.362641    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:09:21.362851 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:21.362833    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 23:09:21.368100 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:21.368077    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-pnjph"]
Jul 16 23:09:21.475750 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:21.475719    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/946acc34-04b1-49c4-9a46-40ce2f660516-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-pnjph\" (UID: \"946acc34-04b1-49c4-9a46-40ce2f660516\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-pnjph"
Jul 16 23:09:21.576358 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:21.576330    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/946acc34-04b1-49c4-9a46-40ce2f660516-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-pnjph\" (UID: \"946acc34-04b1-49c4-9a46-40ce2f660516\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-pnjph"
Jul 16 23:09:21.576878 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:21.576859    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/946acc34-04b1-49c4-9a46-40ce2f660516-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-pnjph\" (UID: \"946acc34-04b1-49c4-9a46-40ce2f660516\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-pnjph"
Jul 16 23:09:21.670994 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:21.670971    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-pnjph"
Jul 16 23:09:21.795007 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:21.794985    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-pnjph"]
Jul 16 23:09:21.797104 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:09:21.797075    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod946acc34_04b1_49c4_9a46_40ce2f660516.slice/crio-b67323c123155831c40ce0f6c471bf74a70353923e2f041065f0912290910780 WatchSource:0}: Error finding container b67323c123155831c40ce0f6c471bf74a70353923e2f041065f0912290910780: Status 404 returned error can't find the container with id b67323c123155831c40ce0f6c471bf74a70353923e2f041065f0912290910780
Jul 16 23:09:22.286801 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:22.286722    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-pnjph" event={"ID":"946acc34-04b1-49c4-9a46-40ce2f660516","Type":"ContainerStarted","Data":"449d1cb3c0bc611e3ea2d48feb82268e0fc926ef3ae32d2020151d180cbed7d6"}
Jul 16 23:09:22.287111 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:22.286816    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-pnjph" event={"ID":"946acc34-04b1-49c4-9a46-40ce2f660516","Type":"ContainerStarted","Data":"b67323c123155831c40ce0f6c471bf74a70353923e2f041065f0912290910780"}
Jul 16 23:09:22.300854 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:22.300808    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-pnjph" podStartSLOduration=1.30079317 podStartE2EDuration="1.30079317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:09:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:09:22.299920683 +0000 UTC m=+2136.310183348" watchObservedRunningTime="2026-07-16 23:09:22.30079317 +0000 UTC m=+2136.311055837"
Jul 16 23:09:23.293214 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:23.293176    2573 generic.go:356] "Generic (PLEG): container finished" podID="946acc34-04b1-49c4-9a46-40ce2f660516" containerID="449d1cb3c0bc611e3ea2d48feb82268e0fc926ef3ae32d2020151d180cbed7d6" exitCode=6
Jul 16 23:09:23.293641 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:23.293275    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-pnjph" event={"ID":"946acc34-04b1-49c4-9a46-40ce2f660516","Type":"ContainerDied","Data":"449d1cb3c0bc611e3ea2d48feb82268e0fc926ef3ae32d2020151d180cbed7d6"}
Jul 16 23:09:23.293641 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:23.293489    2573 scope.go:117] "RemoveContainer" containerID="449d1cb3c0bc611e3ea2d48feb82268e0fc926ef3ae32d2020151d180cbed7d6"
Jul 16 23:09:24.299907 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:24.299827    2573 generic.go:356] "Generic (PLEG): container finished" podID="946acc34-04b1-49c4-9a46-40ce2f660516" containerID="b7c709cbdef141741bb12ec348bd11afa37bae5a5460f05912509aab8c33f6c6" exitCode=6
Jul 16 23:09:24.300277 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:24.299901    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-pnjph" event={"ID":"946acc34-04b1-49c4-9a46-40ce2f660516","Type":"ContainerDied","Data":"b7c709cbdef141741bb12ec348bd11afa37bae5a5460f05912509aab8c33f6c6"}
Jul 16 23:09:24.300277 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:24.299966    2573 scope.go:117] "RemoveContainer" containerID="449d1cb3c0bc611e3ea2d48feb82268e0fc926ef3ae32d2020151d180cbed7d6"
Jul 16 23:09:24.300277 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:24.300074    2573 scope.go:117] "RemoveContainer" containerID="b7c709cbdef141741bb12ec348bd11afa37bae5a5460f05912509aab8c33f6c6"
Jul 16 23:09:24.300277 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:09:24.300254    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-pnjph_odh-ai-gateway-infra(946acc34-04b1-49c4-9a46-40ce2f660516)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-pnjph" podUID="946acc34-04b1-49c4-9a46-40ce2f660516"
Jul 16 23:09:25.306435 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:25.306403    2573 scope.go:117] "RemoveContainer" containerID="b7c709cbdef141741bb12ec348bd11afa37bae5a5460f05912509aab8c33f6c6"
Jul 16 23:09:25.306899 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:09:25.306627    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-pnjph_odh-ai-gateway-infra(946acc34-04b1-49c4-9a46-40ce2f660516)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-pnjph" podUID="946acc34-04b1-49c4-9a46-40ce2f660516"
Jul 16 23:09:30.339898 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:30.339862    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-j4w5n"]
Jul 16 23:09:30.340467 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:30.340347    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="970bcbff-b089-45a2-b77a-7959c1f89169" containerName="revoke-keys"
Jul 16 23:09:30.340467 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:30.340366    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="970bcbff-b089-45a2-b77a-7959c1f89169" containerName="revoke-keys"
Jul 16 23:09:30.340467 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:30.340442    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad391f4f-baed-4d47-a247-fd9c6d8b76e0" containerName="revoke-keys"
Jul 16 23:09:30.344605 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:30.344584    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-j4w5n"
Jul 16 23:09:30.349088 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:30.349067    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-j4w5n"]
Jul 16 23:09:30.537932 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:30.537895    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a41aa9f-fc03-4cb1-82e2-aae7e3a58106-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-j4w5n\" (UID: \"9a41aa9f-fc03-4cb1-82e2-aae7e3a58106\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-j4w5n"
Jul 16 23:09:30.638250 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:30.638171    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a41aa9f-fc03-4cb1-82e2-aae7e3a58106-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-j4w5n\" (UID: \"9a41aa9f-fc03-4cb1-82e2-aae7e3a58106\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-j4w5n"
Jul 16 23:09:30.638812 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:30.638791    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a41aa9f-fc03-4cb1-82e2-aae7e3a58106-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-j4w5n\" (UID: \"9a41aa9f-fc03-4cb1-82e2-aae7e3a58106\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-j4w5n"
Jul 16 23:09:30.656229 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:30.656206    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-j4w5n"
Jul 16 23:09:30.780486 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:30.780460    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-j4w5n"]
Jul 16 23:09:30.783254 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:09:30.783205    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a41aa9f_fc03_4cb1_82e2_aae7e3a58106.slice/crio-ec79964862d3375f4ad3d8ceb1956a015b41c2b4c278c6a5ceba806605922baa WatchSource:0}: Error finding container ec79964862d3375f4ad3d8ceb1956a015b41c2b4c278c6a5ceba806605922baa: Status 404 returned error can't find the container with id ec79964862d3375f4ad3d8ceb1956a015b41c2b4c278c6a5ceba806605922baa
Jul 16 23:09:31.335788 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:31.335756    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-j4w5n" event={"ID":"9a41aa9f-fc03-4cb1-82e2-aae7e3a58106","Type":"ContainerStarted","Data":"e3ae1eb95268c91d940e6a1a49e5e41193ba91a0f7af4173d4d403b3f987f092"}
Jul 16 23:09:31.335925 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:31.335819    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-j4w5n" event={"ID":"9a41aa9f-fc03-4cb1-82e2-aae7e3a58106","Type":"ContainerStarted","Data":"ec79964862d3375f4ad3d8ceb1956a015b41c2b4c278c6a5ceba806605922baa"}
Jul 16 23:09:31.351193 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:31.351148    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-j4w5n" podStartSLOduration=1.351132872 podStartE2EDuration="1.351132872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:09:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:09:31.349935668 +0000 UTC m=+2145.360198334" watchObservedRunningTime="2026-07-16 23:09:31.351132872 +0000 UTC m=+2145.361395536"
Jul 16 23:09:32.341752 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:32.341712    2573 generic.go:356] "Generic (PLEG): container finished" podID="9a41aa9f-fc03-4cb1-82e2-aae7e3a58106" containerID="e3ae1eb95268c91d940e6a1a49e5e41193ba91a0f7af4173d4d403b3f987f092" exitCode=7
Jul 16 23:09:32.341923 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:32.341809    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-j4w5n" event={"ID":"9a41aa9f-fc03-4cb1-82e2-aae7e3a58106","Type":"ContainerDied","Data":"e3ae1eb95268c91d940e6a1a49e5e41193ba91a0f7af4173d4d403b3f987f092"}
Jul 16 23:09:32.341970 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:32.341953    2573 scope.go:117] "RemoveContainer" containerID="e3ae1eb95268c91d940e6a1a49e5e41193ba91a0f7af4173d4d403b3f987f092"
Jul 16 23:09:33.347882 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:33.347845    2573 generic.go:356] "Generic (PLEG): container finished" podID="9a41aa9f-fc03-4cb1-82e2-aae7e3a58106" containerID="49975e25e66754e080b36ac9184a58eeb6bf0738d166b074d45557ce5d26e68a" exitCode=7
Jul 16 23:09:33.348336 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:33.347936    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-j4w5n" event={"ID":"9a41aa9f-fc03-4cb1-82e2-aae7e3a58106","Type":"ContainerDied","Data":"49975e25e66754e080b36ac9184a58eeb6bf0738d166b074d45557ce5d26e68a"}
Jul 16 23:09:33.348336 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:33.348016    2573 scope.go:117] "RemoveContainer" containerID="e3ae1eb95268c91d940e6a1a49e5e41193ba91a0f7af4173d4d403b3f987f092"
Jul 16 23:09:33.348336 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:33.348091    2573 scope.go:117] "RemoveContainer" containerID="49975e25e66754e080b36ac9184a58eeb6bf0738d166b074d45557ce5d26e68a"
Jul 16 23:09:33.348336 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:09:33.348275    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-j4w5n_odh-ai-gateway-infra(9a41aa9f-fc03-4cb1-82e2-aae7e3a58106)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-j4w5n" podUID="9a41aa9f-fc03-4cb1-82e2-aae7e3a58106"
Jul 16 23:09:34.354134 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:34.354107    2573 scope.go:117] "RemoveContainer" containerID="49975e25e66754e080b36ac9184a58eeb6bf0738d166b074d45557ce5d26e68a"
Jul 16 23:09:34.354539 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:09:34.354288    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-j4w5n_odh-ai-gateway-infra(9a41aa9f-fc03-4cb1-82e2-aae7e3a58106)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-j4w5n" podUID="9a41aa9f-fc03-4cb1-82e2-aae7e3a58106"
Jul 16 23:09:37.514122 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:37.514090    2573 scope.go:117] "RemoveContainer" containerID="b7c709cbdef141741bb12ec348bd11afa37bae5a5460f05912509aab8c33f6c6"
Jul 16 23:09:38.376014 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:38.375927    2573 generic.go:356] "Generic (PLEG): container finished" podID="946acc34-04b1-49c4-9a46-40ce2f660516" containerID="d31e49399b9b30306bbcfa7a434f5ed70ea3bc08842f25d5c33abc899284e324" exitCode=6
Jul 16 23:09:38.376171 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:38.376009    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-pnjph" event={"ID":"946acc34-04b1-49c4-9a46-40ce2f660516","Type":"ContainerDied","Data":"d31e49399b9b30306bbcfa7a434f5ed70ea3bc08842f25d5c33abc899284e324"}
Jul 16 23:09:38.376171 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:38.376085    2573 scope.go:117] "RemoveContainer" containerID="b7c709cbdef141741bb12ec348bd11afa37bae5a5460f05912509aab8c33f6c6"
Jul 16 23:09:38.376482 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:38.376461    2573 scope.go:117] "RemoveContainer" containerID="d31e49399b9b30306bbcfa7a434f5ed70ea3bc08842f25d5c33abc899284e324"
Jul 16 23:09:38.376730 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:09:38.376697    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-pnjph_odh-ai-gateway-infra(946acc34-04b1-49c4-9a46-40ce2f660516)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-pnjph" podUID="946acc34-04b1-49c4-9a46-40ce2f660516"
Jul 16 23:09:39.401395 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:39.401358    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-pnjph"]
Jul 16 23:09:39.535682 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:39.535662    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-pnjph"
Jul 16 23:09:39.603476 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:39.603450    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/946acc34-04b1-49c4-9a46-40ce2f660516-maas-api-service-ca\") pod \"946acc34-04b1-49c4-9a46-40ce2f660516\" (UID: \"946acc34-04b1-49c4-9a46-40ce2f660516\") "
Jul 16 23:09:39.603850 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:39.603822    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/946acc34-04b1-49c4-9a46-40ce2f660516-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "946acc34-04b1-49c4-9a46-40ce2f660516" (UID: "946acc34-04b1-49c4-9a46-40ce2f660516"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:09:39.704888 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:39.704830    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/946acc34-04b1-49c4-9a46-40ce2f660516-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:09:40.388892 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:40.388869    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-pnjph"
Jul 16 23:09:40.389048 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:40.388902    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-pnjph" event={"ID":"946acc34-04b1-49c4-9a46-40ce2f660516","Type":"ContainerDied","Data":"b67323c123155831c40ce0f6c471bf74a70353923e2f041065f0912290910780"}
Jul 16 23:09:40.389048 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:40.388965    2573 scope.go:117] "RemoveContainer" containerID="d31e49399b9b30306bbcfa7a434f5ed70ea3bc08842f25d5c33abc899284e324"
Jul 16 23:09:40.411751 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:40.411724    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-pnjph"]
Jul 16 23:09:40.417728 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:40.417705    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-pnjph"]
Jul 16 23:09:40.519443 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:40.519416    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="946acc34-04b1-49c4-9a46-40ce2f660516" path="/var/lib/kubelet/pods/946acc34-04b1-49c4-9a46-40ce2f660516/volumes"
Jul 16 23:09:42.557787 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:42.557752    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-flj26"]
Jul 16 23:09:42.558170 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:42.558037    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="946acc34-04b1-49c4-9a46-40ce2f660516" containerName="revoke-keys"
Jul 16 23:09:42.558170 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:42.558048    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="946acc34-04b1-49c4-9a46-40ce2f660516" containerName="revoke-keys"
Jul 16 23:09:42.558170 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:42.558062    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="946acc34-04b1-49c4-9a46-40ce2f660516" containerName="revoke-keys"
Jul 16 23:09:42.558170 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:42.558068    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="946acc34-04b1-49c4-9a46-40ce2f660516" containerName="revoke-keys"
Jul 16 23:09:42.558170 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:42.558123    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="946acc34-04b1-49c4-9a46-40ce2f660516" containerName="revoke-keys"
Jul 16 23:09:42.558170 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:42.558129    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="946acc34-04b1-49c4-9a46-40ce2f660516" containerName="revoke-keys"
Jul 16 23:09:42.562496 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:42.562478    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-flj26"
Jul 16 23:09:42.568088 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:42.568060    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-flj26"]
Jul 16 23:09:42.626393 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:42.626366    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c13c6db0-7bd2-4a90-9eb0-6b3555414a34-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-flj26\" (UID: \"c13c6db0-7bd2-4a90-9eb0-6b3555414a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-flj26"
Jul 16 23:09:42.727581 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:42.727553    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c13c6db0-7bd2-4a90-9eb0-6b3555414a34-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-flj26\" (UID: \"c13c6db0-7bd2-4a90-9eb0-6b3555414a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-flj26"
Jul 16 23:09:42.728070 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:42.728050    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c13c6db0-7bd2-4a90-9eb0-6b3555414a34-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-flj26\" (UID: \"c13c6db0-7bd2-4a90-9eb0-6b3555414a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-flj26"
Jul 16 23:09:42.874564 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:42.874530    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-flj26"
Jul 16 23:09:43.002593 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:43.002570    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-flj26"]
Jul 16 23:09:43.004141 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:09:43.004116    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13c6db0_7bd2_4a90_9eb0_6b3555414a34.slice/crio-9dfe5ddf00e5a92340b3cbd121b54a3f3161c0b38bf0b0b94e17d8f687a87579 WatchSource:0}: Error finding container 9dfe5ddf00e5a92340b3cbd121b54a3f3161c0b38bf0b0b94e17d8f687a87579: Status 404 returned error can't find the container with id 9dfe5ddf00e5a92340b3cbd121b54a3f3161c0b38bf0b0b94e17d8f687a87579
Jul 16 23:09:43.406382 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:43.406345    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-flj26" event={"ID":"c13c6db0-7bd2-4a90-9eb0-6b3555414a34","Type":"ContainerStarted","Data":"49ddf30b6e815061d32556d9cd63e667bb093e46d762c90147a244783b81b246"}
Jul 16 23:09:43.406553 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:43.406402    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-flj26" event={"ID":"c13c6db0-7bd2-4a90-9eb0-6b3555414a34","Type":"ContainerStarted","Data":"9dfe5ddf00e5a92340b3cbd121b54a3f3161c0b38bf0b0b94e17d8f687a87579"}
Jul 16 23:09:43.421532 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:43.421487    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-flj26" podStartSLOduration=1.421472077 podStartE2EDuration="1.421472077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:09:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:09:43.4192042 +0000 UTC m=+2157.429466881" watchObservedRunningTime="2026-07-16 23:09:43.421472077 +0000 UTC m=+2157.431734741"
Jul 16 23:09:44.412620 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:44.412586    2573 generic.go:356] "Generic (PLEG): container finished" podID="c13c6db0-7bd2-4a90-9eb0-6b3555414a34" containerID="49ddf30b6e815061d32556d9cd63e667bb093e46d762c90147a244783b81b246" exitCode=6
Jul 16 23:09:44.413008 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:44.412676    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-flj26" event={"ID":"c13c6db0-7bd2-4a90-9eb0-6b3555414a34","Type":"ContainerDied","Data":"49ddf30b6e815061d32556d9cd63e667bb093e46d762c90147a244783b81b246"}
Jul 16 23:09:44.413008 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:44.412811    2573 scope.go:117] "RemoveContainer" containerID="49ddf30b6e815061d32556d9cd63e667bb093e46d762c90147a244783b81b246"
Jul 16 23:09:45.420732 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:45.420702    2573 generic.go:356] "Generic (PLEG): container finished" podID="c13c6db0-7bd2-4a90-9eb0-6b3555414a34" containerID="75a94de797f2aaaa88da3c92179d6e26b3e1f3280ace41386b1a3d1b190bbccd" exitCode=6
Jul 16 23:09:45.421087 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:45.420786    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-flj26" event={"ID":"c13c6db0-7bd2-4a90-9eb0-6b3555414a34","Type":"ContainerDied","Data":"75a94de797f2aaaa88da3c92179d6e26b3e1f3280ace41386b1a3d1b190bbccd"}
Jul 16 23:09:45.421087 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:45.420849    2573 scope.go:117] "RemoveContainer" containerID="49ddf30b6e815061d32556d9cd63e667bb093e46d762c90147a244783b81b246"
Jul 16 23:09:45.421087 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:45.421058    2573 scope.go:117] "RemoveContainer" containerID="75a94de797f2aaaa88da3c92179d6e26b3e1f3280ace41386b1a3d1b190bbccd"
Jul 16 23:09:45.421292 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:09:45.421273    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-flj26_odh-ai-gateway-infra(c13c6db0-7bd2-4a90-9eb0-6b3555414a34)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-flj26" podUID="c13c6db0-7bd2-4a90-9eb0-6b3555414a34"
Jul 16 23:09:46.428096 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:46.428068    2573 scope.go:117] "RemoveContainer" containerID="75a94de797f2aaaa88da3c92179d6e26b3e1f3280ace41386b1a3d1b190bbccd"
Jul 16 23:09:46.428480 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:09:46.428228    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-flj26_odh-ai-gateway-infra(c13c6db0-7bd2-4a90-9eb0-6b3555414a34)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-flj26" podUID="c13c6db0-7bd2-4a90-9eb0-6b3555414a34"
Jul 16 23:09:48.514290 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:48.514220    2573 scope.go:117] "RemoveContainer" containerID="49975e25e66754e080b36ac9184a58eeb6bf0738d166b074d45557ce5d26e68a"
Jul 16 23:09:49.445194 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:49.445107    2573 generic.go:356] "Generic (PLEG): container finished" podID="9a41aa9f-fc03-4cb1-82e2-aae7e3a58106" containerID="9063a86ff997bd5815233c893ad46655c5189f6cfc106e5a6fccf07847d4c45f" exitCode=7
Jul 16 23:09:49.445369 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:49.445191    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-j4w5n" event={"ID":"9a41aa9f-fc03-4cb1-82e2-aae7e3a58106","Type":"ContainerDied","Data":"9063a86ff997bd5815233c893ad46655c5189f6cfc106e5a6fccf07847d4c45f"}
Jul 16 23:09:49.445369 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:49.445287    2573 scope.go:117] "RemoveContainer" containerID="49975e25e66754e080b36ac9184a58eeb6bf0738d166b074d45557ce5d26e68a"
Jul 16 23:09:49.445648 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:49.445620    2573 scope.go:117] "RemoveContainer" containerID="9063a86ff997bd5815233c893ad46655c5189f6cfc106e5a6fccf07847d4c45f"
Jul 16 23:09:49.445847 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:09:49.445825    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-j4w5n_odh-ai-gateway-infra(9a41aa9f-fc03-4cb1-82e2-aae7e3a58106)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-j4w5n" podUID="9a41aa9f-fc03-4cb1-82e2-aae7e3a58106"
Jul 16 23:09:50.475747 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:50.475707    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-j4w5n"]
Jul 16 23:09:50.612421 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:50.612386    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-j4w5n"
Jul 16 23:09:50.687160 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:50.687130    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a41aa9f-fc03-4cb1-82e2-aae7e3a58106-maas-api-service-ca\") pod \"9a41aa9f-fc03-4cb1-82e2-aae7e3a58106\" (UID: \"9a41aa9f-fc03-4cb1-82e2-aae7e3a58106\") "
Jul 16 23:09:50.687462 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:50.687440    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a41aa9f-fc03-4cb1-82e2-aae7e3a58106-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9a41aa9f-fc03-4cb1-82e2-aae7e3a58106" (UID: "9a41aa9f-fc03-4cb1-82e2-aae7e3a58106"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:09:50.788184 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:50.788119    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a41aa9f-fc03-4cb1-82e2-aae7e3a58106-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:09:51.460497 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:51.460464    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-j4w5n"
Jul 16 23:09:51.460663 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:51.460468    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-j4w5n" event={"ID":"9a41aa9f-fc03-4cb1-82e2-aae7e3a58106","Type":"ContainerDied","Data":"ec79964862d3375f4ad3d8ceb1956a015b41c2b4c278c6a5ceba806605922baa"}
Jul 16 23:09:51.460663 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:51.460591    2573 scope.go:117] "RemoveContainer" containerID="9063a86ff997bd5815233c893ad46655c5189f6cfc106e5a6fccf07847d4c45f"
Jul 16 23:09:51.480514 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:51.480491    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-j4w5n"]
Jul 16 23:09:51.483818 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:51.483793    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-j4w5n"]
Jul 16 23:09:52.519024 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:52.518984    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a41aa9f-fc03-4cb1-82e2-aae7e3a58106" path="/var/lib/kubelet/pods/9a41aa9f-fc03-4cb1-82e2-aae7e3a58106/volumes"
Jul 16 23:09:58.513589 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:58.513555    2573 scope.go:117] "RemoveContainer" containerID="75a94de797f2aaaa88da3c92179d6e26b3e1f3280ace41386b1a3d1b190bbccd"
Jul 16 23:09:59.506449 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:59.506349    2573 generic.go:356] "Generic (PLEG): container finished" podID="c13c6db0-7bd2-4a90-9eb0-6b3555414a34" containerID="75313f2218ee7f53f95656e9be1dd7b0f5cdb1ee04fa8b0e9adbb7effa7f24d1" exitCode=6
Jul 16 23:09:59.506449 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:59.506422    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-flj26" event={"ID":"c13c6db0-7bd2-4a90-9eb0-6b3555414a34","Type":"ContainerDied","Data":"75313f2218ee7f53f95656e9be1dd7b0f5cdb1ee04fa8b0e9adbb7effa7f24d1"}
Jul 16 23:09:59.506636 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:59.506500    2573 scope.go:117] "RemoveContainer" containerID="75a94de797f2aaaa88da3c92179d6e26b3e1f3280ace41386b1a3d1b190bbccd"
Jul 16 23:09:59.506853 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:09:59.506822    2573 scope.go:117] "RemoveContainer" containerID="75313f2218ee7f53f95656e9be1dd7b0f5cdb1ee04fa8b0e9adbb7effa7f24d1"
Jul 16 23:09:59.507034 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:09:59.507014    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-flj26_odh-ai-gateway-infra(c13c6db0-7bd2-4a90-9eb0-6b3555414a34)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-flj26" podUID="c13c6db0-7bd2-4a90-9eb0-6b3555414a34"
Jul 16 23:10:00.224289 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:00.224246    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:10:00.530668 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:00.530584    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-flj26"]
Jul 16 23:10:00.679479 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:00.679456    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-flj26"
Jul 16 23:10:00.761312 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:00.761286    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c13c6db0-7bd2-4a90-9eb0-6b3555414a34-maas-api-service-ca\") pod \"c13c6db0-7bd2-4a90-9eb0-6b3555414a34\" (UID: \"c13c6db0-7bd2-4a90-9eb0-6b3555414a34\") "
Jul 16 23:10:00.761604 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:00.761583    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c13c6db0-7bd2-4a90-9eb0-6b3555414a34-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c13c6db0-7bd2-4a90-9eb0-6b3555414a34" (UID: "c13c6db0-7bd2-4a90-9eb0-6b3555414a34"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:10:00.862730 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:00.862707    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c13c6db0-7bd2-4a90-9eb0-6b3555414a34-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:10:01.518653 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:01.518607    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-flj26" event={"ID":"c13c6db0-7bd2-4a90-9eb0-6b3555414a34","Type":"ContainerDied","Data":"9dfe5ddf00e5a92340b3cbd121b54a3f3161c0b38bf0b0b94e17d8f687a87579"}
Jul 16 23:10:01.518653 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:01.518625    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-flj26"
Jul 16 23:10:01.519542 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:01.518665    2573 scope.go:117] "RemoveContainer" containerID="75313f2218ee7f53f95656e9be1dd7b0f5cdb1ee04fa8b0e9adbb7effa7f24d1"
Jul 16 23:10:01.539975 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:01.539949    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-flj26"]
Jul 16 23:10:01.547784 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:01.547764    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-flj26"]
Jul 16 23:10:02.519316 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:02.519272    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c13c6db0-7bd2-4a90-9eb0-6b3555414a34" path="/var/lib/kubelet/pods/c13c6db0-7bd2-4a90-9eb0-6b3555414a34/volumes"
Jul 16 23:10:11.395627 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:11.395579    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-9d2df"]
Jul 16 23:10:11.396113 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:11.395899    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a41aa9f-fc03-4cb1-82e2-aae7e3a58106" containerName="revoke-keys"
Jul 16 23:10:11.396113 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:11.395912    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a41aa9f-fc03-4cb1-82e2-aae7e3a58106" containerName="revoke-keys"
Jul 16 23:10:11.396113 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:11.395921    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c13c6db0-7bd2-4a90-9eb0-6b3555414a34" containerName="revoke-keys"
Jul 16 23:10:11.396113 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:11.395926    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c13c6db0-7bd2-4a90-9eb0-6b3555414a34" containerName="revoke-keys"
Jul 16 23:10:11.396113 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:11.395934    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c13c6db0-7bd2-4a90-9eb0-6b3555414a34" containerName="revoke-keys"
Jul 16 23:10:11.396113 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:11.395939    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c13c6db0-7bd2-4a90-9eb0-6b3555414a34" containerName="revoke-keys"
Jul 16 23:10:11.396113 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:11.395950    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="946acc34-04b1-49c4-9a46-40ce2f660516" containerName="revoke-keys"
Jul 16 23:10:11.396113 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:11.395958    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="946acc34-04b1-49c4-9a46-40ce2f660516" containerName="revoke-keys"
Jul 16 23:10:11.396113 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:11.395970    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a41aa9f-fc03-4cb1-82e2-aae7e3a58106" containerName="revoke-keys"
Jul 16 23:10:11.396113 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:11.395976    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a41aa9f-fc03-4cb1-82e2-aae7e3a58106" containerName="revoke-keys"
Jul 16 23:10:11.396113 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:11.395984    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a41aa9f-fc03-4cb1-82e2-aae7e3a58106" containerName="revoke-keys"
Jul 16 23:10:11.396113 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:11.395988    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a41aa9f-fc03-4cb1-82e2-aae7e3a58106" containerName="revoke-keys"
Jul 16 23:10:11.396113 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:11.396039    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c13c6db0-7bd2-4a90-9eb0-6b3555414a34" containerName="revoke-keys"
Jul 16 23:10:11.396113 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:11.396048    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="946acc34-04b1-49c4-9a46-40ce2f660516" containerName="revoke-keys"
Jul 16 23:10:11.396113 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:11.396055    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a41aa9f-fc03-4cb1-82e2-aae7e3a58106" containerName="revoke-keys"
Jul 16 23:10:11.396113 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:11.396061    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c13c6db0-7bd2-4a90-9eb0-6b3555414a34" containerName="revoke-keys"
Jul 16 23:10:11.396113 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:11.396066    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a41aa9f-fc03-4cb1-82e2-aae7e3a58106" containerName="revoke-keys"
Jul 16 23:10:11.400468 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:11.400444    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-9d2df"
Jul 16 23:10:11.402765 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:11.402738    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:10:11.402904 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:11.402794    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 23:10:11.406554 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:11.406520    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-9d2df"]
Jul 16 23:10:11.536162 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:11.536136    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dbe90cf-81bd-4b55-830f-a123420e9f55-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-9d2df\" (UID: \"1dbe90cf-81bd-4b55-830f-a123420e9f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-9d2df"
Jul 16 23:10:11.636450 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:11.636424    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dbe90cf-81bd-4b55-830f-a123420e9f55-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-9d2df\" (UID: \"1dbe90cf-81bd-4b55-830f-a123420e9f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-9d2df"
Jul 16 23:10:11.637003 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:11.636985    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dbe90cf-81bd-4b55-830f-a123420e9f55-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-9d2df\" (UID: \"1dbe90cf-81bd-4b55-830f-a123420e9f55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-9d2df"
Jul 16 23:10:11.712720 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:11.712660    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-9d2df"
Jul 16 23:10:11.837697 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:11.837567    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-9d2df"]
Jul 16 23:10:11.839822 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:10:11.839793    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dbe90cf_81bd_4b55_830f_a123420e9f55.slice/crio-ec097aad5a81122be30d621c64037b77c6c6eb29cb44c02b4ce1a4d2b8e3b7dc WatchSource:0}: Error finding container ec097aad5a81122be30d621c64037b77c6c6eb29cb44c02b4ce1a4d2b8e3b7dc: Status 404 returned error can't find the container with id ec097aad5a81122be30d621c64037b77c6c6eb29cb44c02b4ce1a4d2b8e3b7dc
Jul 16 23:10:12.570145 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:12.570115    2573 generic.go:356] "Generic (PLEG): container finished" podID="1dbe90cf-81bd-4b55-830f-a123420e9f55" containerID="fe3bdf7209a932bc8b51b6fc4a0fe9c3f37fa95398eb77558429869c483c56ce" exitCode=6
Jul 16 23:10:12.570481 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:12.570198    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-9d2df" event={"ID":"1dbe90cf-81bd-4b55-830f-a123420e9f55","Type":"ContainerDied","Data":"fe3bdf7209a932bc8b51b6fc4a0fe9c3f37fa95398eb77558429869c483c56ce"}
Jul 16 23:10:12.570481 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:12.570265    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-9d2df" event={"ID":"1dbe90cf-81bd-4b55-830f-a123420e9f55","Type":"ContainerStarted","Data":"ec097aad5a81122be30d621c64037b77c6c6eb29cb44c02b4ce1a4d2b8e3b7dc"}
Jul 16 23:10:12.570481 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:12.570417    2573 scope.go:117] "RemoveContainer" containerID="fe3bdf7209a932bc8b51b6fc4a0fe9c3f37fa95398eb77558429869c483c56ce"
Jul 16 23:10:13.577975 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:13.577936    2573 generic.go:356] "Generic (PLEG): container finished" podID="1dbe90cf-81bd-4b55-830f-a123420e9f55" containerID="bff9500a23fae948307a98791cc3d5ce27173f1244c597b9e6651792d0e21bd4" exitCode=6
Jul 16 23:10:13.578463 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:13.578020    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-9d2df" event={"ID":"1dbe90cf-81bd-4b55-830f-a123420e9f55","Type":"ContainerDied","Data":"bff9500a23fae948307a98791cc3d5ce27173f1244c597b9e6651792d0e21bd4"}
Jul 16 23:10:13.578463 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:13.578093    2573 scope.go:117] "RemoveContainer" containerID="fe3bdf7209a932bc8b51b6fc4a0fe9c3f37fa95398eb77558429869c483c56ce"
Jul 16 23:10:13.578463 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:13.578197    2573 scope.go:117] "RemoveContainer" containerID="bff9500a23fae948307a98791cc3d5ce27173f1244c597b9e6651792d0e21bd4"
Jul 16 23:10:13.578463 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:10:13.578396    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-9d2df_odh-ai-gateway-infra(1dbe90cf-81bd-4b55-830f-a123420e9f55)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-9d2df" podUID="1dbe90cf-81bd-4b55-830f-a123420e9f55"
Jul 16 23:10:14.585281 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:14.585230    2573 scope.go:117] "RemoveContainer" containerID="bff9500a23fae948307a98791cc3d5ce27173f1244c597b9e6651792d0e21bd4"
Jul 16 23:10:14.585658 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:10:14.585418    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-9d2df_odh-ai-gateway-infra(1dbe90cf-81bd-4b55-830f-a123420e9f55)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-9d2df" podUID="1dbe90cf-81bd-4b55-830f-a123420e9f55"
Jul 16 23:10:28.513935 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:28.513893    2573 scope.go:117] "RemoveContainer" containerID="bff9500a23fae948307a98791cc3d5ce27173f1244c597b9e6651792d0e21bd4"
Jul 16 23:10:29.656343 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:29.656308    2573 generic.go:356] "Generic (PLEG): container finished" podID="1dbe90cf-81bd-4b55-830f-a123420e9f55" containerID="47619cd6b7e3cff2bee38607019a6be0ba707c39e149c89893b9e7ed54b9d19c" exitCode=6
Jul 16 23:10:29.656747 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:29.656364    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-9d2df" event={"ID":"1dbe90cf-81bd-4b55-830f-a123420e9f55","Type":"ContainerDied","Data":"47619cd6b7e3cff2bee38607019a6be0ba707c39e149c89893b9e7ed54b9d19c"}
Jul 16 23:10:29.656747 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:29.656422    2573 scope.go:117] "RemoveContainer" containerID="bff9500a23fae948307a98791cc3d5ce27173f1244c597b9e6651792d0e21bd4"
Jul 16 23:10:29.656747 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:29.656700    2573 scope.go:117] "RemoveContainer" containerID="47619cd6b7e3cff2bee38607019a6be0ba707c39e149c89893b9e7ed54b9d19c"
Jul 16 23:10:29.656909 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:10:29.656888    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-9d2df_odh-ai-gateway-infra(1dbe90cf-81bd-4b55-830f-a123420e9f55)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-9d2df" podUID="1dbe90cf-81bd-4b55-830f-a123420e9f55"
Jul 16 23:10:30.385777 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:30.385739    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-twv4t"]
Jul 16 23:10:30.386141 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:30.386121    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c13c6db0-7bd2-4a90-9eb0-6b3555414a34" containerName="revoke-keys"
Jul 16 23:10:30.386261 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:30.386143    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c13c6db0-7bd2-4a90-9eb0-6b3555414a34" containerName="revoke-keys"
Jul 16 23:10:30.386261 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:30.386218    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c13c6db0-7bd2-4a90-9eb0-6b3555414a34" containerName="revoke-keys"
Jul 16 23:10:30.386261 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:30.386228    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a41aa9f-fc03-4cb1-82e2-aae7e3a58106" containerName="revoke-keys"
Jul 16 23:10:30.390369 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:30.390349    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-twv4t"
Jul 16 23:10:30.396059 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:30.396030    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-twv4t"]
Jul 16 23:10:30.467932 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:30.467907    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d65c64e-4d3a-4a36-b244-b2ba156b6e14-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-twv4t\" (UID: \"4d65c64e-4d3a-4a36-b244-b2ba156b6e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-twv4t"
Jul 16 23:10:30.569029 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:30.569003    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d65c64e-4d3a-4a36-b244-b2ba156b6e14-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-twv4t\" (UID: \"4d65c64e-4d3a-4a36-b244-b2ba156b6e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-twv4t"
Jul 16 23:10:30.569563 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:30.569547    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d65c64e-4d3a-4a36-b244-b2ba156b6e14-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-twv4t\" (UID: \"4d65c64e-4d3a-4a36-b244-b2ba156b6e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-twv4t"
Jul 16 23:10:30.680601 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:30.680531    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-9d2df"]
Jul 16 23:10:30.701403 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:30.701384    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-twv4t"
Jul 16 23:10:30.827989 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:30.827965    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-9d2df"
Jul 16 23:10:30.831516 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:30.831497    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-twv4t"]
Jul 16 23:10:30.833122 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:10:30.833093    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d65c64e_4d3a_4a36_b244_b2ba156b6e14.slice/crio-17b2e8bdf06b10ec19b7fd9ca5770d09cd12ef35f82c49f8eb9b224d4a77c4d6 WatchSource:0}: Error finding container 17b2e8bdf06b10ec19b7fd9ca5770d09cd12ef35f82c49f8eb9b224d4a77c4d6: Status 404 returned error can't find the container with id 17b2e8bdf06b10ec19b7fd9ca5770d09cd12ef35f82c49f8eb9b224d4a77c4d6
Jul 16 23:10:30.972472 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:30.972410    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dbe90cf-81bd-4b55-830f-a123420e9f55-maas-api-service-ca\") pod \"1dbe90cf-81bd-4b55-830f-a123420e9f55\" (UID: \"1dbe90cf-81bd-4b55-830f-a123420e9f55\") "
Jul 16 23:10:30.972731 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:30.972710    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dbe90cf-81bd-4b55-830f-a123420e9f55-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1dbe90cf-81bd-4b55-830f-a123420e9f55" (UID: "1dbe90cf-81bd-4b55-830f-a123420e9f55"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:10:31.073383 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:31.073359    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dbe90cf-81bd-4b55-830f-a123420e9f55-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:10:31.668341 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:31.668269    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-9d2df"
Jul 16 23:10:31.668463 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:31.668271    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-9d2df" event={"ID":"1dbe90cf-81bd-4b55-830f-a123420e9f55","Type":"ContainerDied","Data":"ec097aad5a81122be30d621c64037b77c6c6eb29cb44c02b4ce1a4d2b8e3b7dc"}
Jul 16 23:10:31.668463 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:31.668408    2573 scope.go:117] "RemoveContainer" containerID="47619cd6b7e3cff2bee38607019a6be0ba707c39e149c89893b9e7ed54b9d19c"
Jul 16 23:10:31.669948 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:31.669918    2573 generic.go:356] "Generic (PLEG): container finished" podID="4d65c64e-4d3a-4a36-b244-b2ba156b6e14" containerID="32c1fca7d6852cde284a94199add2f212d31d247443449711606682fa7b969c0" exitCode=7
Jul 16 23:10:31.670086 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:31.669970    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-twv4t" event={"ID":"4d65c64e-4d3a-4a36-b244-b2ba156b6e14","Type":"ContainerDied","Data":"32c1fca7d6852cde284a94199add2f212d31d247443449711606682fa7b969c0"}
Jul 16 23:10:31.670086 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:31.670010    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-twv4t" event={"ID":"4d65c64e-4d3a-4a36-b244-b2ba156b6e14","Type":"ContainerStarted","Data":"17b2e8bdf06b10ec19b7fd9ca5770d09cd12ef35f82c49f8eb9b224d4a77c4d6"}
Jul 16 23:10:31.670213 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:31.670159    2573 scope.go:117] "RemoveContainer" containerID="32c1fca7d6852cde284a94199add2f212d31d247443449711606682fa7b969c0"
Jul 16 23:10:31.698119 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:31.698092    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-9d2df"]
Jul 16 23:10:31.703737 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:31.703715    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-9d2df"]
Jul 16 23:10:32.518354 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:32.518327    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dbe90cf-81bd-4b55-830f-a123420e9f55" path="/var/lib/kubelet/pods/1dbe90cf-81bd-4b55-830f-a123420e9f55/volumes"
Jul 16 23:10:32.597446 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:32.597417    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-dn2hl"]
Jul 16 23:10:32.597730 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:32.597716    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dbe90cf-81bd-4b55-830f-a123420e9f55" containerName="revoke-keys"
Jul 16 23:10:32.597773 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:32.597732    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dbe90cf-81bd-4b55-830f-a123420e9f55" containerName="revoke-keys"
Jul 16 23:10:32.597773 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:32.597744    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dbe90cf-81bd-4b55-830f-a123420e9f55" containerName="revoke-keys"
Jul 16 23:10:32.597773 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:32.597754    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dbe90cf-81bd-4b55-830f-a123420e9f55" containerName="revoke-keys"
Jul 16 23:10:32.597773 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:32.597772    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dbe90cf-81bd-4b55-830f-a123420e9f55" containerName="revoke-keys"
Jul 16 23:10:32.597901 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:32.597779    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dbe90cf-81bd-4b55-830f-a123420e9f55" containerName="revoke-keys"
Jul 16 23:10:32.597901 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:32.597840    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dbe90cf-81bd-4b55-830f-a123420e9f55" containerName="revoke-keys"
Jul 16 23:10:32.597901 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:32.597849    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dbe90cf-81bd-4b55-830f-a123420e9f55" containerName="revoke-keys"
Jul 16 23:10:32.597901 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:32.597855    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dbe90cf-81bd-4b55-830f-a123420e9f55" containerName="revoke-keys"
Jul 16 23:10:32.600535 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:32.600519    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-dn2hl"
Jul 16 23:10:32.606686 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:32.606581    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-dn2hl"]
Jul 16 23:10:32.677053 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:32.677025    2573 generic.go:356] "Generic (PLEG): container finished" podID="4d65c64e-4d3a-4a36-b244-b2ba156b6e14" containerID="1cd208c22f8e275c717c50f1a315d1789dd519075e7ab4a865c8ac2fcd349d68" exitCode=7
Jul 16 23:10:32.677165 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:32.677066    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-twv4t" event={"ID":"4d65c64e-4d3a-4a36-b244-b2ba156b6e14","Type":"ContainerDied","Data":"1cd208c22f8e275c717c50f1a315d1789dd519075e7ab4a865c8ac2fcd349d68"}
Jul 16 23:10:32.677165 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:32.677116    2573 scope.go:117] "RemoveContainer" containerID="32c1fca7d6852cde284a94199add2f212d31d247443449711606682fa7b969c0"
Jul 16 23:10:32.677307 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:32.677292    2573 scope.go:117] "RemoveContainer" containerID="1cd208c22f8e275c717c50f1a315d1789dd519075e7ab4a865c8ac2fcd349d68"
Jul 16 23:10:32.677505 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:10:32.677482    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-twv4t_odh-ai-gateway-infra(4d65c64e-4d3a-4a36-b244-b2ba156b6e14)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-twv4t" podUID="4d65c64e-4d3a-4a36-b244-b2ba156b6e14"
Jul 16 23:10:32.682277 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:32.682231    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaf14f35-8e14-4af1-a2db-06e466508893-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-dn2hl\" (UID: \"eaf14f35-8e14-4af1-a2db-06e466508893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-dn2hl"
Jul 16 23:10:32.782613 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:32.782584    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaf14f35-8e14-4af1-a2db-06e466508893-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-dn2hl\" (UID: \"eaf14f35-8e14-4af1-a2db-06e466508893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-dn2hl"
Jul 16 23:10:32.783169 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:32.783151    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaf14f35-8e14-4af1-a2db-06e466508893-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-dn2hl\" (UID: \"eaf14f35-8e14-4af1-a2db-06e466508893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-dn2hl"
Jul 16 23:10:32.912017 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:32.911957    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-dn2hl"
Jul 16 23:10:33.242292 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:33.242185    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-dn2hl"]
Jul 16 23:10:33.244937 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:10:33.244905    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaf14f35_8e14_4af1_a2db_06e466508893.slice/crio-48cb8f435dee74e1b46a763cb1649bf6f9eb915530df77fd8c7021f6a3c22e66 WatchSource:0}: Error finding container 48cb8f435dee74e1b46a763cb1649bf6f9eb915530df77fd8c7021f6a3c22e66: Status 404 returned error can't find the container with id 48cb8f435dee74e1b46a763cb1649bf6f9eb915530df77fd8c7021f6a3c22e66
Jul 16 23:10:33.684030 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:33.683993    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-dn2hl" event={"ID":"eaf14f35-8e14-4af1-a2db-06e466508893","Type":"ContainerStarted","Data":"0dbc71721b9e73aecc2da9371713620c10e7e85b40466ecdafe59d4bfd478c2c"}
Jul 16 23:10:33.684294 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:33.684055    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-dn2hl" event={"ID":"eaf14f35-8e14-4af1-a2db-06e466508893","Type":"ContainerStarted","Data":"48cb8f435dee74e1b46a763cb1649bf6f9eb915530df77fd8c7021f6a3c22e66"}
Jul 16 23:10:33.685786 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:33.685768    2573 scope.go:117] "RemoveContainer" containerID="1cd208c22f8e275c717c50f1a315d1789dd519075e7ab4a865c8ac2fcd349d68"
Jul 16 23:10:33.685918 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:10:33.685901    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-twv4t_odh-ai-gateway-infra(4d65c64e-4d3a-4a36-b244-b2ba156b6e14)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-twv4t" podUID="4d65c64e-4d3a-4a36-b244-b2ba156b6e14"
Jul 16 23:10:33.699088 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:33.699036    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-dn2hl" podStartSLOduration=1.6990219930000001 podStartE2EDuration="1.699021993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:10:33.697930203 +0000 UTC m=+2207.708192868" watchObservedRunningTime="2026-07-16 23:10:33.699021993 +0000 UTC m=+2207.709284657"
Jul 16 23:10:34.691694 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:34.691661    2573 generic.go:356] "Generic (PLEG): container finished" podID="eaf14f35-8e14-4af1-a2db-06e466508893" containerID="0dbc71721b9e73aecc2da9371713620c10e7e85b40466ecdafe59d4bfd478c2c" exitCode=6
Jul 16 23:10:34.692140 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:34.691711    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-dn2hl" event={"ID":"eaf14f35-8e14-4af1-a2db-06e466508893","Type":"ContainerDied","Data":"0dbc71721b9e73aecc2da9371713620c10e7e85b40466ecdafe59d4bfd478c2c"}
Jul 16 23:10:34.692140 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:34.691962    2573 scope.go:117] "RemoveContainer" containerID="0dbc71721b9e73aecc2da9371713620c10e7e85b40466ecdafe59d4bfd478c2c"
Jul 16 23:10:35.699761 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:35.699730    2573 generic.go:356] "Generic (PLEG): container finished" podID="eaf14f35-8e14-4af1-a2db-06e466508893" containerID="b72cdab1882b7d3cbc4741428531432e7feef5d46b8572771881667929562bfb" exitCode=6
Jul 16 23:10:35.700165 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:35.699828    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-dn2hl" event={"ID":"eaf14f35-8e14-4af1-a2db-06e466508893","Type":"ContainerDied","Data":"b72cdab1882b7d3cbc4741428531432e7feef5d46b8572771881667929562bfb"}
Jul 16 23:10:35.700165 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:35.699914    2573 scope.go:117] "RemoveContainer" containerID="0dbc71721b9e73aecc2da9371713620c10e7e85b40466ecdafe59d4bfd478c2c"
Jul 16 23:10:35.700165 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:35.699967    2573 scope.go:117] "RemoveContainer" containerID="b72cdab1882b7d3cbc4741428531432e7feef5d46b8572771881667929562bfb"
Jul 16 23:10:35.700165 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:10:35.700143    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-dn2hl_odh-ai-gateway-infra(eaf14f35-8e14-4af1-a2db-06e466508893)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-dn2hl" podUID="eaf14f35-8e14-4af1-a2db-06e466508893"
Jul 16 23:10:36.706117 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:36.706085    2573 scope.go:117] "RemoveContainer" containerID="b72cdab1882b7d3cbc4741428531432e7feef5d46b8572771881667929562bfb"
Jul 16 23:10:36.706608 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:10:36.706327    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-dn2hl_odh-ai-gateway-infra(eaf14f35-8e14-4af1-a2db-06e466508893)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-dn2hl" podUID="eaf14f35-8e14-4af1-a2db-06e466508893"
Jul 16 23:10:47.513378 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:47.513343    2573 scope.go:117] "RemoveContainer" containerID="1cd208c22f8e275c717c50f1a315d1789dd519075e7ab4a865c8ac2fcd349d68"
Jul 16 23:10:47.758964 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:47.758942    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-twv4t" event={"ID":"4d65c64e-4d3a-4a36-b244-b2ba156b6e14","Type":"ContainerStarted","Data":"32c174e5b83c4479bf9d898a794646663c7fb7761fc31b3ea6871df8c77037b9"}
Jul 16 23:10:47.775843 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:47.775771    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-twv4t" podStartSLOduration=17.77575706 podStartE2EDuration="17.77575706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:10:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:10:47.773227964 +0000 UTC m=+2221.783490629" watchObservedRunningTime="2026-07-16 23:10:47.77575706 +0000 UTC m=+2221.786019723"
Jul 16 23:10:48.766135 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:48.766100    2573 generic.go:356] "Generic (PLEG): container finished" podID="4d65c64e-4d3a-4a36-b244-b2ba156b6e14" containerID="32c174e5b83c4479bf9d898a794646663c7fb7761fc31b3ea6871df8c77037b9" exitCode=7
Jul 16 23:10:48.766650 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:48.766196    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-twv4t" event={"ID":"4d65c64e-4d3a-4a36-b244-b2ba156b6e14","Type":"ContainerDied","Data":"32c174e5b83c4479bf9d898a794646663c7fb7761fc31b3ea6871df8c77037b9"}
Jul 16 23:10:48.766650 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:48.766286    2573 scope.go:117] "RemoveContainer" containerID="1cd208c22f8e275c717c50f1a315d1789dd519075e7ab4a865c8ac2fcd349d68"
Jul 16 23:10:48.766650 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:48.766565    2573 scope.go:117] "RemoveContainer" containerID="32c174e5b83c4479bf9d898a794646663c7fb7761fc31b3ea6871df8c77037b9"
Jul 16 23:10:48.766790 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:10:48.766752    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-twv4t_odh-ai-gateway-infra(4d65c64e-4d3a-4a36-b244-b2ba156b6e14)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-twv4t" podUID="4d65c64e-4d3a-4a36-b244-b2ba156b6e14"
Jul 16 23:10:48.789795 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:48.789770    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-twv4t"]
Jul 16 23:10:49.900193 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:49.900168    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-twv4t"
Jul 16 23:10:50.004642 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:50.004616    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d65c64e-4d3a-4a36-b244-b2ba156b6e14-maas-api-service-ca\") pod \"4d65c64e-4d3a-4a36-b244-b2ba156b6e14\" (UID: \"4d65c64e-4d3a-4a36-b244-b2ba156b6e14\") "
Jul 16 23:10:50.004932 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:50.004911    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d65c64e-4d3a-4a36-b244-b2ba156b6e14-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4d65c64e-4d3a-4a36-b244-b2ba156b6e14" (UID: "4d65c64e-4d3a-4a36-b244-b2ba156b6e14"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:10:50.105543 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:50.105516    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d65c64e-4d3a-4a36-b244-b2ba156b6e14-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:10:50.777746 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:50.777693    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-twv4t"
Jul 16 23:10:50.777850 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:50.777685    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-twv4t" event={"ID":"4d65c64e-4d3a-4a36-b244-b2ba156b6e14","Type":"ContainerDied","Data":"17b2e8bdf06b10ec19b7fd9ca5770d09cd12ef35f82c49f8eb9b224d4a77c4d6"}
Jul 16 23:10:50.777850 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:50.777837    2573 scope.go:117] "RemoveContainer" containerID="32c174e5b83c4479bf9d898a794646663c7fb7761fc31b3ea6871df8c77037b9"
Jul 16 23:10:50.792470 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:50.792449    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-twv4t"]
Jul 16 23:10:50.798840 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:50.798817    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-twv4t"]
Jul 16 23:10:51.514147 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:51.514121    2573 scope.go:117] "RemoveContainer" containerID="b72cdab1882b7d3cbc4741428531432e7feef5d46b8572771881667929562bfb"
Jul 16 23:10:51.783997 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:51.783936    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-dn2hl" event={"ID":"eaf14f35-8e14-4af1-a2db-06e466508893","Type":"ContainerStarted","Data":"76164f15cf3f5f49d741d4ec40cc2dc976d307ae543c0196b8b90b516ff5e709"}
Jul 16 23:10:52.518800 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:52.518725    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d65c64e-4d3a-4a36-b244-b2ba156b6e14" path="/var/lib/kubelet/pods/4d65c64e-4d3a-4a36-b244-b2ba156b6e14/volumes"
Jul 16 23:10:52.792279 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:52.792189    2573 generic.go:356] "Generic (PLEG): container finished" podID="eaf14f35-8e14-4af1-a2db-06e466508893" containerID="76164f15cf3f5f49d741d4ec40cc2dc976d307ae543c0196b8b90b516ff5e709" exitCode=6
Jul 16 23:10:52.792279 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:52.792264    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-dn2hl" event={"ID":"eaf14f35-8e14-4af1-a2db-06e466508893","Type":"ContainerDied","Data":"76164f15cf3f5f49d741d4ec40cc2dc976d307ae543c0196b8b90b516ff5e709"}
Jul 16 23:10:52.792489 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:52.792333    2573 scope.go:117] "RemoveContainer" containerID="b72cdab1882b7d3cbc4741428531432e7feef5d46b8572771881667929562bfb"
Jul 16 23:10:52.792655 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:52.792637    2573 scope.go:117] "RemoveContainer" containerID="76164f15cf3f5f49d741d4ec40cc2dc976d307ae543c0196b8b90b516ff5e709"
Jul 16 23:10:52.792842 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:10:52.792818    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-dn2hl_odh-ai-gateway-infra(eaf14f35-8e14-4af1-a2db-06e466508893)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-dn2hl" podUID="eaf14f35-8e14-4af1-a2db-06e466508893"
Jul 16 23:10:52.808823 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:52.808798    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-dn2hl"]
Jul 16 23:10:53.924663 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:53.924642    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-dn2hl"
Jul 16 23:10:53.937190 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:53.937168    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaf14f35-8e14-4af1-a2db-06e466508893-maas-api-service-ca\") pod \"eaf14f35-8e14-4af1-a2db-06e466508893\" (UID: \"eaf14f35-8e14-4af1-a2db-06e466508893\") "
Jul 16 23:10:53.937484 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:53.937461    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaf14f35-8e14-4af1-a2db-06e466508893-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eaf14f35-8e14-4af1-a2db-06e466508893" (UID: "eaf14f35-8e14-4af1-a2db-06e466508893"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:10:54.038342 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:54.038311    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaf14f35-8e14-4af1-a2db-06e466508893-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:10:54.804370 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:54.804303    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-dn2hl" event={"ID":"eaf14f35-8e14-4af1-a2db-06e466508893","Type":"ContainerDied","Data":"48cb8f435dee74e1b46a763cb1649bf6f9eb915530df77fd8c7021f6a3c22e66"}
Jul 16 23:10:54.804370 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:54.804360    2573 scope.go:117] "RemoveContainer" containerID="76164f15cf3f5f49d741d4ec40cc2dc976d307ae543c0196b8b90b516ff5e709"
Jul 16 23:10:54.804556 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:54.804317    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-dn2hl"
Jul 16 23:10:54.820879 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:54.820854    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-dn2hl"]
Jul 16 23:10:54.825711 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:54.825690    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-dn2hl"]
Jul 16 23:10:56.518485 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:10:56.518453    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eaf14f35-8e14-4af1-a2db-06e466508893" path="/var/lib/kubelet/pods/eaf14f35-8e14-4af1-a2db-06e466508893/volumes"
Jul 16 23:11:11.456828 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:11.456749    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-2tqmp"]
Jul 16 23:11:11.457253 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:11.457020    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaf14f35-8e14-4af1-a2db-06e466508893" containerName="revoke-keys"
Jul 16 23:11:11.457253 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:11.457030    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaf14f35-8e14-4af1-a2db-06e466508893" containerName="revoke-keys"
Jul 16 23:11:11.457253 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:11.457039    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaf14f35-8e14-4af1-a2db-06e466508893" containerName="revoke-keys"
Jul 16 23:11:11.457253 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:11.457045    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaf14f35-8e14-4af1-a2db-06e466508893" containerName="revoke-keys"
Jul 16 23:11:11.457253 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:11.457058    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d65c64e-4d3a-4a36-b244-b2ba156b6e14" containerName="revoke-keys"
Jul 16 23:11:11.457253 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:11.457064    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d65c64e-4d3a-4a36-b244-b2ba156b6e14" containerName="revoke-keys"
Jul 16 23:11:11.457253 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:11.457074    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d65c64e-4d3a-4a36-b244-b2ba156b6e14" containerName="revoke-keys"
Jul 16 23:11:11.457253 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:11.457079    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d65c64e-4d3a-4a36-b244-b2ba156b6e14" containerName="revoke-keys"
Jul 16 23:11:11.457253 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:11.457085    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d65c64e-4d3a-4a36-b244-b2ba156b6e14" containerName="revoke-keys"
Jul 16 23:11:11.457253 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:11.457090    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d65c64e-4d3a-4a36-b244-b2ba156b6e14" containerName="revoke-keys"
Jul 16 23:11:11.457253 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:11.457137    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="eaf14f35-8e14-4af1-a2db-06e466508893" containerName="revoke-keys"
Jul 16 23:11:11.457253 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:11.457144    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="eaf14f35-8e14-4af1-a2db-06e466508893" containerName="revoke-keys"
Jul 16 23:11:11.457253 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:11.457149    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d65c64e-4d3a-4a36-b244-b2ba156b6e14" containerName="revoke-keys"
Jul 16 23:11:11.457253 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:11.457157    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d65c64e-4d3a-4a36-b244-b2ba156b6e14" containerName="revoke-keys"
Jul 16 23:11:11.461626 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:11.461605    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-2tqmp"
Jul 16 23:11:11.462706 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:11.462681    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-2tqmp"]
Jul 16 23:11:11.463955 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:11.463933    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:11:11.464072 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:11.463965    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 23:11:11.566875 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:11.566848    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e90b869c-386a-4995-a5d1-acb368644454-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-2tqmp\" (UID: \"e90b869c-386a-4995-a5d1-acb368644454\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-2tqmp"
Jul 16 23:11:11.668137 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:11.668111    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e90b869c-386a-4995-a5d1-acb368644454-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-2tqmp\" (UID: \"e90b869c-386a-4995-a5d1-acb368644454\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-2tqmp"
Jul 16 23:11:11.668727 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:11.668710    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e90b869c-386a-4995-a5d1-acb368644454-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-2tqmp\" (UID: \"e90b869c-386a-4995-a5d1-acb368644454\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-2tqmp"
Jul 16 23:11:11.772892 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:11.772828    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-2tqmp"
Jul 16 23:11:11.899271 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:11.899228    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-2tqmp"]
Jul 16 23:11:11.901139 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:11:11.901115    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode90b869c_386a_4995_a5d1_acb368644454.slice/crio-d1306da22910e0fcfabeb9dde344daab53071beee935094705f39a8e327b2d48 WatchSource:0}: Error finding container d1306da22910e0fcfabeb9dde344daab53071beee935094705f39a8e327b2d48: Status 404 returned error can't find the container with id d1306da22910e0fcfabeb9dde344daab53071beee935094705f39a8e327b2d48
Jul 16 23:11:12.892719 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:12.892679    2573 generic.go:356] "Generic (PLEG): container finished" podID="e90b869c-386a-4995-a5d1-acb368644454" containerID="8db39e43d54af50c2db7568303a4d1f5f27a355a84e99478854b9320e41a5bf8" exitCode=6
Jul 16 23:11:12.893155 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:12.892764    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-2tqmp" event={"ID":"e90b869c-386a-4995-a5d1-acb368644454","Type":"ContainerDied","Data":"8db39e43d54af50c2db7568303a4d1f5f27a355a84e99478854b9320e41a5bf8"}
Jul 16 23:11:12.893155 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:12.892821    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-2tqmp" event={"ID":"e90b869c-386a-4995-a5d1-acb368644454","Type":"ContainerStarted","Data":"d1306da22910e0fcfabeb9dde344daab53071beee935094705f39a8e327b2d48"}
Jul 16 23:11:12.893155 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:12.892989    2573 scope.go:117] "RemoveContainer" containerID="8db39e43d54af50c2db7568303a4d1f5f27a355a84e99478854b9320e41a5bf8"
Jul 16 23:11:13.901558 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:13.901517    2573 generic.go:356] "Generic (PLEG): container finished" podID="e90b869c-386a-4995-a5d1-acb368644454" containerID="01cd386deee2c85b7074af5d0466ae2924b69b6e42e2563e85aa59648e0e18e9" exitCode=6
Jul 16 23:11:13.901986 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:13.901588    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-2tqmp" event={"ID":"e90b869c-386a-4995-a5d1-acb368644454","Type":"ContainerDied","Data":"01cd386deee2c85b7074af5d0466ae2924b69b6e42e2563e85aa59648e0e18e9"}
Jul 16 23:11:13.901986 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:13.901658    2573 scope.go:117] "RemoveContainer" containerID="8db39e43d54af50c2db7568303a4d1f5f27a355a84e99478854b9320e41a5bf8"
Jul 16 23:11:13.901986 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:13.901849    2573 scope.go:117] "RemoveContainer" containerID="01cd386deee2c85b7074af5d0466ae2924b69b6e42e2563e85aa59648e0e18e9"
Jul 16 23:11:13.902152 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:11:13.902066    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-2tqmp_odh-ai-gateway-infra(e90b869c-386a-4995-a5d1-acb368644454)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-2tqmp" podUID="e90b869c-386a-4995-a5d1-acb368644454"
Jul 16 23:11:14.907623 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:14.907596    2573 scope.go:117] "RemoveContainer" containerID="01cd386deee2c85b7074af5d0466ae2924b69b6e42e2563e85aa59648e0e18e9"
Jul 16 23:11:14.908011 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:11:14.907762    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-2tqmp_odh-ai-gateway-infra(e90b869c-386a-4995-a5d1-acb368644454)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-2tqmp" podUID="e90b869c-386a-4995-a5d1-acb368644454"
Jul 16 23:11:26.524148 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:26.524120    2573 scope.go:117] "RemoveContainer" containerID="01cd386deee2c85b7074af5d0466ae2924b69b6e42e2563e85aa59648e0e18e9"
Jul 16 23:11:26.971503 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:26.971475    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-2tqmp" event={"ID":"e90b869c-386a-4995-a5d1-acb368644454","Type":"ContainerStarted","Data":"d8770fd9c283e5f3725dcbec9aaa5658b02b6f9c036ce86956505fc896a38340"}
Jul 16 23:11:26.987640 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:26.987594    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-2tqmp" podStartSLOduration=15.987580799 podStartE2EDuration="15.987580799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:11:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:11:26.985840019 +0000 UTC m=+2260.996102697" watchObservedRunningTime="2026-07-16 23:11:26.987580799 +0000 UTC m=+2260.997843510"
Jul 16 23:11:27.978185 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:27.978149    2573 generic.go:356] "Generic (PLEG): container finished" podID="e90b869c-386a-4995-a5d1-acb368644454" containerID="d8770fd9c283e5f3725dcbec9aaa5658b02b6f9c036ce86956505fc896a38340" exitCode=6
Jul 16 23:11:27.978666 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:27.978228    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-2tqmp" event={"ID":"e90b869c-386a-4995-a5d1-acb368644454","Type":"ContainerDied","Data":"d8770fd9c283e5f3725dcbec9aaa5658b02b6f9c036ce86956505fc896a38340"}
Jul 16 23:11:27.978666 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:27.978329    2573 scope.go:117] "RemoveContainer" containerID="01cd386deee2c85b7074af5d0466ae2924b69b6e42e2563e85aa59648e0e18e9"
Jul 16 23:11:27.978666 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:27.978604    2573 scope.go:117] "RemoveContainer" containerID="d8770fd9c283e5f3725dcbec9aaa5658b02b6f9c036ce86956505fc896a38340"
Jul 16 23:11:27.978823 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:11:27.978801    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-2tqmp_odh-ai-gateway-infra(e90b869c-386a-4995-a5d1-acb368644454)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-2tqmp" podUID="e90b869c-386a-4995-a5d1-acb368644454"
Jul 16 23:11:28.000816 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:28.000791    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-2tqmp"]
Jul 16 23:11:29.114709 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:29.114682    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-2tqmp"
Jul 16 23:11:29.190125 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:29.190100    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e90b869c-386a-4995-a5d1-acb368644454-maas-api-service-ca\") pod \"e90b869c-386a-4995-a5d1-acb368644454\" (UID: \"e90b869c-386a-4995-a5d1-acb368644454\") "
Jul 16 23:11:29.190444 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:29.190423    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e90b869c-386a-4995-a5d1-acb368644454-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e90b869c-386a-4995-a5d1-acb368644454" (UID: "e90b869c-386a-4995-a5d1-acb368644454"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:11:29.291367 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:29.291310    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e90b869c-386a-4995-a5d1-acb368644454-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:11:29.993066 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:29.993034    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-2tqmp"
Jul 16 23:11:29.993270 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:29.993034    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-2tqmp" event={"ID":"e90b869c-386a-4995-a5d1-acb368644454","Type":"ContainerDied","Data":"d1306da22910e0fcfabeb9dde344daab53071beee935094705f39a8e327b2d48"}
Jul 16 23:11:29.993270 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:29.993182    2573 scope.go:117] "RemoveContainer" containerID="d8770fd9c283e5f3725dcbec9aaa5658b02b6f9c036ce86956505fc896a38340"
Jul 16 23:11:30.015343 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:30.015315    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-2tqmp"]
Jul 16 23:11:30.020854 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:30.020827    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-2tqmp"]
Jul 16 23:11:30.429828 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:30.429798    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-qqr4d"]
Jul 16 23:11:30.430142 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:30.430085    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e90b869c-386a-4995-a5d1-acb368644454" containerName="revoke-keys"
Jul 16 23:11:30.430142 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:30.430097    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e90b869c-386a-4995-a5d1-acb368644454" containerName="revoke-keys"
Jul 16 23:11:30.430142 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:30.430106    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaf14f35-8e14-4af1-a2db-06e466508893" containerName="revoke-keys"
Jul 16 23:11:30.430142 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:30.430112    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaf14f35-8e14-4af1-a2db-06e466508893" containerName="revoke-keys"
Jul 16 23:11:30.430142 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:30.430130    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e90b869c-386a-4995-a5d1-acb368644454" containerName="revoke-keys"
Jul 16 23:11:30.430142 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:30.430135    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e90b869c-386a-4995-a5d1-acb368644454" containerName="revoke-keys"
Jul 16 23:11:30.430376 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:30.430201    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e90b869c-386a-4995-a5d1-acb368644454" containerName="revoke-keys"
Jul 16 23:11:30.430376 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:30.430208    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e90b869c-386a-4995-a5d1-acb368644454" containerName="revoke-keys"
Jul 16 23:11:30.430376 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:30.430215    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d65c64e-4d3a-4a36-b244-b2ba156b6e14" containerName="revoke-keys"
Jul 16 23:11:30.430376 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:30.430222    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="eaf14f35-8e14-4af1-a2db-06e466508893" containerName="revoke-keys"
Jul 16 23:11:30.434452 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:30.434433    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-qqr4d"
Jul 16 23:11:30.436606 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:30.436586    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 23:11:30.436778 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:30.436761    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:11:30.440803 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:30.440777    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-qqr4d"]
Jul 16 23:11:30.501001 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:30.500975    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88ab634f-73e2-4bdf-812c-30c584386c71-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-qqr4d\" (UID: \"88ab634f-73e2-4bdf-812c-30c584386c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-qqr4d"
Jul 16 23:11:30.517976 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:30.517946    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e90b869c-386a-4995-a5d1-acb368644454" path="/var/lib/kubelet/pods/e90b869c-386a-4995-a5d1-acb368644454/volumes"
Jul 16 23:11:30.601420 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:30.601384    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88ab634f-73e2-4bdf-812c-30c584386c71-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-qqr4d\" (UID: \"88ab634f-73e2-4bdf-812c-30c584386c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-qqr4d"
Jul 16 23:11:30.601975 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:30.601956    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88ab634f-73e2-4bdf-812c-30c584386c71-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-qqr4d\" (UID: \"88ab634f-73e2-4bdf-812c-30c584386c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-qqr4d"
Jul 16 23:11:30.746254 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:30.746178    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-qqr4d"
Jul 16 23:11:30.872109 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:30.871993    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-qqr4d"]
Jul 16 23:11:30.875021 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:11:30.874986    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88ab634f_73e2_4bdf_812c_30c584386c71.slice/crio-33b354441f36012e51f674a11fcbfe781a1e4a1b87f94a808bf5c22c4465b5e2 WatchSource:0}: Error finding container 33b354441f36012e51f674a11fcbfe781a1e4a1b87f94a808bf5c22c4465b5e2: Status 404 returned error can't find the container with id 33b354441f36012e51f674a11fcbfe781a1e4a1b87f94a808bf5c22c4465b5e2
Jul 16 23:11:30.999945 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:30.999889    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-qqr4d" event={"ID":"88ab634f-73e2-4bdf-812c-30c584386c71","Type":"ContainerStarted","Data":"33b354441f36012e51f674a11fcbfe781a1e4a1b87f94a808bf5c22c4465b5e2"}
Jul 16 23:11:32.006437 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:32.006334    2573 generic.go:356] "Generic (PLEG): container finished" podID="88ab634f-73e2-4bdf-812c-30c584386c71" containerID="3ef07735739272df1438059faaa4285397d4e1b1456380690ccaa4b0b35fe73a" exitCode=7
Jul 16 23:11:32.006437 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:32.006379    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-qqr4d" event={"ID":"88ab634f-73e2-4bdf-812c-30c584386c71","Type":"ContainerDied","Data":"3ef07735739272df1438059faaa4285397d4e1b1456380690ccaa4b0b35fe73a"}
Jul 16 23:11:32.007020 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:32.006669    2573 scope.go:117] "RemoveContainer" containerID="3ef07735739272df1438059faaa4285397d4e1b1456380690ccaa4b0b35fe73a"
Jul 16 23:11:32.644406 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:32.644376    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-7psth"]
Jul 16 23:11:32.644677 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:32.644664    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e90b869c-386a-4995-a5d1-acb368644454" containerName="revoke-keys"
Jul 16 23:11:32.644718 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:32.644680    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e90b869c-386a-4995-a5d1-acb368644454" containerName="revoke-keys"
Jul 16 23:11:32.644753 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:32.644745    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e90b869c-386a-4995-a5d1-acb368644454" containerName="revoke-keys"
Jul 16 23:11:32.647574 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:32.647559    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-7psth"
Jul 16 23:11:32.654604 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:32.654543    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-7psth"]
Jul 16 23:11:32.717254 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:32.717216    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ab9dd02-10fd-4387-a146-599286b8fa30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-7psth\" (UID: \"8ab9dd02-10fd-4387-a146-599286b8fa30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-7psth"
Jul 16 23:11:32.817624 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:32.817597    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ab9dd02-10fd-4387-a146-599286b8fa30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-7psth\" (UID: \"8ab9dd02-10fd-4387-a146-599286b8fa30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-7psth"
Jul 16 23:11:32.818145 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:32.818128    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ab9dd02-10fd-4387-a146-599286b8fa30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-7psth\" (UID: \"8ab9dd02-10fd-4387-a146-599286b8fa30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-7psth"
Jul 16 23:11:32.958544 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:32.958481    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-7psth"
Jul 16 23:11:33.017473 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:33.017057    2573 generic.go:356] "Generic (PLEG): container finished" podID="88ab634f-73e2-4bdf-812c-30c584386c71" containerID="e92e82bb3c0a387d4815f1c2f360ecadf3b4a197360e7adb1b346a46a167e476" exitCode=7
Jul 16 23:11:33.017473 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:33.017194    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-qqr4d" event={"ID":"88ab634f-73e2-4bdf-812c-30c584386c71","Type":"ContainerDied","Data":"e92e82bb3c0a387d4815f1c2f360ecadf3b4a197360e7adb1b346a46a167e476"}
Jul 16 23:11:33.017473 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:33.017285    2573 scope.go:117] "RemoveContainer" containerID="3ef07735739272df1438059faaa4285397d4e1b1456380690ccaa4b0b35fe73a"
Jul 16 23:11:33.017947 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:33.017504    2573 scope.go:117] "RemoveContainer" containerID="e92e82bb3c0a387d4815f1c2f360ecadf3b4a197360e7adb1b346a46a167e476"
Jul 16 23:11:33.017947 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:11:33.017713    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-qqr4d_odh-ai-gateway-infra(88ab634f-73e2-4bdf-812c-30c584386c71)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-qqr4d" podUID="88ab634f-73e2-4bdf-812c-30c584386c71"
Jul 16 23:11:33.085132 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:33.085109    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-7psth"]
Jul 16 23:11:33.087190 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:11:33.087162    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ab9dd02_10fd_4387_a146_599286b8fa30.slice/crio-931586220b7590bd5fd635a73252774a443bea2f6ac56db16d1fb79beff558f4 WatchSource:0}: Error finding container 931586220b7590bd5fd635a73252774a443bea2f6ac56db16d1fb79beff558f4: Status 404 returned error can't find the container with id 931586220b7590bd5fd635a73252774a443bea2f6ac56db16d1fb79beff558f4
Jul 16 23:11:34.023932 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:34.023865    2573 scope.go:117] "RemoveContainer" containerID="e92e82bb3c0a387d4815f1c2f360ecadf3b4a197360e7adb1b346a46a167e476"
Jul 16 23:11:34.024351 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:11:34.024042    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-qqr4d_odh-ai-gateway-infra(88ab634f-73e2-4bdf-812c-30c584386c71)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-qqr4d" podUID="88ab634f-73e2-4bdf-812c-30c584386c71"
Jul 16 23:11:34.025298 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:34.025269    2573 generic.go:356] "Generic (PLEG): container finished" podID="8ab9dd02-10fd-4387-a146-599286b8fa30" containerID="72fdec84648817c15144bf9c1ad4e3edaf975be9a8916d942f4c9aa9f4e5ec57" exitCode=6
Jul 16 23:11:34.025424 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:34.025349    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-7psth" event={"ID":"8ab9dd02-10fd-4387-a146-599286b8fa30","Type":"ContainerDied","Data":"72fdec84648817c15144bf9c1ad4e3edaf975be9a8916d942f4c9aa9f4e5ec57"}
Jul 16 23:11:34.025424 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:34.025397    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-7psth" event={"ID":"8ab9dd02-10fd-4387-a146-599286b8fa30","Type":"ContainerStarted","Data":"931586220b7590bd5fd635a73252774a443bea2f6ac56db16d1fb79beff558f4"}
Jul 16 23:11:34.025543 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:34.025525    2573 scope.go:117] "RemoveContainer" containerID="72fdec84648817c15144bf9c1ad4e3edaf975be9a8916d942f4c9aa9f4e5ec57"
Jul 16 23:11:35.031704 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:35.031611    2573 generic.go:356] "Generic (PLEG): container finished" podID="8ab9dd02-10fd-4387-a146-599286b8fa30" containerID="3d4616f5c4f475eaff4035ac485dbd467963e653b3dd90bea08d9052e06f3545" exitCode=6
Jul 16 23:11:35.031704 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:35.031653    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-7psth" event={"ID":"8ab9dd02-10fd-4387-a146-599286b8fa30","Type":"ContainerDied","Data":"3d4616f5c4f475eaff4035ac485dbd467963e653b3dd90bea08d9052e06f3545"}
Jul 16 23:11:35.032157 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:35.031719    2573 scope.go:117] "RemoveContainer" containerID="72fdec84648817c15144bf9c1ad4e3edaf975be9a8916d942f4c9aa9f4e5ec57"
Jul 16 23:11:35.032157 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:35.031885    2573 scope.go:117] "RemoveContainer" containerID="3d4616f5c4f475eaff4035ac485dbd467963e653b3dd90bea08d9052e06f3545"
Jul 16 23:11:35.032157 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:11:35.032038    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-7psth_odh-ai-gateway-infra(8ab9dd02-10fd-4387-a146-599286b8fa30)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-7psth" podUID="8ab9dd02-10fd-4387-a146-599286b8fa30"
Jul 16 23:11:36.037696 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:36.037667    2573 scope.go:117] "RemoveContainer" containerID="3d4616f5c4f475eaff4035ac485dbd467963e653b3dd90bea08d9052e06f3545"
Jul 16 23:11:36.038064 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:11:36.037866    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-7psth_odh-ai-gateway-infra(8ab9dd02-10fd-4387-a146-599286b8fa30)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-7psth" podUID="8ab9dd02-10fd-4387-a146-599286b8fa30"
Jul 16 23:11:36.971881 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:36.971840    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:11:46.516363 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:46.516318    2573 scope.go:117] "RemoveContainer" containerID="e92e82bb3c0a387d4815f1c2f360ecadf3b4a197360e7adb1b346a46a167e476"
Jul 16 23:11:47.093576 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:47.093547    2573 generic.go:356] "Generic (PLEG): container finished" podID="88ab634f-73e2-4bdf-812c-30c584386c71" containerID="0273b109927511e48bc4e2f74cc0b996bfee449c010cf908244de537ddaa6da1" exitCode=7
Jul 16 23:11:47.093709 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:47.093589    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-qqr4d" event={"ID":"88ab634f-73e2-4bdf-812c-30c584386c71","Type":"ContainerDied","Data":"0273b109927511e48bc4e2f74cc0b996bfee449c010cf908244de537ddaa6da1"}
Jul 16 23:11:47.093709 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:47.093646    2573 scope.go:117] "RemoveContainer" containerID="e92e82bb3c0a387d4815f1c2f360ecadf3b4a197360e7adb1b346a46a167e476"
Jul 16 23:11:47.093918 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:47.093897    2573 scope.go:117] "RemoveContainer" containerID="0273b109927511e48bc4e2f74cc0b996bfee449c010cf908244de537ddaa6da1"
Jul 16 23:11:47.094128 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:11:47.094107    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-qqr4d_odh-ai-gateway-infra(88ab634f-73e2-4bdf-812c-30c584386c71)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-qqr4d" podUID="88ab634f-73e2-4bdf-812c-30c584386c71"
Jul 16 23:11:48.117893 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:48.117856    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-qqr4d"]
Jul 16 23:11:48.242030 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:48.242006    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-qqr4d"
Jul 16 23:11:48.330649 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:48.330620    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88ab634f-73e2-4bdf-812c-30c584386c71-maas-api-service-ca\") pod \"88ab634f-73e2-4bdf-812c-30c584386c71\" (UID: \"88ab634f-73e2-4bdf-812c-30c584386c71\") "
Jul 16 23:11:48.330963 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:48.330938    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88ab634f-73e2-4bdf-812c-30c584386c71-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "88ab634f-73e2-4bdf-812c-30c584386c71" (UID: "88ab634f-73e2-4bdf-812c-30c584386c71"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:11:48.431573 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:48.431522    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88ab634f-73e2-4bdf-812c-30c584386c71-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:11:49.105186 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:49.105160    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-qqr4d"
Jul 16 23:11:49.105362 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:49.105201    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-qqr4d" event={"ID":"88ab634f-73e2-4bdf-812c-30c584386c71","Type":"ContainerDied","Data":"33b354441f36012e51f674a11fcbfe781a1e4a1b87f94a808bf5c22c4465b5e2"}
Jul 16 23:11:49.105362 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:49.105310    2573 scope.go:117] "RemoveContainer" containerID="0273b109927511e48bc4e2f74cc0b996bfee449c010cf908244de537ddaa6da1"
Jul 16 23:11:49.120369 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:49.120345    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-qqr4d"]
Jul 16 23:11:49.126362 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:49.126341    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-qqr4d"]
Jul 16 23:11:49.513193 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:49.513138    2573 scope.go:117] "RemoveContainer" containerID="3d4616f5c4f475eaff4035ac485dbd467963e653b3dd90bea08d9052e06f3545"
Jul 16 23:11:50.112702 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:50.112674    2573 generic.go:356] "Generic (PLEG): container finished" podID="8ab9dd02-10fd-4387-a146-599286b8fa30" containerID="6f3e22662687e11b981af4a84f64350bcb74c829aebcddddf3b368e4b6af7e68" exitCode=6
Jul 16 23:11:50.112819 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:50.112750    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-7psth" event={"ID":"8ab9dd02-10fd-4387-a146-599286b8fa30","Type":"ContainerDied","Data":"6f3e22662687e11b981af4a84f64350bcb74c829aebcddddf3b368e4b6af7e68"}
Jul 16 23:11:50.112819 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:50.112812    2573 scope.go:117] "RemoveContainer" containerID="3d4616f5c4f475eaff4035ac485dbd467963e653b3dd90bea08d9052e06f3545"
Jul 16 23:11:50.113083 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:50.113063    2573 scope.go:117] "RemoveContainer" containerID="6f3e22662687e11b981af4a84f64350bcb74c829aebcddddf3b368e4b6af7e68"
Jul 16 23:11:50.113257 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:11:50.113220    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-7psth_odh-ai-gateway-infra(8ab9dd02-10fd-4387-a146-599286b8fa30)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-7psth" podUID="8ab9dd02-10fd-4387-a146-599286b8fa30"
Jul 16 23:11:50.518270 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:50.518185    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88ab634f-73e2-4bdf-812c-30c584386c71" path="/var/lib/kubelet/pods/88ab634f-73e2-4bdf-812c-30c584386c71/volumes"
Jul 16 23:11:51.137110 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:51.137072    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-7psth"]
Jul 16 23:11:51.271262 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:51.271211    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-7psth"
Jul 16 23:11:51.350903 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:51.350877    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ab9dd02-10fd-4387-a146-599286b8fa30-maas-api-service-ca\") pod \"8ab9dd02-10fd-4387-a146-599286b8fa30\" (UID: \"8ab9dd02-10fd-4387-a146-599286b8fa30\") "
Jul 16 23:11:51.351218 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:51.351198    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ab9dd02-10fd-4387-a146-599286b8fa30-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8ab9dd02-10fd-4387-a146-599286b8fa30" (UID: "8ab9dd02-10fd-4387-a146-599286b8fa30"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:11:51.452089 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:51.452021    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ab9dd02-10fd-4387-a146-599286b8fa30-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:11:52.124057 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:52.124025    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-7psth"
Jul 16 23:11:52.124057 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:52.124043    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-7psth" event={"ID":"8ab9dd02-10fd-4387-a146-599286b8fa30","Type":"ContainerDied","Data":"931586220b7590bd5fd635a73252774a443bea2f6ac56db16d1fb79beff558f4"}
Jul 16 23:11:52.124543 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:52.124101    2573 scope.go:117] "RemoveContainer" containerID="6f3e22662687e11b981af4a84f64350bcb74c829aebcddddf3b368e4b6af7e68"
Jul 16 23:11:52.145501 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:52.145477    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-7psth"]
Jul 16 23:11:52.151085 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:52.151064    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-7psth"]
Jul 16 23:11:52.518282 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:11:52.518186    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ab9dd02-10fd-4387-a146-599286b8fa30" path="/var/lib/kubelet/pods/8ab9dd02-10fd-4387-a146-599286b8fa30/volumes"
Jul 16 23:12:00.457371 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:00.457328    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:12:00.556423 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:00.556390    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:12:01.492821 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:01.492788    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-q8svj"]
Jul 16 23:12:01.493164 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:01.493055    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ab9dd02-10fd-4387-a146-599286b8fa30" containerName="revoke-keys"
Jul 16 23:12:01.493164 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:01.493065    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ab9dd02-10fd-4387-a146-599286b8fa30" containerName="revoke-keys"
Jul 16 23:12:01.493164 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:01.493075    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ab9dd02-10fd-4387-a146-599286b8fa30" containerName="revoke-keys"
Jul 16 23:12:01.493164 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:01.493081    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ab9dd02-10fd-4387-a146-599286b8fa30" containerName="revoke-keys"
Jul 16 23:12:01.493164 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:01.493088    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88ab634f-73e2-4bdf-812c-30c584386c71" containerName="revoke-keys"
Jul 16 23:12:01.493164 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:01.493094    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="88ab634f-73e2-4bdf-812c-30c584386c71" containerName="revoke-keys"
Jul 16 23:12:01.493164 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:01.493106    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88ab634f-73e2-4bdf-812c-30c584386c71" containerName="revoke-keys"
Jul 16 23:12:01.493164 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:01.493111    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="88ab634f-73e2-4bdf-812c-30c584386c71" containerName="revoke-keys"
Jul 16 23:12:01.493164 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:01.493120    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88ab634f-73e2-4bdf-812c-30c584386c71" containerName="revoke-keys"
Jul 16 23:12:01.493164 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:01.493125    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="88ab634f-73e2-4bdf-812c-30c584386c71" containerName="revoke-keys"
Jul 16 23:12:01.493164 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:01.493166    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ab9dd02-10fd-4387-a146-599286b8fa30" containerName="revoke-keys"
Jul 16 23:12:01.493513 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:01.493175    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ab9dd02-10fd-4387-a146-599286b8fa30" containerName="revoke-keys"
Jul 16 23:12:01.493513 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:01.493180    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="88ab634f-73e2-4bdf-812c-30c584386c71" containerName="revoke-keys"
Jul 16 23:12:01.493513 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:01.493188    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="88ab634f-73e2-4bdf-812c-30c584386c71" containerName="revoke-keys"
Jul 16 23:12:01.497100 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:01.497083    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-q8svj"
Jul 16 23:12:01.499181 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:01.499162    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:12:01.499336 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:01.499163    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 23:12:01.503848 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:01.503802    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-q8svj"]
Jul 16 23:12:01.624563 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:01.624538    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/526b1e3e-b375-4d05-9e85-542c944af91d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-q8svj\" (UID: \"526b1e3e-b375-4d05-9e85-542c944af91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-q8svj"
Jul 16 23:12:01.725699 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:01.725677    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/526b1e3e-b375-4d05-9e85-542c944af91d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-q8svj\" (UID: \"526b1e3e-b375-4d05-9e85-542c944af91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-q8svj"
Jul 16 23:12:01.726256 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:01.726222    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/526b1e3e-b375-4d05-9e85-542c944af91d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-q8svj\" (UID: \"526b1e3e-b375-4d05-9e85-542c944af91d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-q8svj"
Jul 16 23:12:01.808531 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:01.808473    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-q8svj"
Jul 16 23:12:01.928278 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:01.928253    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-q8svj"]
Jul 16 23:12:01.930328 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:12:01.930303    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod526b1e3e_b375_4d05_9e85_542c944af91d.slice/crio-75c479b5a8ceffe7038baca228e458032853ea044867dd67635f6dcfdc4b6b96 WatchSource:0}: Error finding container 75c479b5a8ceffe7038baca228e458032853ea044867dd67635f6dcfdc4b6b96: Status 404 returned error can't find the container with id 75c479b5a8ceffe7038baca228e458032853ea044867dd67635f6dcfdc4b6b96
Jul 16 23:12:02.171484 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:02.171448    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-q8svj" event={"ID":"526b1e3e-b375-4d05-9e85-542c944af91d","Type":"ContainerStarted","Data":"7ea5deb750667761c6a1a1a91a0a118576e66c5b6bf5f937b1948b2f10bb1a63"}
Jul 16 23:12:02.171597 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:02.171526    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-q8svj" event={"ID":"526b1e3e-b375-4d05-9e85-542c944af91d","Type":"ContainerStarted","Data":"75c479b5a8ceffe7038baca228e458032853ea044867dd67635f6dcfdc4b6b96"}
Jul 16 23:12:02.186891 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:02.186851    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-q8svj" podStartSLOduration=1.186837214 podStartE2EDuration="1.186837214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:12:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:12:02.185564455 +0000 UTC m=+2296.195827120" watchObservedRunningTime="2026-07-16 23:12:02.186837214 +0000 UTC m=+2296.197099877"
Jul 16 23:12:03.176984 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:03.176949    2573 generic.go:356] "Generic (PLEG): container finished" podID="526b1e3e-b375-4d05-9e85-542c944af91d" containerID="7ea5deb750667761c6a1a1a91a0a118576e66c5b6bf5f937b1948b2f10bb1a63" exitCode=6
Jul 16 23:12:03.177330 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:03.177038    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-q8svj" event={"ID":"526b1e3e-b375-4d05-9e85-542c944af91d","Type":"ContainerDied","Data":"7ea5deb750667761c6a1a1a91a0a118576e66c5b6bf5f937b1948b2f10bb1a63"}
Jul 16 23:12:03.177330 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:03.177168    2573 scope.go:117] "RemoveContainer" containerID="7ea5deb750667761c6a1a1a91a0a118576e66c5b6bf5f937b1948b2f10bb1a63"
Jul 16 23:12:03.331541 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:03.331512    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:12:04.183852 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:04.183824    2573 generic.go:356] "Generic (PLEG): container finished" podID="526b1e3e-b375-4d05-9e85-542c944af91d" containerID="5be8b238964bc51ce38d41b32c7da6ccec6c9bb7f9f4d3a4ce724ef801e61b00" exitCode=6
Jul 16 23:12:04.184199 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:04.183910    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-q8svj" event={"ID":"526b1e3e-b375-4d05-9e85-542c944af91d","Type":"ContainerDied","Data":"5be8b238964bc51ce38d41b32c7da6ccec6c9bb7f9f4d3a4ce724ef801e61b00"}
Jul 16 23:12:04.184199 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:04.183973    2573 scope.go:117] "RemoveContainer" containerID="7ea5deb750667761c6a1a1a91a0a118576e66c5b6bf5f937b1948b2f10bb1a63"
Jul 16 23:12:04.184199 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:04.184035    2573 scope.go:117] "RemoveContainer" containerID="5be8b238964bc51ce38d41b32c7da6ccec6c9bb7f9f4d3a4ce724ef801e61b00"
Jul 16 23:12:04.184352 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:12:04.184197    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-q8svj_odh-ai-gateway-infra(526b1e3e-b375-4d05-9e85-542c944af91d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-q8svj" podUID="526b1e3e-b375-4d05-9e85-542c944af91d"
Jul 16 23:12:05.190357 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:05.190329    2573 scope.go:117] "RemoveContainer" containerID="5be8b238964bc51ce38d41b32c7da6ccec6c9bb7f9f4d3a4ce724ef801e61b00"
Jul 16 23:12:05.190739 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:12:05.190481    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-q8svj_odh-ai-gateway-infra(526b1e3e-b375-4d05-9e85-542c944af91d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-q8svj" podUID="526b1e3e-b375-4d05-9e85-542c944af91d"
Jul 16 23:12:05.743700 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:05.743665    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:12:06.379205 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:06.379179    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:12:11.831814 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:11.831784    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:12:11.892828 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:11.892803    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:12:12.114947 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:12.114919    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:12:13.272750 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:13.272700    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:12:13.356442 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:13.356411    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:12:13.417032 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:13.417007    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:12:13.559226 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:13.559161    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:12:13.904324 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:13.904298    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:12:14.009337 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:14.009313    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:12:15.926283 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:15.926254    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:12:16.516973 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:16.516946    2573 scope.go:117] "RemoveContainer" containerID="5be8b238964bc51ce38d41b32c7da6ccec6c9bb7f9f4d3a4ce724ef801e61b00"
Jul 16 23:12:17.248586 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:17.248516    2573 generic.go:356] "Generic (PLEG): container finished" podID="526b1e3e-b375-4d05-9e85-542c944af91d" containerID="df8a38ad346edcfec9ccc2c51bdb093bb6b306cdbec6fb471b4f568d5dc5e8be" exitCode=6
Jul 16 23:12:17.248586 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:17.248568    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-q8svj" event={"ID":"526b1e3e-b375-4d05-9e85-542c944af91d","Type":"ContainerDied","Data":"df8a38ad346edcfec9ccc2c51bdb093bb6b306cdbec6fb471b4f568d5dc5e8be"}
Jul 16 23:12:17.249017 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:17.248627    2573 scope.go:117] "RemoveContainer" containerID="5be8b238964bc51ce38d41b32c7da6ccec6c9bb7f9f4d3a4ce724ef801e61b00"
Jul 16 23:12:17.249017 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:17.248907    2573 scope.go:117] "RemoveContainer" containerID="df8a38ad346edcfec9ccc2c51bdb093bb6b306cdbec6fb471b4f568d5dc5e8be"
Jul 16 23:12:17.249107 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:12:17.249088    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-q8svj_odh-ai-gateway-infra(526b1e3e-b375-4d05-9e85-542c944af91d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-q8svj" podUID="526b1e3e-b375-4d05-9e85-542c944af91d"
Jul 16 23:12:17.639194 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:17.639159    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-q7mqr"]
Jul 16 23:12:17.639588 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:17.639569    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ab9dd02-10fd-4387-a146-599286b8fa30" containerName="revoke-keys"
Jul 16 23:12:17.639678 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:17.639591    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ab9dd02-10fd-4387-a146-599286b8fa30" containerName="revoke-keys"
Jul 16 23:12:17.639732 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:17.639677    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ab9dd02-10fd-4387-a146-599286b8fa30" containerName="revoke-keys"
Jul 16 23:12:17.639732 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:17.639690    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="88ab634f-73e2-4bdf-812c-30c584386c71" containerName="revoke-keys"
Jul 16 23:12:17.643726 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:17.643706    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-q7mqr"
Jul 16 23:12:17.648975 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:17.648951    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-q7mqr"]
Jul 16 23:12:17.735381 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:17.735355    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de89d05d-2130-43bc-93d1-efd489c89b77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-q7mqr\" (UID: \"de89d05d-2130-43bc-93d1-efd489c89b77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-q7mqr"
Jul 16 23:12:17.836623 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:17.836599    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de89d05d-2130-43bc-93d1-efd489c89b77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-q7mqr\" (UID: \"de89d05d-2130-43bc-93d1-efd489c89b77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-q7mqr"
Jul 16 23:12:17.837194 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:17.837172    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de89d05d-2130-43bc-93d1-efd489c89b77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-q7mqr\" (UID: \"de89d05d-2130-43bc-93d1-efd489c89b77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-q7mqr"
Jul 16 23:12:17.955529 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:17.955471    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-q7mqr"
Jul 16 23:12:18.078703 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:18.078677    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-q7mqr"]
Jul 16 23:12:18.080435 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:12:18.080409    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde89d05d_2130_43bc_93d1_efd489c89b77.slice/crio-dd9dc66f5ac4a430a65a895b7cb455be76b10226b467b6ad7056e5e9437dfafc WatchSource:0}: Error finding container dd9dc66f5ac4a430a65a895b7cb455be76b10226b467b6ad7056e5e9437dfafc: Status 404 returned error can't find the container with id dd9dc66f5ac4a430a65a895b7cb455be76b10226b467b6ad7056e5e9437dfafc
Jul 16 23:12:18.255057 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:18.254994    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-q7mqr" event={"ID":"de89d05d-2130-43bc-93d1-efd489c89b77","Type":"ContainerStarted","Data":"dd9dc66f5ac4a430a65a895b7cb455be76b10226b467b6ad7056e5e9437dfafc"}
Jul 16 23:12:18.275918 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:18.275886    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-q8svj"]
Jul 16 23:12:18.481962 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:18.481943    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-q8svj"
Jul 16 23:12:18.642953 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:18.642930    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/526b1e3e-b375-4d05-9e85-542c944af91d-maas-api-service-ca\") pod \"526b1e3e-b375-4d05-9e85-542c944af91d\" (UID: \"526b1e3e-b375-4d05-9e85-542c944af91d\") "
Jul 16 23:12:18.643275 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:18.643232    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/526b1e3e-b375-4d05-9e85-542c944af91d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "526b1e3e-b375-4d05-9e85-542c944af91d" (UID: "526b1e3e-b375-4d05-9e85-542c944af91d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:12:18.743952 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:18.743931    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/526b1e3e-b375-4d05-9e85-542c944af91d-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:12:19.262041 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:19.261983    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-q8svj"
Jul 16 23:12:19.262041 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:19.262023    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-q8svj" event={"ID":"526b1e3e-b375-4d05-9e85-542c944af91d","Type":"ContainerDied","Data":"75c479b5a8ceffe7038baca228e458032853ea044867dd67635f6dcfdc4b6b96"}
Jul 16 23:12:19.262486 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:19.262081    2573 scope.go:117] "RemoveContainer" containerID="df8a38ad346edcfec9ccc2c51bdb093bb6b306cdbec6fb471b4f568d5dc5e8be"
Jul 16 23:12:19.263909 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:19.263885    2573 generic.go:356] "Generic (PLEG): container finished" podID="de89d05d-2130-43bc-93d1-efd489c89b77" containerID="b6d0a4d1f809c1a229cfc318db71cfbfe29ec80bb5a7e34727d3e357bafa3f9d" exitCode=7
Jul 16 23:12:19.264041 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:19.263940    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-q7mqr" event={"ID":"de89d05d-2130-43bc-93d1-efd489c89b77","Type":"ContainerDied","Data":"b6d0a4d1f809c1a229cfc318db71cfbfe29ec80bb5a7e34727d3e357bafa3f9d"}
Jul 16 23:12:19.264148 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:19.264134    2573 scope.go:117] "RemoveContainer" containerID="b6d0a4d1f809c1a229cfc318db71cfbfe29ec80bb5a7e34727d3e357bafa3f9d"
Jul 16 23:12:19.292091 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:19.292067    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-q8svj"]
Jul 16 23:12:19.297782 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:19.297760    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-q8svj"]
Jul 16 23:12:20.270763 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:20.270687    2573 generic.go:356] "Generic (PLEG): container finished" podID="de89d05d-2130-43bc-93d1-efd489c89b77" containerID="8af829940c64280762b3b6e61da287dff73ababfa4935cdcac37361e977036a2" exitCode=7
Jul 16 23:12:20.271127 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:20.270772    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-q7mqr" event={"ID":"de89d05d-2130-43bc-93d1-efd489c89b77","Type":"ContainerDied","Data":"8af829940c64280762b3b6e61da287dff73ababfa4935cdcac37361e977036a2"}
Jul 16 23:12:20.271127 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:20.270839    2573 scope.go:117] "RemoveContainer" containerID="b6d0a4d1f809c1a229cfc318db71cfbfe29ec80bb5a7e34727d3e357bafa3f9d"
Jul 16 23:12:20.271127 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:20.271044    2573 scope.go:117] "RemoveContainer" containerID="8af829940c64280762b3b6e61da287dff73ababfa4935cdcac37361e977036a2"
Jul 16 23:12:20.271292 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:12:20.271215    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-q7mqr_odh-ai-gateway-infra(de89d05d-2130-43bc-93d1-efd489c89b77)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-q7mqr" podUID="de89d05d-2130-43bc-93d1-efd489c89b77"
Jul 16 23:12:20.473530 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:20.473506    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-n4cl5"]
Jul 16 23:12:20.473798 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:20.473785    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="526b1e3e-b375-4d05-9e85-542c944af91d" containerName="revoke-keys"
Jul 16 23:12:20.473840 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:20.473800    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="526b1e3e-b375-4d05-9e85-542c944af91d" containerName="revoke-keys"
Jul 16 23:12:20.473840 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:20.473811    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="526b1e3e-b375-4d05-9e85-542c944af91d" containerName="revoke-keys"
Jul 16 23:12:20.473840 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:20.473817    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="526b1e3e-b375-4d05-9e85-542c944af91d" containerName="revoke-keys"
Jul 16 23:12:20.473840 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:20.473825    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="526b1e3e-b375-4d05-9e85-542c944af91d" containerName="revoke-keys"
Jul 16 23:12:20.473840 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:20.473831    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="526b1e3e-b375-4d05-9e85-542c944af91d" containerName="revoke-keys"
Jul 16 23:12:20.473986 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:20.473884    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="526b1e3e-b375-4d05-9e85-542c944af91d" containerName="revoke-keys"
Jul 16 23:12:20.473986 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:20.473894    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="526b1e3e-b375-4d05-9e85-542c944af91d" containerName="revoke-keys"
Jul 16 23:12:20.477580 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:20.477566    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-n4cl5"
Jul 16 23:12:20.487526 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:20.487498    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-n4cl5"]
Jul 16 23:12:20.519193 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:20.519172    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="526b1e3e-b375-4d05-9e85-542c944af91d" path="/var/lib/kubelet/pods/526b1e3e-b375-4d05-9e85-542c944af91d/volumes"
Jul 16 23:12:20.557612 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:20.557562    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd423e1b-d7ed-4c1e-be18-4cc95a2e896a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-n4cl5\" (UID: \"cd423e1b-d7ed-4c1e-be18-4cc95a2e896a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-n4cl5"
Jul 16 23:12:20.658546 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:20.658522    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd423e1b-d7ed-4c1e-be18-4cc95a2e896a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-n4cl5\" (UID: \"cd423e1b-d7ed-4c1e-be18-4cc95a2e896a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-n4cl5"
Jul 16 23:12:20.659123 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:20.659104    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd423e1b-d7ed-4c1e-be18-4cc95a2e896a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-n4cl5\" (UID: \"cd423e1b-d7ed-4c1e-be18-4cc95a2e896a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-n4cl5"
Jul 16 23:12:20.794755 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:20.794727    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-n4cl5"
Jul 16 23:12:20.919981 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:20.919956    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-n4cl5"]
Jul 16 23:12:20.922118 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:12:20.922091    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd423e1b_d7ed_4c1e_be18_4cc95a2e896a.slice/crio-c5db590e0c66ef1b8e1cea9b756978327c7c54423e9fbdc92c799b741f00500b WatchSource:0}: Error finding container c5db590e0c66ef1b8e1cea9b756978327c7c54423e9fbdc92c799b741f00500b: Status 404 returned error can't find the container with id c5db590e0c66ef1b8e1cea9b756978327c7c54423e9fbdc92c799b741f00500b
Jul 16 23:12:21.277969 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:21.277941    2573 scope.go:117] "RemoveContainer" containerID="8af829940c64280762b3b6e61da287dff73ababfa4935cdcac37361e977036a2"
Jul 16 23:12:21.278379 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:12:21.278146    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-q7mqr_odh-ai-gateway-infra(de89d05d-2130-43bc-93d1-efd489c89b77)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-q7mqr" podUID="de89d05d-2130-43bc-93d1-efd489c89b77"
Jul 16 23:12:21.279353 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:21.279321    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-n4cl5" event={"ID":"cd423e1b-d7ed-4c1e-be18-4cc95a2e896a","Type":"ContainerStarted","Data":"2cfb5e71e8bb54be84baa4164768a0ee6b9e51f59848137e43955cb031c64ef1"}
Jul 16 23:12:21.279455 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:21.279365    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-n4cl5" event={"ID":"cd423e1b-d7ed-4c1e-be18-4cc95a2e896a","Type":"ContainerStarted","Data":"c5db590e0c66ef1b8e1cea9b756978327c7c54423e9fbdc92c799b741f00500b"}
Jul 16 23:12:21.304472 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:21.304426    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-n4cl5" podStartSLOduration=1.30439276 podStartE2EDuration="1.30439276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:12:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:12:21.304380139 +0000 UTC m=+2315.314642834" watchObservedRunningTime="2026-07-16 23:12:21.30439276 +0000 UTC m=+2315.314655425"
Jul 16 23:12:22.284911 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:22.284882    2573 generic.go:356] "Generic (PLEG): container finished" podID="cd423e1b-d7ed-4c1e-be18-4cc95a2e896a" containerID="2cfb5e71e8bb54be84baa4164768a0ee6b9e51f59848137e43955cb031c64ef1" exitCode=6
Jul 16 23:12:22.285284 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:22.284979    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-n4cl5" event={"ID":"cd423e1b-d7ed-4c1e-be18-4cc95a2e896a","Type":"ContainerDied","Data":"2cfb5e71e8bb54be84baa4164768a0ee6b9e51f59848137e43955cb031c64ef1"}
Jul 16 23:12:22.285284 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:22.285120    2573 scope.go:117] "RemoveContainer" containerID="2cfb5e71e8bb54be84baa4164768a0ee6b9e51f59848137e43955cb031c64ef1"
Jul 16 23:12:22.719609 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:22.719578    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-82w7k"]
Jul 16 23:12:22.719930 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:22.719918    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="526b1e3e-b375-4d05-9e85-542c944af91d" containerName="revoke-keys"
Jul 16 23:12:22.722817 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:22.722802    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-82w7k"
Jul 16 23:12:22.735930 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:22.735907    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-82w7k"]
Jul 16 23:12:22.875491 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:22.875468    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6cb72d1-86e9-4294-bc63-d69b04df4d37-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-82w7k\" (UID: \"f6cb72d1-86e9-4294-bc63-d69b04df4d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-82w7k"
Jul 16 23:12:22.975790 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:22.975735    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6cb72d1-86e9-4294-bc63-d69b04df4d37-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-82w7k\" (UID: \"f6cb72d1-86e9-4294-bc63-d69b04df4d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-82w7k"
Jul 16 23:12:22.976319 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:22.976302    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6cb72d1-86e9-4294-bc63-d69b04df4d37-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-82w7k\" (UID: \"f6cb72d1-86e9-4294-bc63-d69b04df4d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-82w7k"
Jul 16 23:12:23.031880 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:23.031851    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-82w7k"
Jul 16 23:12:23.152353 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:23.152330    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-82w7k"]
Jul 16 23:12:23.154003 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:12:23.153976    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6cb72d1_86e9_4294_bc63_d69b04df4d37.slice/crio-0a72fc4acb50ee274251016d28b43b0602d310172a10781ecce805f6afa05fc9 WatchSource:0}: Error finding container 0a72fc4acb50ee274251016d28b43b0602d310172a10781ecce805f6afa05fc9: Status 404 returned error can't find the container with id 0a72fc4acb50ee274251016d28b43b0602d310172a10781ecce805f6afa05fc9
Jul 16 23:12:23.291333 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:23.291256    2573 generic.go:356] "Generic (PLEG): container finished" podID="cd423e1b-d7ed-4c1e-be18-4cc95a2e896a" containerID="996a23477468abff83eab7b1d86a123ef782fb646529de68235d4c95ffd1f487" exitCode=6
Jul 16 23:12:23.291718 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:23.291341    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-n4cl5" event={"ID":"cd423e1b-d7ed-4c1e-be18-4cc95a2e896a","Type":"ContainerDied","Data":"996a23477468abff83eab7b1d86a123ef782fb646529de68235d4c95ffd1f487"}
Jul 16 23:12:23.291718 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:23.291394    2573 scope.go:117] "RemoveContainer" containerID="2cfb5e71e8bb54be84baa4164768a0ee6b9e51f59848137e43955cb031c64ef1"
Jul 16 23:12:23.291718 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:23.291546    2573 scope.go:117] "RemoveContainer" containerID="996a23477468abff83eab7b1d86a123ef782fb646529de68235d4c95ffd1f487"
Jul 16 23:12:23.291919 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:12:23.291743    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-n4cl5_odh-ai-gateway-infra(cd423e1b-d7ed-4c1e-be18-4cc95a2e896a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-n4cl5" podUID="cd423e1b-d7ed-4c1e-be18-4cc95a2e896a"
Jul 16 23:12:23.292563 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:23.292543    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-82w7k" event={"ID":"f6cb72d1-86e9-4294-bc63-d69b04df4d37","Type":"ContainerStarted","Data":"0a72fc4acb50ee274251016d28b43b0602d310172a10781ecce805f6afa05fc9"}
Jul 16 23:12:24.304709 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:24.304683    2573 scope.go:117] "RemoveContainer" containerID="996a23477468abff83eab7b1d86a123ef782fb646529de68235d4c95ffd1f487"
Jul 16 23:12:24.305108 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:12:24.304853    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-n4cl5_odh-ai-gateway-infra(cd423e1b-d7ed-4c1e-be18-4cc95a2e896a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-n4cl5" podUID="cd423e1b-d7ed-4c1e-be18-4cc95a2e896a"
Jul 16 23:12:24.306027 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:24.305996    2573 generic.go:356] "Generic (PLEG): container finished" podID="f6cb72d1-86e9-4294-bc63-d69b04df4d37" containerID="3a407f7e301945ecc4b02f5f64bcbdc64934e7c327b4275fe0502439e8e314db" exitCode=6
Jul 16 23:12:24.306136 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:24.306045    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-82w7k" event={"ID":"f6cb72d1-86e9-4294-bc63-d69b04df4d37","Type":"ContainerDied","Data":"3a407f7e301945ecc4b02f5f64bcbdc64934e7c327b4275fe0502439e8e314db"}
Jul 16 23:12:24.306181 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:24.306152    2573 scope.go:117] "RemoveContainer" containerID="3a407f7e301945ecc4b02f5f64bcbdc64934e7c327b4275fe0502439e8e314db"
Jul 16 23:12:25.312301 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:25.312265    2573 generic.go:356] "Generic (PLEG): container finished" podID="f6cb72d1-86e9-4294-bc63-d69b04df4d37" containerID="76df14c6b4f026ce2ac024dbf5b4fa74a5c3358ac4e915d2d0854106a42abb1c" exitCode=6
Jul 16 23:12:25.312636 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:25.312346    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-82w7k" event={"ID":"f6cb72d1-86e9-4294-bc63-d69b04df4d37","Type":"ContainerDied","Data":"76df14c6b4f026ce2ac024dbf5b4fa74a5c3358ac4e915d2d0854106a42abb1c"}
Jul 16 23:12:25.312636 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:25.312412    2573 scope.go:117] "RemoveContainer" containerID="3a407f7e301945ecc4b02f5f64bcbdc64934e7c327b4275fe0502439e8e314db"
Jul 16 23:12:25.312636 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:25.312582    2573 scope.go:117] "RemoveContainer" containerID="76df14c6b4f026ce2ac024dbf5b4fa74a5c3358ac4e915d2d0854106a42abb1c"
Jul 16 23:12:25.312772 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:12:25.312750    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-82w7k_odh-ai-gateway-infra(f6cb72d1-86e9-4294-bc63-d69b04df4d37)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-82w7k" podUID="f6cb72d1-86e9-4294-bc63-d69b04df4d37"
Jul 16 23:12:26.319127 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:26.319052    2573 scope.go:117] "RemoveContainer" containerID="76df14c6b4f026ce2ac024dbf5b4fa74a5c3358ac4e915d2d0854106a42abb1c"
Jul 16 23:12:26.319471 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:12:26.319276    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-82w7k_odh-ai-gateway-infra(f6cb72d1-86e9-4294-bc63-d69b04df4d37)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-82w7k" podUID="f6cb72d1-86e9-4294-bc63-d69b04df4d37"
Jul 16 23:12:31.867556 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:31.867523    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:12:31.969561 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:31.969529    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:12:35.513455 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:35.513418    2573 scope.go:117] "RemoveContainer" containerID="996a23477468abff83eab7b1d86a123ef782fb646529de68235d4c95ffd1f487"
Jul 16 23:12:35.513881 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:35.513514    2573 scope.go:117] "RemoveContainer" containerID="8af829940c64280762b3b6e61da287dff73ababfa4935cdcac37361e977036a2"
Jul 16 23:12:36.369950 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:36.369888    2573 generic.go:356] "Generic (PLEG): container finished" podID="de89d05d-2130-43bc-93d1-efd489c89b77" containerID="c715154300a4498c625d74804494e3c467e85341bd063aebce0ff39301022974" exitCode=7
Jul 16 23:12:36.370061 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:36.369965    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-q7mqr" event={"ID":"de89d05d-2130-43bc-93d1-efd489c89b77","Type":"ContainerDied","Data":"c715154300a4498c625d74804494e3c467e85341bd063aebce0ff39301022974"}
Jul 16 23:12:36.370061 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:36.370020    2573 scope.go:117] "RemoveContainer" containerID="8af829940c64280762b3b6e61da287dff73ababfa4935cdcac37361e977036a2"
Jul 16 23:12:36.370337 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:36.370318    2573 scope.go:117] "RemoveContainer" containerID="c715154300a4498c625d74804494e3c467e85341bd063aebce0ff39301022974"
Jul 16 23:12:36.370542 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:12:36.370515    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-q7mqr_odh-ai-gateway-infra(de89d05d-2130-43bc-93d1-efd489c89b77)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-q7mqr" podUID="de89d05d-2130-43bc-93d1-efd489c89b77"
Jul 16 23:12:36.371944 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:36.371926    2573 generic.go:356] "Generic (PLEG): container finished" podID="cd423e1b-d7ed-4c1e-be18-4cc95a2e896a" containerID="91116ecdc83cdfae53b9339ee7efa6431e3a0b1e7380c00b72aa528da885e3f9" exitCode=6
Jul 16 23:12:36.372024 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:36.371952    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-n4cl5" event={"ID":"cd423e1b-d7ed-4c1e-be18-4cc95a2e896a","Type":"ContainerDied","Data":"91116ecdc83cdfae53b9339ee7efa6431e3a0b1e7380c00b72aa528da885e3f9"}
Jul 16 23:12:36.372173 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:36.372161    2573 scope.go:117] "RemoveContainer" containerID="91116ecdc83cdfae53b9339ee7efa6431e3a0b1e7380c00b72aa528da885e3f9"
Jul 16 23:12:36.372329 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:12:36.372313    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-n4cl5_odh-ai-gateway-infra(cd423e1b-d7ed-4c1e-be18-4cc95a2e896a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-n4cl5" podUID="cd423e1b-d7ed-4c1e-be18-4cc95a2e896a"
Jul 16 23:12:36.380370 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:36.380351    2573 scope.go:117] "RemoveContainer" containerID="996a23477468abff83eab7b1d86a123ef782fb646529de68235d4c95ffd1f487"
Jul 16 23:12:36.557776 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:36.557753    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:12:37.393898 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:37.393866    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-q7mqr"]
Jul 16 23:12:37.406659 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:37.406630    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-n4cl5"]
Jul 16 23:12:37.524875 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:37.524849    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-q7mqr"
Jul 16 23:12:37.566722 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:37.566703    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-n4cl5"
Jul 16 23:12:37.677085 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:37.677002    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de89d05d-2130-43bc-93d1-efd489c89b77-maas-api-service-ca\") pod \"de89d05d-2130-43bc-93d1-efd489c89b77\" (UID: \"de89d05d-2130-43bc-93d1-efd489c89b77\") "
Jul 16 23:12:37.677085 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:37.677075    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd423e1b-d7ed-4c1e-be18-4cc95a2e896a-maas-api-service-ca\") pod \"cd423e1b-d7ed-4c1e-be18-4cc95a2e896a\" (UID: \"cd423e1b-d7ed-4c1e-be18-4cc95a2e896a\") "
Jul 16 23:12:37.677361 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:37.677340    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de89d05d-2130-43bc-93d1-efd489c89b77-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "de89d05d-2130-43bc-93d1-efd489c89b77" (UID: "de89d05d-2130-43bc-93d1-efd489c89b77"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:12:37.677401 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:37.677383    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd423e1b-d7ed-4c1e-be18-4cc95a2e896a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cd423e1b-d7ed-4c1e-be18-4cc95a2e896a" (UID: "cd423e1b-d7ed-4c1e-be18-4cc95a2e896a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:12:37.777929 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:37.777888    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de89d05d-2130-43bc-93d1-efd489c89b77-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:12:37.777929 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:37.777923    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd423e1b-d7ed-4c1e-be18-4cc95a2e896a-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:12:38.385997 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:38.385963    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-q7mqr"
Jul 16 23:12:38.386153 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:38.386014    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-q7mqr" event={"ID":"de89d05d-2130-43bc-93d1-efd489c89b77","Type":"ContainerDied","Data":"dd9dc66f5ac4a430a65a895b7cb455be76b10226b467b6ad7056e5e9437dfafc"}
Jul 16 23:12:38.386153 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:38.386107    2573 scope.go:117] "RemoveContainer" containerID="c715154300a4498c625d74804494e3c467e85341bd063aebce0ff39301022974"
Jul 16 23:12:38.387375 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:38.387349    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-n4cl5"
Jul 16 23:12:38.387375 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:38.387355    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-n4cl5" event={"ID":"cd423e1b-d7ed-4c1e-be18-4cc95a2e896a","Type":"ContainerDied","Data":"c5db590e0c66ef1b8e1cea9b756978327c7c54423e9fbdc92c799b741f00500b"}
Jul 16 23:12:38.397154 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:38.397041    2573 scope.go:117] "RemoveContainer" containerID="91116ecdc83cdfae53b9339ee7efa6431e3a0b1e7380c00b72aa528da885e3f9"
Jul 16 23:12:38.410832 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:38.410812    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-q7mqr"]
Jul 16 23:12:38.415306 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:38.415287    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-q7mqr"]
Jul 16 23:12:38.424548 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:38.424529    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-n4cl5"]
Jul 16 23:12:38.426533 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:38.426509    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-n4cl5"]
Jul 16 23:12:38.518613 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:38.518592    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd423e1b-d7ed-4c1e-be18-4cc95a2e896a" path="/var/lib/kubelet/pods/cd423e1b-d7ed-4c1e-be18-4cc95a2e896a/volumes"
Jul 16 23:12:38.518890 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:38.518878    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de89d05d-2130-43bc-93d1-efd489c89b77" path="/var/lib/kubelet/pods/de89d05d-2130-43bc-93d1-efd489c89b77/volumes"
Jul 16 23:12:40.513747 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:40.513721    2573 scope.go:117] "RemoveContainer" containerID="76df14c6b4f026ce2ac024dbf5b4fa74a5c3358ac4e915d2d0854106a42abb1c"
Jul 16 23:12:41.406305 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:41.405973    2573 generic.go:356] "Generic (PLEG): container finished" podID="f6cb72d1-86e9-4294-bc63-d69b04df4d37" containerID="e08402906bce2ecd0eed678bd846c1c036efbb1ef830b4d51ae6a5115a9a3080" exitCode=6
Jul 16 23:12:41.406305 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:41.406126    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-82w7k" event={"ID":"f6cb72d1-86e9-4294-bc63-d69b04df4d37","Type":"ContainerDied","Data":"e08402906bce2ecd0eed678bd846c1c036efbb1ef830b4d51ae6a5115a9a3080"}
Jul 16 23:12:41.406305 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:41.406178    2573 scope.go:117] "RemoveContainer" containerID="76df14c6b4f026ce2ac024dbf5b4fa74a5c3358ac4e915d2d0854106a42abb1c"
Jul 16 23:12:41.406597 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:41.406546    2573 scope.go:117] "RemoveContainer" containerID="e08402906bce2ecd0eed678bd846c1c036efbb1ef830b4d51ae6a5115a9a3080"
Jul 16 23:12:41.406783 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:12:41.406761    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-82w7k_odh-ai-gateway-infra(f6cb72d1-86e9-4294-bc63-d69b04df4d37)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-82w7k" podUID="f6cb72d1-86e9-4294-bc63-d69b04df4d37"
Jul 16 23:12:42.430534 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:42.430500    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-82w7k"]
Jul 16 23:12:42.556867 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:42.556845    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-82w7k"
Jul 16 23:12:42.714750 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:42.714676    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6cb72d1-86e9-4294-bc63-d69b04df4d37-maas-api-service-ca\") pod \"f6cb72d1-86e9-4294-bc63-d69b04df4d37\" (UID: \"f6cb72d1-86e9-4294-bc63-d69b04df4d37\") "
Jul 16 23:12:42.715008 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:42.714983    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6cb72d1-86e9-4294-bc63-d69b04df4d37-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f6cb72d1-86e9-4294-bc63-d69b04df4d37" (UID: "f6cb72d1-86e9-4294-bc63-d69b04df4d37"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:12:42.815734 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:42.815689    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6cb72d1-86e9-4294-bc63-d69b04df4d37-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:12:43.418102 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:43.418079    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-82w7k"
Jul 16 23:12:43.418294 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:43.418107    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-82w7k" event={"ID":"f6cb72d1-86e9-4294-bc63-d69b04df4d37","Type":"ContainerDied","Data":"0a72fc4acb50ee274251016d28b43b0602d310172a10781ecce805f6afa05fc9"}
Jul 16 23:12:43.418294 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:43.418166    2573 scope.go:117] "RemoveContainer" containerID="e08402906bce2ecd0eed678bd846c1c036efbb1ef830b4d51ae6a5115a9a3080"
Jul 16 23:12:43.437402 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:43.437380    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-82w7k"]
Jul 16 23:12:43.441547 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:43.441524    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-82w7k"]
Jul 16 23:12:44.518823 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:44.518791    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6cb72d1-86e9-4294-bc63-d69b04df4d37" path="/var/lib/kubelet/pods/f6cb72d1-86e9-4294-bc63-d69b04df4d37/volumes"
Jul 16 23:12:51.529676 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:51.529642    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-kzfs6"]
Jul 16 23:12:51.530103 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:51.529918    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6cb72d1-86e9-4294-bc63-d69b04df4d37" containerName="revoke-keys"
Jul 16 23:12:51.530103 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:51.529930    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6cb72d1-86e9-4294-bc63-d69b04df4d37" containerName="revoke-keys"
Jul 16 23:12:51.530103 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:51.529944    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6cb72d1-86e9-4294-bc63-d69b04df4d37" containerName="revoke-keys"
Jul 16 23:12:51.530103 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:51.529950    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6cb72d1-86e9-4294-bc63-d69b04df4d37" containerName="revoke-keys"
Jul 16 23:12:51.530103 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:51.529958    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de89d05d-2130-43bc-93d1-efd489c89b77" containerName="revoke-keys"
Jul 16 23:12:51.530103 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:51.529963    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="de89d05d-2130-43bc-93d1-efd489c89b77" containerName="revoke-keys"
Jul 16 23:12:51.530103 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:51.529972    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de89d05d-2130-43bc-93d1-efd489c89b77" containerName="revoke-keys"
Jul 16 23:12:51.530103 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:51.529977    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="de89d05d-2130-43bc-93d1-efd489c89b77" containerName="revoke-keys"
Jul 16 23:12:51.530103 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:51.529983    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd423e1b-d7ed-4c1e-be18-4cc95a2e896a" containerName="revoke-keys"
Jul 16 23:12:51.530103 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:51.529988    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd423e1b-d7ed-4c1e-be18-4cc95a2e896a" containerName="revoke-keys"
Jul 16 23:12:51.530103 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:51.529994    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de89d05d-2130-43bc-93d1-efd489c89b77" containerName="revoke-keys"
Jul 16 23:12:51.530103 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:51.530000    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="de89d05d-2130-43bc-93d1-efd489c89b77" containerName="revoke-keys"
Jul 16 23:12:51.530103 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:51.530010    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd423e1b-d7ed-4c1e-be18-4cc95a2e896a" containerName="revoke-keys"
Jul 16 23:12:51.530103 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:51.530015    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd423e1b-d7ed-4c1e-be18-4cc95a2e896a" containerName="revoke-keys"
Jul 16 23:12:51.530103 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:51.530022    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd423e1b-d7ed-4c1e-be18-4cc95a2e896a" containerName="revoke-keys"
Jul 16 23:12:51.530103 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:51.530027    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd423e1b-d7ed-4c1e-be18-4cc95a2e896a" containerName="revoke-keys"
Jul 16 23:12:51.530103 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:51.530073    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd423e1b-d7ed-4c1e-be18-4cc95a2e896a" containerName="revoke-keys"
Jul 16 23:12:51.530103 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:51.530079    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6cb72d1-86e9-4294-bc63-d69b04df4d37" containerName="revoke-keys"
Jul 16 23:12:51.530103 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:51.530084    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd423e1b-d7ed-4c1e-be18-4cc95a2e896a" containerName="revoke-keys"
Jul 16 23:12:51.530103 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:51.530090    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd423e1b-d7ed-4c1e-be18-4cc95a2e896a" containerName="revoke-keys"
Jul 16 23:12:51.530103 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:51.530095    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="de89d05d-2130-43bc-93d1-efd489c89b77" containerName="revoke-keys"
Jul 16 23:12:51.530103 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:51.530101    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="de89d05d-2130-43bc-93d1-efd489c89b77" containerName="revoke-keys"
Jul 16 23:12:51.530103 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:51.530107    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6cb72d1-86e9-4294-bc63-d69b04df4d37" containerName="revoke-keys"
Jul 16 23:12:51.530103 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:51.530111    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6cb72d1-86e9-4294-bc63-d69b04df4d37" containerName="revoke-keys"
Jul 16 23:12:51.534623 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:51.534605    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-kzfs6"
Jul 16 23:12:51.537772 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:51.537377    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 23:12:51.537772 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:51.537607    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:12:51.539285 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:51.539261    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-kzfs6"]
Jul 16 23:12:51.580002 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:51.579966    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e5770bc-df89-4609-983a-2771ae9e02b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-kzfs6\" (UID: \"7e5770bc-df89-4609-983a-2771ae9e02b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-kzfs6"
Jul 16 23:12:51.680845 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:51.680817    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e5770bc-df89-4609-983a-2771ae9e02b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-kzfs6\" (UID: \"7e5770bc-df89-4609-983a-2771ae9e02b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-kzfs6"
Jul 16 23:12:51.681470 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:51.681452    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e5770bc-df89-4609-983a-2771ae9e02b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-kzfs6\" (UID: \"7e5770bc-df89-4609-983a-2771ae9e02b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-kzfs6"
Jul 16 23:12:51.846578 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:51.846556    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-kzfs6"
Jul 16 23:12:51.971138 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:51.971116    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-kzfs6"]
Jul 16 23:12:51.973787 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:12:51.973751    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e5770bc_df89_4609_983a_2771ae9e02b9.slice/crio-3cab0c5bed7623d3e83aef7c08328c6a79492899cf76048374de716336597e7f WatchSource:0}: Error finding container 3cab0c5bed7623d3e83aef7c08328c6a79492899cf76048374de716336597e7f: Status 404 returned error can't find the container with id 3cab0c5bed7623d3e83aef7c08328c6a79492899cf76048374de716336597e7f
Jul 16 23:12:52.464636 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:52.464549    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-kzfs6" event={"ID":"7e5770bc-df89-4609-983a-2771ae9e02b9","Type":"ContainerStarted","Data":"034e4803b296c7b3f837f6063fae8494d275f51f556c4b55554a10326ddf4c45"}
Jul 16 23:12:52.464636 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:52.464597    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-kzfs6" event={"ID":"7e5770bc-df89-4609-983a-2771ae9e02b9","Type":"ContainerStarted","Data":"3cab0c5bed7623d3e83aef7c08328c6a79492899cf76048374de716336597e7f"}
Jul 16 23:12:52.480338 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:52.480288    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-kzfs6" podStartSLOduration=1.480272093 podStartE2EDuration="1.480272093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:12:52.478888801 +0000 UTC m=+2346.489151465" watchObservedRunningTime="2026-07-16 23:12:52.480272093 +0000 UTC m=+2346.490534754"
Jul 16 23:12:53.470306 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:53.470273    2573 generic.go:356] "Generic (PLEG): container finished" podID="7e5770bc-df89-4609-983a-2771ae9e02b9" containerID="034e4803b296c7b3f837f6063fae8494d275f51f556c4b55554a10326ddf4c45" exitCode=6
Jul 16 23:12:53.470676 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:53.470332    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-kzfs6" event={"ID":"7e5770bc-df89-4609-983a-2771ae9e02b9","Type":"ContainerDied","Data":"034e4803b296c7b3f837f6063fae8494d275f51f556c4b55554a10326ddf4c45"}
Jul 16 23:12:53.470676 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:53.470506    2573 scope.go:117] "RemoveContainer" containerID="034e4803b296c7b3f837f6063fae8494d275f51f556c4b55554a10326ddf4c45"
Jul 16 23:12:54.476628 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:54.476550    2573 generic.go:356] "Generic (PLEG): container finished" podID="7e5770bc-df89-4609-983a-2771ae9e02b9" containerID="ce657b086429adb621993f35ed11124693d1bfff9ccc00a6efe61ef9313a7bb0" exitCode=6
Jul 16 23:12:54.476998 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:54.476634    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-kzfs6" event={"ID":"7e5770bc-df89-4609-983a-2771ae9e02b9","Type":"ContainerDied","Data":"ce657b086429adb621993f35ed11124693d1bfff9ccc00a6efe61ef9313a7bb0"}
Jul 16 23:12:54.476998 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:54.476723    2573 scope.go:117] "RemoveContainer" containerID="034e4803b296c7b3f837f6063fae8494d275f51f556c4b55554a10326ddf4c45"
Jul 16 23:12:54.476998 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:54.476782    2573 scope.go:117] "RemoveContainer" containerID="ce657b086429adb621993f35ed11124693d1bfff9ccc00a6efe61ef9313a7bb0"
Jul 16 23:12:54.476998 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:12:54.476954    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-kzfs6_odh-ai-gateway-infra(7e5770bc-df89-4609-983a-2771ae9e02b9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-kzfs6" podUID="7e5770bc-df89-4609-983a-2771ae9e02b9"
Jul 16 23:12:55.482846 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:12:55.482817    2573 scope.go:117] "RemoveContainer" containerID="ce657b086429adb621993f35ed11124693d1bfff9ccc00a6efe61ef9313a7bb0"
Jul 16 23:12:55.483268 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:12:55.482999    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-kzfs6_odh-ai-gateway-infra(7e5770bc-df89-4609-983a-2771ae9e02b9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-kzfs6" podUID="7e5770bc-df89-4609-983a-2771ae9e02b9"
Jul 16 23:13:08.518894 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:08.518863    2573 scope.go:117] "RemoveContainer" containerID="ce657b086429adb621993f35ed11124693d1bfff9ccc00a6efe61ef9313a7bb0"
Jul 16 23:13:09.551871 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:09.551799    2573 generic.go:356] "Generic (PLEG): container finished" podID="7e5770bc-df89-4609-983a-2771ae9e02b9" containerID="70cf6b457d49cc1a72b113281edc149a25ffe7dfe800a8c96d783a1193f8ac39" exitCode=6
Jul 16 23:13:09.552229 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:09.551872    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-kzfs6" event={"ID":"7e5770bc-df89-4609-983a-2771ae9e02b9","Type":"ContainerDied","Data":"70cf6b457d49cc1a72b113281edc149a25ffe7dfe800a8c96d783a1193f8ac39"}
Jul 16 23:13:09.552229 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:09.551937    2573 scope.go:117] "RemoveContainer" containerID="ce657b086429adb621993f35ed11124693d1bfff9ccc00a6efe61ef9313a7bb0"
Jul 16 23:13:09.552229 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:09.552200    2573 scope.go:117] "RemoveContainer" containerID="70cf6b457d49cc1a72b113281edc149a25ffe7dfe800a8c96d783a1193f8ac39"
Jul 16 23:13:09.552434 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:13:09.552416    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-kzfs6_odh-ai-gateway-infra(7e5770bc-df89-4609-983a-2771ae9e02b9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-kzfs6" podUID="7e5770bc-df89-4609-983a-2771ae9e02b9"
Jul 16 23:13:10.508505 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:10.508475    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-dcdvp"]
Jul 16 23:13:10.508775 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:10.508763    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6cb72d1-86e9-4294-bc63-d69b04df4d37" containerName="revoke-keys"
Jul 16 23:13:10.508827 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:10.508776    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6cb72d1-86e9-4294-bc63-d69b04df4d37" containerName="revoke-keys"
Jul 16 23:13:10.508863 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:10.508834    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="de89d05d-2130-43bc-93d1-efd489c89b77" containerName="revoke-keys"
Jul 16 23:13:10.512839 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:10.512821    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-dcdvp"
Jul 16 23:13:10.520459 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:10.520435    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-dcdvp"]
Jul 16 23:13:10.577562 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:10.577538    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-kzfs6"]
Jul 16 23:13:10.610725 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:10.610693    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/feaa5af4-f5ba-4ff9-9bd1-c2ea56c31b73-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-dcdvp\" (UID: \"feaa5af4-f5ba-4ff9-9bd1-c2ea56c31b73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-dcdvp"
Jul 16 23:13:10.716263 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:10.712998    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/feaa5af4-f5ba-4ff9-9bd1-c2ea56c31b73-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-dcdvp\" (UID: \"feaa5af4-f5ba-4ff9-9bd1-c2ea56c31b73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-dcdvp"
Jul 16 23:13:10.716263 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:10.713780    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/feaa5af4-f5ba-4ff9-9bd1-c2ea56c31b73-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-dcdvp\" (UID: \"feaa5af4-f5ba-4ff9-9bd1-c2ea56c31b73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-dcdvp"
Jul 16 23:13:10.726480 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:10.726453    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-kzfs6"
Jul 16 23:13:10.815949 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:10.815861    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e5770bc-df89-4609-983a-2771ae9e02b9-maas-api-service-ca\") pod \"7e5770bc-df89-4609-983a-2771ae9e02b9\" (UID: \"7e5770bc-df89-4609-983a-2771ae9e02b9\") "
Jul 16 23:13:10.816166 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:10.816146    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e5770bc-df89-4609-983a-2771ae9e02b9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7e5770bc-df89-4609-983a-2771ae9e02b9" (UID: "7e5770bc-df89-4609-983a-2771ae9e02b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:13:10.824999 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:10.824978    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-dcdvp"
Jul 16 23:13:10.916753 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:10.916723    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e5770bc-df89-4609-983a-2771ae9e02b9-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:13:10.949844 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:10.949818    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-dcdvp"]
Jul 16 23:13:10.952039 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:13:10.952012    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeaa5af4_f5ba_4ff9_9bd1_c2ea56c31b73.slice/crio-32ba10b522884d71fafc7a6575f92e99c5a4d2a689aa0c42ed93509723cdbe62 WatchSource:0}: Error finding container 32ba10b522884d71fafc7a6575f92e99c5a4d2a689aa0c42ed93509723cdbe62: Status 404 returned error can't find the container with id 32ba10b522884d71fafc7a6575f92e99c5a4d2a689aa0c42ed93509723cdbe62
Jul 16 23:13:11.563528 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:11.563456    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-kzfs6"
Jul 16 23:13:11.563528 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:11.563462    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-kzfs6" event={"ID":"7e5770bc-df89-4609-983a-2771ae9e02b9","Type":"ContainerDied","Data":"3cab0c5bed7623d3e83aef7c08328c6a79492899cf76048374de716336597e7f"}
Jul 16 23:13:11.563528 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:11.563515    2573 scope.go:117] "RemoveContainer" containerID="70cf6b457d49cc1a72b113281edc149a25ffe7dfe800a8c96d783a1193f8ac39"
Jul 16 23:13:11.565077 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:11.565045    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-dcdvp" event={"ID":"feaa5af4-f5ba-4ff9-9bd1-c2ea56c31b73","Type":"ContainerStarted","Data":"899b5ef728ede56c6008fa4b17d65719292f2723672c7472a5f529048259e68b"}
Jul 16 23:13:11.565171 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:11.565099    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-dcdvp" event={"ID":"feaa5af4-f5ba-4ff9-9bd1-c2ea56c31b73","Type":"ContainerStarted","Data":"32ba10b522884d71fafc7a6575f92e99c5a4d2a689aa0c42ed93509723cdbe62"}
Jul 16 23:13:11.581805 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:11.581759    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-dcdvp" podStartSLOduration=1.581745348 podStartE2EDuration="1.581745348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:13:11.580285734 +0000 UTC m=+2365.590548424" watchObservedRunningTime="2026-07-16 23:13:11.581745348 +0000 UTC m=+2365.592008046"
Jul 16 23:13:11.595740 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:11.595720    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-kzfs6"]
Jul 16 23:13:11.598206 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:11.598187    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-kzfs6"]
Jul 16 23:13:12.518219 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:12.518191    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e5770bc-df89-4609-983a-2771ae9e02b9" path="/var/lib/kubelet/pods/7e5770bc-df89-4609-983a-2771ae9e02b9/volumes"
Jul 16 23:13:12.572399 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:12.572368    2573 generic.go:356] "Generic (PLEG): container finished" podID="feaa5af4-f5ba-4ff9-9bd1-c2ea56c31b73" containerID="899b5ef728ede56c6008fa4b17d65719292f2723672c7472a5f529048259e68b" exitCode=6
Jul 16 23:13:12.572540 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:12.572449    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-dcdvp" event={"ID":"feaa5af4-f5ba-4ff9-9bd1-c2ea56c31b73","Type":"ContainerDied","Data":"899b5ef728ede56c6008fa4b17d65719292f2723672c7472a5f529048259e68b"}
Jul 16 23:13:12.572591 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:12.572559    2573 scope.go:117] "RemoveContainer" containerID="899b5ef728ede56c6008fa4b17d65719292f2723672c7472a5f529048259e68b"
Jul 16 23:13:13.578758 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:13.578723    2573 generic.go:356] "Generic (PLEG): container finished" podID="feaa5af4-f5ba-4ff9-9bd1-c2ea56c31b73" containerID="8419a312c16a048b1c2b83bf75b43e4e6eb7f23a36950a50011bf0bca1f9add6" exitCode=6
Jul 16 23:13:13.579163 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:13.578772    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-dcdvp" event={"ID":"feaa5af4-f5ba-4ff9-9bd1-c2ea56c31b73","Type":"ContainerDied","Data":"8419a312c16a048b1c2b83bf75b43e4e6eb7f23a36950a50011bf0bca1f9add6"}
Jul 16 23:13:13.579163 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:13.578829    2573 scope.go:117] "RemoveContainer" containerID="899b5ef728ede56c6008fa4b17d65719292f2723672c7472a5f529048259e68b"
Jul 16 23:13:13.579163 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:13.579012    2573 scope.go:117] "RemoveContainer" containerID="8419a312c16a048b1c2b83bf75b43e4e6eb7f23a36950a50011bf0bca1f9add6"
Jul 16 23:13:13.579303 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:13:13.579222    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-dcdvp_odh-ai-gateway-infra(feaa5af4-f5ba-4ff9-9bd1-c2ea56c31b73)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-dcdvp" podUID="feaa5af4-f5ba-4ff9-9bd1-c2ea56c31b73"
Jul 16 23:13:14.585074 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:14.585050    2573 scope.go:117] "RemoveContainer" containerID="8419a312c16a048b1c2b83bf75b43e4e6eb7f23a36950a50011bf0bca1f9add6"
Jul 16 23:13:14.585457 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:13:14.585269    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-dcdvp_odh-ai-gateway-infra(feaa5af4-f5ba-4ff9-9bd1-c2ea56c31b73)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-dcdvp" podUID="feaa5af4-f5ba-4ff9-9bd1-c2ea56c31b73"
Jul 16 23:13:17.685325 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:17.685291    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-t4pj8"]
Jul 16 23:13:17.685660 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:17.685584    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e5770bc-df89-4609-983a-2771ae9e02b9" containerName="revoke-keys"
Jul 16 23:13:17.685660 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:17.685594    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e5770bc-df89-4609-983a-2771ae9e02b9" containerName="revoke-keys"
Jul 16 23:13:17.685660 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:17.685604    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e5770bc-df89-4609-983a-2771ae9e02b9" containerName="revoke-keys"
Jul 16 23:13:17.685660 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:17.685610    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e5770bc-df89-4609-983a-2771ae9e02b9" containerName="revoke-keys"
Jul 16 23:13:17.685804 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:17.685665    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e5770bc-df89-4609-983a-2771ae9e02b9" containerName="revoke-keys"
Jul 16 23:13:17.685804 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:17.685673    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e5770bc-df89-4609-983a-2771ae9e02b9" containerName="revoke-keys"
Jul 16 23:13:17.689597 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:17.689582    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-t4pj8"
Jul 16 23:13:17.695864 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:17.695834    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-t4pj8"]
Jul 16 23:13:17.763926 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:17.763900    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae858e88-dafd-4bb6-87c9-04342bdde9ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-t4pj8\" (UID: \"ae858e88-dafd-4bb6-87c9-04342bdde9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-t4pj8"
Jul 16 23:13:17.864390 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:17.864366    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae858e88-dafd-4bb6-87c9-04342bdde9ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-t4pj8\" (UID: \"ae858e88-dafd-4bb6-87c9-04342bdde9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-t4pj8"
Jul 16 23:13:17.864863 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:17.864848    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae858e88-dafd-4bb6-87c9-04342bdde9ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-t4pj8\" (UID: \"ae858e88-dafd-4bb6-87c9-04342bdde9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-t4pj8"
Jul 16 23:13:18.002224 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:18.002171    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-t4pj8"
Jul 16 23:13:18.333931 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:18.333901    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-t4pj8"]
Jul 16 23:13:18.335407 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:13:18.335379    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae858e88_dafd_4bb6_87c9_04342bdde9ed.slice/crio-318d1216047798ffe590cd1764e9ce43d47d18d5102769228f29a92f9489f1da WatchSource:0}: Error finding container 318d1216047798ffe590cd1764e9ce43d47d18d5102769228f29a92f9489f1da: Status 404 returned error can't find the container with id 318d1216047798ffe590cd1764e9ce43d47d18d5102769228f29a92f9489f1da
Jul 16 23:13:18.603271 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:18.603184    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-t4pj8" event={"ID":"ae858e88-dafd-4bb6-87c9-04342bdde9ed","Type":"ContainerStarted","Data":"7652fdd68f0ad77dd3cef90f7fd4eb414c65977782cac9f433aa7193f9c1e352"}
Jul 16 23:13:18.603271 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:18.603255    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-t4pj8" event={"ID":"ae858e88-dafd-4bb6-87c9-04342bdde9ed","Type":"ContainerStarted","Data":"318d1216047798ffe590cd1764e9ce43d47d18d5102769228f29a92f9489f1da"}
Jul 16 23:13:18.618626 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:18.618579    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-t4pj8" podStartSLOduration=1.618566181 podStartE2EDuration="1.618566181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:13:18.616766278 +0000 UTC m=+2372.627028976" watchObservedRunningTime="2026-07-16 23:13:18.618566181 +0000 UTC m=+2372.628828844"
Jul 16 23:13:19.609494 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:19.609458    2573 generic.go:356] "Generic (PLEG): container finished" podID="ae858e88-dafd-4bb6-87c9-04342bdde9ed" containerID="7652fdd68f0ad77dd3cef90f7fd4eb414c65977782cac9f433aa7193f9c1e352" exitCode=7
Jul 16 23:13:19.609882 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:19.609539    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-t4pj8" event={"ID":"ae858e88-dafd-4bb6-87c9-04342bdde9ed","Type":"ContainerDied","Data":"7652fdd68f0ad77dd3cef90f7fd4eb414c65977782cac9f433aa7193f9c1e352"}
Jul 16 23:13:19.609882 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:19.609698    2573 scope.go:117] "RemoveContainer" containerID="7652fdd68f0ad77dd3cef90f7fd4eb414c65977782cac9f433aa7193f9c1e352"
Jul 16 23:13:20.615977 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:20.615946    2573 generic.go:356] "Generic (PLEG): container finished" podID="ae858e88-dafd-4bb6-87c9-04342bdde9ed" containerID="76ceb8dc8d597c9a3a75533213fe2b91b9a85fd6690ae3d359af1c9a3e3ad885" exitCode=7
Jul 16 23:13:20.616394 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:20.615991    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-t4pj8" event={"ID":"ae858e88-dafd-4bb6-87c9-04342bdde9ed","Type":"ContainerDied","Data":"76ceb8dc8d597c9a3a75533213fe2b91b9a85fd6690ae3d359af1c9a3e3ad885"}
Jul 16 23:13:20.616394 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:20.616036    2573 scope.go:117] "RemoveContainer" containerID="7652fdd68f0ad77dd3cef90f7fd4eb414c65977782cac9f433aa7193f9c1e352"
Jul 16 23:13:20.616394 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:20.616167    2573 scope.go:117] "RemoveContainer" containerID="76ceb8dc8d597c9a3a75533213fe2b91b9a85fd6690ae3d359af1c9a3e3ad885"
Jul 16 23:13:20.616394 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:13:20.616368    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-t4pj8_odh-ai-gateway-infra(ae858e88-dafd-4bb6-87c9-04342bdde9ed)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-t4pj8" podUID="ae858e88-dafd-4bb6-87c9-04342bdde9ed"
Jul 16 23:13:21.622889 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:21.622865    2573 scope.go:117] "RemoveContainer" containerID="76ceb8dc8d597c9a3a75533213fe2b91b9a85fd6690ae3d359af1c9a3e3ad885"
Jul 16 23:13:21.623258 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:13:21.623035    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-t4pj8_odh-ai-gateway-infra(ae858e88-dafd-4bb6-87c9-04342bdde9ed)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-t4pj8" podUID="ae858e88-dafd-4bb6-87c9-04342bdde9ed"
Jul 16 23:13:22.737537 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:22.737498    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-4tfbc"]
Jul 16 23:13:22.737983 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:22.737970    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e5770bc-df89-4609-983a-2771ae9e02b9" containerName="revoke-keys"
Jul 16 23:13:22.738059 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:22.737989    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e5770bc-df89-4609-983a-2771ae9e02b9" containerName="revoke-keys"
Jul 16 23:13:22.738113 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:22.738072    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e5770bc-df89-4609-983a-2771ae9e02b9" containerName="revoke-keys"
Jul 16 23:13:22.740768 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:22.740747    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-4tfbc"
Jul 16 23:13:22.746993 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:22.746964    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-4tfbc"]
Jul 16 23:13:22.800617 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:22.800588    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea741db3-bc25-4ab2-9ff4-334cd6578223-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-4tfbc\" (UID: \"ea741db3-bc25-4ab2-9ff4-334cd6578223\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-4tfbc"
Jul 16 23:13:22.901150 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:22.901115    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea741db3-bc25-4ab2-9ff4-334cd6578223-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-4tfbc\" (UID: \"ea741db3-bc25-4ab2-9ff4-334cd6578223\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-4tfbc"
Jul 16 23:13:22.901777 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:22.901760    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea741db3-bc25-4ab2-9ff4-334cd6578223-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-4tfbc\" (UID: \"ea741db3-bc25-4ab2-9ff4-334cd6578223\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-4tfbc"
Jul 16 23:13:23.052517 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:23.052439    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-4tfbc"
Jul 16 23:13:23.175562 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:23.175536    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-4tfbc"]
Jul 16 23:13:23.177353 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:13:23.177326    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea741db3_bc25_4ab2_9ff4_334cd6578223.slice/crio-a80197639b5f78662cc885bc7c785fae97799d519a73ec58d23938da4d39e143 WatchSource:0}: Error finding container a80197639b5f78662cc885bc7c785fae97799d519a73ec58d23938da4d39e143: Status 404 returned error can't find the container with id a80197639b5f78662cc885bc7c785fae97799d519a73ec58d23938da4d39e143
Jul 16 23:13:23.635390 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:23.635350    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-4tfbc" event={"ID":"ea741db3-bc25-4ab2-9ff4-334cd6578223","Type":"ContainerStarted","Data":"3df8a481942c783a61b21cb177d4f30251a4e444b173c6bd1a6a68103b05f4f0"}
Jul 16 23:13:23.635586 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:23.635403    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-4tfbc" event={"ID":"ea741db3-bc25-4ab2-9ff4-334cd6578223","Type":"ContainerStarted","Data":"a80197639b5f78662cc885bc7c785fae97799d519a73ec58d23938da4d39e143"}
Jul 16 23:13:23.649675 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:23.649589    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-4tfbc" podStartSLOduration=1.649574562 podStartE2EDuration="1.649574562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:13:23.649101918 +0000 UTC m=+2377.659364973" watchObservedRunningTime="2026-07-16 23:13:23.649574562 +0000 UTC m=+2377.659837204"
Jul 16 23:13:24.641170 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:24.641133    2573 generic.go:356] "Generic (PLEG): container finished" podID="ea741db3-bc25-4ab2-9ff4-334cd6578223" containerID="3df8a481942c783a61b21cb177d4f30251a4e444b173c6bd1a6a68103b05f4f0" exitCode=6
Jul 16 23:13:24.641586 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:24.641195    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-4tfbc" event={"ID":"ea741db3-bc25-4ab2-9ff4-334cd6578223","Type":"ContainerDied","Data":"3df8a481942c783a61b21cb177d4f30251a4e444b173c6bd1a6a68103b05f4f0"}
Jul 16 23:13:24.641586 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:24.641365    2573 scope.go:117] "RemoveContainer" containerID="3df8a481942c783a61b21cb177d4f30251a4e444b173c6bd1a6a68103b05f4f0"
Jul 16 23:13:25.651639 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:25.651608    2573 generic.go:356] "Generic (PLEG): container finished" podID="ea741db3-bc25-4ab2-9ff4-334cd6578223" containerID="55ea65fb55e41cef01d0db5a5cc4cb23d11a1f8711f28fa83d4308f7046213da" exitCode=6
Jul 16 23:13:25.652035 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:25.651694    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-4tfbc" event={"ID":"ea741db3-bc25-4ab2-9ff4-334cd6578223","Type":"ContainerDied","Data":"55ea65fb55e41cef01d0db5a5cc4cb23d11a1f8711f28fa83d4308f7046213da"}
Jul 16 23:13:25.652035 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:25.651750    2573 scope.go:117] "RemoveContainer" containerID="3df8a481942c783a61b21cb177d4f30251a4e444b173c6bd1a6a68103b05f4f0"
Jul 16 23:13:25.652035 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:25.651939    2573 scope.go:117] "RemoveContainer" containerID="55ea65fb55e41cef01d0db5a5cc4cb23d11a1f8711f28fa83d4308f7046213da"
Jul 16 23:13:25.652198 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:13:25.652174    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-4tfbc_odh-ai-gateway-infra(ea741db3-bc25-4ab2-9ff4-334cd6578223)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-4tfbc" podUID="ea741db3-bc25-4ab2-9ff4-334cd6578223"
Jul 16 23:13:26.658414 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:26.658387    2573 scope.go:117] "RemoveContainer" containerID="55ea65fb55e41cef01d0db5a5cc4cb23d11a1f8711f28fa83d4308f7046213da"
Jul 16 23:13:26.658799 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:13:26.658560    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-4tfbc_odh-ai-gateway-infra(ea741db3-bc25-4ab2-9ff4-334cd6578223)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-4tfbc" podUID="ea741db3-bc25-4ab2-9ff4-334cd6578223"
Jul 16 23:13:27.513353 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:27.513322    2573 scope.go:117] "RemoveContainer" containerID="8419a312c16a048b1c2b83bf75b43e4e6eb7f23a36950a50011bf0bca1f9add6"
Jul 16 23:13:28.669609 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:28.669577    2573 generic.go:356] "Generic (PLEG): container finished" podID="feaa5af4-f5ba-4ff9-9bd1-c2ea56c31b73" containerID="2b2e13e342a9e147cc70222f83835114ae84e8d999e001dd1d5fa95f23a983af" exitCode=6
Jul 16 23:13:28.670016 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:28.669620    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-dcdvp" event={"ID":"feaa5af4-f5ba-4ff9-9bd1-c2ea56c31b73","Type":"ContainerDied","Data":"2b2e13e342a9e147cc70222f83835114ae84e8d999e001dd1d5fa95f23a983af"}
Jul 16 23:13:28.670016 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:28.669669    2573 scope.go:117] "RemoveContainer" containerID="8419a312c16a048b1c2b83bf75b43e4e6eb7f23a36950a50011bf0bca1f9add6"
Jul 16 23:13:28.670016 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:28.669934    2573 scope.go:117] "RemoveContainer" containerID="2b2e13e342a9e147cc70222f83835114ae84e8d999e001dd1d5fa95f23a983af"
Jul 16 23:13:28.670147 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:13:28.670125    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-dcdvp_odh-ai-gateway-infra(feaa5af4-f5ba-4ff9-9bd1-c2ea56c31b73)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-dcdvp" podUID="feaa5af4-f5ba-4ff9-9bd1-c2ea56c31b73"
Jul 16 23:13:29.696371 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:29.696330    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-dcdvp"]
Jul 16 23:13:29.822814 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:29.822788    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-dcdvp"
Jul 16 23:13:29.955507 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:29.955413    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/feaa5af4-f5ba-4ff9-9bd1-c2ea56c31b73-maas-api-service-ca\") pod \"feaa5af4-f5ba-4ff9-9bd1-c2ea56c31b73\" (UID: \"feaa5af4-f5ba-4ff9-9bd1-c2ea56c31b73\") "
Jul 16 23:13:29.955795 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:29.955770    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feaa5af4-f5ba-4ff9-9bd1-c2ea56c31b73-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "feaa5af4-f5ba-4ff9-9bd1-c2ea56c31b73" (UID: "feaa5af4-f5ba-4ff9-9bd1-c2ea56c31b73"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:13:30.056389 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:30.056351    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/feaa5af4-f5ba-4ff9-9bd1-c2ea56c31b73-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:13:30.682933 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:30.682897    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-dcdvp" event={"ID":"feaa5af4-f5ba-4ff9-9bd1-c2ea56c31b73","Type":"ContainerDied","Data":"32ba10b522884d71fafc7a6575f92e99c5a4d2a689aa0c42ed93509723cdbe62"}
Jul 16 23:13:30.682933 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:30.682917    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-dcdvp"
Jul 16 23:13:30.683168 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:30.682971    2573 scope.go:117] "RemoveContainer" containerID="2b2e13e342a9e147cc70222f83835114ae84e8d999e001dd1d5fa95f23a983af"
Jul 16 23:13:30.699184 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:30.699156    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-dcdvp"]
Jul 16 23:13:30.702665 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:30.702641    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-dcdvp"]
Jul 16 23:13:32.518615 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:32.518583    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="feaa5af4-f5ba-4ff9-9bd1-c2ea56c31b73" path="/var/lib/kubelet/pods/feaa5af4-f5ba-4ff9-9bd1-c2ea56c31b73/volumes"
Jul 16 23:13:35.513314 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:35.513277    2573 scope.go:117] "RemoveContainer" containerID="76ceb8dc8d597c9a3a75533213fe2b91b9a85fd6690ae3d359af1c9a3e3ad885"
Jul 16 23:13:36.713765 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:36.713734    2573 generic.go:356] "Generic (PLEG): container finished" podID="ae858e88-dafd-4bb6-87c9-04342bdde9ed" containerID="f03f0d8edca7329c4f39b6e27096742371736c1b4d98fb96ad28a768408e5f81" exitCode=7
Jul 16 23:13:36.714205 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:36.713813    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-t4pj8" event={"ID":"ae858e88-dafd-4bb6-87c9-04342bdde9ed","Type":"ContainerDied","Data":"f03f0d8edca7329c4f39b6e27096742371736c1b4d98fb96ad28a768408e5f81"}
Jul 16 23:13:36.714205 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:36.713891    2573 scope.go:117] "RemoveContainer" containerID="76ceb8dc8d597c9a3a75533213fe2b91b9a85fd6690ae3d359af1c9a3e3ad885"
Jul 16 23:13:36.714205 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:36.714138    2573 scope.go:117] "RemoveContainer" containerID="f03f0d8edca7329c4f39b6e27096742371736c1b4d98fb96ad28a768408e5f81"
Jul 16 23:13:36.714383 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:13:36.714313    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-t4pj8_odh-ai-gateway-infra(ae858e88-dafd-4bb6-87c9-04342bdde9ed)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-t4pj8" podUID="ae858e88-dafd-4bb6-87c9-04342bdde9ed"
Jul 16 23:13:37.740288 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:37.740256    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-t4pj8"]
Jul 16 23:13:37.868306 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:37.868283    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-t4pj8"
Jul 16 23:13:38.021449 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:38.021373    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae858e88-dafd-4bb6-87c9-04342bdde9ed-maas-api-service-ca\") pod \"ae858e88-dafd-4bb6-87c9-04342bdde9ed\" (UID: \"ae858e88-dafd-4bb6-87c9-04342bdde9ed\") "
Jul 16 23:13:38.021752 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:38.021727    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae858e88-dafd-4bb6-87c9-04342bdde9ed-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ae858e88-dafd-4bb6-87c9-04342bdde9ed" (UID: "ae858e88-dafd-4bb6-87c9-04342bdde9ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:13:38.122063 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:38.122036    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae858e88-dafd-4bb6-87c9-04342bdde9ed-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:13:38.729387 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:38.729358    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-t4pj8" event={"ID":"ae858e88-dafd-4bb6-87c9-04342bdde9ed","Type":"ContainerDied","Data":"318d1216047798ffe590cd1764e9ce43d47d18d5102769228f29a92f9489f1da"}
Jul 16 23:13:38.729387 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:38.729384    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-t4pj8"
Jul 16 23:13:38.729551 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:38.729417    2573 scope.go:117] "RemoveContainer" containerID="f03f0d8edca7329c4f39b6e27096742371736c1b4d98fb96ad28a768408e5f81"
Jul 16 23:13:38.746781 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:38.746753    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-t4pj8"]
Jul 16 23:13:38.751825 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:38.751804    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-t4pj8"]
Jul 16 23:13:39.514035 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:39.513997    2573 scope.go:117] "RemoveContainer" containerID="55ea65fb55e41cef01d0db5a5cc4cb23d11a1f8711f28fa83d4308f7046213da"
Jul 16 23:13:40.523751 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:40.523659    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae858e88-dafd-4bb6-87c9-04342bdde9ed" path="/var/lib/kubelet/pods/ae858e88-dafd-4bb6-87c9-04342bdde9ed/volumes"
Jul 16 23:13:40.740726 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:40.740692    2573 generic.go:356] "Generic (PLEG): container finished" podID="ea741db3-bc25-4ab2-9ff4-334cd6578223" containerID="d011fe7c0aa42bdecba6815366c39265a19b869d61cd558a358a675fd4bbf439" exitCode=6
Jul 16 23:13:40.740868 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:40.740760    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-4tfbc" event={"ID":"ea741db3-bc25-4ab2-9ff4-334cd6578223","Type":"ContainerDied","Data":"d011fe7c0aa42bdecba6815366c39265a19b869d61cd558a358a675fd4bbf439"}
Jul 16 23:13:40.740868 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:40.740819    2573 scope.go:117] "RemoveContainer" containerID="55ea65fb55e41cef01d0db5a5cc4cb23d11a1f8711f28fa83d4308f7046213da"
Jul 16 23:13:40.741077 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:40.741049    2573 scope.go:117] "RemoveContainer" containerID="d011fe7c0aa42bdecba6815366c39265a19b869d61cd558a358a675fd4bbf439"
Jul 16 23:13:40.741248 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:13:40.741216    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-4tfbc_odh-ai-gateway-infra(ea741db3-bc25-4ab2-9ff4-334cd6578223)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-4tfbc" podUID="ea741db3-bc25-4ab2-9ff4-334cd6578223"
Jul 16 23:13:41.766289 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:41.766250    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-4tfbc"]
Jul 16 23:13:41.899678 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:41.899656    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-4tfbc"
Jul 16 23:13:42.048803 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:42.048737    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea741db3-bc25-4ab2-9ff4-334cd6578223-maas-api-service-ca\") pod \"ea741db3-bc25-4ab2-9ff4-334cd6578223\" (UID: \"ea741db3-bc25-4ab2-9ff4-334cd6578223\") "
Jul 16 23:13:42.049091 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:42.049064    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea741db3-bc25-4ab2-9ff4-334cd6578223-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ea741db3-bc25-4ab2-9ff4-334cd6578223" (UID: "ea741db3-bc25-4ab2-9ff4-334cd6578223"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:13:42.149271 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:42.149224    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea741db3-bc25-4ab2-9ff4-334cd6578223-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:13:42.754113 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:42.754089    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-4tfbc"
Jul 16 23:13:42.754295 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:42.754130    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-4tfbc" event={"ID":"ea741db3-bc25-4ab2-9ff4-334cd6578223","Type":"ContainerDied","Data":"a80197639b5f78662cc885bc7c785fae97799d519a73ec58d23938da4d39e143"}
Jul 16 23:13:42.754295 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:42.754186    2573 scope.go:117] "RemoveContainer" containerID="d011fe7c0aa42bdecba6815366c39265a19b869d61cd558a358a675fd4bbf439"
Jul 16 23:13:42.772314 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:42.772277    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-4tfbc"]
Jul 16 23:13:42.775456 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:42.775434    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-4tfbc"]
Jul 16 23:13:44.519970 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:44.519937    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea741db3-bc25-4ab2-9ff4-334cd6578223" path="/var/lib/kubelet/pods/ea741db3-bc25-4ab2-9ff4-334cd6578223/volumes"
Jul 16 23:13:46.574312 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:46.574289    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r9jm4_506f904a-3272-4586-8528-146884e728f9/ovn-acl-logging/0.log"
Jul 16 23:13:46.580480 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:46.580460    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r9jm4_506f904a-3272-4586-8528-146884e728f9/ovn-acl-logging/0.log"
Jul 16 23:13:51.564507 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:51.564472    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-qchhw"]
Jul 16 23:13:51.564965 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:51.564745    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feaa5af4-f5ba-4ff9-9bd1-c2ea56c31b73" containerName="revoke-keys"
Jul 16 23:13:51.564965 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:51.564757    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="feaa5af4-f5ba-4ff9-9bd1-c2ea56c31b73" containerName="revoke-keys"
Jul 16 23:13:51.564965 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:51.564764    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feaa5af4-f5ba-4ff9-9bd1-c2ea56c31b73" containerName="revoke-keys"
Jul 16 23:13:51.564965 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:51.564770    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="feaa5af4-f5ba-4ff9-9bd1-c2ea56c31b73" containerName="revoke-keys"
Jul 16 23:13:51.564965 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:51.564778    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea741db3-bc25-4ab2-9ff4-334cd6578223" containerName="revoke-keys"
Jul 16 23:13:51.564965 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:51.564784    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea741db3-bc25-4ab2-9ff4-334cd6578223" containerName="revoke-keys"
Jul 16 23:13:51.564965 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:51.564794    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea741db3-bc25-4ab2-9ff4-334cd6578223" containerName="revoke-keys"
Jul 16 23:13:51.564965 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:51.564799    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea741db3-bc25-4ab2-9ff4-334cd6578223" containerName="revoke-keys"
Jul 16 23:13:51.564965 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:51.564808    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae858e88-dafd-4bb6-87c9-04342bdde9ed" containerName="revoke-keys"
Jul 16 23:13:51.564965 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:51.564813    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae858e88-dafd-4bb6-87c9-04342bdde9ed" containerName="revoke-keys"
Jul 16 23:13:51.564965 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:51.564827    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae858e88-dafd-4bb6-87c9-04342bdde9ed" containerName="revoke-keys"
Jul 16 23:13:51.564965 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:51.564832    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae858e88-dafd-4bb6-87c9-04342bdde9ed" containerName="revoke-keys"
Jul 16 23:13:51.564965 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:51.564881    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="feaa5af4-f5ba-4ff9-9bd1-c2ea56c31b73" containerName="revoke-keys"
Jul 16 23:13:51.564965 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:51.564887    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae858e88-dafd-4bb6-87c9-04342bdde9ed" containerName="revoke-keys"
Jul 16 23:13:51.564965 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:51.564892    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea741db3-bc25-4ab2-9ff4-334cd6578223" containerName="revoke-keys"
Jul 16 23:13:51.564965 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:51.564899    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea741db3-bc25-4ab2-9ff4-334cd6578223" containerName="revoke-keys"
Jul 16 23:13:51.564965 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:51.564904    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="feaa5af4-f5ba-4ff9-9bd1-c2ea56c31b73" containerName="revoke-keys"
Jul 16 23:13:51.564965 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:51.564909    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea741db3-bc25-4ab2-9ff4-334cd6578223" containerName="revoke-keys"
Jul 16 23:13:51.564965 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:51.564917    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae858e88-dafd-4bb6-87c9-04342bdde9ed" containerName="revoke-keys"
Jul 16 23:13:51.569244 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:51.569217    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-qchhw"
Jul 16 23:13:51.571716 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:51.571693    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 23:13:51.571716 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:51.571707    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:13:51.576600 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:51.576258    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-qchhw"]
Jul 16 23:13:51.714531 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:51.714498    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28c87fa6-03d0-437c-ae73-97ad82595e93-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-qchhw\" (UID: \"28c87fa6-03d0-437c-ae73-97ad82595e93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-qchhw"
Jul 16 23:13:51.815524 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:51.815451    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28c87fa6-03d0-437c-ae73-97ad82595e93-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-qchhw\" (UID: \"28c87fa6-03d0-437c-ae73-97ad82595e93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-qchhw"
Jul 16 23:13:51.816015 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:51.815995    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28c87fa6-03d0-437c-ae73-97ad82595e93-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-qchhw\" (UID: \"28c87fa6-03d0-437c-ae73-97ad82595e93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-qchhw"
Jul 16 23:13:51.880371 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:51.880347    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-qchhw"
Jul 16 23:13:52.207981 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:52.207954    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-qchhw"]
Jul 16 23:13:52.209998 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:13:52.209972    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28c87fa6_03d0_437c_ae73_97ad82595e93.slice/crio-8996b57d70bb23a8e826d72a93803d15547d7172a512963137ec79f6313fc302 WatchSource:0}: Error finding container 8996b57d70bb23a8e826d72a93803d15547d7172a512963137ec79f6313fc302: Status 404 returned error can't find the container with id 8996b57d70bb23a8e826d72a93803d15547d7172a512963137ec79f6313fc302
Jul 16 23:13:52.800528 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:52.800503    2573 generic.go:356] "Generic (PLEG): container finished" podID="28c87fa6-03d0-437c-ae73-97ad82595e93" containerID="40c25baa5f4e03c5712a146a8c51f251d3fd7bb9eb4b93d1eff8344eeb4f33f6" exitCode=6
Jul 16 23:13:52.800834 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:52.800581    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-qchhw" event={"ID":"28c87fa6-03d0-437c-ae73-97ad82595e93","Type":"ContainerDied","Data":"40c25baa5f4e03c5712a146a8c51f251d3fd7bb9eb4b93d1eff8344eeb4f33f6"}
Jul 16 23:13:52.800834 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:52.800633    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-qchhw" event={"ID":"28c87fa6-03d0-437c-ae73-97ad82595e93","Type":"ContainerStarted","Data":"8996b57d70bb23a8e826d72a93803d15547d7172a512963137ec79f6313fc302"}
Jul 16 23:13:52.800834 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:52.800731    2573 scope.go:117] "RemoveContainer" containerID="40c25baa5f4e03c5712a146a8c51f251d3fd7bb9eb4b93d1eff8344eeb4f33f6"
Jul 16 23:13:53.807827 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:53.807794    2573 generic.go:356] "Generic (PLEG): container finished" podID="28c87fa6-03d0-437c-ae73-97ad82595e93" containerID="8b56bc252a6d8a2e8dae32172047c59d43584e08c7e6ef04947dc533c65cca1f" exitCode=6
Jul 16 23:13:53.808247 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:53.807858    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-qchhw" event={"ID":"28c87fa6-03d0-437c-ae73-97ad82595e93","Type":"ContainerDied","Data":"8b56bc252a6d8a2e8dae32172047c59d43584e08c7e6ef04947dc533c65cca1f"}
Jul 16 23:13:53.808247 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:53.807903    2573 scope.go:117] "RemoveContainer" containerID="40c25baa5f4e03c5712a146a8c51f251d3fd7bb9eb4b93d1eff8344eeb4f33f6"
Jul 16 23:13:53.808247 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:53.808051    2573 scope.go:117] "RemoveContainer" containerID="8b56bc252a6d8a2e8dae32172047c59d43584e08c7e6ef04947dc533c65cca1f"
Jul 16 23:13:53.808436 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:13:53.808261    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-qchhw_odh-ai-gateway-infra(28c87fa6-03d0-437c-ae73-97ad82595e93)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-qchhw" podUID="28c87fa6-03d0-437c-ae73-97ad82595e93"
Jul 16 23:13:54.814861 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:13:54.814829    2573 scope.go:117] "RemoveContainer" containerID="8b56bc252a6d8a2e8dae32172047c59d43584e08c7e6ef04947dc533c65cca1f"
Jul 16 23:13:54.815228 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:13:54.814999    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-qchhw_odh-ai-gateway-infra(28c87fa6-03d0-437c-ae73-97ad82595e93)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-qchhw" podUID="28c87fa6-03d0-437c-ae73-97ad82595e93"
Jul 16 23:14:07.513359 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:07.513327    2573 scope.go:117] "RemoveContainer" containerID="8b56bc252a6d8a2e8dae32172047c59d43584e08c7e6ef04947dc533c65cca1f"
Jul 16 23:14:07.875515 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:07.875479    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-qchhw" event={"ID":"28c87fa6-03d0-437c-ae73-97ad82595e93","Type":"ContainerStarted","Data":"02385428cb53889515375f40902ae92ad992bcd45386795b45980f44dda11a04"}
Jul 16 23:14:07.894456 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:07.894401    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-qchhw" podStartSLOduration=16.894386048 podStartE2EDuration="16.894386048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:13:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:14:07.893131341 +0000 UTC m=+2421.903394022" watchObservedRunningTime="2026-07-16 23:14:07.894386048 +0000 UTC m=+2421.904648712"
Jul 16 23:14:08.881513 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:08.881424    2573 generic.go:356] "Generic (PLEG): container finished" podID="28c87fa6-03d0-437c-ae73-97ad82595e93" containerID="02385428cb53889515375f40902ae92ad992bcd45386795b45980f44dda11a04" exitCode=6
Jul 16 23:14:08.881513 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:08.881500    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-qchhw" event={"ID":"28c87fa6-03d0-437c-ae73-97ad82595e93","Type":"ContainerDied","Data":"02385428cb53889515375f40902ae92ad992bcd45386795b45980f44dda11a04"}
Jul 16 23:14:08.881951 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:08.881562    2573 scope.go:117] "RemoveContainer" containerID="8b56bc252a6d8a2e8dae32172047c59d43584e08c7e6ef04947dc533c65cca1f"
Jul 16 23:14:08.881951 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:08.881853    2573 scope.go:117] "RemoveContainer" containerID="02385428cb53889515375f40902ae92ad992bcd45386795b45980f44dda11a04"
Jul 16 23:14:08.882107 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:14:08.882089    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-qchhw_odh-ai-gateway-infra(28c87fa6-03d0-437c-ae73-97ad82595e93)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-qchhw" podUID="28c87fa6-03d0-437c-ae73-97ad82595e93"
Jul 16 23:14:08.903993 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:08.903970    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-qchhw"]
Jul 16 23:14:10.024915 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:10.024890    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-qchhw"
Jul 16 23:14:10.149728 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:10.149625    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28c87fa6-03d0-437c-ae73-97ad82595e93-maas-api-service-ca\") pod \"28c87fa6-03d0-437c-ae73-97ad82595e93\" (UID: \"28c87fa6-03d0-437c-ae73-97ad82595e93\") "
Jul 16 23:14:10.150002 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:10.149978    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28c87fa6-03d0-437c-ae73-97ad82595e93-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "28c87fa6-03d0-437c-ae73-97ad82595e93" (UID: "28c87fa6-03d0-437c-ae73-97ad82595e93"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:14:10.251118 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:10.251063    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28c87fa6-03d0-437c-ae73-97ad82595e93-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:14:10.551223 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:10.551192    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8g5pk"]
Jul 16 23:14:10.551498 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:10.551483    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28c87fa6-03d0-437c-ae73-97ad82595e93" containerName="revoke-keys"
Jul 16 23:14:10.551572 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:10.551499    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="28c87fa6-03d0-437c-ae73-97ad82595e93" containerName="revoke-keys"
Jul 16 23:14:10.551572 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:10.551518    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feaa5af4-f5ba-4ff9-9bd1-c2ea56c31b73" containerName="revoke-keys"
Jul 16 23:14:10.551572 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:10.551526    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="feaa5af4-f5ba-4ff9-9bd1-c2ea56c31b73" containerName="revoke-keys"
Jul 16 23:14:10.551572 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:10.551540    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea741db3-bc25-4ab2-9ff4-334cd6578223" containerName="revoke-keys"
Jul 16 23:14:10.551572 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:10.551548    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea741db3-bc25-4ab2-9ff4-334cd6578223" containerName="revoke-keys"
Jul 16 23:14:10.551572 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:10.551560    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae858e88-dafd-4bb6-87c9-04342bdde9ed" containerName="revoke-keys"
Jul 16 23:14:10.551572 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:10.551569    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae858e88-dafd-4bb6-87c9-04342bdde9ed" containerName="revoke-keys"
Jul 16 23:14:10.551882 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:10.551578    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28c87fa6-03d0-437c-ae73-97ad82595e93" containerName="revoke-keys"
Jul 16 23:14:10.551882 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:10.551586    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="28c87fa6-03d0-437c-ae73-97ad82595e93" containerName="revoke-keys"
Jul 16 23:14:10.551882 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:10.551636    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae858e88-dafd-4bb6-87c9-04342bdde9ed" containerName="revoke-keys"
Jul 16 23:14:10.551882 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:10.551643    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="28c87fa6-03d0-437c-ae73-97ad82595e93" containerName="revoke-keys"
Jul 16 23:14:10.551882 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:10.551649    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="28c87fa6-03d0-437c-ae73-97ad82595e93" containerName="revoke-keys"
Jul 16 23:14:10.551882 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:10.551656    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="28c87fa6-03d0-437c-ae73-97ad82595e93" containerName="revoke-keys"
Jul 16 23:14:10.551882 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:10.551661    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="feaa5af4-f5ba-4ff9-9bd1-c2ea56c31b73" containerName="revoke-keys"
Jul 16 23:14:10.554456 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:10.554439    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8g5pk"
Jul 16 23:14:10.562558 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:10.562529    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8g5pk"]
Jul 16 23:14:10.654657 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:10.654619    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5c402c1-fc35-47c6-a418-4860ad898cf9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-8g5pk\" (UID: \"a5c402c1-fc35-47c6-a418-4860ad898cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8g5pk"
Jul 16 23:14:10.755495 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:10.755453    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5c402c1-fc35-47c6-a418-4860ad898cf9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-8g5pk\" (UID: \"a5c402c1-fc35-47c6-a418-4860ad898cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8g5pk"
Jul 16 23:14:10.756091 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:10.756072    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5c402c1-fc35-47c6-a418-4860ad898cf9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-8g5pk\" (UID: \"a5c402c1-fc35-47c6-a418-4860ad898cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8g5pk"
Jul 16 23:14:10.865743 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:10.865716    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8g5pk"
Jul 16 23:14:10.894200 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:10.894167    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-qchhw" event={"ID":"28c87fa6-03d0-437c-ae73-97ad82595e93","Type":"ContainerDied","Data":"8996b57d70bb23a8e826d72a93803d15547d7172a512963137ec79f6313fc302"}
Jul 16 23:14:10.894200 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:10.894188    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-qchhw"
Jul 16 23:14:10.894450 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:10.894262    2573 scope.go:117] "RemoveContainer" containerID="02385428cb53889515375f40902ae92ad992bcd45386795b45980f44dda11a04"
Jul 16 23:14:10.911111 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:10.911077    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-qchhw"]
Jul 16 23:14:10.915971 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:10.915948    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-qchhw"]
Jul 16 23:14:10.997565 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:10.997496    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8g5pk"]
Jul 16 23:14:10.999645 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:14:10.999620    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5c402c1_fc35_47c6_a418_4860ad898cf9.slice/crio-0ed806cb8b7830a39435c11d3c17ea17b0a991435241fae88439f67480d5da57 WatchSource:0}: Error finding container 0ed806cb8b7830a39435c11d3c17ea17b0a991435241fae88439f67480d5da57: Status 404 returned error can't find the container with id 0ed806cb8b7830a39435c11d3c17ea17b0a991435241fae88439f67480d5da57
Jul 16 23:14:11.900880 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:11.900840    2573 generic.go:356] "Generic (PLEG): container finished" podID="a5c402c1-fc35-47c6-a418-4860ad898cf9" containerID="a7241d5fcadbb73965d162a0dcb803b64495dddd59242be4e28b162a841a6b17" exitCode=6
Jul 16 23:14:11.901394 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:11.900929    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8g5pk" event={"ID":"a5c402c1-fc35-47c6-a418-4860ad898cf9","Type":"ContainerDied","Data":"a7241d5fcadbb73965d162a0dcb803b64495dddd59242be4e28b162a841a6b17"}
Jul 16 23:14:11.901394 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:11.900977    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8g5pk" event={"ID":"a5c402c1-fc35-47c6-a418-4860ad898cf9","Type":"ContainerStarted","Data":"0ed806cb8b7830a39435c11d3c17ea17b0a991435241fae88439f67480d5da57"}
Jul 16 23:14:11.901394 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:11.901158    2573 scope.go:117] "RemoveContainer" containerID="a7241d5fcadbb73965d162a0dcb803b64495dddd59242be4e28b162a841a6b17"
Jul 16 23:14:12.519164 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:12.519135    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28c87fa6-03d0-437c-ae73-97ad82595e93" path="/var/lib/kubelet/pods/28c87fa6-03d0-437c-ae73-97ad82595e93/volumes"
Jul 16 23:14:12.907182 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:12.907153    2573 generic.go:356] "Generic (PLEG): container finished" podID="a5c402c1-fc35-47c6-a418-4860ad898cf9" containerID="72e2592d9bbb4d4574f4009246626b6e7c9ee32da66b81d248cfb6ae96aa57a5" exitCode=6
Jul 16 23:14:12.907573 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:12.907255    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8g5pk" event={"ID":"a5c402c1-fc35-47c6-a418-4860ad898cf9","Type":"ContainerDied","Data":"72e2592d9bbb4d4574f4009246626b6e7c9ee32da66b81d248cfb6ae96aa57a5"}
Jul 16 23:14:12.907573 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:12.907314    2573 scope.go:117] "RemoveContainer" containerID="a7241d5fcadbb73965d162a0dcb803b64495dddd59242be4e28b162a841a6b17"
Jul 16 23:14:12.907573 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:12.907529    2573 scope.go:117] "RemoveContainer" containerID="72e2592d9bbb4d4574f4009246626b6e7c9ee32da66b81d248cfb6ae96aa57a5"
Jul 16 23:14:12.907734 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:14:12.907717    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-8g5pk_odh-ai-gateway-infra(a5c402c1-fc35-47c6-a418-4860ad898cf9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8g5pk" podUID="a5c402c1-fc35-47c6-a418-4860ad898cf9"
Jul 16 23:14:13.913882 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:13.913851    2573 scope.go:117] "RemoveContainer" containerID="72e2592d9bbb4d4574f4009246626b6e7c9ee32da66b81d248cfb6ae96aa57a5"
Jul 16 23:14:13.914262 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:14:13.914029    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-8g5pk_odh-ai-gateway-infra(a5c402c1-fc35-47c6-a418-4860ad898cf9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8g5pk" podUID="a5c402c1-fc35-47c6-a418-4860ad898cf9"
Jul 16 23:14:17.296304 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:17.296268    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:14:17.382123 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:17.382094    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:14:17.724895 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:17.724861    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-r878t"]
Jul 16 23:14:17.725198 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:17.725185    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28c87fa6-03d0-437c-ae73-97ad82595e93" containerName="revoke-keys"
Jul 16 23:14:17.725262 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:17.725200    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="28c87fa6-03d0-437c-ae73-97ad82595e93" containerName="revoke-keys"
Jul 16 23:14:17.729132 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:17.729116    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-r878t"
Jul 16 23:14:17.735256 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:17.735218    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-r878t"]
Jul 16 23:14:17.809158 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:17.809127    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4601af56-94c8-4837-aa79-dcbd5458f3da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-r878t\" (UID: \"4601af56-94c8-4837-aa79-dcbd5458f3da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-r878t"
Jul 16 23:14:17.910223 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:17.910191    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4601af56-94c8-4837-aa79-dcbd5458f3da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-r878t\" (UID: \"4601af56-94c8-4837-aa79-dcbd5458f3da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-r878t"
Jul 16 23:14:17.910810 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:17.910789    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4601af56-94c8-4837-aa79-dcbd5458f3da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-r878t\" (UID: \"4601af56-94c8-4837-aa79-dcbd5458f3da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-r878t"
Jul 16 23:14:18.040907 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:18.040828    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-r878t"
Jul 16 23:14:18.172640 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:18.172610    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-r878t"]
Jul 16 23:14:18.359180 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:18.359146    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:14:18.938302 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:18.938265    2573 generic.go:356] "Generic (PLEG): container finished" podID="4601af56-94c8-4837-aa79-dcbd5458f3da" containerID="01a144c301dce70161f9772be747bd91105958726245bc852ece72f6223afe54" exitCode=7
Jul 16 23:14:18.938302 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:18.938304    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-r878t" event={"ID":"4601af56-94c8-4837-aa79-dcbd5458f3da","Type":"ContainerDied","Data":"01a144c301dce70161f9772be747bd91105958726245bc852ece72f6223afe54"}
Jul 16 23:14:18.938504 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:18.938343    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-r878t" event={"ID":"4601af56-94c8-4837-aa79-dcbd5458f3da","Type":"ContainerStarted","Data":"e7fd745e36be8d441b5e34c61773abe41aeae79b51403c59796cb701fd6e0718"}
Jul 16 23:14:18.938540 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:18.938533    2573 scope.go:117] "RemoveContainer" containerID="01a144c301dce70161f9772be747bd91105958726245bc852ece72f6223afe54"
Jul 16 23:14:19.549074 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:19.549051    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:14:19.945415 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:19.945370    2573 generic.go:356] "Generic (PLEG): container finished" podID="4601af56-94c8-4837-aa79-dcbd5458f3da" containerID="167833eb01c553120841bbf6d53c3d1c795060fdaa69771925103575cb0c0313" exitCode=7
Jul 16 23:14:19.945581 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:19.945444    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-r878t" event={"ID":"4601af56-94c8-4837-aa79-dcbd5458f3da","Type":"ContainerDied","Data":"167833eb01c553120841bbf6d53c3d1c795060fdaa69771925103575cb0c0313"}
Jul 16 23:14:19.945581 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:19.945507    2573 scope.go:117] "RemoveContainer" containerID="01a144c301dce70161f9772be747bd91105958726245bc852ece72f6223afe54"
Jul 16 23:14:19.945761 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:19.945748    2573 scope.go:117] "RemoveContainer" containerID="167833eb01c553120841bbf6d53c3d1c795060fdaa69771925103575cb0c0313"
Jul 16 23:14:19.945982 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:14:19.945962    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-r878t_odh-ai-gateway-infra(4601af56-94c8-4837-aa79-dcbd5458f3da)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-r878t" podUID="4601af56-94c8-4837-aa79-dcbd5458f3da"
Jul 16 23:14:20.952217 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:20.952187    2573 scope.go:117] "RemoveContainer" containerID="167833eb01c553120841bbf6d53c3d1c795060fdaa69771925103575cb0c0313"
Jul 16 23:14:20.952626 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:14:20.952375    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-r878t_odh-ai-gateway-infra(4601af56-94c8-4837-aa79-dcbd5458f3da)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-r878t" podUID="4601af56-94c8-4837-aa79-dcbd5458f3da"
Jul 16 23:14:22.779173 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:22.779144    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-cvzk4"]
Jul 16 23:14:22.782137 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:22.782121    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-cvzk4"
Jul 16 23:14:22.789571 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:22.789548    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-cvzk4"]
Jul 16 23:14:22.847430 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:22.847405    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05ed95cc-b8d0-46ad-b07a-f50c38542ca9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-cvzk4\" (UID: \"05ed95cc-b8d0-46ad-b07a-f50c38542ca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-cvzk4"
Jul 16 23:14:22.948518 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:22.948498    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05ed95cc-b8d0-46ad-b07a-f50c38542ca9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-cvzk4\" (UID: \"05ed95cc-b8d0-46ad-b07a-f50c38542ca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-cvzk4"
Jul 16 23:14:22.949012 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:22.948994    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05ed95cc-b8d0-46ad-b07a-f50c38542ca9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-cvzk4\" (UID: \"05ed95cc-b8d0-46ad-b07a-f50c38542ca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-cvzk4"
Jul 16 23:14:23.093293 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:23.093270    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-cvzk4"
Jul 16 23:14:23.231910 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:23.231882    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-cvzk4"]
Jul 16 23:14:23.233827 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:14:23.233796    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05ed95cc_b8d0_46ad_b07a_f50c38542ca9.slice/crio-6f5b1b51d22046d3497b07ae6db5113bacd1838c0fac76e01d69b77bbd057cf2 WatchSource:0}: Error finding container 6f5b1b51d22046d3497b07ae6db5113bacd1838c0fac76e01d69b77bbd057cf2: Status 404 returned error can't find the container with id 6f5b1b51d22046d3497b07ae6db5113bacd1838c0fac76e01d69b77bbd057cf2
Jul 16 23:14:23.970173 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:23.970139    2573 generic.go:356] "Generic (PLEG): container finished" podID="05ed95cc-b8d0-46ad-b07a-f50c38542ca9" containerID="5c8e9fbeee8b86e46b4248a13e9ade5abf67b313bf6fb91cf8803dafd2d7b127" exitCode=6
Jul 16 23:14:23.970506 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:23.970219    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-cvzk4" event={"ID":"05ed95cc-b8d0-46ad-b07a-f50c38542ca9","Type":"ContainerDied","Data":"5c8e9fbeee8b86e46b4248a13e9ade5abf67b313bf6fb91cf8803dafd2d7b127"}
Jul 16 23:14:23.970506 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:23.970293    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-cvzk4" event={"ID":"05ed95cc-b8d0-46ad-b07a-f50c38542ca9","Type":"ContainerStarted","Data":"6f5b1b51d22046d3497b07ae6db5113bacd1838c0fac76e01d69b77bbd057cf2"}
Jul 16 23:14:23.970506 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:23.970404    2573 scope.go:117] "RemoveContainer" containerID="5c8e9fbeee8b86e46b4248a13e9ade5abf67b313bf6fb91cf8803dafd2d7b127"
Jul 16 23:14:24.976947 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:24.976871    2573 generic.go:356] "Generic (PLEG): container finished" podID="05ed95cc-b8d0-46ad-b07a-f50c38542ca9" containerID="2fecb3dce16c727deac6c12c2bbefed8acf5919166ff80cfcb488af460c9a397" exitCode=6
Jul 16 23:14:24.977334 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:24.976958    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-cvzk4" event={"ID":"05ed95cc-b8d0-46ad-b07a-f50c38542ca9","Type":"ContainerDied","Data":"2fecb3dce16c727deac6c12c2bbefed8acf5919166ff80cfcb488af460c9a397"}
Jul 16 23:14:24.977334 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:24.977015    2573 scope.go:117] "RemoveContainer" containerID="5c8e9fbeee8b86e46b4248a13e9ade5abf67b313bf6fb91cf8803dafd2d7b127"
Jul 16 23:14:24.977334 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:24.977214    2573 scope.go:117] "RemoveContainer" containerID="2fecb3dce16c727deac6c12c2bbefed8acf5919166ff80cfcb488af460c9a397"
Jul 16 23:14:24.977517 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:14:24.977423    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-cvzk4_odh-ai-gateway-infra(05ed95cc-b8d0-46ad-b07a-f50c38542ca9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-cvzk4" podUID="05ed95cc-b8d0-46ad-b07a-f50c38542ca9"
Jul 16 23:14:25.513877 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:25.513854    2573 scope.go:117] "RemoveContainer" containerID="72e2592d9bbb4d4574f4009246626b6e7c9ee32da66b81d248cfb6ae96aa57a5"
Jul 16 23:14:25.991990 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:25.991960    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8g5pk" event={"ID":"a5c402c1-fc35-47c6-a418-4860ad898cf9","Type":"ContainerStarted","Data":"65b536080a1f9990863e2baeb057b1c06490179142eef77aa09326ea6120a91d"}
Jul 16 23:14:25.993759 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:25.993735    2573 scope.go:117] "RemoveContainer" containerID="2fecb3dce16c727deac6c12c2bbefed8acf5919166ff80cfcb488af460c9a397"
Jul 16 23:14:25.993903 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:14:25.993885    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-cvzk4_odh-ai-gateway-infra(05ed95cc-b8d0-46ad-b07a-f50c38542ca9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-cvzk4" podUID="05ed95cc-b8d0-46ad-b07a-f50c38542ca9"
Jul 16 23:14:26.010013 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:26.009970    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8g5pk" podStartSLOduration=16.009955611 podStartE2EDuration="16.009955611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:14:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:14:26.008319533 +0000 UTC m=+2440.018582197" watchObservedRunningTime="2026-07-16 23:14:26.009955611 +0000 UTC m=+2440.020218274"
Jul 16 23:14:26.999814 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:26.999785    2573 generic.go:356] "Generic (PLEG): container finished" podID="a5c402c1-fc35-47c6-a418-4860ad898cf9" containerID="65b536080a1f9990863e2baeb057b1c06490179142eef77aa09326ea6120a91d" exitCode=6
Jul 16 23:14:27.000291 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:26.999868    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8g5pk" event={"ID":"a5c402c1-fc35-47c6-a418-4860ad898cf9","Type":"ContainerDied","Data":"65b536080a1f9990863e2baeb057b1c06490179142eef77aa09326ea6120a91d"}
Jul 16 23:14:27.000291 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:26.999951    2573 scope.go:117] "RemoveContainer" containerID="72e2592d9bbb4d4574f4009246626b6e7c9ee32da66b81d248cfb6ae96aa57a5"
Jul 16 23:14:27.000291 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:27.000219    2573 scope.go:117] "RemoveContainer" containerID="65b536080a1f9990863e2baeb057b1c06490179142eef77aa09326ea6120a91d"
Jul 16 23:14:27.000436 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:14:27.000417    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-8g5pk_odh-ai-gateway-infra(a5c402c1-fc35-47c6-a418-4860ad898cf9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8g5pk" podUID="a5c402c1-fc35-47c6-a418-4860ad898cf9"
Jul 16 23:14:27.025337 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:27.025317    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8g5pk"]
Jul 16 23:14:28.137883 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:28.137862    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8g5pk"
Jul 16 23:14:28.186128 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:28.186101    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5c402c1-fc35-47c6-a418-4860ad898cf9-maas-api-service-ca\") pod \"a5c402c1-fc35-47c6-a418-4860ad898cf9\" (UID: \"a5c402c1-fc35-47c6-a418-4860ad898cf9\") "
Jul 16 23:14:28.186522 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:28.186498    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5c402c1-fc35-47c6-a418-4860ad898cf9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a5c402c1-fc35-47c6-a418-4860ad898cf9" (UID: "a5c402c1-fc35-47c6-a418-4860ad898cf9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:14:28.287206 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:28.287141    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5c402c1-fc35-47c6-a418-4860ad898cf9-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:14:29.011079 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:29.011044    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8g5pk" event={"ID":"a5c402c1-fc35-47c6-a418-4860ad898cf9","Type":"ContainerDied","Data":"0ed806cb8b7830a39435c11d3c17ea17b0a991435241fae88439f67480d5da57"}
Jul 16 23:14:29.011253 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:29.011105    2573 scope.go:117] "RemoveContainer" containerID="65b536080a1f9990863e2baeb057b1c06490179142eef77aa09326ea6120a91d"
Jul 16 23:14:29.011253 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:29.011060    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8g5pk"
Jul 16 23:14:29.028514 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:29.028492    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8g5pk"]
Jul 16 23:14:29.034554 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:29.034534    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8g5pk"]
Jul 16 23:14:30.519078 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:30.519035    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5c402c1-fc35-47c6-a418-4860ad898cf9" path="/var/lib/kubelet/pods/a5c402c1-fc35-47c6-a418-4860ad898cf9/volumes"
Jul 16 23:14:33.513159 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:33.513131    2573 scope.go:117] "RemoveContainer" containerID="167833eb01c553120841bbf6d53c3d1c795060fdaa69771925103575cb0c0313"
Jul 16 23:14:34.037801 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:34.037719    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-r878t" event={"ID":"4601af56-94c8-4837-aa79-dcbd5458f3da","Type":"ContainerStarted","Data":"24f488bd6205fe87f2c1a92ec1adf7a2c8b934bcc4edbfee90ed1b0e8844e4e6"}
Jul 16 23:14:34.055594 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:34.055546    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-r878t" podStartSLOduration=17.055533118 podStartE2EDuration="17.055533118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:14:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:14:34.055107124 +0000 UTC m=+2448.065369803" watchObservedRunningTime="2026-07-16 23:14:34.055533118 +0000 UTC m=+2448.065795782"
Jul 16 23:14:35.044734 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:35.044698    2573 generic.go:356] "Generic (PLEG): container finished" podID="4601af56-94c8-4837-aa79-dcbd5458f3da" containerID="24f488bd6205fe87f2c1a92ec1adf7a2c8b934bcc4edbfee90ed1b0e8844e4e6" exitCode=7
Jul 16 23:14:35.045159 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:35.044777    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-r878t" event={"ID":"4601af56-94c8-4837-aa79-dcbd5458f3da","Type":"ContainerDied","Data":"24f488bd6205fe87f2c1a92ec1adf7a2c8b934bcc4edbfee90ed1b0e8844e4e6"}
Jul 16 23:14:35.045159 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:35.044838    2573 scope.go:117] "RemoveContainer" containerID="167833eb01c553120841bbf6d53c3d1c795060fdaa69771925103575cb0c0313"
Jul 16 23:14:35.045159 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:35.045085    2573 scope.go:117] "RemoveContainer" containerID="24f488bd6205fe87f2c1a92ec1adf7a2c8b934bcc4edbfee90ed1b0e8844e4e6"
Jul 16 23:14:35.045323 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:14:35.045299    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-r878t_odh-ai-gateway-infra(4601af56-94c8-4837-aa79-dcbd5458f3da)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-r878t" podUID="4601af56-94c8-4837-aa79-dcbd5458f3da"
Jul 16 23:14:35.075061 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:35.075036    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-r878t"]
Jul 16 23:14:36.180465 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:36.180441    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-r878t"
Jul 16 23:14:36.249594 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:36.249569    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4601af56-94c8-4837-aa79-dcbd5458f3da-maas-api-service-ca\") pod \"4601af56-94c8-4837-aa79-dcbd5458f3da\" (UID: \"4601af56-94c8-4837-aa79-dcbd5458f3da\") "
Jul 16 23:14:36.249900 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:36.249877    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4601af56-94c8-4837-aa79-dcbd5458f3da-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4601af56-94c8-4837-aa79-dcbd5458f3da" (UID: "4601af56-94c8-4837-aa79-dcbd5458f3da"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:14:36.350204 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:36.350179    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4601af56-94c8-4837-aa79-dcbd5458f3da-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:14:37.056866 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:37.056838    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-r878t"
Jul 16 23:14:37.057036 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:37.056839    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-r878t" event={"ID":"4601af56-94c8-4837-aa79-dcbd5458f3da","Type":"ContainerDied","Data":"e7fd745e36be8d441b5e34c61773abe41aeae79b51403c59796cb701fd6e0718"}
Jul 16 23:14:37.057036 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:37.056972    2573 scope.go:117] "RemoveContainer" containerID="24f488bd6205fe87f2c1a92ec1adf7a2c8b934bcc4edbfee90ed1b0e8844e4e6"
Jul 16 23:14:37.080206 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:37.080179    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-r878t"]
Jul 16 23:14:37.083650 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:37.083629    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-r878t"]
Jul 16 23:14:37.513338 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:37.513308    2573 scope.go:117] "RemoveContainer" containerID="2fecb3dce16c727deac6c12c2bbefed8acf5919166ff80cfcb488af460c9a397"
Jul 16 23:14:38.064070 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:38.064040    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-cvzk4" event={"ID":"05ed95cc-b8d0-46ad-b07a-f50c38542ca9","Type":"ContainerStarted","Data":"76285216a8f67a47b4a3b9dd9c74ee12da6e705247d8f7919183026608049e40"}
Jul 16 23:14:38.082229 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:38.082167    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-cvzk4" podStartSLOduration=16.082147116 podStartE2EDuration="16.082147116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:14:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:14:38.080493111 +0000 UTC m=+2452.090755775" watchObservedRunningTime="2026-07-16 23:14:38.082147116 +0000 UTC m=+2452.092409781"
Jul 16 23:14:38.518384 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:38.518353    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4601af56-94c8-4837-aa79-dcbd5458f3da" path="/var/lib/kubelet/pods/4601af56-94c8-4837-aa79-dcbd5458f3da/volumes"
Jul 16 23:14:39.070291 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:39.070261    2573 generic.go:356] "Generic (PLEG): container finished" podID="05ed95cc-b8d0-46ad-b07a-f50c38542ca9" containerID="76285216a8f67a47b4a3b9dd9c74ee12da6e705247d8f7919183026608049e40" exitCode=6
Jul 16 23:14:39.070486 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:39.070304    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-cvzk4" event={"ID":"05ed95cc-b8d0-46ad-b07a-f50c38542ca9","Type":"ContainerDied","Data":"76285216a8f67a47b4a3b9dd9c74ee12da6e705247d8f7919183026608049e40"}
Jul 16 23:14:39.070486 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:39.070351    2573 scope.go:117] "RemoveContainer" containerID="2fecb3dce16c727deac6c12c2bbefed8acf5919166ff80cfcb488af460c9a397"
Jul 16 23:14:39.070636 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:39.070618    2573 scope.go:117] "RemoveContainer" containerID="76285216a8f67a47b4a3b9dd9c74ee12da6e705247d8f7919183026608049e40"
Jul 16 23:14:39.070834 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:14:39.070815    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-cvzk4_odh-ai-gateway-infra(05ed95cc-b8d0-46ad-b07a-f50c38542ca9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-cvzk4" podUID="05ed95cc-b8d0-46ad-b07a-f50c38542ca9"
Jul 16 23:14:39.094493 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:39.094459    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-cvzk4"]
Jul 16 23:14:40.208073 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:40.208046    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-cvzk4"
Jul 16 23:14:40.277748 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:40.277714    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05ed95cc-b8d0-46ad-b07a-f50c38542ca9-maas-api-service-ca\") pod \"05ed95cc-b8d0-46ad-b07a-f50c38542ca9\" (UID: \"05ed95cc-b8d0-46ad-b07a-f50c38542ca9\") "
Jul 16 23:14:40.278091 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:40.278066    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05ed95cc-b8d0-46ad-b07a-f50c38542ca9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "05ed95cc-b8d0-46ad-b07a-f50c38542ca9" (UID: "05ed95cc-b8d0-46ad-b07a-f50c38542ca9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:14:40.379212 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:40.379151    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05ed95cc-b8d0-46ad-b07a-f50c38542ca9-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:14:41.082293 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:41.082254    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-cvzk4"
Jul 16 23:14:41.082442 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:41.082254    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-cvzk4" event={"ID":"05ed95cc-b8d0-46ad-b07a-f50c38542ca9","Type":"ContainerDied","Data":"6f5b1b51d22046d3497b07ae6db5113bacd1838c0fac76e01d69b77bbd057cf2"}
Jul 16 23:14:41.082442 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:41.082388    2573 scope.go:117] "RemoveContainer" containerID="76285216a8f67a47b4a3b9dd9c74ee12da6e705247d8f7919183026608049e40"
Jul 16 23:14:41.099118 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:41.099085    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-cvzk4"]
Jul 16 23:14:41.102891 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:41.102870    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-cvzk4"]
Jul 16 23:14:41.611627 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:41.611595    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-jt6tl"]
Jul 16 23:14:41.611948 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:41.611856    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05ed95cc-b8d0-46ad-b07a-f50c38542ca9" containerName="revoke-keys"
Jul 16 23:14:41.611948 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:41.611867    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="05ed95cc-b8d0-46ad-b07a-f50c38542ca9" containerName="revoke-keys"
Jul 16 23:14:41.611948 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:41.611876    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5c402c1-fc35-47c6-a418-4860ad898cf9" containerName="revoke-keys"
Jul 16 23:14:41.611948 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:41.611881    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5c402c1-fc35-47c6-a418-4860ad898cf9" containerName="revoke-keys"
Jul 16 23:14:41.611948 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:41.611893    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5c402c1-fc35-47c6-a418-4860ad898cf9" containerName="revoke-keys"
Jul 16 23:14:41.611948 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:41.611899    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5c402c1-fc35-47c6-a418-4860ad898cf9" containerName="revoke-keys"
Jul 16 23:14:41.611948 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:41.611907    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4601af56-94c8-4837-aa79-dcbd5458f3da" containerName="revoke-keys"
Jul 16 23:14:41.611948 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:41.611912    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4601af56-94c8-4837-aa79-dcbd5458f3da" containerName="revoke-keys"
Jul 16 23:14:41.611948 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:41.611918    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4601af56-94c8-4837-aa79-dcbd5458f3da" containerName="revoke-keys"
Jul 16 23:14:41.611948 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:41.611923    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4601af56-94c8-4837-aa79-dcbd5458f3da" containerName="revoke-keys"
Jul 16 23:14:41.611948 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:41.611932    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05ed95cc-b8d0-46ad-b07a-f50c38542ca9" containerName="revoke-keys"
Jul 16 23:14:41.611948 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:41.611936    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="05ed95cc-b8d0-46ad-b07a-f50c38542ca9" containerName="revoke-keys"
Jul 16 23:14:41.612328 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:41.611981    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5c402c1-fc35-47c6-a418-4860ad898cf9" containerName="revoke-keys"
Jul 16 23:14:41.612328 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:41.611988    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="05ed95cc-b8d0-46ad-b07a-f50c38542ca9" containerName="revoke-keys"
Jul 16 23:14:41.612328 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:41.611994    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4601af56-94c8-4837-aa79-dcbd5458f3da" containerName="revoke-keys"
Jul 16 23:14:41.612328 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:41.612001    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5c402c1-fc35-47c6-a418-4860ad898cf9" containerName="revoke-keys"
Jul 16 23:14:41.612328 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:41.612007    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4601af56-94c8-4837-aa79-dcbd5458f3da" containerName="revoke-keys"
Jul 16 23:14:41.612328 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:41.612012    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="05ed95cc-b8d0-46ad-b07a-f50c38542ca9" containerName="revoke-keys"
Jul 16 23:14:41.616134 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:41.616115    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-jt6tl"
Jul 16 23:14:41.618375 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:41.618355    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:14:41.618485 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:41.618355    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 23:14:41.622727 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:41.622699    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-jt6tl"]
Jul 16 23:14:41.687510 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:41.687483    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfaf1b5c-27c4-4d3a-966f-e8a97f3e8331-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-jt6tl\" (UID: \"dfaf1b5c-27c4-4d3a-966f-e8a97f3e8331\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-jt6tl"
Jul 16 23:14:41.788383 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:41.788360    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfaf1b5c-27c4-4d3a-966f-e8a97f3e8331-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-jt6tl\" (UID: \"dfaf1b5c-27c4-4d3a-966f-e8a97f3e8331\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-jt6tl"
Jul 16 23:14:41.788937 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:41.788913    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfaf1b5c-27c4-4d3a-966f-e8a97f3e8331-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-jt6tl\" (UID: \"dfaf1b5c-27c4-4d3a-966f-e8a97f3e8331\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-jt6tl"
Jul 16 23:14:41.927519 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:41.927466    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-jt6tl"
Jul 16 23:14:42.056271 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:42.056224    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-jt6tl"]
Jul 16 23:14:42.058494 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:14:42.058458    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfaf1b5c_27c4_4d3a_966f_e8a97f3e8331.slice/crio-433746780fb89705456e039366c54b886012db688238f036b6adfff45bf6cc23 WatchSource:0}: Error finding container 433746780fb89705456e039366c54b886012db688238f036b6adfff45bf6cc23: Status 404 returned error can't find the container with id 433746780fb89705456e039366c54b886012db688238f036b6adfff45bf6cc23
Jul 16 23:14:42.088324 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:42.088300    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-jt6tl" event={"ID":"dfaf1b5c-27c4-4d3a-966f-e8a97f3e8331","Type":"ContainerStarted","Data":"433746780fb89705456e039366c54b886012db688238f036b6adfff45bf6cc23"}
Jul 16 23:14:42.518964 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:42.518934    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05ed95cc-b8d0-46ad-b07a-f50c38542ca9" path="/var/lib/kubelet/pods/05ed95cc-b8d0-46ad-b07a-f50c38542ca9/volumes"
Jul 16 23:14:43.093965 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:43.093932    2573 generic.go:356] "Generic (PLEG): container finished" podID="dfaf1b5c-27c4-4d3a-966f-e8a97f3e8331" containerID="1f63840841af81b254926f90d47b393c8bb9d2c6301c8f7db5901b10bc0c1ae0" exitCode=6
Jul 16 23:14:43.094345 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:43.094018    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-jt6tl" event={"ID":"dfaf1b5c-27c4-4d3a-966f-e8a97f3e8331","Type":"ContainerDied","Data":"1f63840841af81b254926f90d47b393c8bb9d2c6301c8f7db5901b10bc0c1ae0"}
Jul 16 23:14:43.094345 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:43.094134    2573 scope.go:117] "RemoveContainer" containerID="1f63840841af81b254926f90d47b393c8bb9d2c6301c8f7db5901b10bc0c1ae0"
Jul 16 23:14:44.100500 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:44.100469    2573 generic.go:356] "Generic (PLEG): container finished" podID="dfaf1b5c-27c4-4d3a-966f-e8a97f3e8331" containerID="368ebfe945148e86cd8a4e2b0402077d78ab61ee2121c008ba1197f0dfe7a2ba" exitCode=6
Jul 16 23:14:44.100876 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:44.100543    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-jt6tl" event={"ID":"dfaf1b5c-27c4-4d3a-966f-e8a97f3e8331","Type":"ContainerDied","Data":"368ebfe945148e86cd8a4e2b0402077d78ab61ee2121c008ba1197f0dfe7a2ba"}
Jul 16 23:14:44.100876 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:44.100603    2573 scope.go:117] "RemoveContainer" containerID="1f63840841af81b254926f90d47b393c8bb9d2c6301c8f7db5901b10bc0c1ae0"
Jul 16 23:14:44.100876 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:44.100791    2573 scope.go:117] "RemoveContainer" containerID="368ebfe945148e86cd8a4e2b0402077d78ab61ee2121c008ba1197f0dfe7a2ba"
Jul 16 23:14:44.101027 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:14:44.101004    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-jt6tl_odh-ai-gateway-infra(dfaf1b5c-27c4-4d3a-966f-e8a97f3e8331)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-jt6tl" podUID="dfaf1b5c-27c4-4d3a-966f-e8a97f3e8331"
Jul 16 23:14:45.107260 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:45.107211    2573 scope.go:117] "RemoveContainer" containerID="368ebfe945148e86cd8a4e2b0402077d78ab61ee2121c008ba1197f0dfe7a2ba"
Jul 16 23:14:45.107712 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:14:45.107483    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-jt6tl_odh-ai-gateway-infra(dfaf1b5c-27c4-4d3a-966f-e8a97f3e8331)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-jt6tl" podUID="dfaf1b5c-27c4-4d3a-966f-e8a97f3e8331"
Jul 16 23:14:56.515779 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:56.515751    2573 scope.go:117] "RemoveContainer" containerID="368ebfe945148e86cd8a4e2b0402077d78ab61ee2121c008ba1197f0dfe7a2ba"
Jul 16 23:14:57.161452 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:57.161419    2573 generic.go:356] "Generic (PLEG): container finished" podID="dfaf1b5c-27c4-4d3a-966f-e8a97f3e8331" containerID="3da528ee1f8619a1935ea3a3330c3a6bc59c49b741224fdd57445ab60a553659" exitCode=6
Jul 16 23:14:57.161595 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:57.161496    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-jt6tl" event={"ID":"dfaf1b5c-27c4-4d3a-966f-e8a97f3e8331","Type":"ContainerDied","Data":"3da528ee1f8619a1935ea3a3330c3a6bc59c49b741224fdd57445ab60a553659"}
Jul 16 23:14:57.161595 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:57.161570    2573 scope.go:117] "RemoveContainer" containerID="368ebfe945148e86cd8a4e2b0402077d78ab61ee2121c008ba1197f0dfe7a2ba"
Jul 16 23:14:57.161868 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:57.161854    2573 scope.go:117] "RemoveContainer" containerID="3da528ee1f8619a1935ea3a3330c3a6bc59c49b741224fdd57445ab60a553659"
Jul 16 23:14:57.162079 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:14:57.162058    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-jt6tl_odh-ai-gateway-infra(dfaf1b5c-27c4-4d3a-966f-e8a97f3e8331)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-jt6tl" podUID="dfaf1b5c-27c4-4d3a-966f-e8a97f3e8331"
Jul 16 23:14:58.187397 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:58.187363    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-jt6tl"]
Jul 16 23:14:58.317857 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:58.317826    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-jt6tl"
Jul 16 23:14:58.405453 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:58.405425    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfaf1b5c-27c4-4d3a-966f-e8a97f3e8331-maas-api-service-ca\") pod \"dfaf1b5c-27c4-4d3a-966f-e8a97f3e8331\" (UID: \"dfaf1b5c-27c4-4d3a-966f-e8a97f3e8331\") "
Jul 16 23:14:58.405745 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:58.405723    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfaf1b5c-27c4-4d3a-966f-e8a97f3e8331-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dfaf1b5c-27c4-4d3a-966f-e8a97f3e8331" (UID: "dfaf1b5c-27c4-4d3a-966f-e8a97f3e8331"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:14:58.506759 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:58.506712    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfaf1b5c-27c4-4d3a-966f-e8a97f3e8331-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:14:59.173945 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:59.173915    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-jt6tl"
Jul 16 23:14:59.174117 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:59.173945    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-jt6tl" event={"ID":"dfaf1b5c-27c4-4d3a-966f-e8a97f3e8331","Type":"ContainerDied","Data":"433746780fb89705456e039366c54b886012db688238f036b6adfff45bf6cc23"}
Jul 16 23:14:59.174117 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:59.174006    2573 scope.go:117] "RemoveContainer" containerID="3da528ee1f8619a1935ea3a3330c3a6bc59c49b741224fdd57445ab60a553659"
Jul 16 23:14:59.189674 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:59.189652    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-jt6tl"]
Jul 16 23:14:59.191431 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:14:59.191411    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-jt6tl"]
Jul 16 23:15:00.138712 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:00.138668    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737395-kg7ct"]
Jul 16 23:15:00.139096 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:00.139080    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5c402c1-fc35-47c6-a418-4860ad898cf9" containerName="revoke-keys"
Jul 16 23:15:00.139140 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:00.139100    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5c402c1-fc35-47c6-a418-4860ad898cf9" containerName="revoke-keys"
Jul 16 23:15:00.139140 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:00.139117    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4601af56-94c8-4837-aa79-dcbd5458f3da" containerName="revoke-keys"
Jul 16 23:15:00.139140 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:00.139126    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4601af56-94c8-4837-aa79-dcbd5458f3da" containerName="revoke-keys"
Jul 16 23:15:00.139261 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:00.139144    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05ed95cc-b8d0-46ad-b07a-f50c38542ca9" containerName="revoke-keys"
Jul 16 23:15:00.139261 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:00.139153    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="05ed95cc-b8d0-46ad-b07a-f50c38542ca9" containerName="revoke-keys"
Jul 16 23:15:00.139261 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:00.139171    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfaf1b5c-27c4-4d3a-966f-e8a97f3e8331" containerName="revoke-keys"
Jul 16 23:15:00.139261 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:00.139180    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfaf1b5c-27c4-4d3a-966f-e8a97f3e8331" containerName="revoke-keys"
Jul 16 23:15:00.139261 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:00.139195    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfaf1b5c-27c4-4d3a-966f-e8a97f3e8331" containerName="revoke-keys"
Jul 16 23:15:00.139261 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:00.139203    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfaf1b5c-27c4-4d3a-966f-e8a97f3e8331" containerName="revoke-keys"
Jul 16 23:15:00.139261 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:00.139217    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfaf1b5c-27c4-4d3a-966f-e8a97f3e8331" containerName="revoke-keys"
Jul 16 23:15:00.139261 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:00.139225    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfaf1b5c-27c4-4d3a-966f-e8a97f3e8331" containerName="revoke-keys"
Jul 16 23:15:00.139498 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:00.139313    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfaf1b5c-27c4-4d3a-966f-e8a97f3e8331" containerName="revoke-keys"
Jul 16 23:15:00.139498 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:00.139323    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5c402c1-fc35-47c6-a418-4860ad898cf9" containerName="revoke-keys"
Jul 16 23:15:00.139498 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:00.139331    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4601af56-94c8-4837-aa79-dcbd5458f3da" containerName="revoke-keys"
Jul 16 23:15:00.139498 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:00.139343    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfaf1b5c-27c4-4d3a-966f-e8a97f3e8331" containerName="revoke-keys"
Jul 16 23:15:00.139498 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:00.139353    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfaf1b5c-27c4-4d3a-966f-e8a97f3e8331" containerName="revoke-keys"
Jul 16 23:15:00.139498 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:00.139362    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="05ed95cc-b8d0-46ad-b07a-f50c38542ca9" containerName="revoke-keys"
Jul 16 23:15:00.143767 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:00.143750    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-f263f2f2-29737395-gkj2v"]
Jul 16 23:15:00.143887 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:00.143872    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737395-kg7ct"
Jul 16 23:15:00.146678 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:00.146656    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:15:00.146874 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:00.146859    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-f263f2f2-29737395-gkj2v"
Jul 16 23:15:00.149426 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:00.149400    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-f263f2f2-29737395-gkj2v"]
Jul 16 23:15:00.152705 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:00.152682    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737395-kg7ct"]
Jul 16 23:15:00.156683 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:00.156662    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737395-kg7ct"
Jul 16 23:15:00.165179 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:00.165153    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-f263f2f2-29737395-gkj2v"
Jul 16 23:15:00.296307 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:00.296272    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737395-kg7ct"]
Jul 16 23:15:00.300508 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:15:00.300482    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59453ec6_6eac_47fe_add4_63aa8b128063.slice/crio-636ece8d6e0be74e2f755f91ec264c3e4dc7fb53afd86cf713392d745216771f WatchSource:0}: Error finding container 636ece8d6e0be74e2f755f91ec264c3e4dc7fb53afd86cf713392d745216771f: Status 404 returned error can't find the container with id 636ece8d6e0be74e2f755f91ec264c3e4dc7fb53afd86cf713392d745216771f
Jul 16 23:15:00.318867 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:00.318844    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-f263f2f2-29737395-gkj2v"]
Jul 16 23:15:00.320536 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:15:00.320503    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fc1b63b_e0f0_4ae9_8ce2_893c4581ff55.slice/crio-4a510c6a515bf369af5ea1740675b9ca7819c4c19aec227c38e6810a36b1d5ea WatchSource:0}: Error finding container 4a510c6a515bf369af5ea1740675b9ca7819c4c19aec227c38e6810a36b1d5ea: Status 404 returned error can't find the container with id 4a510c6a515bf369af5ea1740675b9ca7819c4c19aec227c38e6810a36b1d5ea
Jul 16 23:15:00.518396 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:00.518374    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfaf1b5c-27c4-4d3a-966f-e8a97f3e8331" path="/var/lib/kubelet/pods/dfaf1b5c-27c4-4d3a-966f-e8a97f3e8331/volumes"
Jul 16 23:15:00.587456 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:00.587429    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-f4wlp"]
Jul 16 23:15:00.590624 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:00.590609    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-f4wlp"
Jul 16 23:15:00.593014 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:00.592997    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 23:15:00.598967 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:00.598946    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-f4wlp"]
Jul 16 23:15:00.621746 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:00.621721    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c99341a-aa15-4f99-a45a-05829e73b7cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-f4wlp\" (UID: \"1c99341a-aa15-4f99-a45a-05829e73b7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-f4wlp"
Jul 16 23:15:00.722493 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:00.722452    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c99341a-aa15-4f99-a45a-05829e73b7cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-f4wlp\" (UID: \"1c99341a-aa15-4f99-a45a-05829e73b7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-f4wlp"
Jul 16 23:15:00.723097 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:00.723070    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c99341a-aa15-4f99-a45a-05829e73b7cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-f4wlp\" (UID: \"1c99341a-aa15-4f99-a45a-05829e73b7cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-f4wlp"
Jul 16 23:15:00.901325 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:00.901298    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-f4wlp"
Jul 16 23:15:01.027915 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:01.027891    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-f4wlp"]
Jul 16 23:15:01.033137 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:15:01.033111    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c99341a_aa15_4f99_a45a_05829e73b7cf.slice/crio-e8e36f7b104774683bec4866d44f615742134b2a146a8f4e887d789dcf48e279 WatchSource:0}: Error finding container e8e36f7b104774683bec4866d44f615742134b2a146a8f4e887d789dcf48e279: Status 404 returned error can't find the container with id e8e36f7b104774683bec4866d44f615742134b2a146a8f4e887d789dcf48e279
Jul 16 23:15:01.189137 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:01.189057    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737395-kg7ct" event={"ID":"59453ec6-6eac-47fe-add4-63aa8b128063","Type":"ContainerStarted","Data":"1ea1b1a2796143a381275e233f7fe46750d1a3bfa55c06a9bbc89ede32fda52d"}
Jul 16 23:15:01.189137 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:01.189113    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737395-kg7ct" event={"ID":"59453ec6-6eac-47fe-add4-63aa8b128063","Type":"ContainerStarted","Data":"636ece8d6e0be74e2f755f91ec264c3e4dc7fb53afd86cf713392d745216771f"}
Jul 16 23:15:01.190185 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:01.190162    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-f4wlp" event={"ID":"1c99341a-aa15-4f99-a45a-05829e73b7cf","Type":"ContainerStarted","Data":"e8e36f7b104774683bec4866d44f615742134b2a146a8f4e887d789dcf48e279"}
Jul 16 23:15:01.191763 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:01.191471    2573 generic.go:356] "Generic (PLEG): container finished" podID="8fc1b63b-e0f0-4ae9-8ce2-893c4581ff55" containerID="71df1e2072b7fa7f60b2c9024bfa6ac31b201aa21dc8d8a3567c369f9d19725f" exitCode=7
Jul 16 23:15:01.191763 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:01.191519    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-f263f2f2-29737395-gkj2v" event={"ID":"8fc1b63b-e0f0-4ae9-8ce2-893c4581ff55","Type":"ContainerDied","Data":"71df1e2072b7fa7f60b2c9024bfa6ac31b201aa21dc8d8a3567c369f9d19725f"}
Jul 16 23:15:01.191763 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:01.191565    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-f263f2f2-29737395-gkj2v" event={"ID":"8fc1b63b-e0f0-4ae9-8ce2-893c4581ff55","Type":"ContainerStarted","Data":"4a510c6a515bf369af5ea1740675b9ca7819c4c19aec227c38e6810a36b1d5ea"}
Jul 16 23:15:01.191763 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:01.191725    2573 scope.go:117] "RemoveContainer" containerID="71df1e2072b7fa7f60b2c9024bfa6ac31b201aa21dc8d8a3567c369f9d19725f"
Jul 16 23:15:01.204267 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:01.204195    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737395-kg7ct" podStartSLOduration=1.2041791609999999 podStartE2EDuration="1.204179161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:15:01.202729373 +0000 UTC m=+2475.212992039" watchObservedRunningTime="2026-07-16 23:15:01.204179161 +0000 UTC m=+2475.214441826"
Jul 16 23:15:02.198971 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:02.198890    2573 generic.go:356] "Generic (PLEG): container finished" podID="59453ec6-6eac-47fe-add4-63aa8b128063" containerID="1ea1b1a2796143a381275e233f7fe46750d1a3bfa55c06a9bbc89ede32fda52d" exitCode=0
Jul 16 23:15:02.199445 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:02.198982    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737395-kg7ct" event={"ID":"59453ec6-6eac-47fe-add4-63aa8b128063","Type":"ContainerDied","Data":"1ea1b1a2796143a381275e233f7fe46750d1a3bfa55c06a9bbc89ede32fda52d"}
Jul 16 23:15:02.200550 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:02.200525    2573 generic.go:356] "Generic (PLEG): container finished" podID="1c99341a-aa15-4f99-a45a-05829e73b7cf" containerID="e548dc23ed07da3117195216c801940e0fe6b59580ee76fa59067c2dcc6ac202" exitCode=6
Jul 16 23:15:02.200637 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:02.200614    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-f4wlp" event={"ID":"1c99341a-aa15-4f99-a45a-05829e73b7cf","Type":"ContainerDied","Data":"e548dc23ed07da3117195216c801940e0fe6b59580ee76fa59067c2dcc6ac202"}
Jul 16 23:15:02.200751 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:02.200736    2573 scope.go:117] "RemoveContainer" containerID="e548dc23ed07da3117195216c801940e0fe6b59580ee76fa59067c2dcc6ac202"
Jul 16 23:15:02.202293 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:02.202275    2573 generic.go:356] "Generic (PLEG): container finished" podID="8fc1b63b-e0f0-4ae9-8ce2-893c4581ff55" containerID="96b9ac14e4c6ce5fbc401f234876ffbcff072d5c42087d0ea6f395fada221db2" exitCode=7
Jul 16 23:15:02.202381 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:02.202303    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-f263f2f2-29737395-gkj2v" event={"ID":"8fc1b63b-e0f0-4ae9-8ce2-893c4581ff55","Type":"ContainerDied","Data":"96b9ac14e4c6ce5fbc401f234876ffbcff072d5c42087d0ea6f395fada221db2"}
Jul 16 23:15:02.202381 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:02.202341    2573 scope.go:117] "RemoveContainer" containerID="71df1e2072b7fa7f60b2c9024bfa6ac31b201aa21dc8d8a3567c369f9d19725f"
Jul 16 23:15:02.202496 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:02.202470    2573 scope.go:117] "RemoveContainer" containerID="96b9ac14e4c6ce5fbc401f234876ffbcff072d5c42087d0ea6f395fada221db2"
Jul 16 23:15:02.202708 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:15:02.202681    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-f263f2f2-29737395-gkj2v_odh-ai-gateway-infra(8fc1b63b-e0f0-4ae9-8ce2-893c4581ff55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-f263f2f2-29737395-gkj2v" podUID="8fc1b63b-e0f0-4ae9-8ce2-893c4581ff55"
Jul 16 23:15:03.210131 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:03.210098    2573 generic.go:356] "Generic (PLEG): container finished" podID="1c99341a-aa15-4f99-a45a-05829e73b7cf" containerID="99b3260b9b9c6b2ae60411a07ba981ac8c439bc0eb5999e590d93f09f42b58c3" exitCode=6
Jul 16 23:15:03.210586 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:03.210188    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-f4wlp" event={"ID":"1c99341a-aa15-4f99-a45a-05829e73b7cf","Type":"ContainerDied","Data":"99b3260b9b9c6b2ae60411a07ba981ac8c439bc0eb5999e590d93f09f42b58c3"}
Jul 16 23:15:03.210586 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:03.210279    2573 scope.go:117] "RemoveContainer" containerID="e548dc23ed07da3117195216c801940e0fe6b59580ee76fa59067c2dcc6ac202"
Jul 16 23:15:03.210586 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:03.210416    2573 scope.go:117] "RemoveContainer" containerID="99b3260b9b9c6b2ae60411a07ba981ac8c439bc0eb5999e590d93f09f42b58c3"
Jul 16 23:15:03.210771 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:15:03.210659    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-f4wlp_odh-ai-gateway-infra(1c99341a-aa15-4f99-a45a-05829e73b7cf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-f4wlp" podUID="1c99341a-aa15-4f99-a45a-05829e73b7cf"
Jul 16 23:15:03.211934 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:03.211917    2573 scope.go:117] "RemoveContainer" containerID="96b9ac14e4c6ce5fbc401f234876ffbcff072d5c42087d0ea6f395fada221db2"
Jul 16 23:15:03.212156 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:15:03.212095    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-f263f2f2-29737395-gkj2v_odh-ai-gateway-infra(8fc1b63b-e0f0-4ae9-8ce2-893c4581ff55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-f263f2f2-29737395-gkj2v" podUID="8fc1b63b-e0f0-4ae9-8ce2-893c4581ff55"
Jul 16 23:15:03.329153 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:03.329129    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737395-kg7ct"
Jul 16 23:15:04.218688 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:04.218662    2573 scope.go:117] "RemoveContainer" containerID="99b3260b9b9c6b2ae60411a07ba981ac8c439bc0eb5999e590d93f09f42b58c3"
Jul 16 23:15:04.219099 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:15:04.218838    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-f4wlp_odh-ai-gateway-infra(1c99341a-aa15-4f99-a45a-05829e73b7cf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-f4wlp" podUID="1c99341a-aa15-4f99-a45a-05829e73b7cf"
Jul 16 23:15:04.219870 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:04.219853    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737395-kg7ct"
Jul 16 23:15:04.219950 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:04.219880    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737395-kg7ct" event={"ID":"59453ec6-6eac-47fe-add4-63aa8b128063","Type":"ContainerDied","Data":"636ece8d6e0be74e2f755f91ec264c3e4dc7fb53afd86cf713392d745216771f"}
Jul 16 23:15:04.219950 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:04.219919    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="636ece8d6e0be74e2f755f91ec264c3e4dc7fb53afd86cf713392d745216771f"
Jul 16 23:15:07.782315 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:07.782283    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-q5mv2"]
Jul 16 23:15:07.782699 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:07.782578    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59453ec6-6eac-47fe-add4-63aa8b128063" containerName="cleanup"
Jul 16 23:15:07.782699 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:07.782589    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="59453ec6-6eac-47fe-add4-63aa8b128063" containerName="cleanup"
Jul 16 23:15:07.782699 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:07.782649    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="59453ec6-6eac-47fe-add4-63aa8b128063" containerName="cleanup"
Jul 16 23:15:07.785550 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:07.785533    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-q5mv2"
Jul 16 23:15:07.794166 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:07.794139    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-q5mv2"]
Jul 16 23:15:07.881296 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:07.881263    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b34b5f9-f42d-4341-bfec-a662f6ee188c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-q5mv2\" (UID: \"5b34b5f9-f42d-4341-bfec-a662f6ee188c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-q5mv2"
Jul 16 23:15:07.981956 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:07.981930    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b34b5f9-f42d-4341-bfec-a662f6ee188c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-q5mv2\" (UID: \"5b34b5f9-f42d-4341-bfec-a662f6ee188c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-q5mv2"
Jul 16 23:15:07.982524 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:07.982503    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b34b5f9-f42d-4341-bfec-a662f6ee188c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-q5mv2\" (UID: \"5b34b5f9-f42d-4341-bfec-a662f6ee188c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-q5mv2"
Jul 16 23:15:08.097611 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:08.097593    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-q5mv2"
Jul 16 23:15:08.220811 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:08.220786    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-q5mv2"]
Jul 16 23:15:08.222683 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:15:08.222653    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b34b5f9_f42d_4341_bfec_a662f6ee188c.slice/crio-fb3073eea620d675186236fa57d31b560e301dfa04d624bef21f70e1f2b94576 WatchSource:0}: Error finding container fb3073eea620d675186236fa57d31b560e301dfa04d624bef21f70e1f2b94576: Status 404 returned error can't find the container with id fb3073eea620d675186236fa57d31b560e301dfa04d624bef21f70e1f2b94576
Jul 16 23:15:08.240296 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:08.240265    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-q5mv2" event={"ID":"5b34b5f9-f42d-4341-bfec-a662f6ee188c","Type":"ContainerStarted","Data":"fb3073eea620d675186236fa57d31b560e301dfa04d624bef21f70e1f2b94576"}
Jul 16 23:15:09.247109 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:09.247070    2573 generic.go:356] "Generic (PLEG): container finished" podID="5b34b5f9-f42d-4341-bfec-a662f6ee188c" containerID="9cc8a7d75a809530912eb2bfbac4eacd9abe9aa77cd3fdd115cb838e7fceec46" exitCode=7
Jul 16 23:15:09.247528 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:09.247148    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-q5mv2" event={"ID":"5b34b5f9-f42d-4341-bfec-a662f6ee188c","Type":"ContainerDied","Data":"9cc8a7d75a809530912eb2bfbac4eacd9abe9aa77cd3fdd115cb838e7fceec46"}
Jul 16 23:15:09.247528 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:09.247289    2573 scope.go:117] "RemoveContainer" containerID="9cc8a7d75a809530912eb2bfbac4eacd9abe9aa77cd3fdd115cb838e7fceec46"
Jul 16 23:15:10.253477 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:10.253405    2573 generic.go:356] "Generic (PLEG): container finished" podID="5b34b5f9-f42d-4341-bfec-a662f6ee188c" containerID="964442086bbbd4f2075509e6ce619be3a5d6ecfb3ddd4ed911b6e828d2ac4886" exitCode=7
Jul 16 23:15:10.253477 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:10.253451    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-q5mv2" event={"ID":"5b34b5f9-f42d-4341-bfec-a662f6ee188c","Type":"ContainerDied","Data":"964442086bbbd4f2075509e6ce619be3a5d6ecfb3ddd4ed911b6e828d2ac4886"}
Jul 16 23:15:10.253877 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:10.253510    2573 scope.go:117] "RemoveContainer" containerID="9cc8a7d75a809530912eb2bfbac4eacd9abe9aa77cd3fdd115cb838e7fceec46"
Jul 16 23:15:10.253877 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:10.253690    2573 scope.go:117] "RemoveContainer" containerID="964442086bbbd4f2075509e6ce619be3a5d6ecfb3ddd4ed911b6e828d2ac4886"
Jul 16 23:15:10.253877 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:15:10.253870    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-q5mv2_odh-ai-gateway-infra(5b34b5f9-f42d-4341-bfec-a662f6ee188c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-q5mv2" podUID="5b34b5f9-f42d-4341-bfec-a662f6ee188c"
Jul 16 23:15:11.260179 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:11.260147    2573 scope.go:117] "RemoveContainer" containerID="964442086bbbd4f2075509e6ce619be3a5d6ecfb3ddd4ed911b6e828d2ac4886"
Jul 16 23:15:11.260575 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:15:11.260323    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-q5mv2_odh-ai-gateway-infra(5b34b5f9-f42d-4341-bfec-a662f6ee188c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-q5mv2" podUID="5b34b5f9-f42d-4341-bfec-a662f6ee188c"
Jul 16 23:15:12.843348 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:12.843309    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-bb7rc"]
Jul 16 23:15:12.847474 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:12.847455    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-bb7rc"
Jul 16 23:15:12.864045 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:12.864014    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-bb7rc"]
Jul 16 23:15:12.919174 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:12.919136    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a26ba45-401d-487e-a3a0-38e76343c093-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-bb7rc\" (UID: \"6a26ba45-401d-487e-a3a0-38e76343c093\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-bb7rc"
Jul 16 23:15:13.020544 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:13.020497    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a26ba45-401d-487e-a3a0-38e76343c093-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-bb7rc\" (UID: \"6a26ba45-401d-487e-a3a0-38e76343c093\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-bb7rc"
Jul 16 23:15:13.021124 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:13.021103    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a26ba45-401d-487e-a3a0-38e76343c093-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-bb7rc\" (UID: \"6a26ba45-401d-487e-a3a0-38e76343c093\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-bb7rc"
Jul 16 23:15:13.156742 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:13.156641    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-bb7rc"
Jul 16 23:15:13.289441 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:13.289416    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-bb7rc"]
Jul 16 23:15:13.292063 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:15:13.292040    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a26ba45_401d_487e_a3a0_38e76343c093.slice/crio-cca0164639cb775fd471fd754792526fe264b19aaf4d8b059a602a35ca1fd70a WatchSource:0}: Error finding container cca0164639cb775fd471fd754792526fe264b19aaf4d8b059a602a35ca1fd70a: Status 404 returned error can't find the container with id cca0164639cb775fd471fd754792526fe264b19aaf4d8b059a602a35ca1fd70a
Jul 16 23:15:14.277183 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:14.277101    2573 generic.go:356] "Generic (PLEG): container finished" podID="6a26ba45-401d-487e-a3a0-38e76343c093" containerID="d58eaf85c96aed8161f29d56e30d660d01814f542f037894cc0e74733d6afb39" exitCode=6
Jul 16 23:15:14.277183 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:14.277160    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-bb7rc" event={"ID":"6a26ba45-401d-487e-a3a0-38e76343c093","Type":"ContainerDied","Data":"d58eaf85c96aed8161f29d56e30d660d01814f542f037894cc0e74733d6afb39"}
Jul 16 23:15:14.277581 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:14.277199    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-bb7rc" event={"ID":"6a26ba45-401d-487e-a3a0-38e76343c093","Type":"ContainerStarted","Data":"cca0164639cb775fd471fd754792526fe264b19aaf4d8b059a602a35ca1fd70a"}
Jul 16 23:15:14.277581 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:14.277369    2573 scope.go:117] "RemoveContainer" containerID="d58eaf85c96aed8161f29d56e30d660d01814f542f037894cc0e74733d6afb39"
Jul 16 23:15:14.513468 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:14.513445    2573 scope.go:117] "RemoveContainer" containerID="99b3260b9b9c6b2ae60411a07ba981ac8c439bc0eb5999e590d93f09f42b58c3"
Jul 16 23:15:15.284262 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:15.284158    2573 generic.go:356] "Generic (PLEG): container finished" podID="6a26ba45-401d-487e-a3a0-38e76343c093" containerID="fa575f8bee23e53941cfeb24e0ea1bae9736b1d730c69cb4bc47eee12e1b6fac" exitCode=6
Jul 16 23:15:15.284684 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:15.284254    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-bb7rc" event={"ID":"6a26ba45-401d-487e-a3a0-38e76343c093","Type":"ContainerDied","Data":"fa575f8bee23e53941cfeb24e0ea1bae9736b1d730c69cb4bc47eee12e1b6fac"}
Jul 16 23:15:15.284684 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:15.284306    2573 scope.go:117] "RemoveContainer" containerID="d58eaf85c96aed8161f29d56e30d660d01814f542f037894cc0e74733d6afb39"
Jul 16 23:15:15.284684 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:15.284552    2573 scope.go:117] "RemoveContainer" containerID="fa575f8bee23e53941cfeb24e0ea1bae9736b1d730c69cb4bc47eee12e1b6fac"
Jul 16 23:15:15.284845 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:15:15.284759    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-bb7rc_odh-ai-gateway-infra(6a26ba45-401d-487e-a3a0-38e76343c093)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-bb7rc" podUID="6a26ba45-401d-487e-a3a0-38e76343c093"
Jul 16 23:15:15.286128 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:15.286104    2573 generic.go:356] "Generic (PLEG): container finished" podID="1c99341a-aa15-4f99-a45a-05829e73b7cf" containerID="57e9af510ddccca1f74eae667252096b4595f3377ac4508e8e76c0a9ec5f8b0a" exitCode=6
Jul 16 23:15:15.286231 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:15.286172    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-f4wlp" event={"ID":"1c99341a-aa15-4f99-a45a-05829e73b7cf","Type":"ContainerDied","Data":"57e9af510ddccca1f74eae667252096b4595f3377ac4508e8e76c0a9ec5f8b0a"}
Jul 16 23:15:15.286485 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:15.286470    2573 scope.go:117] "RemoveContainer" containerID="57e9af510ddccca1f74eae667252096b4595f3377ac4508e8e76c0a9ec5f8b0a"
Jul 16 23:15:15.286633 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:15:15.286617    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-f4wlp_odh-ai-gateway-infra(1c99341a-aa15-4f99-a45a-05829e73b7cf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-f4wlp" podUID="1c99341a-aa15-4f99-a45a-05829e73b7cf"
Jul 16 23:15:15.295046 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:15.295027    2573 scope.go:117] "RemoveContainer" containerID="99b3260b9b9c6b2ae60411a07ba981ac8c439bc0eb5999e590d93f09f42b58c3"
Jul 16 23:15:15.513979 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:15.513945    2573 scope.go:117] "RemoveContainer" containerID="96b9ac14e4c6ce5fbc401f234876ffbcff072d5c42087d0ea6f395fada221db2"
Jul 16 23:15:16.294799 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:16.294759    2573 scope.go:117] "RemoveContainer" containerID="fa575f8bee23e53941cfeb24e0ea1bae9736b1d730c69cb4bc47eee12e1b6fac"
Jul 16 23:15:16.295268 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:15:16.295000    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-bb7rc_odh-ai-gateway-infra(6a26ba45-401d-487e-a3a0-38e76343c093)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-bb7rc" podUID="6a26ba45-401d-487e-a3a0-38e76343c093"
Jul 16 23:15:16.297754 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:16.297729    2573 generic.go:356] "Generic (PLEG): container finished" podID="8fc1b63b-e0f0-4ae9-8ce2-893c4581ff55" containerID="b33562573061f5bdaf96314d477caca962c4df462c33ba5b6497aef1e63385ba" exitCode=7
Jul 16 23:15:16.297842 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:16.297807    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-f263f2f2-29737395-gkj2v" event={"ID":"8fc1b63b-e0f0-4ae9-8ce2-893c4581ff55","Type":"ContainerDied","Data":"b33562573061f5bdaf96314d477caca962c4df462c33ba5b6497aef1e63385ba"}
Jul 16 23:15:16.297906 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:16.297894    2573 scope.go:117] "RemoveContainer" containerID="96b9ac14e4c6ce5fbc401f234876ffbcff072d5c42087d0ea6f395fada221db2"
Jul 16 23:15:16.298115 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:16.298100    2573 scope.go:117] "RemoveContainer" containerID="b33562573061f5bdaf96314d477caca962c4df462c33ba5b6497aef1e63385ba"
Jul 16 23:15:16.298329 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:15:16.298298    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-f263f2f2-29737395-gkj2v_odh-ai-gateway-infra(8fc1b63b-e0f0-4ae9-8ce2-893c4581ff55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-f263f2f2-29737395-gkj2v" podUID="8fc1b63b-e0f0-4ae9-8ce2-893c4581ff55"
Jul 16 23:15:16.351335 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:16.351272    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-f4wlp"]
Jul 16 23:15:16.480099 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:16.480078    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-f4wlp"
Jul 16 23:15:16.549180 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:16.549154    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c99341a-aa15-4f99-a45a-05829e73b7cf-maas-api-service-ca\") pod \"1c99341a-aa15-4f99-a45a-05829e73b7cf\" (UID: \"1c99341a-aa15-4f99-a45a-05829e73b7cf\") "
Jul 16 23:15:16.549514 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:16.549495    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c99341a-aa15-4f99-a45a-05829e73b7cf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1c99341a-aa15-4f99-a45a-05829e73b7cf" (UID: "1c99341a-aa15-4f99-a45a-05829e73b7cf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:15:16.649977 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:16.649911    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c99341a-aa15-4f99-a45a-05829e73b7cf-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:15:17.304131 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:17.304103    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-f4wlp"
Jul 16 23:15:17.304131 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:17.304123    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-f4wlp" event={"ID":"1c99341a-aa15-4f99-a45a-05829e73b7cf","Type":"ContainerDied","Data":"e8e36f7b104774683bec4866d44f615742134b2a146a8f4e887d789dcf48e279"}
Jul 16 23:15:17.304694 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:17.304186    2573 scope.go:117] "RemoveContainer" containerID="57e9af510ddccca1f74eae667252096b4595f3377ac4508e8e76c0a9ec5f8b0a"
Jul 16 23:15:17.347482 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:17.347456    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-f4wlp"]
Jul 16 23:15:17.362923 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:17.362886    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-f263f2f2-29737395-gkj2v"]
Jul 16 23:15:17.365052 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:17.365029    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-f4wlp"]
Jul 16 23:15:17.490809 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:17.490783    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-f263f2f2-29737395-gkj2v"
Jul 16 23:15:17.924095 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:17.924054    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:15:18.029266 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:18.029219    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:15:18.312557 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:18.312477    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-f263f2f2-29737395-gkj2v" event={"ID":"8fc1b63b-e0f0-4ae9-8ce2-893c4581ff55","Type":"ContainerDied","Data":"4a510c6a515bf369af5ea1740675b9ca7819c4c19aec227c38e6810a36b1d5ea"}
Jul 16 23:15:18.312557 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:18.312495    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-f263f2f2-29737395-gkj2v"
Jul 16 23:15:18.312557 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:18.312534    2573 scope.go:117] "RemoveContainer" containerID="b33562573061f5bdaf96314d477caca962c4df462c33ba5b6497aef1e63385ba"
Jul 16 23:15:18.343101 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:18.343076    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-f263f2f2-29737395-gkj2v"]
Jul 16 23:15:18.348447 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:18.348426    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-f263f2f2-29737395-gkj2v"]
Jul 16 23:15:18.518687 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:18.518650    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c99341a-aa15-4f99-a45a-05829e73b7cf" path="/var/lib/kubelet/pods/1c99341a-aa15-4f99-a45a-05829e73b7cf/volumes"
Jul 16 23:15:18.519007 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:18.518987    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fc1b63b-e0f0-4ae9-8ce2-893c4581ff55" path="/var/lib/kubelet/pods/8fc1b63b-e0f0-4ae9-8ce2-893c4581ff55/volumes"
Jul 16 23:15:20.841468 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:20.841434    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:15:21.513118 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:21.513090    2573 scope.go:117] "RemoveContainer" containerID="964442086bbbd4f2075509e6ce619be3a5d6ecfb3ddd4ed911b6e828d2ac4886"
Jul 16 23:15:22.333898 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:22.333812    2573 generic.go:356] "Generic (PLEG): container finished" podID="5b34b5f9-f42d-4341-bfec-a662f6ee188c" containerID="40b176a2719f19d5018a59dd59302cd4f9859483998954000537dc1df443bb9a" exitCode=7
Jul 16 23:15:22.334302 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:22.333892    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-q5mv2" event={"ID":"5b34b5f9-f42d-4341-bfec-a662f6ee188c","Type":"ContainerDied","Data":"40b176a2719f19d5018a59dd59302cd4f9859483998954000537dc1df443bb9a"}
Jul 16 23:15:22.334302 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:22.333957    2573 scope.go:117] "RemoveContainer" containerID="964442086bbbd4f2075509e6ce619be3a5d6ecfb3ddd4ed911b6e828d2ac4886"
Jul 16 23:15:22.334302 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:22.334254    2573 scope.go:117] "RemoveContainer" containerID="40b176a2719f19d5018a59dd59302cd4f9859483998954000537dc1df443bb9a"
Jul 16 23:15:22.334497 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:15:22.334476    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-q5mv2_odh-ai-gateway-infra(5b34b5f9-f42d-4341-bfec-a662f6ee188c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-q5mv2" podUID="5b34b5f9-f42d-4341-bfec-a662f6ee188c"
Jul 16 23:15:23.217156 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:23.217126    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:15:23.362686 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:23.362658    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-q5mv2"]
Jul 16 23:15:23.490034 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:23.490009    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-q5mv2"
Jul 16 23:15:23.606396 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:23.606374    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b34b5f9-f42d-4341-bfec-a662f6ee188c-maas-api-service-ca\") pod \"5b34b5f9-f42d-4341-bfec-a662f6ee188c\" (UID: \"5b34b5f9-f42d-4341-bfec-a662f6ee188c\") "
Jul 16 23:15:23.606697 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:23.606675    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b34b5f9-f42d-4341-bfec-a662f6ee188c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5b34b5f9-f42d-4341-bfec-a662f6ee188c" (UID: "5b34b5f9-f42d-4341-bfec-a662f6ee188c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:15:23.707137 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:23.707078    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b34b5f9-f42d-4341-bfec-a662f6ee188c-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:15:24.346400 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:24.346369    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-q5mv2"
Jul 16 23:15:24.346400 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:24.346389    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-q5mv2" event={"ID":"5b34b5f9-f42d-4341-bfec-a662f6ee188c","Type":"ContainerDied","Data":"fb3073eea620d675186236fa57d31b560e301dfa04d624bef21f70e1f2b94576"}
Jul 16 23:15:24.346635 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:24.346445    2573 scope.go:117] "RemoveContainer" containerID="40b176a2719f19d5018a59dd59302cd4f9859483998954000537dc1df443bb9a"
Jul 16 23:15:24.370880 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:24.370846    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-q5mv2"]
Jul 16 23:15:24.374122 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:24.374095    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-q5mv2"]
Jul 16 23:15:24.407789 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:24.407762    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:15:24.518204 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:24.518171    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b34b5f9-f42d-4341-bfec-a662f6ee188c" path="/var/lib/kubelet/pods/5b34b5f9-f42d-4341-bfec-a662f6ee188c/volumes"
Jul 16 23:15:27.514138 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:27.514110    2573 scope.go:117] "RemoveContainer" containerID="fa575f8bee23e53941cfeb24e0ea1bae9736b1d730c69cb4bc47eee12e1b6fac"
Jul 16 23:15:28.372192 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:28.372110    2573 generic.go:356] "Generic (PLEG): container finished" podID="6a26ba45-401d-487e-a3a0-38e76343c093" containerID="fb1b915eef6db1a8c108abc7acc24ad1b1ef5f41a67e9206eb5986e42b27a074" exitCode=6
Jul 16 23:15:28.372355 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:28.372187    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-bb7rc" event={"ID":"6a26ba45-401d-487e-a3a0-38e76343c093","Type":"ContainerDied","Data":"fb1b915eef6db1a8c108abc7acc24ad1b1ef5f41a67e9206eb5986e42b27a074"}
Jul 16 23:15:28.372355 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:28.372284    2573 scope.go:117] "RemoveContainer" containerID="fa575f8bee23e53941cfeb24e0ea1bae9736b1d730c69cb4bc47eee12e1b6fac"
Jul 16 23:15:28.372510 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:28.372497    2573 scope.go:117] "RemoveContainer" containerID="fb1b915eef6db1a8c108abc7acc24ad1b1ef5f41a67e9206eb5986e42b27a074"
Jul 16 23:15:28.372699 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:15:28.372679    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-bb7rc_odh-ai-gateway-infra(6a26ba45-401d-487e-a3a0-38e76343c093)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-bb7rc" podUID="6a26ba45-401d-487e-a3a0-38e76343c093"
Jul 16 23:15:29.397462 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:29.397423    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-bb7rc"]
Jul 16 23:15:29.529897 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:29.529876    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-bb7rc"
Jul 16 23:15:29.650267 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:29.650176    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a26ba45-401d-487e-a3a0-38e76343c093-maas-api-service-ca\") pod \"6a26ba45-401d-487e-a3a0-38e76343c093\" (UID: \"6a26ba45-401d-487e-a3a0-38e76343c093\") "
Jul 16 23:15:29.650565 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:29.650543    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a26ba45-401d-487e-a3a0-38e76343c093-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6a26ba45-401d-487e-a3a0-38e76343c093" (UID: "6a26ba45-401d-487e-a3a0-38e76343c093"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:15:29.751645 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:29.751618    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a26ba45-401d-487e-a3a0-38e76343c093-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:15:30.218404 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:30.218369    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:15:30.318933 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:30.318909    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:15:30.384110 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:30.384074    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-bb7rc" event={"ID":"6a26ba45-401d-487e-a3a0-38e76343c093","Type":"ContainerDied","Data":"cca0164639cb775fd471fd754792526fe264b19aaf4d8b059a602a35ca1fd70a"}
Jul 16 23:15:30.384220 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:30.384129    2573 scope.go:117] "RemoveContainer" containerID="fb1b915eef6db1a8c108abc7acc24ad1b1ef5f41a67e9206eb5986e42b27a074"
Jul 16 23:15:30.384220 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:30.384080    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-bb7rc"
Jul 16 23:15:30.426505 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:30.426483    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-bb7rc"]
Jul 16 23:15:30.428176 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:30.428158    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-bb7rc"]
Jul 16 23:15:30.518122 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:30.518074    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a26ba45-401d-487e-a3a0-38e76343c093" path="/var/lib/kubelet/pods/6a26ba45-401d-487e-a3a0-38e76343c093/volumes"
Jul 16 23:15:30.723566 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:30.723538    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:15:30.814646 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:30.814592    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:15:30.914871 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:30.914845    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:15:32.345926 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:32.345894    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:15:34.440445 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:34.440415    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:15:35.048026 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:35.047994    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:15:35.148181 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:35.148148    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:15:38.560040 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:38.560011    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:15:38.670796 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:38.670726    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:15:38.782636 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:38.782603    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:15:38.864306 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:38.864278    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:15:39.265805 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:15:39.265723    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:16:05.189390 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:16:05.189349    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:16:10.984591 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:16:10.984560    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:16:11.084504 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:16:11.084468    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:16:11.184354 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:16:11.184316    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:16:27.012048 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:16:27.012016    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:16:32.869602 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:16:32.869564    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:16:33.053127 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:16:33.053095    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:16:33.134164 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:16:33.134087    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:16:43.579410 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:16:43.579377    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:16:44.340463 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:16:44.340429    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:16:50.078511 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:16:50.078479    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:16:50.181139 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:16:50.181108    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:16:50.278397 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:16:50.278367    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:16:50.398099 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:16:50.398031    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:16:51.160119 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:16:51.160073    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:16:51.260079 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:16:51.260053    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:16:51.363695 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:16:51.363668    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:16:52.447590 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:16:52.447528    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:16:52.893532 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:16:52.893496    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:16:58.796390 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:16:58.796359    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:16:58.856681 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:16:58.856654    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:16:59.477221 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:16:59.477185    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:17:01.032192 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:01.032158    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:17:05.298427 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:05.298393    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:17:05.363719 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:05.363683    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:17:05.556522 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:05.556440    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:17:05.866601 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:05.866548    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:17:06.167135 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:06.167048    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:17:09.148163 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:09.148022    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:17:09.241486 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:09.241456    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:17:09.351263 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:09.351209    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:17:09.520955 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:09.520873    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:17:09.809859 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:09.809778    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:17:09.920304 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:09.920267    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:17:11.737565 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:11.737529    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-4l9wf"]
Jul 16 23:17:11.737933 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:11.737817    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fc1b63b-e0f0-4ae9-8ce2-893c4581ff55" containerName="cleanup"
Jul 16 23:17:11.737933 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:11.737828    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fc1b63b-e0f0-4ae9-8ce2-893c4581ff55" containerName="cleanup"
Jul 16 23:17:11.737933 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:11.737841    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a26ba45-401d-487e-a3a0-38e76343c093" containerName="revoke-keys"
Jul 16 23:17:11.737933 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:11.737847    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a26ba45-401d-487e-a3a0-38e76343c093" containerName="revoke-keys"
Jul 16 23:17:11.737933 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:11.737854    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b34b5f9-f42d-4341-bfec-a662f6ee188c" containerName="revoke-keys"
Jul 16 23:17:11.737933 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:11.737859    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b34b5f9-f42d-4341-bfec-a662f6ee188c" containerName="revoke-keys"
Jul 16 23:17:11.737933 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:11.737867    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b34b5f9-f42d-4341-bfec-a662f6ee188c" containerName="revoke-keys"
Jul 16 23:17:11.737933 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:11.737872    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b34b5f9-f42d-4341-bfec-a662f6ee188c" containerName="revoke-keys"
Jul 16 23:17:11.737933 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:11.737877    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a26ba45-401d-487e-a3a0-38e76343c093" containerName="revoke-keys"
Jul 16 23:17:11.737933 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:11.737882    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a26ba45-401d-487e-a3a0-38e76343c093" containerName="revoke-keys"
Jul 16 23:17:11.737933 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:11.737890    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c99341a-aa15-4f99-a45a-05829e73b7cf" containerName="revoke-keys"
Jul 16 23:17:11.737933 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:11.737895    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c99341a-aa15-4f99-a45a-05829e73b7cf" containerName="revoke-keys"
Jul 16 23:17:11.737933 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:11.737905    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c99341a-aa15-4f99-a45a-05829e73b7cf" containerName="revoke-keys"
Jul 16 23:17:11.737933 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:11.737910    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c99341a-aa15-4f99-a45a-05829e73b7cf" containerName="revoke-keys"
Jul 16 23:17:11.737933 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:11.737918    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fc1b63b-e0f0-4ae9-8ce2-893c4581ff55" containerName="cleanup"
Jul 16 23:17:11.737933 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:11.737923    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fc1b63b-e0f0-4ae9-8ce2-893c4581ff55" containerName="cleanup"
Jul 16 23:17:11.737933 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:11.737929    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b34b5f9-f42d-4341-bfec-a662f6ee188c" containerName="revoke-keys"
Jul 16 23:17:11.737933 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:11.737935    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b34b5f9-f42d-4341-bfec-a662f6ee188c" containerName="revoke-keys"
Jul 16 23:17:11.737933 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:11.737940    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c99341a-aa15-4f99-a45a-05829e73b7cf" containerName="revoke-keys"
Jul 16 23:17:11.738535 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:11.737945    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c99341a-aa15-4f99-a45a-05829e73b7cf" containerName="revoke-keys"
Jul 16 23:17:11.738535 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:11.737993    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b34b5f9-f42d-4341-bfec-a662f6ee188c" containerName="revoke-keys"
Jul 16 23:17:11.738535 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:11.738000    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fc1b63b-e0f0-4ae9-8ce2-893c4581ff55" containerName="cleanup"
Jul 16 23:17:11.738535 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:11.738006    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a26ba45-401d-487e-a3a0-38e76343c093" containerName="revoke-keys"
Jul 16 23:17:11.738535 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:11.738011    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fc1b63b-e0f0-4ae9-8ce2-893c4581ff55" containerName="cleanup"
Jul 16 23:17:11.738535 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:11.738017    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c99341a-aa15-4f99-a45a-05829e73b7cf" containerName="revoke-keys"
Jul 16 23:17:11.738535 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:11.738022    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b34b5f9-f42d-4341-bfec-a662f6ee188c" containerName="revoke-keys"
Jul 16 23:17:11.738535 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:11.738027    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a26ba45-401d-487e-a3a0-38e76343c093" containerName="revoke-keys"
Jul 16 23:17:11.738535 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:11.738033    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c99341a-aa15-4f99-a45a-05829e73b7cf" containerName="revoke-keys"
Jul 16 23:17:11.738535 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:11.738038    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a26ba45-401d-487e-a3a0-38e76343c093" containerName="revoke-keys"
Jul 16 23:17:11.741010 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:11.740993    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-4l9wf"
Jul 16 23:17:11.743331 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:11.743310    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:17:11.743331 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:11.743325    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 23:17:11.746949 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:11.746926    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-4l9wf"]
Jul 16 23:17:11.875341 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:11.875304    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4c5a5f1-6de2-448a-88fd-0c1543117229-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-4l9wf\" (UID: \"b4c5a5f1-6de2-448a-88fd-0c1543117229\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-4l9wf"
Jul 16 23:17:11.975819 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:11.975775    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4c5a5f1-6de2-448a-88fd-0c1543117229-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-4l9wf\" (UID: \"b4c5a5f1-6de2-448a-88fd-0c1543117229\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-4l9wf"
Jul 16 23:17:11.976483 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:11.976457    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4c5a5f1-6de2-448a-88fd-0c1543117229-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-4l9wf\" (UID: \"b4c5a5f1-6de2-448a-88fd-0c1543117229\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-4l9wf"
Jul 16 23:17:12.052419 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:12.052328    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-4l9wf"
Jul 16 23:17:12.182489 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:12.182454    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-4l9wf"]
Jul 16 23:17:12.184633 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:17:12.184602    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4c5a5f1_6de2_448a_88fd_0c1543117229.slice/crio-5c47cc55aa03c02d2c9a3e8927497f052b55ac331c3e68390da5b21709ef55ae WatchSource:0}: Error finding container 5c47cc55aa03c02d2c9a3e8927497f052b55ac331c3e68390da5b21709ef55ae: Status 404 returned error can't find the container with id 5c47cc55aa03c02d2c9a3e8927497f052b55ac331c3e68390da5b21709ef55ae
Jul 16 23:17:12.849811 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:12.849774    2573 generic.go:356] "Generic (PLEG): container finished" podID="b4c5a5f1-6de2-448a-88fd-0c1543117229" containerID="a9cad001848356c2bcf2aa7861594fda5530de4d4452354e28b59e99d5b56f14" exitCode=6
Jul 16 23:17:12.850149 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:12.849844    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-4l9wf" event={"ID":"b4c5a5f1-6de2-448a-88fd-0c1543117229","Type":"ContainerDied","Data":"a9cad001848356c2bcf2aa7861594fda5530de4d4452354e28b59e99d5b56f14"}
Jul 16 23:17:12.850149 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:12.849896    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-4l9wf" event={"ID":"b4c5a5f1-6de2-448a-88fd-0c1543117229","Type":"ContainerStarted","Data":"5c47cc55aa03c02d2c9a3e8927497f052b55ac331c3e68390da5b21709ef55ae"}
Jul 16 23:17:12.850258 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:12.850159    2573 scope.go:117] "RemoveContainer" containerID="a9cad001848356c2bcf2aa7861594fda5530de4d4452354e28b59e99d5b56f14"
Jul 16 23:17:13.632198 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:13.632165    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:17:13.857394 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:13.857359    2573 generic.go:356] "Generic (PLEG): container finished" podID="b4c5a5f1-6de2-448a-88fd-0c1543117229" containerID="4315c1ad9734fe5d143bf330251d55240484e3936c70866c95a55623a2de6d72" exitCode=6
Jul 16 23:17:13.857813 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:13.857451    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-4l9wf" event={"ID":"b4c5a5f1-6de2-448a-88fd-0c1543117229","Type":"ContainerDied","Data":"4315c1ad9734fe5d143bf330251d55240484e3936c70866c95a55623a2de6d72"}
Jul 16 23:17:13.857813 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:13.857509    2573 scope.go:117] "RemoveContainer" containerID="a9cad001848356c2bcf2aa7861594fda5530de4d4452354e28b59e99d5b56f14"
Jul 16 23:17:13.857813 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:13.857608    2573 scope.go:117] "RemoveContainer" containerID="4315c1ad9734fe5d143bf330251d55240484e3936c70866c95a55623a2de6d72"
Jul 16 23:17:13.857932 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:17:13.857819    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-4l9wf_odh-ai-gateway-infra(b4c5a5f1-6de2-448a-88fd-0c1543117229)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-4l9wf" podUID="b4c5a5f1-6de2-448a-88fd-0c1543117229"
Jul 16 23:17:14.257625 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:14.257548    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:17:14.357246 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:14.357215    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:17:14.532710 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:14.532630    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:17:14.673117 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:14.673090    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:17:14.863957 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:14.863926    2573 scope.go:117] "RemoveContainer" containerID="4315c1ad9734fe5d143bf330251d55240484e3936c70866c95a55623a2de6d72"
Jul 16 23:17:14.864423 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:17:14.864173    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-4l9wf_odh-ai-gateway-infra(b4c5a5f1-6de2-448a-88fd-0c1543117229)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-4l9wf" podUID="b4c5a5f1-6de2-448a-88fd-0c1543117229"
Jul 16 23:17:20.539324 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:20.539264    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:17:20.596533 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:20.596496    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:17:20.863860 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:20.863820    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4b61afbb-dcvjr"]
Jul 16 23:17:20.864194 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:20.864177    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a26ba45-401d-487e-a3a0-38e76343c093" containerName="revoke-keys"
Jul 16 23:17:20.864261 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:20.864196    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a26ba45-401d-487e-a3a0-38e76343c093" containerName="revoke-keys"
Jul 16 23:17:20.864261 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:20.864214    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fc1b63b-e0f0-4ae9-8ce2-893c4581ff55" containerName="cleanup"
Jul 16 23:17:20.864261 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:20.864220    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fc1b63b-e0f0-4ae9-8ce2-893c4581ff55" containerName="cleanup"
Jul 16 23:17:20.864376 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:20.864287    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fc1b63b-e0f0-4ae9-8ce2-893c4581ff55" containerName="cleanup"
Jul 16 23:17:20.864376 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:20.864297    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b34b5f9-f42d-4341-bfec-a662f6ee188c" containerName="revoke-keys"
Jul 16 23:17:20.864376 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:20.864303    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c99341a-aa15-4f99-a45a-05829e73b7cf" containerName="revoke-keys"
Jul 16 23:17:20.868482 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:20.868464    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4b61afbb-dcvjr"
Jul 16 23:17:20.874281 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:20.874254    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4b61afbb-dcvjr"]
Jul 16 23:17:20.950687 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:20.950653    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/960f3ef4-671f-4f86-8a81-8ac8994b5695-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4b61afbb-dcvjr\" (UID: \"960f3ef4-671f-4f86-8a81-8ac8994b5695\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4b61afbb-dcvjr"
Jul 16 23:17:21.052077 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:21.052035    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/960f3ef4-671f-4f86-8a81-8ac8994b5695-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4b61afbb-dcvjr\" (UID: \"960f3ef4-671f-4f86-8a81-8ac8994b5695\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4b61afbb-dcvjr"
Jul 16 23:17:21.052640 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:21.052621    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/960f3ef4-671f-4f86-8a81-8ac8994b5695-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4b61afbb-dcvjr\" (UID: \"960f3ef4-671f-4f86-8a81-8ac8994b5695\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4b61afbb-dcvjr"
Jul 16 23:17:21.180578 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:21.180460    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4b61afbb-dcvjr"
Jul 16 23:17:21.316182 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:21.316153    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4b61afbb-dcvjr"]
Jul 16 23:17:21.318893 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:17:21.318862    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod960f3ef4_671f_4f86_8a81_8ac8994b5695.slice/crio-d49d05ecccdd2624aaba172cbfa54c2054eb54bfa0dd17d3ab83d46390367d87 WatchSource:0}: Error finding container d49d05ecccdd2624aaba172cbfa54c2054eb54bfa0dd17d3ab83d46390367d87: Status 404 returned error can't find the container with id d49d05ecccdd2624aaba172cbfa54c2054eb54bfa0dd17d3ab83d46390367d87
Jul 16 23:17:21.900162 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:21.900123    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4b61afbb-dcvjr" event={"ID":"960f3ef4-671f-4f86-8a81-8ac8994b5695","Type":"ContainerStarted","Data":"271610c8058b1789ac5810494c2b29aa364e660c38eea42d8bee18f3515ea2b5"}
Jul 16 23:17:21.900551 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:21.900187    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4b61afbb-dcvjr" event={"ID":"960f3ef4-671f-4f86-8a81-8ac8994b5695","Type":"ContainerStarted","Data":"d49d05ecccdd2624aaba172cbfa54c2054eb54bfa0dd17d3ab83d46390367d87"}
Jul 16 23:17:21.913960 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:21.913911    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4b61afbb-dcvjr" podStartSLOduration=1.913892079 podStartE2EDuration="1.913892079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:17:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:17:21.913194374 +0000 UTC m=+2615.923457059" watchObservedRunningTime="2026-07-16 23:17:21.913892079 +0000 UTC m=+2615.924154743"
Jul 16 23:17:22.906645 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:22.906613    2573 generic.go:356] "Generic (PLEG): container finished" podID="960f3ef4-671f-4f86-8a81-8ac8994b5695" containerID="271610c8058b1789ac5810494c2b29aa364e660c38eea42d8bee18f3515ea2b5" exitCode=0
Jul 16 23:17:22.907006 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:22.906664    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4b61afbb-dcvjr" event={"ID":"960f3ef4-671f-4f86-8a81-8ac8994b5695","Type":"ContainerDied","Data":"271610c8058b1789ac5810494c2b29aa364e660c38eea42d8bee18f3515ea2b5"}
Jul 16 23:17:24.036440 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:24.036418    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4b61afbb-dcvjr"
Jul 16 23:17:24.181714 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:24.181651    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/960f3ef4-671f-4f86-8a81-8ac8994b5695-maas-api-service-ca\") pod \"960f3ef4-671f-4f86-8a81-8ac8994b5695\" (UID: \"960f3ef4-671f-4f86-8a81-8ac8994b5695\") "
Jul 16 23:17:24.181984 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:24.181962    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/960f3ef4-671f-4f86-8a81-8ac8994b5695-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "960f3ef4-671f-4f86-8a81-8ac8994b5695" (UID: "960f3ef4-671f-4f86-8a81-8ac8994b5695"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:17:24.282668 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:24.282642    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/960f3ef4-671f-4f86-8a81-8ac8994b5695-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:17:24.917582 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:24.917556    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4b61afbb-dcvjr"
Jul 16 23:17:24.917747 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:24.917556    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4b61afbb-dcvjr" event={"ID":"960f3ef4-671f-4f86-8a81-8ac8994b5695","Type":"ContainerDied","Data":"d49d05ecccdd2624aaba172cbfa54c2054eb54bfa0dd17d3ab83d46390367d87"}
Jul 16 23:17:24.917747 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:24.917682    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d49d05ecccdd2624aaba172cbfa54c2054eb54bfa0dd17d3ab83d46390367d87"
Jul 16 23:17:26.516383 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:26.516357    2573 scope.go:117] "RemoveContainer" containerID="4315c1ad9734fe5d143bf330251d55240484e3936c70866c95a55623a2de6d72"
Jul 16 23:17:26.929317 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:26.929287    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-4l9wf" event={"ID":"b4c5a5f1-6de2-448a-88fd-0c1543117229","Type":"ContainerStarted","Data":"929d64e29d0c4da9929b04b8ab9724cbde3c6592d351173c75381a9f1db88a32"}
Jul 16 23:17:26.944195 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:26.944153    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-4l9wf" podStartSLOduration=15.944140351 podStartE2EDuration="15.944140351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:17:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:17:26.943032425 +0000 UTC m=+2620.953295092" watchObservedRunningTime="2026-07-16 23:17:26.944140351 +0000 UTC m=+2620.954403014"
Jul 16 23:17:27.936566 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:27.936531    2573 generic.go:356] "Generic (PLEG): container finished" podID="b4c5a5f1-6de2-448a-88fd-0c1543117229" containerID="929d64e29d0c4da9929b04b8ab9724cbde3c6592d351173c75381a9f1db88a32" exitCode=6
Jul 16 23:17:27.936995 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:27.936570    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-4l9wf" event={"ID":"b4c5a5f1-6de2-448a-88fd-0c1543117229","Type":"ContainerDied","Data":"929d64e29d0c4da9929b04b8ab9724cbde3c6592d351173c75381a9f1db88a32"}
Jul 16 23:17:27.936995 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:27.936614    2573 scope.go:117] "RemoveContainer" containerID="4315c1ad9734fe5d143bf330251d55240484e3936c70866c95a55623a2de6d72"
Jul 16 23:17:27.936995 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:27.936847    2573 scope.go:117] "RemoveContainer" containerID="929d64e29d0c4da9929b04b8ab9724cbde3c6592d351173c75381a9f1db88a32"
Jul 16 23:17:27.937121 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:17:27.937032    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-4l9wf_odh-ai-gateway-infra(b4c5a5f1-6de2-448a-88fd-0c1543117229)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-4l9wf" podUID="b4c5a5f1-6de2-448a-88fd-0c1543117229"
Jul 16 23:17:27.960725 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:27.960702    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-4l9wf"]
Jul 16 23:17:29.067975 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:29.067954    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-4l9wf"
Jul 16 23:17:29.222610 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:29.222535    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4c5a5f1-6de2-448a-88fd-0c1543117229-maas-api-service-ca\") pod \"b4c5a5f1-6de2-448a-88fd-0c1543117229\" (UID: \"b4c5a5f1-6de2-448a-88fd-0c1543117229\") "
Jul 16 23:17:29.222891 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:29.222864    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4c5a5f1-6de2-448a-88fd-0c1543117229-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b4c5a5f1-6de2-448a-88fd-0c1543117229" (UID: "b4c5a5f1-6de2-448a-88fd-0c1543117229"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:17:29.323964 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:29.323934    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4c5a5f1-6de2-448a-88fd-0c1543117229-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:17:29.953447 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:29.953413    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-4l9wf" event={"ID":"b4c5a5f1-6de2-448a-88fd-0c1543117229","Type":"ContainerDied","Data":"5c47cc55aa03c02d2c9a3e8927497f052b55ac331c3e68390da5b21709ef55ae"}
Jul 16 23:17:29.953447 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:29.953444    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-4l9wf"
Jul 16 23:17:29.953665 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:29.953495    2573 scope.go:117] "RemoveContainer" containerID="929d64e29d0c4da9929b04b8ab9724cbde3c6592d351173c75381a9f1db88a32"
Jul 16 23:17:29.978690 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:29.978606    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-4l9wf"]
Jul 16 23:17:29.980867 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:29.980843    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-4l9wf"]
Jul 16 23:17:30.518549 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:30.518520    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4c5a5f1-6de2-448a-88fd-0c1543117229" path="/var/lib/kubelet/pods/b4c5a5f1-6de2-448a-88fd-0c1543117229/volumes"
Jul 16 23:17:31.516997 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:31.516963    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:17:34.139548 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:34.139516    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:17:35.946722 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:35.946691    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:17:41.822643 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:41.822608    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:17:41.922796 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:41.922769    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:17:46.595500 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:46.595457    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:17:46.816204 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:46.816172    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:17:46.940319 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:46.940204    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:17:47.039468 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:47.039437    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:17:47.165060 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:47.165024    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:17:53.006458 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:53.006417    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:17:53.110013 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:53.109980    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:17:53.207622 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:53.207590    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:17:53.336714 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:53.336680    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:17:59.057381 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:59.057345    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:17:59.156827 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:17:59.156782    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:18:05.714739 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:05.714703    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:18:06.026623 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:06.026538    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:18:06.372621 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:06.372584    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:18:08.466136 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:08.466101    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:18:08.571029 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:08.570997    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:18:08.636071 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:08.636041    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:18:08.802790 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:08.802707    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:18:09.117928 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:09.117893    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:18:09.219637 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:09.219602    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:18:12.938366 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:12.938331    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:18:15.833690 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:15.833655    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:18:22.832177 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:22.832142    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-9752734e/e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2"]
Jul 16 23:18:22.832691 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:22.832434    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="960f3ef4-671f-4f86-8a81-8ac8994b5695" containerName="revoke-keys"
Jul 16 23:18:22.832691 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:22.832447    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="960f3ef4-671f-4f86-8a81-8ac8994b5695" containerName="revoke-keys"
Jul 16 23:18:22.832691 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:22.832458    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4c5a5f1-6de2-448a-88fd-0c1543117229" containerName="revoke-keys"
Jul 16 23:18:22.832691 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:22.832463    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4c5a5f1-6de2-448a-88fd-0c1543117229" containerName="revoke-keys"
Jul 16 23:18:22.832691 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:22.832477    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4c5a5f1-6de2-448a-88fd-0c1543117229" containerName="revoke-keys"
Jul 16 23:18:22.832691 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:22.832483    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4c5a5f1-6de2-448a-88fd-0c1543117229" containerName="revoke-keys"
Jul 16 23:18:22.832691 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:22.832530    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4c5a5f1-6de2-448a-88fd-0c1543117229" containerName="revoke-keys"
Jul 16 23:18:22.832691 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:22.832537    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="960f3ef4-671f-4f86-8a81-8ac8994b5695" containerName="revoke-keys"
Jul 16 23:18:22.832691 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:22.832545    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4c5a5f1-6de2-448a-88fd-0c1543117229" containerName="revoke-keys"
Jul 16 23:18:22.835487 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:22.835471    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9752734e/e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2"
Jul 16 23:18:22.838463 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:22.838432    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-9752734e\"/\"kube-root-ca.crt\""
Jul 16 23:18:22.838591 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:22.838476    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-9752734e\"/\"default-dockercfg-sf79c\""
Jul 16 23:18:22.838591 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:22.838549    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-9752734e\"/\"e2e-lifecycle-model-9752734e-kserve-self-signed-certs\""
Jul 16 23:18:22.838591 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:22.838492    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-9752734e\"/\"openshift-service-ca.crt\""
Jul 16 23:18:22.845185 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:22.845165    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-9752734e/e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2"]
Jul 16 23:18:22.926521 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:22.926484    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:18:22.940746 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:22.940716    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9ff1625-8e38-44a4-8ab4-529c5d16e4e4-home\") pod \"e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2\" (UID: \"c9ff1625-8e38-44a4-8ab4-529c5d16e4e4\") " pod="ai-tenant-e2e-mt-9752734e/e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2"
Jul 16 23:18:22.940891 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:22.940771    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9ff1625-8e38-44a4-8ab4-529c5d16e4e4-tls-certs\") pod \"e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2\" (UID: \"c9ff1625-8e38-44a4-8ab4-529c5d16e4e4\") " pod="ai-tenant-e2e-mt-9752734e/e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2"
Jul 16 23:18:22.940891 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:22.940801    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9ff1625-8e38-44a4-8ab4-529c5d16e4e4-dshm\") pod \"e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2\" (UID: \"c9ff1625-8e38-44a4-8ab4-529c5d16e4e4\") " pod="ai-tenant-e2e-mt-9752734e/e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2"
Jul 16 23:18:22.940891 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:22.940837    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9ff1625-8e38-44a4-8ab4-529c5d16e4e4-tmp-dir\") pod \"e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2\" (UID: \"c9ff1625-8e38-44a4-8ab4-529c5d16e4e4\") " pod="ai-tenant-e2e-mt-9752734e/e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2"
Jul 16 23:18:22.941024 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:22.940930    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9ff1625-8e38-44a4-8ab4-529c5d16e4e4-model-cache\") pod \"e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2\" (UID: \"c9ff1625-8e38-44a4-8ab4-529c5d16e4e4\") " pod="ai-tenant-e2e-mt-9752734e/e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2"
Jul 16 23:18:22.941024 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:22.940965    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wsrk\" (UniqueName: \"kubernetes.io/projected/c9ff1625-8e38-44a4-8ab4-529c5d16e4e4-kube-api-access-8wsrk\") pod \"e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2\" (UID: \"c9ff1625-8e38-44a4-8ab4-529c5d16e4e4\") " pod="ai-tenant-e2e-mt-9752734e/e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2"
Jul 16 23:18:23.041307 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:23.041281    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9ff1625-8e38-44a4-8ab4-529c5d16e4e4-home\") pod \"e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2\" (UID: \"c9ff1625-8e38-44a4-8ab4-529c5d16e4e4\") " pod="ai-tenant-e2e-mt-9752734e/e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2"
Jul 16 23:18:23.041471 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:23.041314    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9ff1625-8e38-44a4-8ab4-529c5d16e4e4-tls-certs\") pod \"e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2\" (UID: \"c9ff1625-8e38-44a4-8ab4-529c5d16e4e4\") " pod="ai-tenant-e2e-mt-9752734e/e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2"
Jul 16 23:18:23.041471 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:23.041334    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9ff1625-8e38-44a4-8ab4-529c5d16e4e4-dshm\") pod \"e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2\" (UID: \"c9ff1625-8e38-44a4-8ab4-529c5d16e4e4\") " pod="ai-tenant-e2e-mt-9752734e/e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2"
Jul 16 23:18:23.041471 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:23.041356    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9ff1625-8e38-44a4-8ab4-529c5d16e4e4-tmp-dir\") pod \"e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2\" (UID: \"c9ff1625-8e38-44a4-8ab4-529c5d16e4e4\") " pod="ai-tenant-e2e-mt-9752734e/e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2"
Jul 16 23:18:23.041471 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:23.041399    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9ff1625-8e38-44a4-8ab4-529c5d16e4e4-model-cache\") pod \"e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2\" (UID: \"c9ff1625-8e38-44a4-8ab4-529c5d16e4e4\") " pod="ai-tenant-e2e-mt-9752734e/e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2"
Jul 16 23:18:23.041471 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:23.041428    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wsrk\" (UniqueName: \"kubernetes.io/projected/c9ff1625-8e38-44a4-8ab4-529c5d16e4e4-kube-api-access-8wsrk\") pod \"e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2\" (UID: \"c9ff1625-8e38-44a4-8ab4-529c5d16e4e4\") " pod="ai-tenant-e2e-mt-9752734e/e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2"
Jul 16 23:18:23.041770 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:23.041741    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9ff1625-8e38-44a4-8ab4-529c5d16e4e4-home\") pod \"e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2\" (UID: \"c9ff1625-8e38-44a4-8ab4-529c5d16e4e4\") " pod="ai-tenant-e2e-mt-9752734e/e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2"
Jul 16 23:18:23.041817 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:23.041776    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9ff1625-8e38-44a4-8ab4-529c5d16e4e4-model-cache\") pod \"e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2\" (UID: \"c9ff1625-8e38-44a4-8ab4-529c5d16e4e4\") " pod="ai-tenant-e2e-mt-9752734e/e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2"
Jul 16 23:18:23.041854 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:23.041816    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9ff1625-8e38-44a4-8ab4-529c5d16e4e4-tmp-dir\") pod \"e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2\" (UID: \"c9ff1625-8e38-44a4-8ab4-529c5d16e4e4\") " pod="ai-tenant-e2e-mt-9752734e/e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2"
Jul 16 23:18:23.043705 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:23.043683    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9ff1625-8e38-44a4-8ab4-529c5d16e4e4-dshm\") pod \"e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2\" (UID: \"c9ff1625-8e38-44a4-8ab4-529c5d16e4e4\") " pod="ai-tenant-e2e-mt-9752734e/e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2"
Jul 16 23:18:23.043830 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:23.043815    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9ff1625-8e38-44a4-8ab4-529c5d16e4e4-tls-certs\") pod \"e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2\" (UID: \"c9ff1625-8e38-44a4-8ab4-529c5d16e4e4\") " pod="ai-tenant-e2e-mt-9752734e/e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2"
Jul 16 23:18:23.048818 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:23.048798    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wsrk\" (UniqueName: \"kubernetes.io/projected/c9ff1625-8e38-44a4-8ab4-529c5d16e4e4-kube-api-access-8wsrk\") pod \"e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2\" (UID: \"c9ff1625-8e38-44a4-8ab4-529c5d16e4e4\") " pod="ai-tenant-e2e-mt-9752734e/e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2"
Jul 16 23:18:23.144944 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:23.144890    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9752734e/e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2"
Jul 16 23:18:23.273692 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:23.273660    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-9752734e/e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2"]
Jul 16 23:18:23.276964 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:18:23.276939    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9ff1625_8e38_44a4_8ab4_529c5d16e4e4.slice/crio-004717573f0f732ff22d2acefa13d76218cc31c1eba55e0f53b0e1a088e22c76 WatchSource:0}: Error finding container 004717573f0f732ff22d2acefa13d76218cc31c1eba55e0f53b0e1a088e22c76: Status 404 returned error can't find the container with id 004717573f0f732ff22d2acefa13d76218cc31c1eba55e0f53b0e1a088e22c76
Jul 16 23:18:24.211409 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:24.211370    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9752734e/e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2" event={"ID":"c9ff1625-8e38-44a4-8ab4-529c5d16e4e4","Type":"ContainerStarted","Data":"4ebd158b7879087e5ad9b230d6e0a8e66b9ca3a4afb96cc1b2475afdaf995f1c"}
Jul 16 23:18:24.211777 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:24.211422    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9752734e/e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2" event={"ID":"c9ff1625-8e38-44a4-8ab4-529c5d16e4e4","Type":"ContainerStarted","Data":"004717573f0f732ff22d2acefa13d76218cc31c1eba55e0f53b0e1a088e22c76"}
Jul 16 23:18:24.230906 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:24.230864    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-9752734e/e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2" podStartSLOduration=2.230848088 podStartE2EDuration="2.230848088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:18:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:18:24.229730796 +0000 UTC m=+2678.239993462" watchObservedRunningTime="2026-07-16 23:18:24.230848088 +0000 UTC m=+2678.241110751"
Jul 16 23:18:27.626033 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:27.625998    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:18:30.748824 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:30.748791    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-dnt5c"]
Jul 16 23:18:30.749176 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:30.749086    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4c5a5f1-6de2-448a-88fd-0c1543117229" containerName="revoke-keys"
Jul 16 23:18:30.749176 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:30.749097    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4c5a5f1-6de2-448a-88fd-0c1543117229" containerName="revoke-keys"
Jul 16 23:18:30.749176 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:30.749159    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4c5a5f1-6de2-448a-88fd-0c1543117229" containerName="revoke-keys"
Jul 16 23:18:30.752024 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:30.752007    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-dnt5c"
Jul 16 23:18:30.754357 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:30.754327    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 23:18:30.754357 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:30.754327    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:18:30.758510 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:30.758485    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-dnt5c"]
Jul 16 23:18:30.902258 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:30.902222    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8a68f23-7b44-407f-9578-24af8d3b3ebb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-dnt5c\" (UID: \"f8a68f23-7b44-407f-9578-24af8d3b3ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-dnt5c"
Jul 16 23:18:31.002788 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:31.002722    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8a68f23-7b44-407f-9578-24af8d3b3ebb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-dnt5c\" (UID: \"f8a68f23-7b44-407f-9578-24af8d3b3ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-dnt5c"
Jul 16 23:18:31.003339 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:31.003320    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8a68f23-7b44-407f-9578-24af8d3b3ebb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-dnt5c\" (UID: \"f8a68f23-7b44-407f-9578-24af8d3b3ebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-dnt5c"
Jul 16 23:18:31.064907 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:31.064878    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-dnt5c"
Jul 16 23:18:31.398321 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:31.398296    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-dnt5c"]
Jul 16 23:18:31.400481 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:18:31.400456    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8a68f23_7b44_407f_9578_24af8d3b3ebb.slice/crio-085578bc9c60c9870341a79f132dedfe8bae01193430bb3bdd2a27bea2bb9a93 WatchSource:0}: Error finding container 085578bc9c60c9870341a79f132dedfe8bae01193430bb3bdd2a27bea2bb9a93: Status 404 returned error can't find the container with id 085578bc9c60c9870341a79f132dedfe8bae01193430bb3bdd2a27bea2bb9a93
Jul 16 23:18:32.247805 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:32.247718    2573 generic.go:356] "Generic (PLEG): container finished" podID="f8a68f23-7b44-407f-9578-24af8d3b3ebb" containerID="0be6f411feee4de97de6e7a702d90301c721bff9be05a10522d1db9e701bb3f1" exitCode=6
Jul 16 23:18:32.248255 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:32.247805    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-dnt5c" event={"ID":"f8a68f23-7b44-407f-9578-24af8d3b3ebb","Type":"ContainerDied","Data":"0be6f411feee4de97de6e7a702d90301c721bff9be05a10522d1db9e701bb3f1"}
Jul 16 23:18:32.248255 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:32.247883    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-dnt5c" event={"ID":"f8a68f23-7b44-407f-9578-24af8d3b3ebb","Type":"ContainerStarted","Data":"085578bc9c60c9870341a79f132dedfe8bae01193430bb3bdd2a27bea2bb9a93"}
Jul 16 23:18:32.248255 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:32.248028    2573 scope.go:117] "RemoveContainer" containerID="0be6f411feee4de97de6e7a702d90301c721bff9be05a10522d1db9e701bb3f1"
Jul 16 23:18:33.145526 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:33.145444    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-9752734e/e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2"
Jul 16 23:18:33.145526 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:33.145488    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-9752734e/e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2"
Jul 16 23:18:33.158228 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:33.158204    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-9752734e/e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2"
Jul 16 23:18:33.255389 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:33.255356    2573 generic.go:356] "Generic (PLEG): container finished" podID="f8a68f23-7b44-407f-9578-24af8d3b3ebb" containerID="7d0dc1e23ab16faaae71879c3a98cb7b07cf29a33f2e76b027a82fa41ae5af9f" exitCode=6
Jul 16 23:18:33.255753 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:33.255442    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-dnt5c" event={"ID":"f8a68f23-7b44-407f-9578-24af8d3b3ebb","Type":"ContainerDied","Data":"7d0dc1e23ab16faaae71879c3a98cb7b07cf29a33f2e76b027a82fa41ae5af9f"}
Jul 16 23:18:33.255753 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:33.255500    2573 scope.go:117] "RemoveContainer" containerID="0be6f411feee4de97de6e7a702d90301c721bff9be05a10522d1db9e701bb3f1"
Jul 16 23:18:33.255753 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:33.255621    2573 scope.go:117] "RemoveContainer" containerID="7d0dc1e23ab16faaae71879c3a98cb7b07cf29a33f2e76b027a82fa41ae5af9f"
Jul 16 23:18:33.255926 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:18:33.255812    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-dnt5c_odh-ai-gateway-infra(f8a68f23-7b44-407f-9578-24af8d3b3ebb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-dnt5c" podUID="f8a68f23-7b44-407f-9578-24af8d3b3ebb"
Jul 16 23:18:33.267339 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:33.267323    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-9752734e/e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2"
Jul 16 23:18:34.261855 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:34.261829    2573 scope.go:117] "RemoveContainer" containerID="7d0dc1e23ab16faaae71879c3a98cb7b07cf29a33f2e76b027a82fa41ae5af9f"
Jul 16 23:18:34.262227 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:18:34.261972    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-dnt5c_odh-ai-gateway-infra(f8a68f23-7b44-407f-9578-24af8d3b3ebb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-dnt5c" podUID="f8a68f23-7b44-407f-9578-24af8d3b3ebb"
Jul 16 23:18:35.933524 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:35.933493    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:18:39.834285 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:39.834170    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:18:42.086044 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:42.086007    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:18:46.520123 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:46.520096    2573 scope.go:117] "RemoveContainer" containerID="7d0dc1e23ab16faaae71879c3a98cb7b07cf29a33f2e76b027a82fa41ae5af9f"
Jul 16 23:18:46.605351 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:46.605325    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r9jm4_506f904a-3272-4586-8528-146884e728f9/ovn-acl-logging/0.log"
Jul 16 23:18:46.615817 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:46.615796    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r9jm4_506f904a-3272-4586-8528-146884e728f9/ovn-acl-logging/0.log"
Jul 16 23:18:47.330582 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:47.330515    2573 generic.go:356] "Generic (PLEG): container finished" podID="f8a68f23-7b44-407f-9578-24af8d3b3ebb" containerID="d91525826329a1becf14cfb210bb4854a911156b2f8203ba14fec312881b6677" exitCode=6
Jul 16 23:18:47.330701 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:47.330597    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-dnt5c" event={"ID":"f8a68f23-7b44-407f-9578-24af8d3b3ebb","Type":"ContainerDied","Data":"d91525826329a1becf14cfb210bb4854a911156b2f8203ba14fec312881b6677"}
Jul 16 23:18:47.330701 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:47.330694    2573 scope.go:117] "RemoveContainer" containerID="7d0dc1e23ab16faaae71879c3a98cb7b07cf29a33f2e76b027a82fa41ae5af9f"
Jul 16 23:18:47.330950 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:47.330932    2573 scope.go:117] "RemoveContainer" containerID="d91525826329a1becf14cfb210bb4854a911156b2f8203ba14fec312881b6677"
Jul 16 23:18:47.331139 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:18:47.331121    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-dnt5c_odh-ai-gateway-infra(f8a68f23-7b44-407f-9578-24af8d3b3ebb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-dnt5c" podUID="f8a68f23-7b44-407f-9578-24af8d3b3ebb"
Jul 16 23:18:48.358899 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:48.358868    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-dnt5c"]
Jul 16 23:18:48.483789 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:48.483768    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-dnt5c"
Jul 16 23:18:48.542370 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:48.542345    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8a68f23-7b44-407f-9578-24af8d3b3ebb-maas-api-service-ca\") pod \"f8a68f23-7b44-407f-9578-24af8d3b3ebb\" (UID: \"f8a68f23-7b44-407f-9578-24af8d3b3ebb\") "
Jul 16 23:18:48.542696 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:48.542675    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8a68f23-7b44-407f-9578-24af8d3b3ebb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f8a68f23-7b44-407f-9578-24af8d3b3ebb" (UID: "f8a68f23-7b44-407f-9578-24af8d3b3ebb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:18:48.642858 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:48.642807    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8a68f23-7b44-407f-9578-24af8d3b3ebb-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:18:49.344142 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:49.344115    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-dnt5c"
Jul 16 23:18:49.344343 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:49.344150    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-dnt5c" event={"ID":"f8a68f23-7b44-407f-9578-24af8d3b3ebb","Type":"ContainerDied","Data":"085578bc9c60c9870341a79f132dedfe8bae01193430bb3bdd2a27bea2bb9a93"}
Jul 16 23:18:49.344343 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:49.344207    2573 scope.go:117] "RemoveContainer" containerID="d91525826329a1becf14cfb210bb4854a911156b2f8203ba14fec312881b6677"
Jul 16 23:18:49.367020 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:49.366994    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-dnt5c"]
Jul 16 23:18:49.372172 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:49.372150    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-dnt5c"]
Jul 16 23:18:50.257250 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:50.257203    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:18:50.518268 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:50.518178    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8a68f23-7b44-407f-9578-24af8d3b3ebb" path="/var/lib/kubelet/pods/f8a68f23-7b44-407f-9578-24af8d3b3ebb/volumes"
Jul 16 23:18:51.815951 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:51.815896    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-xztgf"]
Jul 16 23:18:51.816463 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:51.816435    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8a68f23-7b44-407f-9578-24af8d3b3ebb" containerName="revoke-keys"
Jul 16 23:18:51.816463 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:51.816459    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8a68f23-7b44-407f-9578-24af8d3b3ebb" containerName="revoke-keys"
Jul 16 23:18:51.816607 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:51.816480    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8a68f23-7b44-407f-9578-24af8d3b3ebb" containerName="revoke-keys"
Jul 16 23:18:51.816607 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:51.816491    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8a68f23-7b44-407f-9578-24af8d3b3ebb" containerName="revoke-keys"
Jul 16 23:18:51.816607 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:51.816500    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8a68f23-7b44-407f-9578-24af8d3b3ebb" containerName="revoke-keys"
Jul 16 23:18:51.816607 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:51.816511    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8a68f23-7b44-407f-9578-24af8d3b3ebb" containerName="revoke-keys"
Jul 16 23:18:51.816607 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:51.816605    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8a68f23-7b44-407f-9578-24af8d3b3ebb" containerName="revoke-keys"
Jul 16 23:18:51.816843 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:51.816618    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8a68f23-7b44-407f-9578-24af8d3b3ebb" containerName="revoke-keys"
Jul 16 23:18:51.824723 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:51.822351    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-xztgf"
Jul 16 23:18:51.825757 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:51.825712    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:18:51.825757 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:51.825715    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 23:18:51.828588 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:51.828549    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-xztgf"]
Jul 16 23:18:51.864867 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:51.864839    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb0897ab-b199-4de4-b2c9-69769ca4455a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-xztgf\" (UID: \"bb0897ab-b199-4de4-b2c9-69769ca4455a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-xztgf"
Jul 16 23:18:51.965337 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:51.965302    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb0897ab-b199-4de4-b2c9-69769ca4455a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-xztgf\" (UID: \"bb0897ab-b199-4de4-b2c9-69769ca4455a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-xztgf"
Jul 16 23:18:51.965961 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:51.965932    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb0897ab-b199-4de4-b2c9-69769ca4455a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-xztgf\" (UID: \"bb0897ab-b199-4de4-b2c9-69769ca4455a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-xztgf"
Jul 16 23:18:52.139472 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:52.139446    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-xztgf"
Jul 16 23:18:52.263765 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:52.263741    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-xztgf"]
Jul 16 23:18:52.265719 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:18:52.265690    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb0897ab_b199_4de4_b2c9_69769ca4455a.slice/crio-0ee83d942cc5b8f86f983e5b03d34831aecfff462bbdd81a1e05076beb394cc6 WatchSource:0}: Error finding container 0ee83d942cc5b8f86f983e5b03d34831aecfff462bbdd81a1e05076beb394cc6: Status 404 returned error can't find the container with id 0ee83d942cc5b8f86f983e5b03d34831aecfff462bbdd81a1e05076beb394cc6
Jul 16 23:18:52.359292 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:52.359263    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-xztgf" event={"ID":"bb0897ab-b199-4de4-b2c9-69769ca4455a","Type":"ContainerStarted","Data":"0ee83d942cc5b8f86f983e5b03d34831aecfff462bbdd81a1e05076beb394cc6"}
Jul 16 23:18:53.007807 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:53.007777    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-78js2"]
Jul 16 23:18:53.008133 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:53.008119    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8a68f23-7b44-407f-9578-24af8d3b3ebb" containerName="revoke-keys"
Jul 16 23:18:53.011053 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:53.011036    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-78js2"
Jul 16 23:18:53.017695 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:53.017666    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-78js2"]
Jul 16 23:18:53.074365 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:53.074337    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8173a39-39c9-4a25-aa66-ceafe419d4a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-78js2\" (UID: \"f8173a39-39c9-4a25-aa66-ceafe419d4a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-78js2"
Jul 16 23:18:53.175024 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:53.174951    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8173a39-39c9-4a25-aa66-ceafe419d4a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-78js2\" (UID: \"f8173a39-39c9-4a25-aa66-ceafe419d4a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-78js2"
Jul 16 23:18:53.175597 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:53.175578    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8173a39-39c9-4a25-aa66-ceafe419d4a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-78js2\" (UID: \"f8173a39-39c9-4a25-aa66-ceafe419d4a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-78js2"
Jul 16 23:18:53.323068 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:53.323025    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-78js2"
Jul 16 23:18:53.367170 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:53.367138    2573 generic.go:356] "Generic (PLEG): container finished" podID="bb0897ab-b199-4de4-b2c9-69769ca4455a" containerID="c36ea957ed4a985e0018edcba43b173b8bb0310fa48369c2f864c066b26f1c35" exitCode=6
Jul 16 23:18:53.367331 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:53.367230    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-xztgf" event={"ID":"bb0897ab-b199-4de4-b2c9-69769ca4455a","Type":"ContainerDied","Data":"c36ea957ed4a985e0018edcba43b173b8bb0310fa48369c2f864c066b26f1c35"}
Jul 16 23:18:53.367414 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:53.367380    2573 scope.go:117] "RemoveContainer" containerID="c36ea957ed4a985e0018edcba43b173b8bb0310fa48369c2f864c066b26f1c35"
Jul 16 23:18:53.451981 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:53.451957    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-78js2"]
Jul 16 23:18:53.453698 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:18:53.453672    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8173a39_39c9_4a25_aa66_ceafe419d4a3.slice/crio-c4218463c00450b075984ed65d0c2eab24aae36b08c0d0437f71df83793eb184 WatchSource:0}: Error finding container c4218463c00450b075984ed65d0c2eab24aae36b08c0d0437f71df83793eb184: Status 404 returned error can't find the container with id c4218463c00450b075984ed65d0c2eab24aae36b08c0d0437f71df83793eb184
Jul 16 23:18:54.373596 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:54.373566    2573 generic.go:356] "Generic (PLEG): container finished" podID="f8173a39-39c9-4a25-aa66-ceafe419d4a3" containerID="2b81113b62899f496bb3ce9688321194b093e0bcd2e1f56a6161950a9e179088" exitCode=6
Jul 16 23:18:54.373988 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:54.373650    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-78js2" event={"ID":"f8173a39-39c9-4a25-aa66-ceafe419d4a3","Type":"ContainerDied","Data":"2b81113b62899f496bb3ce9688321194b093e0bcd2e1f56a6161950a9e179088"}
Jul 16 23:18:54.373988 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:54.373707    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-78js2" event={"ID":"f8173a39-39c9-4a25-aa66-ceafe419d4a3","Type":"ContainerStarted","Data":"c4218463c00450b075984ed65d0c2eab24aae36b08c0d0437f71df83793eb184"}
Jul 16 23:18:54.373988 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:54.373805    2573 scope.go:117] "RemoveContainer" containerID="2b81113b62899f496bb3ce9688321194b093e0bcd2e1f56a6161950a9e179088"
Jul 16 23:18:54.375672 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:54.375647    2573 generic.go:356] "Generic (PLEG): container finished" podID="bb0897ab-b199-4de4-b2c9-69769ca4455a" containerID="f114b32a2ebe660152584a4483a30b06527c5efb0b250d56e85ddb61e5ff6153" exitCode=6
Jul 16 23:18:54.375777 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:54.375714    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-xztgf" event={"ID":"bb0897ab-b199-4de4-b2c9-69769ca4455a","Type":"ContainerDied","Data":"f114b32a2ebe660152584a4483a30b06527c5efb0b250d56e85ddb61e5ff6153"}
Jul 16 23:18:54.375777 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:54.375769    2573 scope.go:117] "RemoveContainer" containerID="c36ea957ed4a985e0018edcba43b173b8bb0310fa48369c2f864c066b26f1c35"
Jul 16 23:18:54.375953 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:54.375909    2573 scope.go:117] "RemoveContainer" containerID="f114b32a2ebe660152584a4483a30b06527c5efb0b250d56e85ddb61e5ff6153"
Jul 16 23:18:54.376081 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:18:54.376062    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-xztgf_odh-ai-gateway-infra(bb0897ab-b199-4de4-b2c9-69769ca4455a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-xztgf" podUID="bb0897ab-b199-4de4-b2c9-69769ca4455a"
Jul 16 23:18:55.387436 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:55.387399    2573 generic.go:356] "Generic (PLEG): container finished" podID="f8173a39-39c9-4a25-aa66-ceafe419d4a3" containerID="55e9a7373961a7e1d1f5477c4c1a00e376d7becf69cb89b3e07bb69108b1007c" exitCode=6
Jul 16 23:18:55.387879 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:55.387488    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-78js2" event={"ID":"f8173a39-39c9-4a25-aa66-ceafe419d4a3","Type":"ContainerDied","Data":"55e9a7373961a7e1d1f5477c4c1a00e376d7becf69cb89b3e07bb69108b1007c"}
Jul 16 23:18:55.387879 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:55.387604    2573 scope.go:117] "RemoveContainer" containerID="2b81113b62899f496bb3ce9688321194b093e0bcd2e1f56a6161950a9e179088"
Jul 16 23:18:55.387879 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:55.387669    2573 scope.go:117] "RemoveContainer" containerID="55e9a7373961a7e1d1f5477c4c1a00e376d7becf69cb89b3e07bb69108b1007c"
Jul 16 23:18:55.388021 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:18:55.387878    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-78js2_odh-ai-gateway-infra(f8173a39-39c9-4a25-aa66-ceafe419d4a3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-78js2" podUID="f8173a39-39c9-4a25-aa66-ceafe419d4a3"
Jul 16 23:18:55.389417 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:55.389399    2573 scope.go:117] "RemoveContainer" containerID="f114b32a2ebe660152584a4483a30b06527c5efb0b250d56e85ddb61e5ff6153"
Jul 16 23:18:55.389558 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:18:55.389541    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-xztgf_odh-ai-gateway-infra(bb0897ab-b199-4de4-b2c9-69769ca4455a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-xztgf" podUID="bb0897ab-b199-4de4-b2c9-69769ca4455a"
Jul 16 23:18:56.334084 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:56.334050    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:18:56.396312 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:18:56.396290    2573 scope.go:117] "RemoveContainer" containerID="55e9a7373961a7e1d1f5477c4c1a00e376d7becf69cb89b3e07bb69108b1007c"
Jul 16 23:18:56.396638 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:18:56.396449    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-78js2_odh-ai-gateway-infra(f8173a39-39c9-4a25-aa66-ceafe419d4a3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-78js2" podUID="f8173a39-39c9-4a25-aa66-ceafe419d4a3"
Jul 16 23:19:00.327109 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:00.327071    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:19:00.974908 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:00.974875    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:19:05.961571 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:05.961526    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-9752734e/e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2"]
Jul 16 23:19:06.055630 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:06.055599    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:19:08.514330 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:08.514301    2573 scope.go:117] "RemoveContainer" containerID="55e9a7373961a7e1d1f5477c4c1a00e376d7becf69cb89b3e07bb69108b1007c"
Jul 16 23:19:09.459182 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:09.459109    2573 generic.go:356] "Generic (PLEG): container finished" podID="f8173a39-39c9-4a25-aa66-ceafe419d4a3" containerID="2270614c79ee8014475ae93cbfa7f89c0dc2e5ec02a7d59dc1584b865a8e4ebb" exitCode=6
Jul 16 23:19:09.459329 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:09.459178    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-78js2" event={"ID":"f8173a39-39c9-4a25-aa66-ceafe419d4a3","Type":"ContainerDied","Data":"2270614c79ee8014475ae93cbfa7f89c0dc2e5ec02a7d59dc1584b865a8e4ebb"}
Jul 16 23:19:09.459329 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:09.459254    2573 scope.go:117] "RemoveContainer" containerID="55e9a7373961a7e1d1f5477c4c1a00e376d7becf69cb89b3e07bb69108b1007c"
Jul 16 23:19:09.459523 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:09.459503    2573 scope.go:117] "RemoveContainer" containerID="2270614c79ee8014475ae93cbfa7f89c0dc2e5ec02a7d59dc1584b865a8e4ebb"
Jul 16 23:19:09.459729 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:19:09.459707    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-78js2_odh-ai-gateway-infra(f8173a39-39c9-4a25-aa66-ceafe419d4a3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-78js2" podUID="f8173a39-39c9-4a25-aa66-ceafe419d4a3"
Jul 16 23:19:09.513435 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:09.513413    2573 scope.go:117] "RemoveContainer" containerID="f114b32a2ebe660152584a4483a30b06527c5efb0b250d56e85ddb61e5ff6153"
Jul 16 23:19:10.467745 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:10.467663    2573 generic.go:356] "Generic (PLEG): container finished" podID="bb0897ab-b199-4de4-b2c9-69769ca4455a" containerID="083a90d0fa27a24d226fe7891e5b175de99ae9de314c6725256730ee3a8d8ee4" exitCode=6
Jul 16 23:19:10.467745 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:10.467714    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-xztgf" event={"ID":"bb0897ab-b199-4de4-b2c9-69769ca4455a","Type":"ContainerDied","Data":"083a90d0fa27a24d226fe7891e5b175de99ae9de314c6725256730ee3a8d8ee4"}
Jul 16 23:19:10.468195 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:10.467761    2573 scope.go:117] "RemoveContainer" containerID="f114b32a2ebe660152584a4483a30b06527c5efb0b250d56e85ddb61e5ff6153"
Jul 16 23:19:10.468195 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:10.468020    2573 scope.go:117] "RemoveContainer" containerID="083a90d0fa27a24d226fe7891e5b175de99ae9de314c6725256730ee3a8d8ee4"
Jul 16 23:19:10.468330 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:19:10.468198    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-xztgf_odh-ai-gateway-infra(bb0897ab-b199-4de4-b2c9-69769ca4455a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-xztgf" podUID="bb0897ab-b199-4de4-b2c9-69769ca4455a"
Jul 16 23:19:10.483575 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:10.483540    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-78js2"]
Jul 16 23:19:10.609051 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:10.609023    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-78js2"
Jul 16 23:19:10.698205 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:10.698177    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8173a39-39c9-4a25-aa66-ceafe419d4a3-maas-api-service-ca\") pod \"f8173a39-39c9-4a25-aa66-ceafe419d4a3\" (UID: \"f8173a39-39c9-4a25-aa66-ceafe419d4a3\") "
Jul 16 23:19:10.698562 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:10.698531    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8173a39-39c9-4a25-aa66-ceafe419d4a3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f8173a39-39c9-4a25-aa66-ceafe419d4a3" (UID: "f8173a39-39c9-4a25-aa66-ceafe419d4a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:19:10.799459 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:10.799377    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8173a39-39c9-4a25-aa66-ceafe419d4a3-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:19:11.473967 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:11.473936    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-78js2"
Jul 16 23:19:11.473967 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:11.473958    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-78js2" event={"ID":"f8173a39-39c9-4a25-aa66-ceafe419d4a3","Type":"ContainerDied","Data":"c4218463c00450b075984ed65d0c2eab24aae36b08c0d0437f71df83793eb184"}
Jul 16 23:19:11.474527 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:11.474008    2573 scope.go:117] "RemoveContainer" containerID="2270614c79ee8014475ae93cbfa7f89c0dc2e5ec02a7d59dc1584b865a8e4ebb"
Jul 16 23:19:11.495126 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:11.495100    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-xztgf"]
Jul 16 23:19:11.498307 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:11.498285    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-78js2"]
Jul 16 23:19:11.505441 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:11.505407    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-78js2"]
Jul 16 23:19:11.626059 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:11.626039    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-xztgf"
Jul 16 23:19:11.707730 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:11.707707    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb0897ab-b199-4de4-b2c9-69769ca4455a-maas-api-service-ca\") pod \"bb0897ab-b199-4de4-b2c9-69769ca4455a\" (UID: \"bb0897ab-b199-4de4-b2c9-69769ca4455a\") "
Jul 16 23:19:11.708039 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:11.708016    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb0897ab-b199-4de4-b2c9-69769ca4455a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bb0897ab-b199-4de4-b2c9-69769ca4455a" (UID: "bb0897ab-b199-4de4-b2c9-69769ca4455a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:19:11.808437 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:11.808377    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb0897ab-b199-4de4-b2c9-69769ca4455a-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:19:12.481476 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:12.481439    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-xztgf"
Jul 16 23:19:12.481909 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:12.481481    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-xztgf" event={"ID":"bb0897ab-b199-4de4-b2c9-69769ca4455a","Type":"ContainerDied","Data":"0ee83d942cc5b8f86f983e5b03d34831aecfff462bbdd81a1e05076beb394cc6"}
Jul 16 23:19:12.481909 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:12.481539    2573 scope.go:117] "RemoveContainer" containerID="083a90d0fa27a24d226fe7891e5b175de99ae9de314c6725256730ee3a8d8ee4"
Jul 16 23:19:12.501438 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:12.501411    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-xztgf"]
Jul 16 23:19:12.507887 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:12.507856    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-xztgf"]
Jul 16 23:19:12.517699 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:12.517677    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb0897ab-b199-4de4-b2c9-69769ca4455a" path="/var/lib/kubelet/pods/bb0897ab-b199-4de4-b2c9-69769ca4455a/volumes"
Jul 16 23:19:12.517990 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:12.517977    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8173a39-39c9-4a25-aa66-ceafe419d4a3" path="/var/lib/kubelet/pods/f8173a39-39c9-4a25-aa66-ceafe419d4a3/volumes"
Jul 16 23:19:18.057913 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:18.057876    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:19:18.155102 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:18.155049    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:19:20.982882 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:20.982845    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-9752734e/e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2" podUID="c9ff1625-8e38-44a4-8ab4-529c5d16e4e4" containerName="main" containerID="cri-o://4ebd158b7879087e5ad9b230d6e0a8e66b9ca3a4afb96cc1b2475afdaf995f1c" gracePeriod=45
Jul 16 23:19:21.226542 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:21.226520    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9752734e/e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2"
Jul 16 23:19:21.275768 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:21.275704    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wsrk\" (UniqueName: \"kubernetes.io/projected/c9ff1625-8e38-44a4-8ab4-529c5d16e4e4-kube-api-access-8wsrk\") pod \"c9ff1625-8e38-44a4-8ab4-529c5d16e4e4\" (UID: \"c9ff1625-8e38-44a4-8ab4-529c5d16e4e4\") "
Jul 16 23:19:21.275768 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:21.275742    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9ff1625-8e38-44a4-8ab4-529c5d16e4e4-model-cache\") pod \"c9ff1625-8e38-44a4-8ab4-529c5d16e4e4\" (UID: \"c9ff1625-8e38-44a4-8ab4-529c5d16e4e4\") "
Jul 16 23:19:21.275960 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:21.275785    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9ff1625-8e38-44a4-8ab4-529c5d16e4e4-tls-certs\") pod \"c9ff1625-8e38-44a4-8ab4-529c5d16e4e4\" (UID: \"c9ff1625-8e38-44a4-8ab4-529c5d16e4e4\") "
Jul 16 23:19:21.275960 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:21.275808    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9ff1625-8e38-44a4-8ab4-529c5d16e4e4-home\") pod \"c9ff1625-8e38-44a4-8ab4-529c5d16e4e4\" (UID: \"c9ff1625-8e38-44a4-8ab4-529c5d16e4e4\") "
Jul 16 23:19:21.275960 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:21.275829    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9ff1625-8e38-44a4-8ab4-529c5d16e4e4-tmp-dir\") pod \"c9ff1625-8e38-44a4-8ab4-529c5d16e4e4\" (UID: \"c9ff1625-8e38-44a4-8ab4-529c5d16e4e4\") "
Jul 16 23:19:21.275960 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:21.275856    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9ff1625-8e38-44a4-8ab4-529c5d16e4e4-dshm\") pod \"c9ff1625-8e38-44a4-8ab4-529c5d16e4e4\" (UID: \"c9ff1625-8e38-44a4-8ab4-529c5d16e4e4\") "
Jul 16 23:19:21.276175 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:21.276087    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9ff1625-8e38-44a4-8ab4-529c5d16e4e4-home" (OuterVolumeSpecName: "home") pod "c9ff1625-8e38-44a4-8ab4-529c5d16e4e4" (UID: "c9ff1625-8e38-44a4-8ab4-529c5d16e4e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:19:21.276175 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:21.276107    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9ff1625-8e38-44a4-8ab4-529c5d16e4e4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c9ff1625-8e38-44a4-8ab4-529c5d16e4e4" (UID: "c9ff1625-8e38-44a4-8ab4-529c5d16e4e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:19:21.276311 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:21.276286    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9ff1625-8e38-44a4-8ab4-529c5d16e4e4-model-cache" (OuterVolumeSpecName: "model-cache") pod "c9ff1625-8e38-44a4-8ab4-529c5d16e4e4" (UID: "c9ff1625-8e38-44a4-8ab4-529c5d16e4e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:19:21.277967 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:21.277939    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9ff1625-8e38-44a4-8ab4-529c5d16e4e4-dshm" (OuterVolumeSpecName: "dshm") pod "c9ff1625-8e38-44a4-8ab4-529c5d16e4e4" (UID: "c9ff1625-8e38-44a4-8ab4-529c5d16e4e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:19:21.278112 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:21.278011    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9ff1625-8e38-44a4-8ab4-529c5d16e4e4-kube-api-access-8wsrk" (OuterVolumeSpecName: "kube-api-access-8wsrk") pod "c9ff1625-8e38-44a4-8ab4-529c5d16e4e4" (UID: "c9ff1625-8e38-44a4-8ab4-529c5d16e4e4"). InnerVolumeSpecName "kube-api-access-8wsrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:19:21.278383 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:21.278365    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9ff1625-8e38-44a4-8ab4-529c5d16e4e4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c9ff1625-8e38-44a4-8ab4-529c5d16e4e4" (UID: "c9ff1625-8e38-44a4-8ab4-529c5d16e4e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 23:19:21.377368 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:21.377342    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9ff1625-8e38-44a4-8ab4-529c5d16e4e4-tmp-dir\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:19:21.377368 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:21.377364    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9ff1625-8e38-44a4-8ab4-529c5d16e4e4-dshm\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:19:21.377505 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:21.377373    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wsrk\" (UniqueName: \"kubernetes.io/projected/c9ff1625-8e38-44a4-8ab4-529c5d16e4e4-kube-api-access-8wsrk\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:19:21.377505 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:21.377382    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9ff1625-8e38-44a4-8ab4-529c5d16e4e4-model-cache\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:19:21.377505 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:21.377391    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9ff1625-8e38-44a4-8ab4-529c5d16e4e4-tls-certs\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:19:21.377505 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:21.377398    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9ff1625-8e38-44a4-8ab4-529c5d16e4e4-home\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:19:21.527252 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:21.527169    2573 generic.go:356] "Generic (PLEG): container finished" podID="c9ff1625-8e38-44a4-8ab4-529c5d16e4e4" containerID="4ebd158b7879087e5ad9b230d6e0a8e66b9ca3a4afb96cc1b2475afdaf995f1c" exitCode=0
Jul 16 23:19:21.527252 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:21.527210    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9752734e/e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2"
Jul 16 23:19:21.527415 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:21.527264    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9752734e/e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2" event={"ID":"c9ff1625-8e38-44a4-8ab4-529c5d16e4e4","Type":"ContainerDied","Data":"4ebd158b7879087e5ad9b230d6e0a8e66b9ca3a4afb96cc1b2475afdaf995f1c"}
Jul 16 23:19:21.527415 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:21.527307    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9752734e/e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2" event={"ID":"c9ff1625-8e38-44a4-8ab4-529c5d16e4e4","Type":"ContainerDied","Data":"004717573f0f732ff22d2acefa13d76218cc31c1eba55e0f53b0e1a088e22c76"}
Jul 16 23:19:21.527415 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:21.527325    2573 scope.go:117] "RemoveContainer" containerID="4ebd158b7879087e5ad9b230d6e0a8e66b9ca3a4afb96cc1b2475afdaf995f1c"
Jul 16 23:19:21.536433 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:21.536418    2573 scope.go:117] "RemoveContainer" containerID="4ebd158b7879087e5ad9b230d6e0a8e66b9ca3a4afb96cc1b2475afdaf995f1c"
Jul 16 23:19:21.536699 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:19:21.536673    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ebd158b7879087e5ad9b230d6e0a8e66b9ca3a4afb96cc1b2475afdaf995f1c\": container with ID starting with 4ebd158b7879087e5ad9b230d6e0a8e66b9ca3a4afb96cc1b2475afdaf995f1c not found: ID does not exist" containerID="4ebd158b7879087e5ad9b230d6e0a8e66b9ca3a4afb96cc1b2475afdaf995f1c"
Jul 16 23:19:21.536801 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:21.536704    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ebd158b7879087e5ad9b230d6e0a8e66b9ca3a4afb96cc1b2475afdaf995f1c"} err="failed to get container status \"4ebd158b7879087e5ad9b230d6e0a8e66b9ca3a4afb96cc1b2475afdaf995f1c\": rpc error: code = NotFound desc = could not find container \"4ebd158b7879087e5ad9b230d6e0a8e66b9ca3a4afb96cc1b2475afdaf995f1c\": container with ID starting with 4ebd158b7879087e5ad9b230d6e0a8e66b9ca3a4afb96cc1b2475afdaf995f1c not found: ID does not exist"
Jul 16 23:19:21.550315 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:21.550286    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-9752734e/e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2"]
Jul 16 23:19:21.554916 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:21.554890    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-9752734e/e2e-lifecycle-model-9752734e-kserve-86699c4bf5-ql2b2"]
Jul 16 23:19:22.519621 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:22.519583    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9ff1625-8e38-44a4-8ab4-529c5d16e4e4" path="/var/lib/kubelet/pods/c9ff1625-8e38-44a4-8ab4-529c5d16e4e4/volumes"
Jul 16 23:19:42.678230 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:42.678192    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:19:44.035327 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:44.035286    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:19:48.421177 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:48.421135    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:19:48.486879 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:48.486848    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:19:53.095755 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:53.095722    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:19:53.312379 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:53.312336    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:19:54.312682 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:54.312650    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:19:54.410549 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:54.410508    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:19:54.510818 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:54.510788    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:19:55.166947 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:55.166915    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:19:55.259078 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:55.259048    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:19:55.358987 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:55.358962    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:19:56.675943 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:56.675905    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:19:57.338726 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:19:57.338681    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:20:02.404066 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:02.404017    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:20:02.462249 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:02.462211    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:20:03.051387 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:03.051330    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:20:08.722106 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:08.722026    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:20:08.790957 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:08.790929    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:20:08.877454 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:08.877435    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:20:08.978655 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:08.978598    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:20:09.403021 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:09.402984    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:20:19.428183 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:19.428137    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:20:19.772433 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:19.772337    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:20:40.701295 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:40.701260    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:20:46.470039 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:46.469998    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:20:46.530271 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:46.530226    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:20:46.811453 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:46.811385    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:20:47.100623 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:47.100584    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:20:47.414329 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:47.414252    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:20:50.390963 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:50.390923    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:20:50.481038 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:50.481005    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:20:50.635612 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:50.635577    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:20:50.965663 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:50.965632    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:20:51.066352 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:51.066301    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:20:51.170763 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:51.170737    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:20:51.932183 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:51.932149    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-vfslp"]
Jul 16 23:20:51.932581 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:51.932472    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8173a39-39c9-4a25-aa66-ceafe419d4a3" containerName="revoke-keys"
Jul 16 23:20:51.932581 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:51.932483    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8173a39-39c9-4a25-aa66-ceafe419d4a3" containerName="revoke-keys"
Jul 16 23:20:51.932581 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:51.932493    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb0897ab-b199-4de4-b2c9-69769ca4455a" containerName="revoke-keys"
Jul 16 23:20:51.932581 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:51.932499    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb0897ab-b199-4de4-b2c9-69769ca4455a" containerName="revoke-keys"
Jul 16 23:20:51.932581 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:51.932508    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb0897ab-b199-4de4-b2c9-69769ca4455a" containerName="revoke-keys"
Jul 16 23:20:51.932581 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:51.932514    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb0897ab-b199-4de4-b2c9-69769ca4455a" containerName="revoke-keys"
Jul 16 23:20:51.932581 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:51.932521    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb0897ab-b199-4de4-b2c9-69769ca4455a" containerName="revoke-keys"
Jul 16 23:20:51.932581 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:51.932526    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb0897ab-b199-4de4-b2c9-69769ca4455a" containerName="revoke-keys"
Jul 16 23:20:51.932581 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:51.932533    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8173a39-39c9-4a25-aa66-ceafe419d4a3" containerName="revoke-keys"
Jul 16 23:20:51.932581 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:51.932537    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8173a39-39c9-4a25-aa66-ceafe419d4a3" containerName="revoke-keys"
Jul 16 23:20:51.932581 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:51.932546    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8173a39-39c9-4a25-aa66-ceafe419d4a3" containerName="revoke-keys"
Jul 16 23:20:51.932581 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:51.932551    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8173a39-39c9-4a25-aa66-ceafe419d4a3" containerName="revoke-keys"
Jul 16 23:20:51.932581 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:51.932562    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9ff1625-8e38-44a4-8ab4-529c5d16e4e4" containerName="main"
Jul 16 23:20:51.932581 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:51.932568    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9ff1625-8e38-44a4-8ab4-529c5d16e4e4" containerName="main"
Jul 16 23:20:51.932994 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:51.932614    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8173a39-39c9-4a25-aa66-ceafe419d4a3" containerName="revoke-keys"
Jul 16 23:20:51.932994 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:51.932620    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9ff1625-8e38-44a4-8ab4-529c5d16e4e4" containerName="main"
Jul 16 23:20:51.932994 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:51.932629    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb0897ab-b199-4de4-b2c9-69769ca4455a" containerName="revoke-keys"
Jul 16 23:20:51.932994 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:51.932633    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb0897ab-b199-4de4-b2c9-69769ca4455a" containerName="revoke-keys"
Jul 16 23:20:51.932994 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:51.932638    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb0897ab-b199-4de4-b2c9-69769ca4455a" containerName="revoke-keys"
Jul 16 23:20:51.932994 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:51.932643    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8173a39-39c9-4a25-aa66-ceafe419d4a3" containerName="revoke-keys"
Jul 16 23:20:51.935496 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:51.935480    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-vfslp"
Jul 16 23:20:51.937709 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:51.937686    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:20:51.937827 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:51.937696    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 23:20:51.941808 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:51.941780    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-vfslp"]
Jul 16 23:20:52.113176 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:52.113144    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62f4ac42-7db4-44da-a09c-7e9977e6d05e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-vfslp\" (UID: \"62f4ac42-7db4-44da-a09c-7e9977e6d05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-vfslp"
Jul 16 23:20:52.213924 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:52.213866    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62f4ac42-7db4-44da-a09c-7e9977e6d05e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-vfslp\" (UID: \"62f4ac42-7db4-44da-a09c-7e9977e6d05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-vfslp"
Jul 16 23:20:52.214477 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:52.214459    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62f4ac42-7db4-44da-a09c-7e9977e6d05e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-vfslp\" (UID: \"62f4ac42-7db4-44da-a09c-7e9977e6d05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-vfslp"
Jul 16 23:20:52.247010 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:52.246989    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-vfslp"
Jul 16 23:20:52.576277 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:52.576231    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-vfslp"]
Jul 16 23:20:52.579331 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:20:52.579292    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62f4ac42_7db4_44da_a09c_7e9977e6d05e.slice/crio-36b4b0a0c412d511cfd6551b9829f7d9c2daa54ba46ffde8ba56eac83d0f10e9 WatchSource:0}: Error finding container 36b4b0a0c412d511cfd6551b9829f7d9c2daa54ba46ffde8ba56eac83d0f10e9: Status 404 returned error can't find the container with id 36b4b0a0c412d511cfd6551b9829f7d9c2daa54ba46ffde8ba56eac83d0f10e9
Jul 16 23:20:52.950584 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:52.950505    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-vfslp" event={"ID":"62f4ac42-7db4-44da-a09c-7e9977e6d05e","Type":"ContainerStarted","Data":"4081cab448781ff35bce101d76a092e786307b7ec16b089d6eac1e593468d0e5"}
Jul 16 23:20:52.950584 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:52.950557    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-vfslp" event={"ID":"62f4ac42-7db4-44da-a09c-7e9977e6d05e","Type":"ContainerStarted","Data":"36b4b0a0c412d511cfd6551b9829f7d9c2daa54ba46ffde8ba56eac83d0f10e9"}
Jul 16 23:20:52.967763 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:52.967720    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-vfslp" podStartSLOduration=1.967705665 podStartE2EDuration="1.967705665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:20:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:20:52.965967549 +0000 UTC m=+2826.976230211" watchObservedRunningTime="2026-07-16 23:20:52.967705665 +0000 UTC m=+2826.977968328"
Jul 16 23:20:53.956385 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:53.956353    2573 generic.go:356] "Generic (PLEG): container finished" podID="62f4ac42-7db4-44da-a09c-7e9977e6d05e" containerID="4081cab448781ff35bce101d76a092e786307b7ec16b089d6eac1e593468d0e5" exitCode=6
Jul 16 23:20:53.956734 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:53.956404    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-vfslp" event={"ID":"62f4ac42-7db4-44da-a09c-7e9977e6d05e","Type":"ContainerDied","Data":"4081cab448781ff35bce101d76a092e786307b7ec16b089d6eac1e593468d0e5"}
Jul 16 23:20:53.956734 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:53.956606    2573 scope.go:117] "RemoveContainer" containerID="4081cab448781ff35bce101d76a092e786307b7ec16b089d6eac1e593468d0e5"
Jul 16 23:20:54.860168 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:54.860101    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:20:54.961989 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:54.961958    2573 generic.go:356] "Generic (PLEG): container finished" podID="62f4ac42-7db4-44da-a09c-7e9977e6d05e" containerID="8702f08444f28c91598e644a9c4cb6a129dfa36d7118130cfe59eecfa72798d2" exitCode=6
Jul 16 23:20:54.962354 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:54.962029    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-vfslp" event={"ID":"62f4ac42-7db4-44da-a09c-7e9977e6d05e","Type":"ContainerDied","Data":"8702f08444f28c91598e644a9c4cb6a129dfa36d7118130cfe59eecfa72798d2"}
Jul 16 23:20:54.962354 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:54.962090    2573 scope.go:117] "RemoveContainer" containerID="4081cab448781ff35bce101d76a092e786307b7ec16b089d6eac1e593468d0e5"
Jul 16 23:20:54.962354 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:54.962271    2573 scope.go:117] "RemoveContainer" containerID="8702f08444f28c91598e644a9c4cb6a129dfa36d7118130cfe59eecfa72798d2"
Jul 16 23:20:54.962522 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:20:54.962476    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-vfslp_odh-ai-gateway-infra(62f4ac42-7db4-44da-a09c-7e9977e6d05e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-vfslp" podUID="62f4ac42-7db4-44da-a09c-7e9977e6d05e"
Jul 16 23:20:55.190898 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:55.190839    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:20:55.484903 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:55.484840    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:20:55.787163 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:55.787086    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:20:55.968760 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:55.968734    2573 scope.go:117] "RemoveContainer" containerID="8702f08444f28c91598e644a9c4cb6a129dfa36d7118130cfe59eecfa72798d2"
Jul 16 23:20:55.969137 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:20:55.968902    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-vfslp_odh-ai-gateway-infra(62f4ac42-7db4-44da-a09c-7e9977e6d05e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-vfslp" podUID="62f4ac42-7db4-44da-a09c-7e9977e6d05e"
Jul 16 23:20:58.231721 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:58.231678    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:20:58.443523 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:58.443486    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:20:58.540164 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:58.540101    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:20:58.670696 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:58.670669    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:20:58.934917 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:58.934886    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:20:59.106070 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:59.106040    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:20:59.206190 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:20:59.206112    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:21:02.943527 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:02.943494    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:21:06.245631 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:06.245601    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:21:06.303647 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:06.303621    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:21:09.334364 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:09.334329    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:21:11.514145 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:11.514116    2573 scope.go:117] "RemoveContainer" containerID="8702f08444f28c91598e644a9c4cb6a129dfa36d7118130cfe59eecfa72798d2"
Jul 16 23:21:12.044772 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:12.044735    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-vfslp" event={"ID":"62f4ac42-7db4-44da-a09c-7e9977e6d05e","Type":"ContainerStarted","Data":"bea4ed7a8bdbe7a26588c3dfdba9249a34200123d31662a470e2ca011401c076"}
Jul 16 23:21:12.846032 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:12.845996    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:21:12.946991 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:12.946956    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:21:13.051105 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:13.051069    2573 generic.go:356] "Generic (PLEG): container finished" podID="62f4ac42-7db4-44da-a09c-7e9977e6d05e" containerID="bea4ed7a8bdbe7a26588c3dfdba9249a34200123d31662a470e2ca011401c076" exitCode=6
Jul 16 23:21:13.051306 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:13.051110    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-vfslp" event={"ID":"62f4ac42-7db4-44da-a09c-7e9977e6d05e","Type":"ContainerDied","Data":"bea4ed7a8bdbe7a26588c3dfdba9249a34200123d31662a470e2ca011401c076"}
Jul 16 23:21:13.051306 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:13.051166    2573 scope.go:117] "RemoveContainer" containerID="8702f08444f28c91598e644a9c4cb6a129dfa36d7118130cfe59eecfa72798d2"
Jul 16 23:21:13.051480 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:13.051459    2573 scope.go:117] "RemoveContainer" containerID="bea4ed7a8bdbe7a26588c3dfdba9249a34200123d31662a470e2ca011401c076"
Jul 16 23:21:13.051696 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:21:13.051675    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-vfslp_odh-ai-gateway-infra(62f4ac42-7db4-44da-a09c-7e9977e6d05e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-vfslp" podUID="62f4ac42-7db4-44da-a09c-7e9977e6d05e"
Jul 16 23:21:13.075421 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:13.075387    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-vfslp"]
Jul 16 23:21:13.435486 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:13.435456    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-8a2515/test-model-8a2515-kserve-854c565bd5-f8mwj"]
Jul 16 23:21:13.435966 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:13.435952    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8173a39-39c9-4a25-aa66-ceafe419d4a3" containerName="revoke-keys"
Jul 16 23:21:13.440300 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:13.440278    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8a2515/test-model-8a2515-kserve-854c565bd5-f8mwj"
Jul 16 23:21:13.443727 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:13.443704    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8a2515\"/\"kube-root-ca.crt\""
Jul 16 23:21:13.443860 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:13.443706    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8a2515\"/\"default-dockercfg-hmwkv\""
Jul 16 23:21:13.443860 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:13.443739    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8a2515\"/\"openshift-service-ca.crt\""
Jul 16 23:21:13.443860 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:13.443706    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8a2515\"/\"test-model-8a2515-kserve-self-signed-certs\""
Jul 16 23:21:13.451130 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:13.451106    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8a2515/test-model-8a2515-kserve-854c565bd5-f8mwj"]
Jul 16 23:21:13.461537 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:13.461515    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f114885-fb59-4f81-b94e-d4b65f148073-dshm\") pod \"test-model-8a2515-kserve-854c565bd5-f8mwj\" (UID: \"2f114885-fb59-4f81-b94e-d4b65f148073\") " pod="ai-tenant-e2e-inf-a-8a2515/test-model-8a2515-kserve-854c565bd5-f8mwj"
Jul 16 23:21:13.461654 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:13.461547    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f114885-fb59-4f81-b94e-d4b65f148073-tls-certs\") pod \"test-model-8a2515-kserve-854c565bd5-f8mwj\" (UID: \"2f114885-fb59-4f81-b94e-d4b65f148073\") " pod="ai-tenant-e2e-inf-a-8a2515/test-model-8a2515-kserve-854c565bd5-f8mwj"
Jul 16 23:21:13.461654 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:13.461570    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f114885-fb59-4f81-b94e-d4b65f148073-home\") pod \"test-model-8a2515-kserve-854c565bd5-f8mwj\" (UID: \"2f114885-fb59-4f81-b94e-d4b65f148073\") " pod="ai-tenant-e2e-inf-a-8a2515/test-model-8a2515-kserve-854c565bd5-f8mwj"
Jul 16 23:21:13.461800 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:13.461677    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f114885-fb59-4f81-b94e-d4b65f148073-tmp-dir\") pod \"test-model-8a2515-kserve-854c565bd5-f8mwj\" (UID: \"2f114885-fb59-4f81-b94e-d4b65f148073\") " pod="ai-tenant-e2e-inf-a-8a2515/test-model-8a2515-kserve-854c565bd5-f8mwj"
Jul 16 23:21:13.461800 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:13.461702    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tpzq\" (UniqueName: \"kubernetes.io/projected/2f114885-fb59-4f81-b94e-d4b65f148073-kube-api-access-4tpzq\") pod \"test-model-8a2515-kserve-854c565bd5-f8mwj\" (UID: \"2f114885-fb59-4f81-b94e-d4b65f148073\") " pod="ai-tenant-e2e-inf-a-8a2515/test-model-8a2515-kserve-854c565bd5-f8mwj"
Jul 16 23:21:13.461800 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:13.461736    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f114885-fb59-4f81-b94e-d4b65f148073-model-cache\") pod \"test-model-8a2515-kserve-854c565bd5-f8mwj\" (UID: \"2f114885-fb59-4f81-b94e-d4b65f148073\") " pod="ai-tenant-e2e-inf-a-8a2515/test-model-8a2515-kserve-854c565bd5-f8mwj"
Jul 16 23:21:13.477833 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:13.477796    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:21:13.562791 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:13.562748    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f114885-fb59-4f81-b94e-d4b65f148073-tmp-dir\") pod \"test-model-8a2515-kserve-854c565bd5-f8mwj\" (UID: \"2f114885-fb59-4f81-b94e-d4b65f148073\") " pod="ai-tenant-e2e-inf-a-8a2515/test-model-8a2515-kserve-854c565bd5-f8mwj"
Jul 16 23:21:13.562791 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:13.562798    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tpzq\" (UniqueName: \"kubernetes.io/projected/2f114885-fb59-4f81-b94e-d4b65f148073-kube-api-access-4tpzq\") pod \"test-model-8a2515-kserve-854c565bd5-f8mwj\" (UID: \"2f114885-fb59-4f81-b94e-d4b65f148073\") " pod="ai-tenant-e2e-inf-a-8a2515/test-model-8a2515-kserve-854c565bd5-f8mwj"
Jul 16 23:21:13.563044 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:13.562825    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f114885-fb59-4f81-b94e-d4b65f148073-model-cache\") pod \"test-model-8a2515-kserve-854c565bd5-f8mwj\" (UID: \"2f114885-fb59-4f81-b94e-d4b65f148073\") " pod="ai-tenant-e2e-inf-a-8a2515/test-model-8a2515-kserve-854c565bd5-f8mwj"
Jul 16 23:21:13.563044 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:13.562841    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f114885-fb59-4f81-b94e-d4b65f148073-dshm\") pod \"test-model-8a2515-kserve-854c565bd5-f8mwj\" (UID: \"2f114885-fb59-4f81-b94e-d4b65f148073\") " pod="ai-tenant-e2e-inf-a-8a2515/test-model-8a2515-kserve-854c565bd5-f8mwj"
Jul 16 23:21:13.563044 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:13.562863    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f114885-fb59-4f81-b94e-d4b65f148073-tls-certs\") pod \"test-model-8a2515-kserve-854c565bd5-f8mwj\" (UID: \"2f114885-fb59-4f81-b94e-d4b65f148073\") " pod="ai-tenant-e2e-inf-a-8a2515/test-model-8a2515-kserve-854c565bd5-f8mwj"
Jul 16 23:21:13.563044 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:13.562899    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f114885-fb59-4f81-b94e-d4b65f148073-home\") pod \"test-model-8a2515-kserve-854c565bd5-f8mwj\" (UID: \"2f114885-fb59-4f81-b94e-d4b65f148073\") " pod="ai-tenant-e2e-inf-a-8a2515/test-model-8a2515-kserve-854c565bd5-f8mwj"
Jul 16 23:21:13.563294 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:13.563217    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f114885-fb59-4f81-b94e-d4b65f148073-tmp-dir\") pod \"test-model-8a2515-kserve-854c565bd5-f8mwj\" (UID: \"2f114885-fb59-4f81-b94e-d4b65f148073\") " pod="ai-tenant-e2e-inf-a-8a2515/test-model-8a2515-kserve-854c565bd5-f8mwj"
Jul 16 23:21:13.563294 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:13.563230    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f114885-fb59-4f81-b94e-d4b65f148073-model-cache\") pod \"test-model-8a2515-kserve-854c565bd5-f8mwj\" (UID: \"2f114885-fb59-4f81-b94e-d4b65f148073\") " pod="ai-tenant-e2e-inf-a-8a2515/test-model-8a2515-kserve-854c565bd5-f8mwj"
Jul 16 23:21:13.563410 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:13.563339    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f114885-fb59-4f81-b94e-d4b65f148073-home\") pod \"test-model-8a2515-kserve-854c565bd5-f8mwj\" (UID: \"2f114885-fb59-4f81-b94e-d4b65f148073\") " pod="ai-tenant-e2e-inf-a-8a2515/test-model-8a2515-kserve-854c565bd5-f8mwj"
Jul 16 23:21:13.565315 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:13.565294    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f114885-fb59-4f81-b94e-d4b65f148073-dshm\") pod \"test-model-8a2515-kserve-854c565bd5-f8mwj\" (UID: \"2f114885-fb59-4f81-b94e-d4b65f148073\") " pod="ai-tenant-e2e-inf-a-8a2515/test-model-8a2515-kserve-854c565bd5-f8mwj"
Jul 16 23:21:13.565429 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:13.565385    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f114885-fb59-4f81-b94e-d4b65f148073-tls-certs\") pod \"test-model-8a2515-kserve-854c565bd5-f8mwj\" (UID: \"2f114885-fb59-4f81-b94e-d4b65f148073\") " pod="ai-tenant-e2e-inf-a-8a2515/test-model-8a2515-kserve-854c565bd5-f8mwj"
Jul 16 23:21:13.573862 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:13.573835    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tpzq\" (UniqueName: \"kubernetes.io/projected/2f114885-fb59-4f81-b94e-d4b65f148073-kube-api-access-4tpzq\") pod \"test-model-8a2515-kserve-854c565bd5-f8mwj\" (UID: \"2f114885-fb59-4f81-b94e-d4b65f148073\") " pod="ai-tenant-e2e-inf-a-8a2515/test-model-8a2515-kserve-854c565bd5-f8mwj"
Jul 16 23:21:13.749541 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:13.749442    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8a2515/test-model-8a2515-kserve-854c565bd5-f8mwj"
Jul 16 23:21:13.880976 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:13.880951    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8a2515/test-model-8a2515-kserve-854c565bd5-f8mwj"]
Jul 16 23:21:13.882898 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:21:13.882860    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f114885_fb59_4f81_b94e_d4b65f148073.slice/crio-db48a25e08ddc2920b671da374f9a9f07cac2afb00cb83f82fd42badfb6c503a WatchSource:0}: Error finding container db48a25e08ddc2920b671da374f9a9f07cac2afb00cb83f82fd42badfb6c503a: Status 404 returned error can't find the container with id db48a25e08ddc2920b671da374f9a9f07cac2afb00cb83f82fd42badfb6c503a
Jul 16 23:21:14.058871 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:14.058841    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8a2515/test-model-8a2515-kserve-854c565bd5-f8mwj" event={"ID":"2f114885-fb59-4f81-b94e-d4b65f148073","Type":"ContainerStarted","Data":"aa0778385fe20b979a9d270fe30881dd45a8f220dd55b0374c5d7f508844363a"}
Jul 16 23:21:14.059000 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:14.058884    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8a2515/test-model-8a2515-kserve-854c565bd5-f8mwj" event={"ID":"2f114885-fb59-4f81-b94e-d4b65f148073","Type":"ContainerStarted","Data":"db48a25e08ddc2920b671da374f9a9f07cac2afb00cb83f82fd42badfb6c503a"}
Jul 16 23:21:14.088427 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:14.088362    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-8a2515/test-model-8a2515-kserve-854c565bd5-f8mwj" podStartSLOduration=1.088343337 podStartE2EDuration="1.088343337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:21:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:21:14.087521531 +0000 UTC m=+2848.097784196" watchObservedRunningTime="2026-07-16 23:21:14.088343337 +0000 UTC m=+2848.098606020"
Jul 16 23:21:14.174561 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:14.174534    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-vfslp"
Jul 16 23:21:14.268793 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:14.268753    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62f4ac42-7db4-44da-a09c-7e9977e6d05e-maas-api-service-ca\") pod \"62f4ac42-7db4-44da-a09c-7e9977e6d05e\" (UID: \"62f4ac42-7db4-44da-a09c-7e9977e6d05e\") "
Jul 16 23:21:14.269116 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:14.269093    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62f4ac42-7db4-44da-a09c-7e9977e6d05e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "62f4ac42-7db4-44da-a09c-7e9977e6d05e" (UID: "62f4ac42-7db4-44da-a09c-7e9977e6d05e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:21:14.369459 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:14.369429    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62f4ac42-7db4-44da-a09c-7e9977e6d05e-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:21:15.064507 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:15.064480    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-vfslp"
Jul 16 23:21:15.065024 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:15.064522    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-vfslp" event={"ID":"62f4ac42-7db4-44da-a09c-7e9977e6d05e","Type":"ContainerDied","Data":"36b4b0a0c412d511cfd6551b9829f7d9c2daa54ba46ffde8ba56eac83d0f10e9"}
Jul 16 23:21:15.065024 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:15.064592    2573 scope.go:117] "RemoveContainer" containerID="bea4ed7a8bdbe7a26588c3dfdba9249a34200123d31662a470e2ca011401c076"
Jul 16 23:21:15.080158 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:15.080133    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-vfslp"]
Jul 16 23:21:15.087747 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:15.087684    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-vfslp"]
Jul 16 23:21:16.519354 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:16.519321    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62f4ac42-7db4-44da-a09c-7e9977e6d05e" path="/var/lib/kubelet/pods/62f4ac42-7db4-44da-a09c-7e9977e6d05e/volumes"
Jul 16 23:21:17.641867 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:17.641833    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:21:20.716631 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:20.716592    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:21:20.816079 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:20.816049    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:21:20.869505 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:20.869473    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-5trvh"]
Jul 16 23:21:20.869814 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:20.869796    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62f4ac42-7db4-44da-a09c-7e9977e6d05e" containerName="revoke-keys"
Jul 16 23:21:20.869894 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:20.869817    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="62f4ac42-7db4-44da-a09c-7e9977e6d05e" containerName="revoke-keys"
Jul 16 23:21:20.869894 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:20.869843    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62f4ac42-7db4-44da-a09c-7e9977e6d05e" containerName="revoke-keys"
Jul 16 23:21:20.869894 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:20.869852    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="62f4ac42-7db4-44da-a09c-7e9977e6d05e" containerName="revoke-keys"
Jul 16 23:21:20.869894 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:20.869863    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62f4ac42-7db4-44da-a09c-7e9977e6d05e" containerName="revoke-keys"
Jul 16 23:21:20.869894 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:20.869871    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="62f4ac42-7db4-44da-a09c-7e9977e6d05e" containerName="revoke-keys"
Jul 16 23:21:20.870142 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:20.869958    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="62f4ac42-7db4-44da-a09c-7e9977e6d05e" containerName="revoke-keys"
Jul 16 23:21:20.870142 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:20.869972    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="62f4ac42-7db4-44da-a09c-7e9977e6d05e" containerName="revoke-keys"
Jul 16 23:21:20.874373 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:20.874353    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-5trvh"
Jul 16 23:21:20.883202 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:20.883183    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 23:21:20.883378 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:20.883362    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:21:20.886063 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:20.886040    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-5trvh"]
Jul 16 23:21:20.921309 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:20.921283    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2df16e94-cebb-4c12-ac8e-6a0b190a8dc9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-5trvh\" (UID: \"2df16e94-cebb-4c12-ac8e-6a0b190a8dc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-5trvh"
Jul 16 23:21:21.021854 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:21.021797    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2df16e94-cebb-4c12-ac8e-6a0b190a8dc9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-5trvh\" (UID: \"2df16e94-cebb-4c12-ac8e-6a0b190a8dc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-5trvh"
Jul 16 23:21:21.022450 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:21.022424    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2df16e94-cebb-4c12-ac8e-6a0b190a8dc9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-5trvh\" (UID: \"2df16e94-cebb-4c12-ac8e-6a0b190a8dc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-5trvh"
Jul 16 23:21:21.183576 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:21.183553    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-5trvh"
Jul 16 23:21:21.306504 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:21.306476    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-5trvh"]
Jul 16 23:21:21.308308 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:21:21.308281    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2df16e94_cebb_4c12_ac8e_6a0b190a8dc9.slice/crio-86e1ed9f68d4e6e11183f29fb892e99dfa71601bd5b083bacef29c9a58adbcce WatchSource:0}: Error finding container 86e1ed9f68d4e6e11183f29fb892e99dfa71601bd5b083bacef29c9a58adbcce: Status 404 returned error can't find the container with id 86e1ed9f68d4e6e11183f29fb892e99dfa71601bd5b083bacef29c9a58adbcce
Jul 16 23:21:22.098681 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:22.098649    2573 generic.go:356] "Generic (PLEG): container finished" podID="2df16e94-cebb-4c12-ac8e-6a0b190a8dc9" containerID="e45f35dd44c6701ee011a58aaa808b68c2b6abc9012841450c0d7ee6405ae1a1" exitCode=6
Jul 16 23:21:22.099016 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:22.098740    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-5trvh" event={"ID":"2df16e94-cebb-4c12-ac8e-6a0b190a8dc9","Type":"ContainerDied","Data":"e45f35dd44c6701ee011a58aaa808b68c2b6abc9012841450c0d7ee6405ae1a1"}
Jul 16 23:21:22.099016 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:22.098792    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-5trvh" event={"ID":"2df16e94-cebb-4c12-ac8e-6a0b190a8dc9","Type":"ContainerStarted","Data":"86e1ed9f68d4e6e11183f29fb892e99dfa71601bd5b083bacef29c9a58adbcce"}
Jul 16 23:21:22.099016 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:22.098895    2573 scope.go:117] "RemoveContainer" containerID="e45f35dd44c6701ee011a58aaa808b68c2b6abc9012841450c0d7ee6405ae1a1"
Jul 16 23:21:22.795292 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:21:22.795262    2573 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2df16e94_cebb_4c12_ac8e_6a0b190a8dc9.slice/crio-5474c9f1c5f6e2751a9327918ab25a94a2393ef7c62a7fc49608c980d7972674.scope\": RecentStats: unable to find data in memory cache]"
Jul 16 23:21:23.106127 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:23.106089    2573 generic.go:356] "Generic (PLEG): container finished" podID="2df16e94-cebb-4c12-ac8e-6a0b190a8dc9" containerID="5474c9f1c5f6e2751a9327918ab25a94a2393ef7c62a7fc49608c980d7972674" exitCode=6
Jul 16 23:21:23.106581 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:23.106170    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-5trvh" event={"ID":"2df16e94-cebb-4c12-ac8e-6a0b190a8dc9","Type":"ContainerDied","Data":"5474c9f1c5f6e2751a9327918ab25a94a2393ef7c62a7fc49608c980d7972674"}
Jul 16 23:21:23.106581 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:23.106254    2573 scope.go:117] "RemoveContainer" containerID="e45f35dd44c6701ee011a58aaa808b68c2b6abc9012841450c0d7ee6405ae1a1"
Jul 16 23:21:23.106581 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:23.106519    2573 scope.go:117] "RemoveContainer" containerID="5474c9f1c5f6e2751a9327918ab25a94a2393ef7c62a7fc49608c980d7972674"
Jul 16 23:21:23.106749 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:21:23.106686    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-5trvh_odh-ai-gateway-infra(2df16e94-cebb-4c12-ac8e-6a0b190a8dc9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-5trvh" podUID="2df16e94-cebb-4c12-ac8e-6a0b190a8dc9"
Jul 16 23:21:23.126811 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:23.126790    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6swjx"]
Jul 16 23:21:23.127168 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:23.127153    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="62f4ac42-7db4-44da-a09c-7e9977e6d05e" containerName="revoke-keys"
Jul 16 23:21:23.131016 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:23.130999    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6swjx"
Jul 16 23:21:23.136588 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:23.136568    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6swjx"]
Jul 16 23:21:23.240006 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:23.239973    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62d53248-592d-43ef-a80e-0e805f156395-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-6swjx\" (UID: \"62d53248-592d-43ef-a80e-0e805f156395\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6swjx"
Jul 16 23:21:23.340891 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:23.340860    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62d53248-592d-43ef-a80e-0e805f156395-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-6swjx\" (UID: \"62d53248-592d-43ef-a80e-0e805f156395\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6swjx"
Jul 16 23:21:23.341473 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:23.341455    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62d53248-592d-43ef-a80e-0e805f156395-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-6swjx\" (UID: \"62d53248-592d-43ef-a80e-0e805f156395\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6swjx"
Jul 16 23:21:23.441974 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:23.441913    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6swjx"
Jul 16 23:21:23.568605 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:23.568581    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6swjx"]
Jul 16 23:21:23.570518 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:21:23.570476    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62d53248_592d_43ef_a80e_0e805f156395.slice/crio-d8fbc1ea7f03b297288374bfae98aef9b4e47b576533132fc89f7fd77dfede3f WatchSource:0}: Error finding container d8fbc1ea7f03b297288374bfae98aef9b4e47b576533132fc89f7fd77dfede3f: Status 404 returned error can't find the container with id d8fbc1ea7f03b297288374bfae98aef9b4e47b576533132fc89f7fd77dfede3f
Jul 16 23:21:23.749811 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:23.749749    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-8a2515/test-model-8a2515-kserve-854c565bd5-f8mwj"
Jul 16 23:21:23.749811 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:23.749800    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-8a2515/test-model-8a2515-kserve-854c565bd5-f8mwj"
Jul 16 23:21:23.762042 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:23.762021    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-8a2515/test-model-8a2515-kserve-854c565bd5-f8mwj"
Jul 16 23:21:24.113442 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:24.113409    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6swjx" event={"ID":"62d53248-592d-43ef-a80e-0e805f156395","Type":"ContainerStarted","Data":"338c1e97fbad5bc656bb8e4dcec6464242622dba6a96e365a4239f66e6a72b99"}
Jul 16 23:21:24.113830 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:24.113462    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6swjx" event={"ID":"62d53248-592d-43ef-a80e-0e805f156395","Type":"ContainerStarted","Data":"d8fbc1ea7f03b297288374bfae98aef9b4e47b576533132fc89f7fd77dfede3f"}
Jul 16 23:21:24.115116 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:24.115095    2573 scope.go:117] "RemoveContainer" containerID="5474c9f1c5f6e2751a9327918ab25a94a2393ef7c62a7fc49608c980d7972674"
Jul 16 23:21:24.115335 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:21:24.115314    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-5trvh_odh-ai-gateway-infra(2df16e94-cebb-4c12-ac8e-6a0b190a8dc9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-5trvh" podUID="2df16e94-cebb-4c12-ac8e-6a0b190a8dc9"
Jul 16 23:21:24.126409 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:24.126390    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-8a2515/test-model-8a2515-kserve-854c565bd5-f8mwj"
Jul 16 23:21:24.130945 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:24.130909    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6swjx" podStartSLOduration=1.130894793 podStartE2EDuration="1.130894793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:21:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:21:24.129329683 +0000 UTC m=+2858.139592347" watchObservedRunningTime="2026-07-16 23:21:24.130894793 +0000 UTC m=+2858.141157458"
Jul 16 23:21:25.121203 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:25.121169    2573 generic.go:356] "Generic (PLEG): container finished" podID="62d53248-592d-43ef-a80e-0e805f156395" containerID="338c1e97fbad5bc656bb8e4dcec6464242622dba6a96e365a4239f66e6a72b99" exitCode=6
Jul 16 23:21:25.121677 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:25.121269    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6swjx" event={"ID":"62d53248-592d-43ef-a80e-0e805f156395","Type":"ContainerDied","Data":"338c1e97fbad5bc656bb8e4dcec6464242622dba6a96e365a4239f66e6a72b99"}
Jul 16 23:21:25.121677 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:25.121421    2573 scope.go:117] "RemoveContainer" containerID="338c1e97fbad5bc656bb8e4dcec6464242622dba6a96e365a4239f66e6a72b99"
Jul 16 23:21:26.127552 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:26.127514    2573 generic.go:356] "Generic (PLEG): container finished" podID="62d53248-592d-43ef-a80e-0e805f156395" containerID="e8af67fe1bff3b34be53570146af3a285dce2acda69a125e375c6688acbe74d6" exitCode=6
Jul 16 23:21:26.128012 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:26.127557    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6swjx" event={"ID":"62d53248-592d-43ef-a80e-0e805f156395","Type":"ContainerDied","Data":"e8af67fe1bff3b34be53570146af3a285dce2acda69a125e375c6688acbe74d6"}
Jul 16 23:21:26.128012 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:26.127627    2573 scope.go:117] "RemoveContainer" containerID="338c1e97fbad5bc656bb8e4dcec6464242622dba6a96e365a4239f66e6a72b99"
Jul 16 23:21:26.128012 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:26.127829    2573 scope.go:117] "RemoveContainer" containerID="e8af67fe1bff3b34be53570146af3a285dce2acda69a125e375c6688acbe74d6"
Jul 16 23:21:26.128183 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:21:26.128036    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-6swjx_odh-ai-gateway-infra(62d53248-592d-43ef-a80e-0e805f156395)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6swjx" podUID="62d53248-592d-43ef-a80e-0e805f156395"
Jul 16 23:21:27.133486 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:27.133458    2573 scope.go:117] "RemoveContainer" containerID="e8af67fe1bff3b34be53570146af3a285dce2acda69a125e375c6688acbe74d6"
Jul 16 23:21:27.133831 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:21:27.133600    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-6swjx_odh-ai-gateway-infra(62d53248-592d-43ef-a80e-0e805f156395)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6swjx" podUID="62d53248-592d-43ef-a80e-0e805f156395"
Jul 16 23:21:28.079061 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:28.079030    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-qp8kr"]
Jul 16 23:21:28.083396 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:28.083379    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-qp8kr"
Jul 16 23:21:28.089745 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:28.089716    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-qp8kr"]
Jul 16 23:21:28.175782 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:28.175751    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e8b6e78-d3ea-4e0d-8736-8e7be8d6dc07-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-qp8kr\" (UID: \"2e8b6e78-d3ea-4e0d-8736-8e7be8d6dc07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-qp8kr"
Jul 16 23:21:28.276975 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:28.276948    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e8b6e78-d3ea-4e0d-8736-8e7be8d6dc07-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-qp8kr\" (UID: \"2e8b6e78-d3ea-4e0d-8736-8e7be8d6dc07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-qp8kr"
Jul 16 23:21:28.277555 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:28.277536    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e8b6e78-d3ea-4e0d-8736-8e7be8d6dc07-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-qp8kr\" (UID: \"2e8b6e78-d3ea-4e0d-8736-8e7be8d6dc07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-qp8kr"
Jul 16 23:21:28.395183 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:28.395121    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-qp8kr"
Jul 16 23:21:28.524623 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:28.524600    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-qp8kr"]
Jul 16 23:21:28.526226 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:21:28.526204    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e8b6e78_d3ea_4e0d_8736_8e7be8d6dc07.slice/crio-9cc94380a6b320d9669305031c8c81e73ad72c2e6c52cf1d2971eec15006de4c WatchSource:0}: Error finding container 9cc94380a6b320d9669305031c8c81e73ad72c2e6c52cf1d2971eec15006de4c: Status 404 returned error can't find the container with id 9cc94380a6b320d9669305031c8c81e73ad72c2e6c52cf1d2971eec15006de4c
Jul 16 23:21:29.144655 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:29.144624    2573 generic.go:356] "Generic (PLEG): container finished" podID="2e8b6e78-d3ea-4e0d-8736-8e7be8d6dc07" containerID="6ad4fb7e6d76af49e67a6f3302729cf8da9149261bb0d663b051e866bd820016" exitCode=6
Jul 16 23:21:29.144791 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:29.144691    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-qp8kr" event={"ID":"2e8b6e78-d3ea-4e0d-8736-8e7be8d6dc07","Type":"ContainerDied","Data":"6ad4fb7e6d76af49e67a6f3302729cf8da9149261bb0d663b051e866bd820016"}
Jul 16 23:21:29.144791 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:29.144726    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-qp8kr" event={"ID":"2e8b6e78-d3ea-4e0d-8736-8e7be8d6dc07","Type":"ContainerStarted","Data":"9cc94380a6b320d9669305031c8c81e73ad72c2e6c52cf1d2971eec15006de4c"}
Jul 16 23:21:29.144909 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:29.144870    2573 scope.go:117] "RemoveContainer" containerID="6ad4fb7e6d76af49e67a6f3302729cf8da9149261bb0d663b051e866bd820016"
Jul 16 23:21:30.151292 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:30.151254    2573 generic.go:356] "Generic (PLEG): container finished" podID="2e8b6e78-d3ea-4e0d-8736-8e7be8d6dc07" containerID="0156a174a4bcf267e9dde86af3be61376edf8c39399f4e0150e9dc719e8c27a9" exitCode=6
Jul 16 23:21:30.151760 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:30.151321    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-qp8kr" event={"ID":"2e8b6e78-d3ea-4e0d-8736-8e7be8d6dc07","Type":"ContainerDied","Data":"0156a174a4bcf267e9dde86af3be61376edf8c39399f4e0150e9dc719e8c27a9"}
Jul 16 23:21:30.151760 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:30.151377    2573 scope.go:117] "RemoveContainer" containerID="6ad4fb7e6d76af49e67a6f3302729cf8da9149261bb0d663b051e866bd820016"
Jul 16 23:21:30.151760 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:30.151576    2573 scope.go:117] "RemoveContainer" containerID="0156a174a4bcf267e9dde86af3be61376edf8c39399f4e0150e9dc719e8c27a9"
Jul 16 23:21:30.151926 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:21:30.151808    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-qp8kr_odh-ai-gateway-infra(2e8b6e78-d3ea-4e0d-8736-8e7be8d6dc07)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-qp8kr" podUID="2e8b6e78-d3ea-4e0d-8736-8e7be8d6dc07"
Jul 16 23:21:31.158541 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:31.158515    2573 scope.go:117] "RemoveContainer" containerID="0156a174a4bcf267e9dde86af3be61376edf8c39399f4e0150e9dc719e8c27a9"
Jul 16 23:21:31.158876 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:21:31.158660    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-qp8kr_odh-ai-gateway-infra(2e8b6e78-d3ea-4e0d-8736-8e7be8d6dc07)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-qp8kr" podUID="2e8b6e78-d3ea-4e0d-8736-8e7be8d6dc07"
Jul 16 23:21:33.637705 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:33.637668    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-4d775e/test-model-4d775e-kserve-5679b4d6cc-vh8vl"]
Jul 16 23:21:33.641939 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:33.641923    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4d775e/test-model-4d775e-kserve-5679b4d6cc-vh8vl"
Jul 16 23:21:33.644390 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:33.644369    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4d775e\"/\"openshift-service-ca.crt\""
Jul 16 23:21:33.645209 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:33.645192    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4d775e\"/\"default-dockercfg-6t47j\""
Jul 16 23:21:33.645272 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:33.645252    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4d775e\"/\"kube-root-ca.crt\""
Jul 16 23:21:33.645384 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:33.645362    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4d775e\"/\"test-model-4d775e-kserve-self-signed-certs\""
Jul 16 23:21:33.649618 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:33.649593    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4d775e/test-model-4d775e-kserve-5679b4d6cc-vh8vl"]
Jul 16 23:21:33.715005 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:33.714973    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/711e6a1c-5d7c-4e0b-9830-ec204086a72f-tmp-dir\") pod \"test-model-4d775e-kserve-5679b4d6cc-vh8vl\" (UID: \"711e6a1c-5d7c-4e0b-9830-ec204086a72f\") " pod="ai-tenant-e2e-inf-b-4d775e/test-model-4d775e-kserve-5679b4d6cc-vh8vl"
Jul 16 23:21:33.715005 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:33.715005    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjg8r\" (UniqueName: \"kubernetes.io/projected/711e6a1c-5d7c-4e0b-9830-ec204086a72f-kube-api-access-mjg8r\") pod \"test-model-4d775e-kserve-5679b4d6cc-vh8vl\" (UID: \"711e6a1c-5d7c-4e0b-9830-ec204086a72f\") " pod="ai-tenant-e2e-inf-b-4d775e/test-model-4d775e-kserve-5679b4d6cc-vh8vl"
Jul 16 23:21:33.715270 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:33.715039    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/711e6a1c-5d7c-4e0b-9830-ec204086a72f-model-cache\") pod \"test-model-4d775e-kserve-5679b4d6cc-vh8vl\" (UID: \"711e6a1c-5d7c-4e0b-9830-ec204086a72f\") " pod="ai-tenant-e2e-inf-b-4d775e/test-model-4d775e-kserve-5679b4d6cc-vh8vl"
Jul 16 23:21:33.715270 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:33.715108    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/711e6a1c-5d7c-4e0b-9830-ec204086a72f-home\") pod \"test-model-4d775e-kserve-5679b4d6cc-vh8vl\" (UID: \"711e6a1c-5d7c-4e0b-9830-ec204086a72f\") " pod="ai-tenant-e2e-inf-b-4d775e/test-model-4d775e-kserve-5679b4d6cc-vh8vl"
Jul 16 23:21:33.715270 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:33.715151    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/711e6a1c-5d7c-4e0b-9830-ec204086a72f-dshm\") pod \"test-model-4d775e-kserve-5679b4d6cc-vh8vl\" (UID: \"711e6a1c-5d7c-4e0b-9830-ec204086a72f\") " pod="ai-tenant-e2e-inf-b-4d775e/test-model-4d775e-kserve-5679b4d6cc-vh8vl"
Jul 16 23:21:33.715270 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:33.715196    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/711e6a1c-5d7c-4e0b-9830-ec204086a72f-tls-certs\") pod \"test-model-4d775e-kserve-5679b4d6cc-vh8vl\" (UID: \"711e6a1c-5d7c-4e0b-9830-ec204086a72f\") " pod="ai-tenant-e2e-inf-b-4d775e/test-model-4d775e-kserve-5679b4d6cc-vh8vl"
Jul 16 23:21:33.738748 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:33.738693    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:21:33.816206 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:33.816177    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/711e6a1c-5d7c-4e0b-9830-ec204086a72f-tls-certs\") pod \"test-model-4d775e-kserve-5679b4d6cc-vh8vl\" (UID: \"711e6a1c-5d7c-4e0b-9830-ec204086a72f\") " pod="ai-tenant-e2e-inf-b-4d775e/test-model-4d775e-kserve-5679b4d6cc-vh8vl"
Jul 16 23:21:33.816374 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:33.816225    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/711e6a1c-5d7c-4e0b-9830-ec204086a72f-tmp-dir\") pod \"test-model-4d775e-kserve-5679b4d6cc-vh8vl\" (UID: \"711e6a1c-5d7c-4e0b-9830-ec204086a72f\") " pod="ai-tenant-e2e-inf-b-4d775e/test-model-4d775e-kserve-5679b4d6cc-vh8vl"
Jul 16 23:21:33.816374 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:33.816256    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjg8r\" (UniqueName: \"kubernetes.io/projected/711e6a1c-5d7c-4e0b-9830-ec204086a72f-kube-api-access-mjg8r\") pod \"test-model-4d775e-kserve-5679b4d6cc-vh8vl\" (UID: \"711e6a1c-5d7c-4e0b-9830-ec204086a72f\") " pod="ai-tenant-e2e-inf-b-4d775e/test-model-4d775e-kserve-5679b4d6cc-vh8vl"
Jul 16 23:21:33.816374 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:33.816300    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/711e6a1c-5d7c-4e0b-9830-ec204086a72f-model-cache\") pod \"test-model-4d775e-kserve-5679b4d6cc-vh8vl\" (UID: \"711e6a1c-5d7c-4e0b-9830-ec204086a72f\") " pod="ai-tenant-e2e-inf-b-4d775e/test-model-4d775e-kserve-5679b4d6cc-vh8vl"
Jul 16 23:21:33.816374 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:33.816319    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/711e6a1c-5d7c-4e0b-9830-ec204086a72f-home\") pod \"test-model-4d775e-kserve-5679b4d6cc-vh8vl\" (UID: \"711e6a1c-5d7c-4e0b-9830-ec204086a72f\") " pod="ai-tenant-e2e-inf-b-4d775e/test-model-4d775e-kserve-5679b4d6cc-vh8vl"
Jul 16 23:21:33.816374 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:33.816337    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/711e6a1c-5d7c-4e0b-9830-ec204086a72f-dshm\") pod \"test-model-4d775e-kserve-5679b4d6cc-vh8vl\" (UID: \"711e6a1c-5d7c-4e0b-9830-ec204086a72f\") " pod="ai-tenant-e2e-inf-b-4d775e/test-model-4d775e-kserve-5679b4d6cc-vh8vl"
Jul 16 23:21:33.816666 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:33.816641    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/711e6a1c-5d7c-4e0b-9830-ec204086a72f-tmp-dir\") pod \"test-model-4d775e-kserve-5679b4d6cc-vh8vl\" (UID: \"711e6a1c-5d7c-4e0b-9830-ec204086a72f\") " pod="ai-tenant-e2e-inf-b-4d775e/test-model-4d775e-kserve-5679b4d6cc-vh8vl"
Jul 16 23:21:33.816772 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:33.816676    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/711e6a1c-5d7c-4e0b-9830-ec204086a72f-model-cache\") pod \"test-model-4d775e-kserve-5679b4d6cc-vh8vl\" (UID: \"711e6a1c-5d7c-4e0b-9830-ec204086a72f\") " pod="ai-tenant-e2e-inf-b-4d775e/test-model-4d775e-kserve-5679b4d6cc-vh8vl"
Jul 16 23:21:33.816772 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:33.816716    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/711e6a1c-5d7c-4e0b-9830-ec204086a72f-home\") pod \"test-model-4d775e-kserve-5679b4d6cc-vh8vl\" (UID: \"711e6a1c-5d7c-4e0b-9830-ec204086a72f\") " pod="ai-tenant-e2e-inf-b-4d775e/test-model-4d775e-kserve-5679b4d6cc-vh8vl"
Jul 16 23:21:33.818621 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:33.818597    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/711e6a1c-5d7c-4e0b-9830-ec204086a72f-dshm\") pod \"test-model-4d775e-kserve-5679b4d6cc-vh8vl\" (UID: \"711e6a1c-5d7c-4e0b-9830-ec204086a72f\") " pod="ai-tenant-e2e-inf-b-4d775e/test-model-4d775e-kserve-5679b4d6cc-vh8vl"
Jul 16 23:21:33.818885 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:33.818867    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/711e6a1c-5d7c-4e0b-9830-ec204086a72f-tls-certs\") pod \"test-model-4d775e-kserve-5679b4d6cc-vh8vl\" (UID: \"711e6a1c-5d7c-4e0b-9830-ec204086a72f\") " pod="ai-tenant-e2e-inf-b-4d775e/test-model-4d775e-kserve-5679b4d6cc-vh8vl"
Jul 16 23:21:33.824667 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:33.824645    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjg8r\" (UniqueName: \"kubernetes.io/projected/711e6a1c-5d7c-4e0b-9830-ec204086a72f-kube-api-access-mjg8r\") pod \"test-model-4d775e-kserve-5679b4d6cc-vh8vl\" (UID: \"711e6a1c-5d7c-4e0b-9830-ec204086a72f\") " pod="ai-tenant-e2e-inf-b-4d775e/test-model-4d775e-kserve-5679b4d6cc-vh8vl"
Jul 16 23:21:33.951277 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:33.951195    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4d775e/test-model-4d775e-kserve-5679b4d6cc-vh8vl"
Jul 16 23:21:34.080787 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:34.080761    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4d775e/test-model-4d775e-kserve-5679b4d6cc-vh8vl"]
Jul 16 23:21:34.082359 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:21:34.082330    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod711e6a1c_5d7c_4e0b_9830_ec204086a72f.slice/crio-8ef3e72f0409aaa9a066f5f51e97b7950d919d9d0aecfaea5fa5b07954bc905d WatchSource:0}: Error finding container 8ef3e72f0409aaa9a066f5f51e97b7950d919d9d0aecfaea5fa5b07954bc905d: Status 404 returned error can't find the container with id 8ef3e72f0409aaa9a066f5f51e97b7950d919d9d0aecfaea5fa5b07954bc905d
Jul 16 23:21:34.175721 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:34.175686    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4d775e/test-model-4d775e-kserve-5679b4d6cc-vh8vl" event={"ID":"711e6a1c-5d7c-4e0b-9830-ec204086a72f","Type":"ContainerStarted","Data":"7e1988998648d3b4fc0967a08d1eb3c5c43e5441b2475c560e4ea8ad2ebe9143"}
Jul 16 23:21:34.175854 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:34.175734    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4d775e/test-model-4d775e-kserve-5679b4d6cc-vh8vl" event={"ID":"711e6a1c-5d7c-4e0b-9830-ec204086a72f","Type":"ContainerStarted","Data":"8ef3e72f0409aaa9a066f5f51e97b7950d919d9d0aecfaea5fa5b07954bc905d"}
Jul 16 23:21:34.195169 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:34.195115    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-4d775e/test-model-4d775e-kserve-5679b4d6cc-vh8vl" podStartSLOduration=1.195095153 podStartE2EDuration="1.195095153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:21:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:21:34.194844669 +0000 UTC m=+2868.205107334" watchObservedRunningTime="2026-07-16 23:21:34.195095153 +0000 UTC m=+2868.205357818"
Jul 16 23:21:34.513703 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:34.513662    2573 scope.go:117] "RemoveContainer" containerID="5474c9f1c5f6e2751a9327918ab25a94a2393ef7c62a7fc49608c980d7972674"
Jul 16 23:21:35.185454 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:35.185428    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-5trvh" event={"ID":"2df16e94-cebb-4c12-ac8e-6a0b190a8dc9","Type":"ContainerStarted","Data":"b98840a0576aab70a79edb4e096d328a1e647e2ca803328587847521f0eff435"}
Jul 16 23:21:35.202122 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:35.202079    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-5trvh" podStartSLOduration=15.202065 podStartE2EDuration="15.202065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:21:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:21:35.199879964 +0000 UTC m=+2869.210142653" watchObservedRunningTime="2026-07-16 23:21:35.202065 +0000 UTC m=+2869.212327708"
Jul 16 23:21:36.192226 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:36.192188    2573 generic.go:356] "Generic (PLEG): container finished" podID="2df16e94-cebb-4c12-ac8e-6a0b190a8dc9" containerID="b98840a0576aab70a79edb4e096d328a1e647e2ca803328587847521f0eff435" exitCode=6
Jul 16 23:21:36.192731 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:36.192262    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-5trvh" event={"ID":"2df16e94-cebb-4c12-ac8e-6a0b190a8dc9","Type":"ContainerDied","Data":"b98840a0576aab70a79edb4e096d328a1e647e2ca803328587847521f0eff435"}
Jul 16 23:21:36.192731 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:36.192339    2573 scope.go:117] "RemoveContainer" containerID="5474c9f1c5f6e2751a9327918ab25a94a2393ef7c62a7fc49608c980d7972674"
Jul 16 23:21:36.192731 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:36.192636    2573 scope.go:117] "RemoveContainer" containerID="b98840a0576aab70a79edb4e096d328a1e647e2ca803328587847521f0eff435"
Jul 16 23:21:36.192896 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:21:36.192850    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-5trvh_odh-ai-gateway-infra(2df16e94-cebb-4c12-ac8e-6a0b190a8dc9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-5trvh" podUID="2df16e94-cebb-4c12-ac8e-6a0b190a8dc9"
Jul 16 23:21:36.216309 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:36.216286    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-5trvh"]
Jul 16 23:21:37.318228 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:37.318207    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-5trvh"
Jul 16 23:21:37.447400 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:37.447374    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2df16e94-cebb-4c12-ac8e-6a0b190a8dc9-maas-api-service-ca\") pod \"2df16e94-cebb-4c12-ac8e-6a0b190a8dc9\" (UID: \"2df16e94-cebb-4c12-ac8e-6a0b190a8dc9\") "
Jul 16 23:21:37.447720 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:37.447695    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2df16e94-cebb-4c12-ac8e-6a0b190a8dc9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2df16e94-cebb-4c12-ac8e-6a0b190a8dc9" (UID: "2df16e94-cebb-4c12-ac8e-6a0b190a8dc9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:21:37.548042 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:37.547987    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2df16e94-cebb-4c12-ac8e-6a0b190a8dc9-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:21:38.204685 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:38.204638    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-5trvh"
Jul 16 23:21:38.204685 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:38.204676    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-5trvh" event={"ID":"2df16e94-cebb-4c12-ac8e-6a0b190a8dc9","Type":"ContainerDied","Data":"86e1ed9f68d4e6e11183f29fb892e99dfa71601bd5b083bacef29c9a58adbcce"}
Jul 16 23:21:38.204902 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:38.204735    2573 scope.go:117] "RemoveContainer" containerID="b98840a0576aab70a79edb4e096d328a1e647e2ca803328587847521f0eff435"
Jul 16 23:21:38.238326 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:38.238297    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-5trvh"]
Jul 16 23:21:38.242250 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:38.242219    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-5trvh"]
Jul 16 23:21:38.518651 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:38.518584    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2df16e94-cebb-4c12-ac8e-6a0b190a8dc9" path="/var/lib/kubelet/pods/2df16e94-cebb-4c12-ac8e-6a0b190a8dc9/volumes"
Jul 16 23:21:39.154190 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:39.154158    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:21:41.513480 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:41.513449    2573 scope.go:117] "RemoveContainer" containerID="e8af67fe1bff3b34be53570146af3a285dce2acda69a125e375c6688acbe74d6"
Jul 16 23:21:42.227228 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:42.227200    2573 generic.go:356] "Generic (PLEG): container finished" podID="62d53248-592d-43ef-a80e-0e805f156395" containerID="5e581a44206782ef7c11597ff9b138cc78f4e464b82fc490416564c1ae29b295" exitCode=6
Jul 16 23:21:42.227362 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:42.227270    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6swjx" event={"ID":"62d53248-592d-43ef-a80e-0e805f156395","Type":"ContainerDied","Data":"5e581a44206782ef7c11597ff9b138cc78f4e464b82fc490416564c1ae29b295"}
Jul 16 23:21:42.227362 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:42.227325    2573 scope.go:117] "RemoveContainer" containerID="e8af67fe1bff3b34be53570146af3a285dce2acda69a125e375c6688acbe74d6"
Jul 16 23:21:42.227600 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:42.227583    2573 scope.go:117] "RemoveContainer" containerID="5e581a44206782ef7c11597ff9b138cc78f4e464b82fc490416564c1ae29b295"
Jul 16 23:21:42.227766 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:21:42.227744    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-6swjx_odh-ai-gateway-infra(62d53248-592d-43ef-a80e-0e805f156395)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6swjx" podUID="62d53248-592d-43ef-a80e-0e805f156395"
Jul 16 23:21:43.251130 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:43.251086    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6swjx"]
Jul 16 23:21:43.384317 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:43.384293    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6swjx"
Jul 16 23:21:43.496755 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:43.496722    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62d53248-592d-43ef-a80e-0e805f156395-maas-api-service-ca\") pod \"62d53248-592d-43ef-a80e-0e805f156395\" (UID: \"62d53248-592d-43ef-a80e-0e805f156395\") "
Jul 16 23:21:43.497104 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:43.497079    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62d53248-592d-43ef-a80e-0e805f156395-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "62d53248-592d-43ef-a80e-0e805f156395" (UID: "62d53248-592d-43ef-a80e-0e805f156395"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:21:43.514073 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:43.514028    2573 scope.go:117] "RemoveContainer" containerID="0156a174a4bcf267e9dde86af3be61376edf8c39399f4e0150e9dc719e8c27a9"
Jul 16 23:21:43.597573 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:43.597549    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62d53248-592d-43ef-a80e-0e805f156395-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:21:43.951904 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:43.951873    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-4d775e/test-model-4d775e-kserve-5679b4d6cc-vh8vl"
Jul 16 23:21:43.952072 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:43.951930    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-4d775e/test-model-4d775e-kserve-5679b4d6cc-vh8vl"
Jul 16 23:21:43.964403 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:43.964383    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-4d775e/test-model-4d775e-kserve-5679b4d6cc-vh8vl"
Jul 16 23:21:44.240130 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:44.240043    2573 generic.go:356] "Generic (PLEG): container finished" podID="2e8b6e78-d3ea-4e0d-8736-8e7be8d6dc07" containerID="0a905cb0279391ecfe9cb67670c344fcb6989d231e3da9708acd16de9c706995" exitCode=6
Jul 16 23:21:44.240130 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:44.240114    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-qp8kr" event={"ID":"2e8b6e78-d3ea-4e0d-8736-8e7be8d6dc07","Type":"ContainerDied","Data":"0a905cb0279391ecfe9cb67670c344fcb6989d231e3da9708acd16de9c706995"}
Jul 16 23:21:44.240378 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:44.240173    2573 scope.go:117] "RemoveContainer" containerID="0156a174a4bcf267e9dde86af3be61376edf8c39399f4e0150e9dc719e8c27a9"
Jul 16 23:21:44.240504 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:44.240484    2573 scope.go:117] "RemoveContainer" containerID="0a905cb0279391ecfe9cb67670c344fcb6989d231e3da9708acd16de9c706995"
Jul 16 23:21:44.240742 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:21:44.240719    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-qp8kr_odh-ai-gateway-infra(2e8b6e78-d3ea-4e0d-8736-8e7be8d6dc07)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-qp8kr" podUID="2e8b6e78-d3ea-4e0d-8736-8e7be8d6dc07"
Jul 16 23:21:44.241538 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:44.241520    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6swjx"
Jul 16 23:21:44.241614 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:44.241566    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6swjx" event={"ID":"62d53248-592d-43ef-a80e-0e805f156395","Type":"ContainerDied","Data":"d8fbc1ea7f03b297288374bfae98aef9b4e47b576533132fc89f7fd77dfede3f"}
Jul 16 23:21:44.254213 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:44.254191    2573 scope.go:117] "RemoveContainer" containerID="5e581a44206782ef7c11597ff9b138cc78f4e464b82fc490416564c1ae29b295"
Jul 16 23:21:44.255685 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:44.255665    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-4d775e/test-model-4d775e-kserve-5679b4d6cc-vh8vl"
Jul 16 23:21:44.270549 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:44.270528    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6swjx"]
Jul 16 23:21:44.272306 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:44.272285    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6swjx"]
Jul 16 23:21:44.518793 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:44.518712    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62d53248-592d-43ef-a80e-0e805f156395" path="/var/lib/kubelet/pods/62d53248-592d-43ef-a80e-0e805f156395/volumes"
Jul 16 23:21:45.265669 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:45.265642    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-qp8kr"]
Jul 16 23:21:45.394384 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:45.394360    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-qp8kr"
Jul 16 23:21:45.511654 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:45.511624    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e8b6e78-d3ea-4e0d-8736-8e7be8d6dc07-maas-api-service-ca\") pod \"2e8b6e78-d3ea-4e0d-8736-8e7be8d6dc07\" (UID: \"2e8b6e78-d3ea-4e0d-8736-8e7be8d6dc07\") "
Jul 16 23:21:45.511936 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:45.511916    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e8b6e78-d3ea-4e0d-8736-8e7be8d6dc07-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2e8b6e78-d3ea-4e0d-8736-8e7be8d6dc07" (UID: "2e8b6e78-d3ea-4e0d-8736-8e7be8d6dc07"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:21:45.612892 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:45.612864    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e8b6e78-d3ea-4e0d-8736-8e7be8d6dc07-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:21:46.254335 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:46.254302    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-qp8kr"
Jul 16 23:21:46.254335 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:46.254328    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-qp8kr" event={"ID":"2e8b6e78-d3ea-4e0d-8736-8e7be8d6dc07","Type":"ContainerDied","Data":"9cc94380a6b320d9669305031c8c81e73ad72c2e6c52cf1d2971eec15006de4c"}
Jul 16 23:21:46.254558 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:46.254387    2573 scope.go:117] "RemoveContainer" containerID="0a905cb0279391ecfe9cb67670c344fcb6989d231e3da9708acd16de9c706995"
Jul 16 23:21:46.276431 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:46.276402    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-qp8kr"]
Jul 16 23:21:46.282430 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:46.282410    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-qp8kr"]
Jul 16 23:21:46.518340 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:46.518277    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e8b6e78-d3ea-4e0d-8736-8e7be8d6dc07" path="/var/lib/kubelet/pods/2e8b6e78-d3ea-4e0d-8736-8e7be8d6dc07/volumes"
Jul 16 23:21:51.012190 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:51.012160    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8a2515/test-model-8a2515-kserve-854c565bd5-f8mwj"]
Jul 16 23:21:51.037651 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:51.037624    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:21:51.972335 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:51.972306    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4d775e/test-model-4d775e-kserve-5679b4d6cc-vh8vl"]
Jul 16 23:21:51.997216 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:51.997185    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:21:52.673944 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:52.673913    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:21:52.775816 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:52.775787    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:21:58.328477 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:21:58.328436    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:22:02.489794 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:02.489756    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:22:06.031584 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:06.031525    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-8a2515/test-model-8a2515-kserve-854c565bd5-f8mwj" podUID="2f114885-fb59-4f81-b94e-d4b65f148073" containerName="main" containerID="cri-o://aa0778385fe20b979a9d270fe30881dd45a8f220dd55b0374c5d7f508844363a" gracePeriod=45
Jul 16 23:22:06.277972 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:06.277949    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8a2515/test-model-8a2515-kserve-854c565bd5-f8mwj"
Jul 16 23:22:06.347341 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:06.347307    2573 generic.go:356] "Generic (PLEG): container finished" podID="2f114885-fb59-4f81-b94e-d4b65f148073" containerID="aa0778385fe20b979a9d270fe30881dd45a8f220dd55b0374c5d7f508844363a" exitCode=0
Jul 16 23:22:06.347466 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:06.347361    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8a2515/test-model-8a2515-kserve-854c565bd5-f8mwj"
Jul 16 23:22:06.347466 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:06.347378    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8a2515/test-model-8a2515-kserve-854c565bd5-f8mwj" event={"ID":"2f114885-fb59-4f81-b94e-d4b65f148073","Type":"ContainerDied","Data":"aa0778385fe20b979a9d270fe30881dd45a8f220dd55b0374c5d7f508844363a"}
Jul 16 23:22:06.347466 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:06.347416    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8a2515/test-model-8a2515-kserve-854c565bd5-f8mwj" event={"ID":"2f114885-fb59-4f81-b94e-d4b65f148073","Type":"ContainerDied","Data":"db48a25e08ddc2920b671da374f9a9f07cac2afb00cb83f82fd42badfb6c503a"}
Jul 16 23:22:06.347466 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:06.347440    2573 scope.go:117] "RemoveContainer" containerID="aa0778385fe20b979a9d270fe30881dd45a8f220dd55b0374c5d7f508844363a"
Jul 16 23:22:06.355724 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:06.355705    2573 scope.go:117] "RemoveContainer" containerID="aa0778385fe20b979a9d270fe30881dd45a8f220dd55b0374c5d7f508844363a"
Jul 16 23:22:06.355985 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:22:06.355963    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa0778385fe20b979a9d270fe30881dd45a8f220dd55b0374c5d7f508844363a\": container with ID starting with aa0778385fe20b979a9d270fe30881dd45a8f220dd55b0374c5d7f508844363a not found: ID does not exist" containerID="aa0778385fe20b979a9d270fe30881dd45a8f220dd55b0374c5d7f508844363a"
Jul 16 23:22:06.356048 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:06.355996    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa0778385fe20b979a9d270fe30881dd45a8f220dd55b0374c5d7f508844363a"} err="failed to get container status \"aa0778385fe20b979a9d270fe30881dd45a8f220dd55b0374c5d7f508844363a\": rpc error: code = NotFound desc = could not find container \"aa0778385fe20b979a9d270fe30881dd45a8f220dd55b0374c5d7f508844363a\": container with ID starting with aa0778385fe20b979a9d270fe30881dd45a8f220dd55b0374c5d7f508844363a not found: ID does not exist"
Jul 16 23:22:06.368276 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:06.368246    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tpzq\" (UniqueName: \"kubernetes.io/projected/2f114885-fb59-4f81-b94e-d4b65f148073-kube-api-access-4tpzq\") pod \"2f114885-fb59-4f81-b94e-d4b65f148073\" (UID: \"2f114885-fb59-4f81-b94e-d4b65f148073\") "
Jul 16 23:22:06.368348 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:06.368288    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f114885-fb59-4f81-b94e-d4b65f148073-dshm\") pod \"2f114885-fb59-4f81-b94e-d4b65f148073\" (UID: \"2f114885-fb59-4f81-b94e-d4b65f148073\") "
Jul 16 23:22:06.368406 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:06.368345    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f114885-fb59-4f81-b94e-d4b65f148073-tls-certs\") pod \"2f114885-fb59-4f81-b94e-d4b65f148073\" (UID: \"2f114885-fb59-4f81-b94e-d4b65f148073\") "
Jul 16 23:22:06.368406 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:06.368384    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f114885-fb59-4f81-b94e-d4b65f148073-tmp-dir\") pod \"2f114885-fb59-4f81-b94e-d4b65f148073\" (UID: \"2f114885-fb59-4f81-b94e-d4b65f148073\") "
Jul 16 23:22:06.368520 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:06.368411    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f114885-fb59-4f81-b94e-d4b65f148073-home\") pod \"2f114885-fb59-4f81-b94e-d4b65f148073\" (UID: \"2f114885-fb59-4f81-b94e-d4b65f148073\") "
Jul 16 23:22:06.368520 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:06.368458    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f114885-fb59-4f81-b94e-d4b65f148073-model-cache\") pod \"2f114885-fb59-4f81-b94e-d4b65f148073\" (UID: \"2f114885-fb59-4f81-b94e-d4b65f148073\") "
Jul 16 23:22:06.368721 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:06.368663    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f114885-fb59-4f81-b94e-d4b65f148073-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2f114885-fb59-4f81-b94e-d4b65f148073" (UID: "2f114885-fb59-4f81-b94e-d4b65f148073"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:22:06.368835 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:06.368743    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f114885-fb59-4f81-b94e-d4b65f148073-home" (OuterVolumeSpecName: "home") pod "2f114885-fb59-4f81-b94e-d4b65f148073" (UID: "2f114885-fb59-4f81-b94e-d4b65f148073"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:22:06.368835 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:06.368782    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f114885-fb59-4f81-b94e-d4b65f148073-model-cache" (OuterVolumeSpecName: "model-cache") pod "2f114885-fb59-4f81-b94e-d4b65f148073" (UID: "2f114885-fb59-4f81-b94e-d4b65f148073"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:22:06.370399 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:06.370376    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f114885-fb59-4f81-b94e-d4b65f148073-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2f114885-fb59-4f81-b94e-d4b65f148073" (UID: "2f114885-fb59-4f81-b94e-d4b65f148073"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 23:22:06.370487 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:06.370412    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f114885-fb59-4f81-b94e-d4b65f148073-dshm" (OuterVolumeSpecName: "dshm") pod "2f114885-fb59-4f81-b94e-d4b65f148073" (UID: "2f114885-fb59-4f81-b94e-d4b65f148073"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:22:06.370487 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:06.370447    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f114885-fb59-4f81-b94e-d4b65f148073-kube-api-access-4tpzq" (OuterVolumeSpecName: "kube-api-access-4tpzq") pod "2f114885-fb59-4f81-b94e-d4b65f148073" (UID: "2f114885-fb59-4f81-b94e-d4b65f148073"). InnerVolumeSpecName "kube-api-access-4tpzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:22:06.469614 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:06.469578    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f114885-fb59-4f81-b94e-d4b65f148073-dshm\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:22:06.469614 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:06.469611    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f114885-fb59-4f81-b94e-d4b65f148073-tls-certs\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:22:06.469614 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:06.469621    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f114885-fb59-4f81-b94e-d4b65f148073-tmp-dir\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:22:06.469791 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:06.469629    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f114885-fb59-4f81-b94e-d4b65f148073-home\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:22:06.469791 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:06.469637    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f114885-fb59-4f81-b94e-d4b65f148073-model-cache\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:22:06.469791 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:06.469645    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tpzq\" (UniqueName: \"kubernetes.io/projected/2f114885-fb59-4f81-b94e-d4b65f148073-kube-api-access-4tpzq\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:22:06.662726 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:06.662671    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8a2515/test-model-8a2515-kserve-854c565bd5-f8mwj"]
Jul 16 23:22:06.664400 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:06.664384    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-8a2515/test-model-8a2515-kserve-854c565bd5-f8mwj"]
Jul 16 23:22:06.996987 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:06.996882    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-4d775e/test-model-4d775e-kserve-5679b4d6cc-vh8vl" podUID="711e6a1c-5d7c-4e0b-9830-ec204086a72f" containerName="main" containerID="cri-o://7e1988998648d3b4fc0967a08d1eb3c5c43e5441b2475c560e4ea8ad2ebe9143" gracePeriod=45
Jul 16 23:22:07.164018 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:07.163976    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:22:07.247514 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:07.247463    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4d775e/test-model-4d775e-kserve-5679b4d6cc-vh8vl"
Jul 16 23:22:07.355887 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:07.355857    2573 generic.go:356] "Generic (PLEG): container finished" podID="711e6a1c-5d7c-4e0b-9830-ec204086a72f" containerID="7e1988998648d3b4fc0967a08d1eb3c5c43e5441b2475c560e4ea8ad2ebe9143" exitCode=0
Jul 16 23:22:07.356028 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:07.355891    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4d775e/test-model-4d775e-kserve-5679b4d6cc-vh8vl"
Jul 16 23:22:07.356028 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:07.355909    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4d775e/test-model-4d775e-kserve-5679b4d6cc-vh8vl" event={"ID":"711e6a1c-5d7c-4e0b-9830-ec204086a72f","Type":"ContainerDied","Data":"7e1988998648d3b4fc0967a08d1eb3c5c43e5441b2475c560e4ea8ad2ebe9143"}
Jul 16 23:22:07.356028 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:07.355937    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4d775e/test-model-4d775e-kserve-5679b4d6cc-vh8vl" event={"ID":"711e6a1c-5d7c-4e0b-9830-ec204086a72f","Type":"ContainerDied","Data":"8ef3e72f0409aaa9a066f5f51e97b7950d919d9d0aecfaea5fa5b07954bc905d"}
Jul 16 23:22:07.356028 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:07.355955    2573 scope.go:117] "RemoveContainer" containerID="7e1988998648d3b4fc0967a08d1eb3c5c43e5441b2475c560e4ea8ad2ebe9143"
Jul 16 23:22:07.364337 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:07.364313    2573 scope.go:117] "RemoveContainer" containerID="7e1988998648d3b4fc0967a08d1eb3c5c43e5441b2475c560e4ea8ad2ebe9143"
Jul 16 23:22:07.364603 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:22:07.364582    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e1988998648d3b4fc0967a08d1eb3c5c43e5441b2475c560e4ea8ad2ebe9143\": container with ID starting with 7e1988998648d3b4fc0967a08d1eb3c5c43e5441b2475c560e4ea8ad2ebe9143 not found: ID does not exist" containerID="7e1988998648d3b4fc0967a08d1eb3c5c43e5441b2475c560e4ea8ad2ebe9143"
Jul 16 23:22:07.364661 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:07.364614    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e1988998648d3b4fc0967a08d1eb3c5c43e5441b2475c560e4ea8ad2ebe9143"} err="failed to get container status \"7e1988998648d3b4fc0967a08d1eb3c5c43e5441b2475c560e4ea8ad2ebe9143\": rpc error: code = NotFound desc = could not find container \"7e1988998648d3b4fc0967a08d1eb3c5c43e5441b2475c560e4ea8ad2ebe9143\": container with ID starting with 7e1988998648d3b4fc0967a08d1eb3c5c43e5441b2475c560e4ea8ad2ebe9143 not found: ID does not exist"
Jul 16 23:22:07.377011 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:07.376989    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/711e6a1c-5d7c-4e0b-9830-ec204086a72f-dshm\") pod \"711e6a1c-5d7c-4e0b-9830-ec204086a72f\" (UID: \"711e6a1c-5d7c-4e0b-9830-ec204086a72f\") "
Jul 16 23:22:07.377098 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:07.377020    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/711e6a1c-5d7c-4e0b-9830-ec204086a72f-home\") pod \"711e6a1c-5d7c-4e0b-9830-ec204086a72f\" (UID: \"711e6a1c-5d7c-4e0b-9830-ec204086a72f\") "
Jul 16 23:22:07.377098 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:07.377057    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/711e6a1c-5d7c-4e0b-9830-ec204086a72f-tls-certs\") pod \"711e6a1c-5d7c-4e0b-9830-ec204086a72f\" (UID: \"711e6a1c-5d7c-4e0b-9830-ec204086a72f\") "
Jul 16 23:22:07.377098 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:07.377089    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/711e6a1c-5d7c-4e0b-9830-ec204086a72f-tmp-dir\") pod \"711e6a1c-5d7c-4e0b-9830-ec204086a72f\" (UID: \"711e6a1c-5d7c-4e0b-9830-ec204086a72f\") "
Jul 16 23:22:07.377216 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:07.377109    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjg8r\" (UniqueName: \"kubernetes.io/projected/711e6a1c-5d7c-4e0b-9830-ec204086a72f-kube-api-access-mjg8r\") pod \"711e6a1c-5d7c-4e0b-9830-ec204086a72f\" (UID: \"711e6a1c-5d7c-4e0b-9830-ec204086a72f\") "
Jul 16 23:22:07.377216 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:07.377128    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/711e6a1c-5d7c-4e0b-9830-ec204086a72f-model-cache\") pod \"711e6a1c-5d7c-4e0b-9830-ec204086a72f\" (UID: \"711e6a1c-5d7c-4e0b-9830-ec204086a72f\") "
Jul 16 23:22:07.377395 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:07.377367    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711e6a1c-5d7c-4e0b-9830-ec204086a72f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "711e6a1c-5d7c-4e0b-9830-ec204086a72f" (UID: "711e6a1c-5d7c-4e0b-9830-ec204086a72f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:22:07.377492 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:07.377374    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711e6a1c-5d7c-4e0b-9830-ec204086a72f-home" (OuterVolumeSpecName: "home") pod "711e6a1c-5d7c-4e0b-9830-ec204086a72f" (UID: "711e6a1c-5d7c-4e0b-9830-ec204086a72f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:22:07.377541 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:07.377508    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711e6a1c-5d7c-4e0b-9830-ec204086a72f-model-cache" (OuterVolumeSpecName: "model-cache") pod "711e6a1c-5d7c-4e0b-9830-ec204086a72f" (UID: "711e6a1c-5d7c-4e0b-9830-ec204086a72f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:22:07.379134 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:07.379108    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/711e6a1c-5d7c-4e0b-9830-ec204086a72f-dshm" (OuterVolumeSpecName: "dshm") pod "711e6a1c-5d7c-4e0b-9830-ec204086a72f" (UID: "711e6a1c-5d7c-4e0b-9830-ec204086a72f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:22:07.379134 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:07.379118    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/711e6a1c-5d7c-4e0b-9830-ec204086a72f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "711e6a1c-5d7c-4e0b-9830-ec204086a72f" (UID: "711e6a1c-5d7c-4e0b-9830-ec204086a72f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 23:22:07.379277 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:07.379173    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/711e6a1c-5d7c-4e0b-9830-ec204086a72f-kube-api-access-mjg8r" (OuterVolumeSpecName: "kube-api-access-mjg8r") pod "711e6a1c-5d7c-4e0b-9830-ec204086a72f" (UID: "711e6a1c-5d7c-4e0b-9830-ec204086a72f"). InnerVolumeSpecName "kube-api-access-mjg8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:22:07.478611 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:07.478586    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/711e6a1c-5d7c-4e0b-9830-ec204086a72f-dshm\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:22:07.478611 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:07.478609    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/711e6a1c-5d7c-4e0b-9830-ec204086a72f-home\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:22:07.478741 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:07.478621    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/711e6a1c-5d7c-4e0b-9830-ec204086a72f-tls-certs\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:22:07.478741 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:07.478634    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/711e6a1c-5d7c-4e0b-9830-ec204086a72f-tmp-dir\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:22:07.478741 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:07.478646    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjg8r\" (UniqueName: \"kubernetes.io/projected/711e6a1c-5d7c-4e0b-9830-ec204086a72f-kube-api-access-mjg8r\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:22:07.478741 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:07.478659    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/711e6a1c-5d7c-4e0b-9830-ec204086a72f-model-cache\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:22:07.677982 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:07.677952    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4d775e/test-model-4d775e-kserve-5679b4d6cc-vh8vl"]
Jul 16 23:22:07.681718 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:07.681695    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-4d775e/test-model-4d775e-kserve-5679b4d6cc-vh8vl"]
Jul 16 23:22:08.518578 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:08.518524    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f114885-fb59-4f81-b94e-d4b65f148073" path="/var/lib/kubelet/pods/2f114885-fb59-4f81-b94e-d4b65f148073/volumes"
Jul 16 23:22:08.518951 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:08.518911    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="711e6a1c-5d7c-4e0b-9830-ec204086a72f" path="/var/lib/kubelet/pods/711e6a1c-5d7c-4e0b-9830-ec204086a72f/volumes"
Jul 16 23:22:10.903282 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:10.903250    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-ccc56"]
Jul 16 23:22:10.903675 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:10.903539    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e8b6e78-d3ea-4e0d-8736-8e7be8d6dc07" containerName="revoke-keys"
Jul 16 23:22:10.903675 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:10.903550    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e8b6e78-d3ea-4e0d-8736-8e7be8d6dc07" containerName="revoke-keys"
Jul 16 23:22:10.903675 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:10.903561    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62d53248-592d-43ef-a80e-0e805f156395" containerName="revoke-keys"
Jul 16 23:22:10.903675 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:10.903567    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="62d53248-592d-43ef-a80e-0e805f156395" containerName="revoke-keys"
Jul 16 23:22:10.903675 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:10.903578    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2df16e94-cebb-4c12-ac8e-6a0b190a8dc9" containerName="revoke-keys"
Jul 16 23:22:10.903675 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:10.903585    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2df16e94-cebb-4c12-ac8e-6a0b190a8dc9" containerName="revoke-keys"
Jul 16 23:22:10.903675 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:10.903594    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="711e6a1c-5d7c-4e0b-9830-ec204086a72f" containerName="main"
Jul 16 23:22:10.903675 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:10.903599    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="711e6a1c-5d7c-4e0b-9830-ec204086a72f" containerName="main"
Jul 16 23:22:10.903675 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:10.903605    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e8b6e78-d3ea-4e0d-8736-8e7be8d6dc07" containerName="revoke-keys"
Jul 16 23:22:10.903675 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:10.903610    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e8b6e78-d3ea-4e0d-8736-8e7be8d6dc07" containerName="revoke-keys"
Jul 16 23:22:10.903675 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:10.903622    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f114885-fb59-4f81-b94e-d4b65f148073" containerName="main"
Jul 16 23:22:10.903675 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:10.903627    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f114885-fb59-4f81-b94e-d4b65f148073" containerName="main"
Jul 16 23:22:10.903675 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:10.903633    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2df16e94-cebb-4c12-ac8e-6a0b190a8dc9" containerName="revoke-keys"
Jul 16 23:22:10.903675 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:10.903638    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2df16e94-cebb-4c12-ac8e-6a0b190a8dc9" containerName="revoke-keys"
Jul 16 23:22:10.903675 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:10.903645    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2df16e94-cebb-4c12-ac8e-6a0b190a8dc9" containerName="revoke-keys"
Jul 16 23:22:10.903675 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:10.903650    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2df16e94-cebb-4c12-ac8e-6a0b190a8dc9" containerName="revoke-keys"
Jul 16 23:22:10.903675 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:10.903655    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62d53248-592d-43ef-a80e-0e805f156395" containerName="revoke-keys"
Jul 16 23:22:10.903675 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:10.903660    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="62d53248-592d-43ef-a80e-0e805f156395" containerName="revoke-keys"
Jul 16 23:22:10.904374 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:10.903706    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="62d53248-592d-43ef-a80e-0e805f156395" containerName="revoke-keys"
Jul 16 23:22:10.904374 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:10.903715    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e8b6e78-d3ea-4e0d-8736-8e7be8d6dc07" containerName="revoke-keys"
Jul 16 23:22:10.904374 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:10.903721    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2df16e94-cebb-4c12-ac8e-6a0b190a8dc9" containerName="revoke-keys"
Jul 16 23:22:10.904374 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:10.903727    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f114885-fb59-4f81-b94e-d4b65f148073" containerName="main"
Jul 16 23:22:10.904374 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:10.903734    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="711e6a1c-5d7c-4e0b-9830-ec204086a72f" containerName="main"
Jul 16 23:22:10.904374 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:10.903739    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e8b6e78-d3ea-4e0d-8736-8e7be8d6dc07" containerName="revoke-keys"
Jul 16 23:22:10.904374 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:10.903748    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2df16e94-cebb-4c12-ac8e-6a0b190a8dc9" containerName="revoke-keys"
Jul 16 23:22:10.904374 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:10.903753    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="62d53248-592d-43ef-a80e-0e805f156395" containerName="revoke-keys"
Jul 16 23:22:10.908034 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:10.908015    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-ccc56"
Jul 16 23:22:10.910501 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:10.910464    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 23:22:10.911092 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:10.911067    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:22:10.913354 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:10.913303    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-ccc56"]
Jul 16 23:22:11.005081 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:11.005055    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b08d2986-c8c9-4371-8303-891f507c1f07-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-ccc56\" (UID: \"b08d2986-c8c9-4371-8303-891f507c1f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-ccc56"
Jul 16 23:22:11.105826 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:11.105801    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b08d2986-c8c9-4371-8303-891f507c1f07-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-ccc56\" (UID: \"b08d2986-c8c9-4371-8303-891f507c1f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-ccc56"
Jul 16 23:22:11.106474 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:11.106451    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b08d2986-c8c9-4371-8303-891f507c1f07-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-ccc56\" (UID: \"b08d2986-c8c9-4371-8303-891f507c1f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-ccc56"
Jul 16 23:22:11.220267 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:11.220191    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-ccc56"
Jul 16 23:22:11.345990 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:11.345969    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-ccc56"]
Jul 16 23:22:11.347986 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:22:11.347961    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb08d2986_c8c9_4371_8303_891f507c1f07.slice/crio-1427450bab2de0b1249439785940ad7da85108ae76910528dbe8f60cf0e54e46 WatchSource:0}: Error finding container 1427450bab2de0b1249439785940ad7da85108ae76910528dbe8f60cf0e54e46: Status 404 returned error can't find the container with id 1427450bab2de0b1249439785940ad7da85108ae76910528dbe8f60cf0e54e46
Jul 16 23:22:11.376834 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:11.376801    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-ccc56" event={"ID":"b08d2986-c8c9-4371-8303-891f507c1f07","Type":"ContainerStarted","Data":"1427450bab2de0b1249439785940ad7da85108ae76910528dbe8f60cf0e54e46"}
Jul 16 23:22:12.382607 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:12.382576    2573 generic.go:356] "Generic (PLEG): container finished" podID="b08d2986-c8c9-4371-8303-891f507c1f07" containerID="0e2c73cda9db7679b3cbd08345b6338ad8a0ecd8de1a3a4a638958c3d141f916" exitCode=6
Jul 16 23:22:12.382954 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:12.382634    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-ccc56" event={"ID":"b08d2986-c8c9-4371-8303-891f507c1f07","Type":"ContainerDied","Data":"0e2c73cda9db7679b3cbd08345b6338ad8a0ecd8de1a3a4a638958c3d141f916"}
Jul 16 23:22:12.382954 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:12.382850    2573 scope.go:117] "RemoveContainer" containerID="0e2c73cda9db7679b3cbd08345b6338ad8a0ecd8de1a3a4a638958c3d141f916"
Jul 16 23:22:12.439891 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:12.439870    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:22:12.877549 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:12.877513    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:22:12.935609 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:12.935584    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:22:13.389186 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:13.389151    2573 generic.go:356] "Generic (PLEG): container finished" podID="b08d2986-c8c9-4371-8303-891f507c1f07" containerID="aece8cf22e14ed9b99e206eb24d2c9eebb804dd4fa2f6cfb81dc4c6c4db45d4c" exitCode=6
Jul 16 23:22:13.389637 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:13.389192    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-ccc56" event={"ID":"b08d2986-c8c9-4371-8303-891f507c1f07","Type":"ContainerDied","Data":"aece8cf22e14ed9b99e206eb24d2c9eebb804dd4fa2f6cfb81dc4c6c4db45d4c"}
Jul 16 23:22:13.389637 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:13.389269    2573 scope.go:117] "RemoveContainer" containerID="0e2c73cda9db7679b3cbd08345b6338ad8a0ecd8de1a3a4a638958c3d141f916"
Jul 16 23:22:13.389637 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:13.389484    2573 scope.go:117] "RemoveContainer" containerID="aece8cf22e14ed9b99e206eb24d2c9eebb804dd4fa2f6cfb81dc4c6c4db45d4c"
Jul 16 23:22:13.389804 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:22:13.389683    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-ccc56_odh-ai-gateway-infra(b08d2986-c8c9-4371-8303-891f507c1f07)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-ccc56" podUID="b08d2986-c8c9-4371-8303-891f507c1f07"
Jul 16 23:22:14.395270 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:14.395225    2573 scope.go:117] "RemoveContainer" containerID="aece8cf22e14ed9b99e206eb24d2c9eebb804dd4fa2f6cfb81dc4c6c4db45d4c"
Jul 16 23:22:14.395748 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:22:14.395467    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-ccc56_odh-ai-gateway-infra(b08d2986-c8c9-4371-8303-891f507c1f07)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-ccc56" podUID="b08d2986-c8c9-4371-8303-891f507c1f07"
Jul 16 23:22:17.905937 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:17.905904    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:22:18.115394 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:18.115355    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-jsl7m"]
Jul 16 23:22:18.115753 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:18.115733    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62d53248-592d-43ef-a80e-0e805f156395" containerName="revoke-keys"
Jul 16 23:22:18.115851 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:18.115756    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="62d53248-592d-43ef-a80e-0e805f156395" containerName="revoke-keys"
Jul 16 23:22:18.115851 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:18.115781    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e8b6e78-d3ea-4e0d-8736-8e7be8d6dc07" containerName="revoke-keys"
Jul 16 23:22:18.115851 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:18.115789    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e8b6e78-d3ea-4e0d-8736-8e7be8d6dc07" containerName="revoke-keys"
Jul 16 23:22:18.116006 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:18.115872    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2df16e94-cebb-4c12-ac8e-6a0b190a8dc9" containerName="revoke-keys"
Jul 16 23:22:18.116006 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:18.115887    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e8b6e78-d3ea-4e0d-8736-8e7be8d6dc07" containerName="revoke-keys"
Jul 16 23:22:18.116006 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:18.115900    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="62d53248-592d-43ef-a80e-0e805f156395" containerName="revoke-keys"
Jul 16 23:22:18.120210 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:18.120188    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-jsl7m"
Jul 16 23:22:18.127108 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:18.127085    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-jsl7m"]
Jul 16 23:22:18.152906 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:18.152868    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b968752-a57e-4951-bb34-093d3cac44fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-jsl7m\" (UID: \"1b968752-a57e-4951-bb34-093d3cac44fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-jsl7m"
Jul 16 23:22:18.154052 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:18.154034    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:22:18.254217 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:18.254152    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b968752-a57e-4951-bb34-093d3cac44fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-jsl7m\" (UID: \"1b968752-a57e-4951-bb34-093d3cac44fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-jsl7m"
Jul 16 23:22:18.254792 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:18.254766    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b968752-a57e-4951-bb34-093d3cac44fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-jsl7m\" (UID: \"1b968752-a57e-4951-bb34-093d3cac44fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-jsl7m"
Jul 16 23:22:18.269385 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:18.269360    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4d775e-nn8l8"]
Jul 16 23:22:18.272578 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:18.272564    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4d775e-nn8l8"
Jul 16 23:22:18.279689 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:18.279667    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4d775e-nn8l8"]
Jul 16 23:22:18.355286 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:18.355264    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/818d5380-3cca-4741-bc74-de88a5a5d701-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4d775e-nn8l8\" (UID: \"818d5380-3cca-4741-bc74-de88a5a5d701\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4d775e-nn8l8"
Jul 16 23:22:18.431416 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:18.431394    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-jsl7m"
Jul 16 23:22:18.456360 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:18.456335    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/818d5380-3cca-4741-bc74-de88a5a5d701-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4d775e-nn8l8\" (UID: \"818d5380-3cca-4741-bc74-de88a5a5d701\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4d775e-nn8l8"
Jul 16 23:22:18.456898 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:18.456879    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/818d5380-3cca-4741-bc74-de88a5a5d701-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4d775e-nn8l8\" (UID: \"818d5380-3cca-4741-bc74-de88a5a5d701\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4d775e-nn8l8"
Jul 16 23:22:18.562134 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:18.562111    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-jsl7m"]
Jul 16 23:22:18.563893 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:22:18.563862    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b968752_a57e_4951_bb34_093d3cac44fc.slice/crio-0d3321176850d3e5835fd6cbb9e9d6a1f4033e4910bdd00127fbc153a869265b WatchSource:0}: Error finding container 0d3321176850d3e5835fd6cbb9e9d6a1f4033e4910bdd00127fbc153a869265b: Status 404 returned error can't find the container with id 0d3321176850d3e5835fd6cbb9e9d6a1f4033e4910bdd00127fbc153a869265b
Jul 16 23:22:18.583991 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:18.583971    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4d775e-nn8l8"
Jul 16 23:22:18.708460 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:18.708432    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4d775e-nn8l8"]
Jul 16 23:22:18.710072 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:22:18.710050    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod818d5380_3cca_4741_bc74_de88a5a5d701.slice/crio-33e7c25bd735e3f349e45120bb643644c3b03369082f4e875306962982f04671 WatchSource:0}: Error finding container 33e7c25bd735e3f349e45120bb643644c3b03369082f4e875306962982f04671: Status 404 returned error can't find the container with id 33e7c25bd735e3f349e45120bb643644c3b03369082f4e875306962982f04671
Jul 16 23:22:18.860784 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:18.860756    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:22:18.966894 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:18.966869    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:22:19.418214 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:19.418186    2573 generic.go:356] "Generic (PLEG): container finished" podID="1b968752-a57e-4951-bb34-093d3cac44fc" containerID="b19b2e5a9d5488358c9656aaa07e107d55a806fa396b3b5be487a8d9287c4782" exitCode=6
Jul 16 23:22:19.418357 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:19.418302    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-jsl7m" event={"ID":"1b968752-a57e-4951-bb34-093d3cac44fc","Type":"ContainerDied","Data":"b19b2e5a9d5488358c9656aaa07e107d55a806fa396b3b5be487a8d9287c4782"}
Jul 16 23:22:19.418357 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:19.418352    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-jsl7m" event={"ID":"1b968752-a57e-4951-bb34-093d3cac44fc","Type":"ContainerStarted","Data":"0d3321176850d3e5835fd6cbb9e9d6a1f4033e4910bdd00127fbc153a869265b"}
Jul 16 23:22:19.418538 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:19.418525    2573 scope.go:117] "RemoveContainer" containerID="b19b2e5a9d5488358c9656aaa07e107d55a806fa396b3b5be487a8d9287c4782"
Jul 16 23:22:19.419978 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:19.419959    2573 generic.go:356] "Generic (PLEG): container finished" podID="818d5380-3cca-4741-bc74-de88a5a5d701" containerID="04881de20960c1e008531ceae8da00b843d9d61dca7b7564e7971699c6566606" exitCode=0
Jul 16 23:22:19.420082 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:19.420038    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4d775e-nn8l8" event={"ID":"818d5380-3cca-4741-bc74-de88a5a5d701","Type":"ContainerDied","Data":"04881de20960c1e008531ceae8da00b843d9d61dca7b7564e7971699c6566606"}
Jul 16 23:22:19.420141 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:19.420090    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4d775e-nn8l8" event={"ID":"818d5380-3cca-4741-bc74-de88a5a5d701","Type":"ContainerStarted","Data":"33e7c25bd735e3f349e45120bb643644c3b03369082f4e875306962982f04671"}
Jul 16 23:22:20.427959 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:20.427924    2573 generic.go:356] "Generic (PLEG): container finished" podID="1b968752-a57e-4951-bb34-093d3cac44fc" containerID="7f0cd6d5d9936e39f445369893f25462a06da9c84971166c8952bf0502c69af3" exitCode=6
Jul 16 23:22:20.428415 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:20.428011    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-jsl7m" event={"ID":"1b968752-a57e-4951-bb34-093d3cac44fc","Type":"ContainerDied","Data":"7f0cd6d5d9936e39f445369893f25462a06da9c84971166c8952bf0502c69af3"}
Jul 16 23:22:20.428415 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:20.428077    2573 scope.go:117] "RemoveContainer" containerID="b19b2e5a9d5488358c9656aaa07e107d55a806fa396b3b5be487a8d9287c4782"
Jul 16 23:22:20.428415 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:20.428167    2573 scope.go:117] "RemoveContainer" containerID="7f0cd6d5d9936e39f445369893f25462a06da9c84971166c8952bf0502c69af3"
Jul 16 23:22:20.428415 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:22:20.428341    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-jsl7m_odh-ai-gateway-infra(1b968752-a57e-4951-bb34-093d3cac44fc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-jsl7m" podUID="1b968752-a57e-4951-bb34-093d3cac44fc"
Jul 16 23:22:20.545116 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:20.545095    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4d775e-nn8l8"
Jul 16 23:22:20.671177 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:20.671148    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/818d5380-3cca-4741-bc74-de88a5a5d701-maas-api-service-ca\") pod \"818d5380-3cca-4741-bc74-de88a5a5d701\" (UID: \"818d5380-3cca-4741-bc74-de88a5a5d701\") "
Jul 16 23:22:20.671461 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:20.671439    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/818d5380-3cca-4741-bc74-de88a5a5d701-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "818d5380-3cca-4741-bc74-de88a5a5d701" (UID: "818d5380-3cca-4741-bc74-de88a5a5d701"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:22:20.772225 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:20.772167    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/818d5380-3cca-4741-bc74-de88a5a5d701-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:22:21.433604 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:21.433575    2573 scope.go:117] "RemoveContainer" containerID="7f0cd6d5d9936e39f445369893f25462a06da9c84971166c8952bf0502c69af3"
Jul 16 23:22:21.433999 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:22:21.433761    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-jsl7m_odh-ai-gateway-infra(1b968752-a57e-4951-bb34-093d3cac44fc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-jsl7m" podUID="1b968752-a57e-4951-bb34-093d3cac44fc"
Jul 16 23:22:21.434617 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:21.434599    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4d775e-nn8l8"
Jul 16 23:22:21.434721 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:21.434640    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4d775e-nn8l8" event={"ID":"818d5380-3cca-4741-bc74-de88a5a5d701","Type":"ContainerDied","Data":"33e7c25bd735e3f349e45120bb643644c3b03369082f4e875306962982f04671"}
Jul 16 23:22:21.434721 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:21.434676    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33e7c25bd735e3f349e45120bb643644c3b03369082f4e875306962982f04671"
Jul 16 23:22:23.170261 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:23.170204    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-jsknh"]
Jul 16 23:22:23.170653 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:23.170560    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="818d5380-3cca-4741-bc74-de88a5a5d701" containerName="revoke-keys"
Jul 16 23:22:23.170653 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:23.170574    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="818d5380-3cca-4741-bc74-de88a5a5d701" containerName="revoke-keys"
Jul 16 23:22:23.170653 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:23.170623    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="818d5380-3cca-4741-bc74-de88a5a5d701" containerName="revoke-keys"
Jul 16 23:22:23.173578 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:23.173548    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-jsknh"
Jul 16 23:22:23.181841 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:23.181817    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-jsknh"]
Jul 16 23:22:23.184931 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:23.184901    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5e00d30-9c72-463c-a84d-7668bcd08e38-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-jsknh\" (UID: \"c5e00d30-9c72-463c-a84d-7668bcd08e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-jsknh"
Jul 16 23:22:23.286067 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:23.286040    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5e00d30-9c72-463c-a84d-7668bcd08e38-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-jsknh\" (UID: \"c5e00d30-9c72-463c-a84d-7668bcd08e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-jsknh"
Jul 16 23:22:23.286659 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:23.286641    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5e00d30-9c72-463c-a84d-7668bcd08e38-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-jsknh\" (UID: \"c5e00d30-9c72-463c-a84d-7668bcd08e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-jsknh"
Jul 16 23:22:23.486398 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:23.486332    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-jsknh"
Jul 16 23:22:23.611776 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:23.611750    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-jsknh"]
Jul 16 23:22:23.614094 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:22:23.614067    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5e00d30_9c72_463c_a84d_7668bcd08e38.slice/crio-e946ea58bb3c14abc701ce778ef9ec55b1c8cdf46b16f8b797e570cf6d92f2d8 WatchSource:0}: Error finding container e946ea58bb3c14abc701ce778ef9ec55b1c8cdf46b16f8b797e570cf6d92f2d8: Status 404 returned error can't find the container with id e946ea58bb3c14abc701ce778ef9ec55b1c8cdf46b16f8b797e570cf6d92f2d8
Jul 16 23:22:24.451697 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:24.451618    2573 generic.go:356] "Generic (PLEG): container finished" podID="c5e00d30-9c72-463c-a84d-7668bcd08e38" containerID="2d842798ade56ccc33e098947f803f479963ffa6e45103e19c6830e50fff3f5a" exitCode=6
Jul 16 23:22:24.451697 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:24.451675    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-jsknh" event={"ID":"c5e00d30-9c72-463c-a84d-7668bcd08e38","Type":"ContainerDied","Data":"2d842798ade56ccc33e098947f803f479963ffa6e45103e19c6830e50fff3f5a"}
Jul 16 23:22:24.452051 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:24.451710    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-jsknh" event={"ID":"c5e00d30-9c72-463c-a84d-7668bcd08e38","Type":"ContainerStarted","Data":"e946ea58bb3c14abc701ce778ef9ec55b1c8cdf46b16f8b797e570cf6d92f2d8"}
Jul 16 23:22:24.452051 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:24.451842    2573 scope.go:117] "RemoveContainer" containerID="2d842798ade56ccc33e098947f803f479963ffa6e45103e19c6830e50fff3f5a"
Jul 16 23:22:25.458159 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:25.458080    2573 generic.go:356] "Generic (PLEG): container finished" podID="c5e00d30-9c72-463c-a84d-7668bcd08e38" containerID="4ffdefa19e637963d13708c9716ca8c08d6fe92deaf7114c8ce5bcba17d7589f" exitCode=6
Jul 16 23:22:25.458516 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:25.458170    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-jsknh" event={"ID":"c5e00d30-9c72-463c-a84d-7668bcd08e38","Type":"ContainerDied","Data":"4ffdefa19e637963d13708c9716ca8c08d6fe92deaf7114c8ce5bcba17d7589f"}
Jul 16 23:22:25.458516 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:25.458224    2573 scope.go:117] "RemoveContainer" containerID="2d842798ade56ccc33e098947f803f479963ffa6e45103e19c6830e50fff3f5a"
Jul 16 23:22:25.458516 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:25.458344    2573 scope.go:117] "RemoveContainer" containerID="4ffdefa19e637963d13708c9716ca8c08d6fe92deaf7114c8ce5bcba17d7589f"
Jul 16 23:22:25.458673 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:22:25.458527    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-jsknh_odh-ai-gateway-infra(c5e00d30-9c72-463c-a84d-7668bcd08e38)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-jsknh" podUID="c5e00d30-9c72-463c-a84d-7668bcd08e38"
Jul 16 23:22:26.464971 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:26.464948    2573 scope.go:117] "RemoveContainer" containerID="4ffdefa19e637963d13708c9716ca8c08d6fe92deaf7114c8ce5bcba17d7589f"
Jul 16 23:22:26.465372 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:22:26.465109    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-jsknh_odh-ai-gateway-infra(c5e00d30-9c72-463c-a84d-7668bcd08e38)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-jsknh" podUID="c5e00d30-9c72-463c-a84d-7668bcd08e38"
Jul 16 23:22:27.513679 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:27.513652    2573 scope.go:117] "RemoveContainer" containerID="aece8cf22e14ed9b99e206eb24d2c9eebb804dd4fa2f6cfb81dc4c6c4db45d4c"
Jul 16 23:22:28.476013 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:28.475929    2573 generic.go:356] "Generic (PLEG): container finished" podID="b08d2986-c8c9-4371-8303-891f507c1f07" containerID="e44b673a49c4742a12c592aba4fbdb5356d187a6db5eb00f13531dc1b8464290" exitCode=6
Jul 16 23:22:28.476143 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:28.476010    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-ccc56" event={"ID":"b08d2986-c8c9-4371-8303-891f507c1f07","Type":"ContainerDied","Data":"e44b673a49c4742a12c592aba4fbdb5356d187a6db5eb00f13531dc1b8464290"}
Jul 16 23:22:28.476143 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:28.476073    2573 scope.go:117] "RemoveContainer" containerID="aece8cf22e14ed9b99e206eb24d2c9eebb804dd4fa2f6cfb81dc4c6c4db45d4c"
Jul 16 23:22:28.476352 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:28.476338    2573 scope.go:117] "RemoveContainer" containerID="e44b673a49c4742a12c592aba4fbdb5356d187a6db5eb00f13531dc1b8464290"
Jul 16 23:22:28.476563 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:22:28.476540    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-ccc56_odh-ai-gateway-infra(b08d2986-c8c9-4371-8303-891f507c1f07)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-ccc56" podUID="b08d2986-c8c9-4371-8303-891f507c1f07"
Jul 16 23:22:28.771645 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:28.771563    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:22:29.499566 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:29.499532    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-ccc56"]
Jul 16 23:22:29.631802 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:29.631781    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-ccc56"
Jul 16 23:22:29.735041 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:29.735018    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b08d2986-c8c9-4371-8303-891f507c1f07-maas-api-service-ca\") pod \"b08d2986-c8c9-4371-8303-891f507c1f07\" (UID: \"b08d2986-c8c9-4371-8303-891f507c1f07\") "
Jul 16 23:22:29.735408 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:29.735384    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b08d2986-c8c9-4371-8303-891f507c1f07-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b08d2986-c8c9-4371-8303-891f507c1f07" (UID: "b08d2986-c8c9-4371-8303-891f507c1f07"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:22:29.835690 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:29.835667    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b08d2986-c8c9-4371-8303-891f507c1f07-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:22:30.489154 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:30.489130    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-ccc56"
Jul 16 23:22:30.489154 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:30.489141    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-ccc56" event={"ID":"b08d2986-c8c9-4371-8303-891f507c1f07","Type":"ContainerDied","Data":"1427450bab2de0b1249439785940ad7da85108ae76910528dbe8f60cf0e54e46"}
Jul 16 23:22:30.489367 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:30.489203    2573 scope.go:117] "RemoveContainer" containerID="e44b673a49c4742a12c592aba4fbdb5356d187a6db5eb00f13531dc1b8464290"
Jul 16 23:22:30.511952 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:30.511929    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-ccc56"]
Jul 16 23:22:30.518732 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:30.518713    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-ccc56"]
Jul 16 23:22:31.630116 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:31.630085    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:22:32.514166 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:32.514133    2573 scope.go:117] "RemoveContainer" containerID="7f0cd6d5d9936e39f445369893f25462a06da9c84971166c8952bf0502c69af3"
Jul 16 23:22:32.519531 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:32.519505    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b08d2986-c8c9-4371-8303-891f507c1f07" path="/var/lib/kubelet/pods/b08d2986-c8c9-4371-8303-891f507c1f07/volumes"
Jul 16 23:22:33.350307 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:33.350276    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:22:33.506496 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:33.506410    2573 generic.go:356] "Generic (PLEG): container finished" podID="1b968752-a57e-4951-bb34-093d3cac44fc" containerID="34ac0ba8b5ebaa2d919cfa87eb406760d738a143eee885fbf64745892e2a06a8" exitCode=6
Jul 16 23:22:33.506496 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:33.506481    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-jsl7m" event={"ID":"1b968752-a57e-4951-bb34-093d3cac44fc","Type":"ContainerDied","Data":"34ac0ba8b5ebaa2d919cfa87eb406760d738a143eee885fbf64745892e2a06a8"}
Jul 16 23:22:33.506692 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:33.506561    2573 scope.go:117] "RemoveContainer" containerID="7f0cd6d5d9936e39f445369893f25462a06da9c84971166c8952bf0502c69af3"
Jul 16 23:22:33.506849 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:33.506828    2573 scope.go:117] "RemoveContainer" containerID="34ac0ba8b5ebaa2d919cfa87eb406760d738a143eee885fbf64745892e2a06a8"
Jul 16 23:22:33.507057 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:22:33.507035    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-jsl7m_odh-ai-gateway-infra(1b968752-a57e-4951-bb34-093d3cac44fc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-jsl7m" podUID="1b968752-a57e-4951-bb34-093d3cac44fc"
Jul 16 23:22:34.531316 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:34.531283    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-jsl7m"]
Jul 16 23:22:34.654673 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:34.654649    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-jsl7m"
Jul 16 23:22:34.673628 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:34.673397    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b968752-a57e-4951-bb34-093d3cac44fc-maas-api-service-ca\") pod \"1b968752-a57e-4951-bb34-093d3cac44fc\" (UID: \"1b968752-a57e-4951-bb34-093d3cac44fc\") "
Jul 16 23:22:34.673774 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:34.673749    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b968752-a57e-4951-bb34-093d3cac44fc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1b968752-a57e-4951-bb34-093d3cac44fc" (UID: "1b968752-a57e-4951-bb34-093d3cac44fc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:22:34.774094 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:34.774070    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b968752-a57e-4951-bb34-093d3cac44fc-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:22:35.519006 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:35.518977    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-jsl7m"
Jul 16 23:22:35.519156 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:35.519021    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-jsl7m" event={"ID":"1b968752-a57e-4951-bb34-093d3cac44fc","Type":"ContainerDied","Data":"0d3321176850d3e5835fd6cbb9e9d6a1f4033e4910bdd00127fbc153a869265b"}
Jul 16 23:22:35.519156 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:35.519079    2573 scope.go:117] "RemoveContainer" containerID="34ac0ba8b5ebaa2d919cfa87eb406760d738a143eee885fbf64745892e2a06a8"
Jul 16 23:22:35.542346 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:35.542322    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-jsl7m"]
Jul 16 23:22:35.548457 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:35.548438    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-jsl7m"]
Jul 16 23:22:36.518050 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:36.518021    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b968752-a57e-4951-bb34-093d3cac44fc" path="/var/lib/kubelet/pods/1b968752-a57e-4951-bb34-093d3cac44fc/volumes"
Jul 16 23:22:39.006627 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:39.006583    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:22:39.107163 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:39.107130    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:22:41.513917 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:41.513893    2573 scope.go:117] "RemoveContainer" containerID="4ffdefa19e637963d13708c9716ca8c08d6fe92deaf7114c8ce5bcba17d7589f"
Jul 16 23:22:42.556647 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:42.556563    2573 generic.go:356] "Generic (PLEG): container finished" podID="c5e00d30-9c72-463c-a84d-7668bcd08e38" containerID="4aad2d51451ee09a3569498160f66bbf894790003183e9c6b7a6a262307df93d" exitCode=6
Jul 16 23:22:42.556647 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:42.556636    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-jsknh" event={"ID":"c5e00d30-9c72-463c-a84d-7668bcd08e38","Type":"ContainerDied","Data":"4aad2d51451ee09a3569498160f66bbf894790003183e9c6b7a6a262307df93d"}
Jul 16 23:22:42.557085 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:42.556689    2573 scope.go:117] "RemoveContainer" containerID="4ffdefa19e637963d13708c9716ca8c08d6fe92deaf7114c8ce5bcba17d7589f"
Jul 16 23:22:42.557085 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:42.556896    2573 scope.go:117] "RemoveContainer" containerID="4aad2d51451ee09a3569498160f66bbf894790003183e9c6b7a6a262307df93d"
Jul 16 23:22:42.557152 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:22:42.557098    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-jsknh_odh-ai-gateway-infra(c5e00d30-9c72-463c-a84d-7668bcd08e38)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-jsknh" podUID="c5e00d30-9c72-463c-a84d-7668bcd08e38"
Jul 16 23:22:43.580634 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:43.580603    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-jsknh"]
Jul 16 23:22:43.707692 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:43.707671    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-jsknh"
Jul 16 23:22:43.737754 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:43.737732    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5e00d30-9c72-463c-a84d-7668bcd08e38-maas-api-service-ca\") pod \"c5e00d30-9c72-463c-a84d-7668bcd08e38\" (UID: \"c5e00d30-9c72-463c-a84d-7668bcd08e38\") "
Jul 16 23:22:43.738077 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:43.738056    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5e00d30-9c72-463c-a84d-7668bcd08e38-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c5e00d30-9c72-463c-a84d-7668bcd08e38" (UID: "c5e00d30-9c72-463c-a84d-7668bcd08e38"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:22:43.838580 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:43.838520    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5e00d30-9c72-463c-a84d-7668bcd08e38-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:22:44.044365 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:44.044332    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:22:44.261633 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:44.261555    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:22:44.568894 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:44.568864    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-jsknh" event={"ID":"c5e00d30-9c72-463c-a84d-7668bcd08e38","Type":"ContainerDied","Data":"e946ea58bb3c14abc701ce778ef9ec55b1c8cdf46b16f8b797e570cf6d92f2d8"}
Jul 16 23:22:44.569055 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:44.568916    2573 scope.go:117] "RemoveContainer" containerID="4aad2d51451ee09a3569498160f66bbf894790003183e9c6b7a6a262307df93d"
Jul 16 23:22:44.569055 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:44.568876    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-jsknh"
Jul 16 23:22:44.584045 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:44.584023    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-jsknh"]
Jul 16 23:22:44.588622 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:44.588600    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:22:44.589808 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:44.589790    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-jsknh"]
Jul 16 23:22:44.894816 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:44.894788    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:22:45.512318 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:45.512293    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:22:46.520005 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:46.519972    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5e00d30-9c72-463c-a84d-7668bcd08e38" path="/var/lib/kubelet/pods/c5e00d30-9c72-463c-a84d-7668bcd08e38/volumes"
Jul 16 23:22:48.201701 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:48.201673    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:22:48.283668 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:48.283641    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:22:48.349914 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:48.349889    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:22:48.777404 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:48.777371    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:22:48.879032 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:48.879001    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:22:48.976222 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:48.976181    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:22:52.014369 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:52.014322    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-xlnrs"]
Jul 16 23:22:52.014808 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:52.014671    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5e00d30-9c72-463c-a84d-7668bcd08e38" containerName="revoke-keys"
Jul 16 23:22:52.014808 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:52.014685    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5e00d30-9c72-463c-a84d-7668bcd08e38" containerName="revoke-keys"
Jul 16 23:22:52.014808 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:52.014698    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5e00d30-9c72-463c-a84d-7668bcd08e38" containerName="revoke-keys"
Jul 16 23:22:52.014808 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:52.014704    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5e00d30-9c72-463c-a84d-7668bcd08e38" containerName="revoke-keys"
Jul 16 23:22:52.014808 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:52.014711    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b08d2986-c8c9-4371-8303-891f507c1f07" containerName="revoke-keys"
Jul 16 23:22:52.014808 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:52.014719    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b08d2986-c8c9-4371-8303-891f507c1f07" containerName="revoke-keys"
Jul 16 23:22:52.014808 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:52.014728    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5e00d30-9c72-463c-a84d-7668bcd08e38" containerName="revoke-keys"
Jul 16 23:22:52.014808 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:52.014733    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5e00d30-9c72-463c-a84d-7668bcd08e38" containerName="revoke-keys"
Jul 16 23:22:52.014808 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:52.014743    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b968752-a57e-4951-bb34-093d3cac44fc" containerName="revoke-keys"
Jul 16 23:22:52.014808 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:52.014748    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b968752-a57e-4951-bb34-093d3cac44fc" containerName="revoke-keys"
Jul 16 23:22:52.014808 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:52.014759    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b08d2986-c8c9-4371-8303-891f507c1f07" containerName="revoke-keys"
Jul 16 23:22:52.014808 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:52.014764    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b08d2986-c8c9-4371-8303-891f507c1f07" containerName="revoke-keys"
Jul 16 23:22:52.014808 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:52.014774    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b968752-a57e-4951-bb34-093d3cac44fc" containerName="revoke-keys"
Jul 16 23:22:52.014808 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:52.014779    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b968752-a57e-4951-bb34-093d3cac44fc" containerName="revoke-keys"
Jul 16 23:22:52.014808 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:52.014785    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b968752-a57e-4951-bb34-093d3cac44fc" containerName="revoke-keys"
Jul 16 23:22:52.014808 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:52.014790    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b968752-a57e-4951-bb34-093d3cac44fc" containerName="revoke-keys"
Jul 16 23:22:52.015333 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:52.014843    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b08d2986-c8c9-4371-8303-891f507c1f07" containerName="revoke-keys"
Jul 16 23:22:52.015333 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:52.014851    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5e00d30-9c72-463c-a84d-7668bcd08e38" containerName="revoke-keys"
Jul 16 23:22:52.015333 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:52.014858    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5e00d30-9c72-463c-a84d-7668bcd08e38" containerName="revoke-keys"
Jul 16 23:22:52.015333 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:52.014863    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b968752-a57e-4951-bb34-093d3cac44fc" containerName="revoke-keys"
Jul 16 23:22:52.015333 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:52.014868    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b968752-a57e-4951-bb34-093d3cac44fc" containerName="revoke-keys"
Jul 16 23:22:52.015333 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:52.014875    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b08d2986-c8c9-4371-8303-891f507c1f07" containerName="revoke-keys"
Jul 16 23:22:52.015333 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:52.014880    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b968752-a57e-4951-bb34-093d3cac44fc" containerName="revoke-keys"
Jul 16 23:22:52.019689 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:52.019666    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-xlnrs"
Jul 16 23:22:52.022301 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:52.022269    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 23:22:52.022477 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:52.022269    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:22:52.024523 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:52.024495    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-xlnrs"]
Jul 16 23:22:52.098330 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:52.098297    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/737600a3-de2b-4c05-8052-bd5a6eb2f482-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-xlnrs\" (UID: \"737600a3-de2b-4c05-8052-bd5a6eb2f482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-xlnrs"
Jul 16 23:22:52.198727 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:52.198702    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/737600a3-de2b-4c05-8052-bd5a6eb2f482-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-xlnrs\" (UID: \"737600a3-de2b-4c05-8052-bd5a6eb2f482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-xlnrs"
Jul 16 23:22:52.199275 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:52.199226    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/737600a3-de2b-4c05-8052-bd5a6eb2f482-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-xlnrs\" (UID: \"737600a3-de2b-4c05-8052-bd5a6eb2f482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-xlnrs"
Jul 16 23:22:52.333814 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:52.333790    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-xlnrs"
Jul 16 23:22:52.462608 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:52.462538    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-xlnrs"]
Jul 16 23:22:52.464752 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:22:52.464722    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod737600a3_de2b_4c05_8052_bd5a6eb2f482.slice/crio-2731d897464259704872fb1923fd2289a9e5ed72e8a088af0f8f6b714bdbaf89 WatchSource:0}: Error finding container 2731d897464259704872fb1923fd2289a9e5ed72e8a088af0f8f6b714bdbaf89: Status 404 returned error can't find the container with id 2731d897464259704872fb1923fd2289a9e5ed72e8a088af0f8f6b714bdbaf89
Jul 16 23:22:52.581602 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:52.581571    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:22:52.605051 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:52.604992    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-xlnrs" event={"ID":"737600a3-de2b-4c05-8052-bd5a6eb2f482","Type":"ContainerStarted","Data":"2731d897464259704872fb1923fd2289a9e5ed72e8a088af0f8f6b714bdbaf89"}
Jul 16 23:22:53.019328 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:53.019255    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:22:53.331634 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:53.331605    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:22:53.564325 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:53.564297    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:22:53.610885 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:53.610809    2573 generic.go:356] "Generic (PLEG): container finished" podID="737600a3-de2b-4c05-8052-bd5a6eb2f482" containerID="7104d3859879b505702db1b140d79a2c21ddb960d754eb2be4f26f4cc7732147" exitCode=6
Jul 16 23:22:53.611008 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:53.610900    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-xlnrs" event={"ID":"737600a3-de2b-4c05-8052-bd5a6eb2f482","Type":"ContainerDied","Data":"7104d3859879b505702db1b140d79a2c21ddb960d754eb2be4f26f4cc7732147"}
Jul 16 23:22:53.611060 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:53.611048    2573 scope.go:117] "RemoveContainer" containerID="7104d3859879b505702db1b140d79a2c21ddb960d754eb2be4f26f4cc7732147"
Jul 16 23:22:53.630383 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:53.630364    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:22:54.617482 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:54.617449    2573 generic.go:356] "Generic (PLEG): container finished" podID="737600a3-de2b-4c05-8052-bd5a6eb2f482" containerID="716d2c08b02f65d612fd20580387d790cabecdc3144429caf067559094d5e7d2" exitCode=6
Jul 16 23:22:54.617928 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:54.617532    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-xlnrs" event={"ID":"737600a3-de2b-4c05-8052-bd5a6eb2f482","Type":"ContainerDied","Data":"716d2c08b02f65d612fd20580387d790cabecdc3144429caf067559094d5e7d2"}
Jul 16 23:22:54.617928 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:54.617584    2573 scope.go:117] "RemoveContainer" containerID="7104d3859879b505702db1b140d79a2c21ddb960d754eb2be4f26f4cc7732147"
Jul 16 23:22:54.617928 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:54.617801    2573 scope.go:117] "RemoveContainer" containerID="716d2c08b02f65d612fd20580387d790cabecdc3144429caf067559094d5e7d2"
Jul 16 23:22:54.618092 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:22:54.617971    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-xlnrs_odh-ai-gateway-infra(737600a3-de2b-4c05-8052-bd5a6eb2f482)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-xlnrs" podUID="737600a3-de2b-4c05-8052-bd5a6eb2f482"
Jul 16 23:22:55.625169 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:55.625140    2573 scope.go:117] "RemoveContainer" containerID="716d2c08b02f65d612fd20580387d790cabecdc3144429caf067559094d5e7d2"
Jul 16 23:22:55.625641 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:22:55.625345    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-xlnrs_odh-ai-gateway-infra(737600a3-de2b-4c05-8052-bd5a6eb2f482)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-xlnrs" podUID="737600a3-de2b-4c05-8052-bd5a6eb2f482"
Jul 16 23:22:56.101671 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:56.101625    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:22:56.200276 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:56.200223    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:22:56.262663 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:56.262631    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:22:56.380108 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:56.380042    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:22:56.798376 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:56.798305    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:22:56.897262 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:22:56.897200    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:23:00.564536 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:00.564482    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:23:00.941912 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:00.941880    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-wfwzt"]
Jul 16 23:23:00.942177 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:00.942164    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b08d2986-c8c9-4371-8303-891f507c1f07" containerName="revoke-keys"
Jul 16 23:23:00.942223 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:00.942178    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b08d2986-c8c9-4371-8303-891f507c1f07" containerName="revoke-keys"
Jul 16 23:23:00.942278 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:00.942260    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b08d2986-c8c9-4371-8303-891f507c1f07" containerName="revoke-keys"
Jul 16 23:23:00.942278 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:00.942271    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5e00d30-9c72-463c-a84d-7668bcd08e38" containerName="revoke-keys"
Jul 16 23:23:00.944549 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:00.944533    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-wfwzt"
Jul 16 23:23:00.951861 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:00.951836    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-wfwzt"]
Jul 16 23:23:01.060750 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:01.060721    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aa5ad08-be80-4fdd-b3e9-eb8a58e8c6c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-wfwzt\" (UID: \"4aa5ad08-be80-4fdd-b3e9-eb8a58e8c6c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-wfwzt"
Jul 16 23:23:01.161373 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:01.161339    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aa5ad08-be80-4fdd-b3e9-eb8a58e8c6c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-wfwzt\" (UID: \"4aa5ad08-be80-4fdd-b3e9-eb8a58e8c6c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-wfwzt"
Jul 16 23:23:01.161878 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:01.161859    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aa5ad08-be80-4fdd-b3e9-eb8a58e8c6c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-wfwzt\" (UID: \"4aa5ad08-be80-4fdd-b3e9-eb8a58e8c6c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-wfwzt"
Jul 16 23:23:01.255043 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:01.254981    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-wfwzt"
Jul 16 23:23:01.378456 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:01.378425    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-wfwzt"]
Jul 16 23:23:01.380147 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:23:01.380116    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aa5ad08_be80_4fdd_b3e9_eb8a58e8c6c4.slice/crio-2fb83d9de9754841e5b0991263552c336a704898e496a3545fb4e6299c480347 WatchSource:0}: Error finding container 2fb83d9de9754841e5b0991263552c336a704898e496a3545fb4e6299c480347: Status 404 returned error can't find the container with id 2fb83d9de9754841e5b0991263552c336a704898e496a3545fb4e6299c480347
Jul 16 23:23:01.653423 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:01.653396    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-wfwzt" event={"ID":"4aa5ad08-be80-4fdd-b3e9-eb8a58e8c6c4","Type":"ContainerStarted","Data":"1ffe70e8c4e7faba0b58afd0f9fdb6f37ce61d1a6675af9bf4515984ae33be78"}
Jul 16 23:23:01.653695 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:01.653450    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-wfwzt" event={"ID":"4aa5ad08-be80-4fdd-b3e9-eb8a58e8c6c4","Type":"ContainerStarted","Data":"2fb83d9de9754841e5b0991263552c336a704898e496a3545fb4e6299c480347"}
Jul 16 23:23:01.670537 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:01.670327    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-wfwzt" podStartSLOduration=1.6703077039999998 podStartE2EDuration="1.670307704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:23:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:23:01.67017527 +0000 UTC m=+2955.680437937" watchObservedRunningTime="2026-07-16 23:23:01.670307704 +0000 UTC m=+2955.680570370"
Jul 16 23:23:02.659527 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:02.659489    2573 generic.go:356] "Generic (PLEG): container finished" podID="4aa5ad08-be80-4fdd-b3e9-eb8a58e8c6c4" containerID="1ffe70e8c4e7faba0b58afd0f9fdb6f37ce61d1a6675af9bf4515984ae33be78" exitCode=6
Jul 16 23:23:02.659890 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:02.659574    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-wfwzt" event={"ID":"4aa5ad08-be80-4fdd-b3e9-eb8a58e8c6c4","Type":"ContainerDied","Data":"1ffe70e8c4e7faba0b58afd0f9fdb6f37ce61d1a6675af9bf4515984ae33be78"}
Jul 16 23:23:02.659890 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:02.659708    2573 scope.go:117] "RemoveContainer" containerID="1ffe70e8c4e7faba0b58afd0f9fdb6f37ce61d1a6675af9bf4515984ae33be78"
Jul 16 23:23:03.665871 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:03.665839    2573 generic.go:356] "Generic (PLEG): container finished" podID="4aa5ad08-be80-4fdd-b3e9-eb8a58e8c6c4" containerID="7e19942af893b3ea5ec98841d50335e5bd92a0ea49696ea56562755e8f2e9d41" exitCode=6
Jul 16 23:23:03.666223 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:03.665919    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-wfwzt" event={"ID":"4aa5ad08-be80-4fdd-b3e9-eb8a58e8c6c4","Type":"ContainerDied","Data":"7e19942af893b3ea5ec98841d50335e5bd92a0ea49696ea56562755e8f2e9d41"}
Jul 16 23:23:03.666223 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:03.665975    2573 scope.go:117] "RemoveContainer" containerID="1ffe70e8c4e7faba0b58afd0f9fdb6f37ce61d1a6675af9bf4515984ae33be78"
Jul 16 23:23:03.666223 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:03.666193    2573 scope.go:117] "RemoveContainer" containerID="7e19942af893b3ea5ec98841d50335e5bd92a0ea49696ea56562755e8f2e9d41"
Jul 16 23:23:03.666416 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:23:03.666394    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-wfwzt_odh-ai-gateway-infra(4aa5ad08-be80-4fdd-b3e9-eb8a58e8c6c4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-wfwzt" podUID="4aa5ad08-be80-4fdd-b3e9-eb8a58e8c6c4"
Jul 16 23:23:04.671571 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:04.671550    2573 scope.go:117] "RemoveContainer" containerID="7e19942af893b3ea5ec98841d50335e5bd92a0ea49696ea56562755e8f2e9d41"
Jul 16 23:23:04.671902 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:23:04.671695    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-wfwzt_odh-ai-gateway-infra(4aa5ad08-be80-4fdd-b3e9-eb8a58e8c6c4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-wfwzt" podUID="4aa5ad08-be80-4fdd-b3e9-eb8a58e8c6c4"
Jul 16 23:23:07.431471 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:07.431434    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:23:07.513285 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:07.513259    2573 scope.go:117] "RemoveContainer" containerID="716d2c08b02f65d612fd20580387d790cabecdc3144429caf067559094d5e7d2"
Jul 16 23:23:08.157094 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:08.157064    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7w8ps"]
Jul 16 23:23:08.159273 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:08.159255    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7w8ps"
Jul 16 23:23:08.167541 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:08.167515    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7w8ps"]
Jul 16 23:23:08.213505 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:08.213477    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a303957-01da-4600-b9e3-b3699478e472-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-7w8ps\" (UID: \"6a303957-01da-4600-b9e3-b3699478e472\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7w8ps"
Jul 16 23:23:08.314502 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:08.314462    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a303957-01da-4600-b9e3-b3699478e472-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-7w8ps\" (UID: \"6a303957-01da-4600-b9e3-b3699478e472\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7w8ps"
Jul 16 23:23:08.315080 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:08.315061    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a303957-01da-4600-b9e3-b3699478e472-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-7w8ps\" (UID: \"6a303957-01da-4600-b9e3-b3699478e472\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7w8ps"
Jul 16 23:23:08.466356 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:08.466292    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d302e4-klvz4"]
Jul 16 23:23:08.468522 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:08.468507    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d302e4-klvz4"
Jul 16 23:23:08.469511 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:08.469492    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7w8ps"
Jul 16 23:23:08.476594 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:08.476560    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d302e4-klvz4"]
Jul 16 23:23:08.516384 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:08.516359    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bba0259b-9239-4f63-b8e6-4cce88092ac0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d302e4-klvz4\" (UID: \"bba0259b-9239-4f63-b8e6-4cce88092ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d302e4-klvz4"
Jul 16 23:23:08.596299 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:08.596272    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7w8ps"]
Jul 16 23:23:08.598173 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:23:08.598136    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a303957_01da_4600_b9e3_b3699478e472.slice/crio-eba147be2a7765bba59b5ddce6cd1f9b0291156d2bca6de505091593b9bce56e WatchSource:0}: Error finding container eba147be2a7765bba59b5ddce6cd1f9b0291156d2bca6de505091593b9bce56e: Status 404 returned error can't find the container with id eba147be2a7765bba59b5ddce6cd1f9b0291156d2bca6de505091593b9bce56e
Jul 16 23:23:08.617750 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:08.617727    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bba0259b-9239-4f63-b8e6-4cce88092ac0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d302e4-klvz4\" (UID: \"bba0259b-9239-4f63-b8e6-4cce88092ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d302e4-klvz4"
Jul 16 23:23:08.618640 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:08.618614    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bba0259b-9239-4f63-b8e6-4cce88092ac0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d302e4-klvz4\" (UID: \"bba0259b-9239-4f63-b8e6-4cce88092ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d302e4-klvz4"
Jul 16 23:23:08.691897 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:08.691876    2573 generic.go:356] "Generic (PLEG): container finished" podID="737600a3-de2b-4c05-8052-bd5a6eb2f482" containerID="0580e0df0d8c9666dc8706ce1588efb2ea4847f3cfb909d536863ca4927ab61f" exitCode=6
Jul 16 23:23:08.691992 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:08.691949    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-xlnrs" event={"ID":"737600a3-de2b-4c05-8052-bd5a6eb2f482","Type":"ContainerDied","Data":"0580e0df0d8c9666dc8706ce1588efb2ea4847f3cfb909d536863ca4927ab61f"}
Jul 16 23:23:08.692041 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:08.692002    2573 scope.go:117] "RemoveContainer" containerID="716d2c08b02f65d612fd20580387d790cabecdc3144429caf067559094d5e7d2"
Jul 16 23:23:08.692307 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:08.692287    2573 scope.go:117] "RemoveContainer" containerID="0580e0df0d8c9666dc8706ce1588efb2ea4847f3cfb909d536863ca4927ab61f"
Jul 16 23:23:08.692492 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:23:08.692473    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-xlnrs_odh-ai-gateway-infra(737600a3-de2b-4c05-8052-bd5a6eb2f482)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-xlnrs" podUID="737600a3-de2b-4c05-8052-bd5a6eb2f482"
Jul 16 23:23:08.693229 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:08.693211    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7w8ps" event={"ID":"6a303957-01da-4600-b9e3-b3699478e472","Type":"ContainerStarted","Data":"eba147be2a7765bba59b5ddce6cd1f9b0291156d2bca6de505091593b9bce56e"}
Jul 16 23:23:08.779638 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:08.779596    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d302e4-klvz4"
Jul 16 23:23:08.900991 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:08.900968    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d302e4-klvz4"]
Jul 16 23:23:08.902336 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:23:08.902308    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbba0259b_9239_4f63_b8e6_4cce88092ac0.slice/crio-9296b502fd7079ebfa2f43d6e6d64bdb1264baaf19410a82f1315e776b8ab872 WatchSource:0}: Error finding container 9296b502fd7079ebfa2f43d6e6d64bdb1264baaf19410a82f1315e776b8ab872: Status 404 returned error can't find the container with id 9296b502fd7079ebfa2f43d6e6d64bdb1264baaf19410a82f1315e776b8ab872
Jul 16 23:23:09.085950 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:09.085921    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:23:09.185363 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:09.185339    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:23:09.700137 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:09.700103    2573 generic.go:356] "Generic (PLEG): container finished" podID="6a303957-01da-4600-b9e3-b3699478e472" containerID="fb9a07ab61941702a88e41bbba8f4b633eb3694777b331d998987646ef838904" exitCode=6
Jul 16 23:23:09.700535 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:09.700185    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7w8ps" event={"ID":"6a303957-01da-4600-b9e3-b3699478e472","Type":"ContainerDied","Data":"fb9a07ab61941702a88e41bbba8f4b633eb3694777b331d998987646ef838904"}
Jul 16 23:23:09.700535 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:09.700370    2573 scope.go:117] "RemoveContainer" containerID="fb9a07ab61941702a88e41bbba8f4b633eb3694777b331d998987646ef838904"
Jul 16 23:23:09.701769 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:09.701745    2573 generic.go:356] "Generic (PLEG): container finished" podID="bba0259b-9239-4f63-b8e6-4cce88092ac0" containerID="17d0b0f585aadd26b03d5ca3be0c37bb9867dab483bf23fab6756edea7e85fa3" exitCode=0
Jul 16 23:23:09.701867 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:09.701814    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d302e4-klvz4" event={"ID":"bba0259b-9239-4f63-b8e6-4cce88092ac0","Type":"ContainerDied","Data":"17d0b0f585aadd26b03d5ca3be0c37bb9867dab483bf23fab6756edea7e85fa3"}
Jul 16 23:23:09.701867 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:09.701857    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d302e4-klvz4" event={"ID":"bba0259b-9239-4f63-b8e6-4cce88092ac0","Type":"ContainerStarted","Data":"9296b502fd7079ebfa2f43d6e6d64bdb1264baaf19410a82f1315e776b8ab872"}
Jul 16 23:23:09.722581 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:09.722555    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-xlnrs"]
Jul 16 23:23:09.851661 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:09.851629    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-xlnrs"
Jul 16 23:23:09.926818 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:09.926791    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/737600a3-de2b-4c05-8052-bd5a6eb2f482-maas-api-service-ca\") pod \"737600a3-de2b-4c05-8052-bd5a6eb2f482\" (UID: \"737600a3-de2b-4c05-8052-bd5a6eb2f482\") "
Jul 16 23:23:09.927124 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:09.927102    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/737600a3-de2b-4c05-8052-bd5a6eb2f482-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "737600a3-de2b-4c05-8052-bd5a6eb2f482" (UID: "737600a3-de2b-4c05-8052-bd5a6eb2f482"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:23:10.027369 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:10.027319    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/737600a3-de2b-4c05-8052-bd5a6eb2f482-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:23:10.709909 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:10.709846    2573 generic.go:356] "Generic (PLEG): container finished" podID="6a303957-01da-4600-b9e3-b3699478e472" containerID="4dd3cd69fbcc4e77542873c53e067f97497d35f001192f3dd12942380eec5098" exitCode=6
Jul 16 23:23:10.710287 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:10.709928    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7w8ps" event={"ID":"6a303957-01da-4600-b9e3-b3699478e472","Type":"ContainerDied","Data":"4dd3cd69fbcc4e77542873c53e067f97497d35f001192f3dd12942380eec5098"}
Jul 16 23:23:10.710287 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:10.709991    2573 scope.go:117] "RemoveContainer" containerID="fb9a07ab61941702a88e41bbba8f4b633eb3694777b331d998987646ef838904"
Jul 16 23:23:10.710287 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:10.710163    2573 scope.go:117] "RemoveContainer" containerID="4dd3cd69fbcc4e77542873c53e067f97497d35f001192f3dd12942380eec5098"
Jul 16 23:23:10.710444 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:23:10.710356    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-7w8ps_odh-ai-gateway-infra(6a303957-01da-4600-b9e3-b3699478e472)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7w8ps" podUID="6a303957-01da-4600-b9e3-b3699478e472"
Jul 16 23:23:10.711369 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:10.711350    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-xlnrs" event={"ID":"737600a3-de2b-4c05-8052-bd5a6eb2f482","Type":"ContainerDied","Data":"2731d897464259704872fb1923fd2289a9e5ed72e8a088af0f8f6b714bdbaf89"}
Jul 16 23:23:10.711481 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:10.711355    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-xlnrs"
Jul 16 23:23:10.720879 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:10.720863    2573 scope.go:117] "RemoveContainer" containerID="0580e0df0d8c9666dc8706ce1588efb2ea4847f3cfb909d536863ca4927ab61f"
Jul 16 23:23:10.740555 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:10.740527    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-xlnrs"]
Jul 16 23:23:10.743222 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:10.743196    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-xlnrs"]
Jul 16 23:23:10.843556 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:10.843537    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d302e4-klvz4"
Jul 16 23:23:10.934265 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:10.934213    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bba0259b-9239-4f63-b8e6-4cce88092ac0-maas-api-service-ca\") pod \"bba0259b-9239-4f63-b8e6-4cce88092ac0\" (UID: \"bba0259b-9239-4f63-b8e6-4cce88092ac0\") "
Jul 16 23:23:10.934542 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:10.934522    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bba0259b-9239-4f63-b8e6-4cce88092ac0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bba0259b-9239-4f63-b8e6-4cce88092ac0" (UID: "bba0259b-9239-4f63-b8e6-4cce88092ac0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:23:11.035060 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:11.034994    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bba0259b-9239-4f63-b8e6-4cce88092ac0-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:23:11.717666 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:11.717631    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d302e4-klvz4"
Jul 16 23:23:11.718157 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:11.717661    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d302e4-klvz4" event={"ID":"bba0259b-9239-4f63-b8e6-4cce88092ac0","Type":"ContainerDied","Data":"9296b502fd7079ebfa2f43d6e6d64bdb1264baaf19410a82f1315e776b8ab872"}
Jul 16 23:23:11.718157 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:11.717707    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9296b502fd7079ebfa2f43d6e6d64bdb1264baaf19410a82f1315e776b8ab872"
Jul 16 23:23:11.720508 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:11.720486    2573 scope.go:117] "RemoveContainer" containerID="4dd3cd69fbcc4e77542873c53e067f97497d35f001192f3dd12942380eec5098"
Jul 16 23:23:11.720660 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:23:11.720618    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-7w8ps_odh-ai-gateway-infra(6a303957-01da-4600-b9e3-b3699478e472)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7w8ps" podUID="6a303957-01da-4600-b9e3-b3699478e472"
Jul 16 23:23:12.518445 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:12.518407    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="737600a3-de2b-4c05-8052-bd5a6eb2f482" path="/var/lib/kubelet/pods/737600a3-de2b-4c05-8052-bd5a6eb2f482/volumes"
Jul 16 23:23:16.516021 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:16.515984    2573 scope.go:117] "RemoveContainer" containerID="7e19942af893b3ea5ec98841d50335e5bd92a0ea49696ea56562755e8f2e9d41"
Jul 16 23:23:17.750588 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:17.750561    2573 generic.go:356] "Generic (PLEG): container finished" podID="4aa5ad08-be80-4fdd-b3e9-eb8a58e8c6c4" containerID="f773df588f2ebbeaa70337cd9468a67ab88d4f8778a57e7377560b049e4960e6" exitCode=6
Jul 16 23:23:17.750964 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:17.750634    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-wfwzt" event={"ID":"4aa5ad08-be80-4fdd-b3e9-eb8a58e8c6c4","Type":"ContainerDied","Data":"f773df588f2ebbeaa70337cd9468a67ab88d4f8778a57e7377560b049e4960e6"}
Jul 16 23:23:17.750964 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:17.750697    2573 scope.go:117] "RemoveContainer" containerID="7e19942af893b3ea5ec98841d50335e5bd92a0ea49696ea56562755e8f2e9d41"
Jul 16 23:23:17.750964 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:17.750924    2573 scope.go:117] "RemoveContainer" containerID="f773df588f2ebbeaa70337cd9468a67ab88d4f8778a57e7377560b049e4960e6"
Jul 16 23:23:17.751132 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:23:17.751090    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-wfwzt_odh-ai-gateway-infra(4aa5ad08-be80-4fdd-b3e9-eb8a58e8c6c4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-wfwzt" podUID="4aa5ad08-be80-4fdd-b3e9-eb8a58e8c6c4"
Jul 16 23:23:18.365683 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:18.365648    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:23:18.468651 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:18.468622    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:23:18.625980 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:18.625874    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:23:18.775347 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:18.775314    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-wfwzt"]
Jul 16 23:23:18.905445 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:18.905421    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-wfwzt"
Jul 16 23:23:18.992784 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:18.992762    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aa5ad08-be80-4fdd-b3e9-eb8a58e8c6c4-maas-api-service-ca\") pod \"4aa5ad08-be80-4fdd-b3e9-eb8a58e8c6c4\" (UID: \"4aa5ad08-be80-4fdd-b3e9-eb8a58e8c6c4\") "
Jul 16 23:23:18.993063 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:18.993043    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aa5ad08-be80-4fdd-b3e9-eb8a58e8c6c4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4aa5ad08-be80-4fdd-b3e9-eb8a58e8c6c4" (UID: "4aa5ad08-be80-4fdd-b3e9-eb8a58e8c6c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:23:19.094102 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:19.094078    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aa5ad08-be80-4fdd-b3e9-eb8a58e8c6c4-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:23:19.762518 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:19.762485    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-wfwzt"
Jul 16 23:23:19.762518 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:19.762494    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-wfwzt" event={"ID":"4aa5ad08-be80-4fdd-b3e9-eb8a58e8c6c4","Type":"ContainerDied","Data":"2fb83d9de9754841e5b0991263552c336a704898e496a3545fb4e6299c480347"}
Jul 16 23:23:19.762829 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:19.762556    2573 scope.go:117] "RemoveContainer" containerID="f773df588f2ebbeaa70337cd9468a67ab88d4f8778a57e7377560b049e4960e6"
Jul 16 23:23:19.784335 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:19.784310    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-wfwzt"]
Jul 16 23:23:19.790281 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:19.790258    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-wfwzt"]
Jul 16 23:23:20.518985 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:20.518951    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4aa5ad08-be80-4fdd-b3e9-eb8a58e8c6c4" path="/var/lib/kubelet/pods/4aa5ad08-be80-4fdd-b3e9-eb8a58e8c6c4/volumes"
Jul 16 23:23:22.629191 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:22.629153    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:23:23.226474 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:23.226441    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-t6z79"]
Jul 16 23:23:23.226725 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:23.226712    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="737600a3-de2b-4c05-8052-bd5a6eb2f482" containerName="revoke-keys"
Jul 16 23:23:23.226779 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:23.226727    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="737600a3-de2b-4c05-8052-bd5a6eb2f482" containerName="revoke-keys"
Jul 16 23:23:23.226779 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:23.226738    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aa5ad08-be80-4fdd-b3e9-eb8a58e8c6c4" containerName="revoke-keys"
Jul 16 23:23:23.226779 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:23.226743    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aa5ad08-be80-4fdd-b3e9-eb8a58e8c6c4" containerName="revoke-keys"
Jul 16 23:23:23.226779 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:23.226751    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="737600a3-de2b-4c05-8052-bd5a6eb2f482" containerName="revoke-keys"
Jul 16 23:23:23.226779 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:23.226756    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="737600a3-de2b-4c05-8052-bd5a6eb2f482" containerName="revoke-keys"
Jul 16 23:23:23.226779 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:23.226766    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bba0259b-9239-4f63-b8e6-4cce88092ac0" containerName="revoke-keys"
Jul 16 23:23:23.226779 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:23.226771    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bba0259b-9239-4f63-b8e6-4cce88092ac0" containerName="revoke-keys"
Jul 16 23:23:23.226779 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:23.226777    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aa5ad08-be80-4fdd-b3e9-eb8a58e8c6c4" containerName="revoke-keys"
Jul 16 23:23:23.226779 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:23.226782    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aa5ad08-be80-4fdd-b3e9-eb8a58e8c6c4" containerName="revoke-keys"
Jul 16 23:23:23.227064 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:23.226788    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aa5ad08-be80-4fdd-b3e9-eb8a58e8c6c4" containerName="revoke-keys"
Jul 16 23:23:23.227064 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:23.226793    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aa5ad08-be80-4fdd-b3e9-eb8a58e8c6c4" containerName="revoke-keys"
Jul 16 23:23:23.227064 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:23.226841    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4aa5ad08-be80-4fdd-b3e9-eb8a58e8c6c4" containerName="revoke-keys"
Jul 16 23:23:23.227064 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:23.226848    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="737600a3-de2b-4c05-8052-bd5a6eb2f482" containerName="revoke-keys"
Jul 16 23:23:23.227064 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:23.226855    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="737600a3-de2b-4c05-8052-bd5a6eb2f482" containerName="revoke-keys"
Jul 16 23:23:23.227064 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:23.226860    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4aa5ad08-be80-4fdd-b3e9-eb8a58e8c6c4" containerName="revoke-keys"
Jul 16 23:23:23.227064 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:23.226865    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="737600a3-de2b-4c05-8052-bd5a6eb2f482" containerName="revoke-keys"
Jul 16 23:23:23.227064 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:23.226870    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4aa5ad08-be80-4fdd-b3e9-eb8a58e8c6c4" containerName="revoke-keys"
Jul 16 23:23:23.227064 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:23.226877    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bba0259b-9239-4f63-b8e6-4cce88092ac0" containerName="revoke-keys"
Jul 16 23:23:23.230776 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:23.230758    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-t6z79"
Jul 16 23:23:23.237754 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:23.237680    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-t6z79"]
Jul 16 23:23:23.327837 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:23.327804    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa931d93-a2d2-4841-8bbd-f96b1022c09e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-t6z79\" (UID: \"aa931d93-a2d2-4841-8bbd-f96b1022c09e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-t6z79"
Jul 16 23:23:23.428405 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:23.428373    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa931d93-a2d2-4841-8bbd-f96b1022c09e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-t6z79\" (UID: \"aa931d93-a2d2-4841-8bbd-f96b1022c09e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-t6z79"
Jul 16 23:23:23.428932 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:23.428913    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa931d93-a2d2-4841-8bbd-f96b1022c09e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-t6z79\" (UID: \"aa931d93-a2d2-4841-8bbd-f96b1022c09e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-t6z79"
Jul 16 23:23:23.542544 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:23.542482    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-t6z79"
Jul 16 23:23:23.544735 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:23.544716    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:23:23.666696 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:23.666669    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-t6z79"]
Jul 16 23:23:23.667977 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:23:23.667951    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa931d93_a2d2_4841_8bbd_f96b1022c09e.slice/crio-fc9dcba8b7b22e4ccebed60f74cf835eb0b2d6f99b96425122f2bc981a58e4e1 WatchSource:0}: Error finding container fc9dcba8b7b22e4ccebed60f74cf835eb0b2d6f99b96425122f2bc981a58e4e1: Status 404 returned error can't find the container with id fc9dcba8b7b22e4ccebed60f74cf835eb0b2d6f99b96425122f2bc981a58e4e1
Jul 16 23:23:23.783836 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:23.783812    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-t6z79" event={"ID":"aa931d93-a2d2-4841-8bbd-f96b1022c09e","Type":"ContainerStarted","Data":"fc9dcba8b7b22e4ccebed60f74cf835eb0b2d6f99b96425122f2bc981a58e4e1"}
Jul 16 23:23:24.790805 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:24.790772    2573 generic.go:356] "Generic (PLEG): container finished" podID="aa931d93-a2d2-4841-8bbd-f96b1022c09e" containerID="2b57159dc760b11e37b1cc645ab4e56acb4f3b0a76404bb6a0f8a0f16fe085b4" exitCode=6
Jul 16 23:23:24.791186 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:24.790819    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-t6z79" event={"ID":"aa931d93-a2d2-4841-8bbd-f96b1022c09e","Type":"ContainerDied","Data":"2b57159dc760b11e37b1cc645ab4e56acb4f3b0a76404bb6a0f8a0f16fe085b4"}
Jul 16 23:23:24.791186 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:24.790996    2573 scope.go:117] "RemoveContainer" containerID="2b57159dc760b11e37b1cc645ab4e56acb4f3b0a76404bb6a0f8a0f16fe085b4"
Jul 16 23:23:25.513191 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:25.513172    2573 scope.go:117] "RemoveContainer" containerID="4dd3cd69fbcc4e77542873c53e067f97497d35f001192f3dd12942380eec5098"
Jul 16 23:23:25.797907 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:25.797879    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7w8ps" event={"ID":"6a303957-01da-4600-b9e3-b3699478e472","Type":"ContainerStarted","Data":"064989f514fd03283b8584f48e9880bd6b0ef01b877fb3151fea1ab90fa752e8"}
Jul 16 23:23:25.799633 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:25.799607    2573 generic.go:356] "Generic (PLEG): container finished" podID="aa931d93-a2d2-4841-8bbd-f96b1022c09e" containerID="a3a81db3286964ab81b32cf1a0108b2726cf7fc538484c2eb7f1e165aef60dc0" exitCode=6
Jul 16 23:23:25.799721 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:25.799668    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-t6z79" event={"ID":"aa931d93-a2d2-4841-8bbd-f96b1022c09e","Type":"ContainerDied","Data":"a3a81db3286964ab81b32cf1a0108b2726cf7fc538484c2eb7f1e165aef60dc0"}
Jul 16 23:23:25.799721 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:25.799710    2573 scope.go:117] "RemoveContainer" containerID="2b57159dc760b11e37b1cc645ab4e56acb4f3b0a76404bb6a0f8a0f16fe085b4"
Jul 16 23:23:25.799898 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:25.799883    2573 scope.go:117] "RemoveContainer" containerID="a3a81db3286964ab81b32cf1a0108b2726cf7fc538484c2eb7f1e165aef60dc0"
Jul 16 23:23:25.800033 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:23:25.800018    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-t6z79_odh-ai-gateway-infra(aa931d93-a2d2-4841-8bbd-f96b1022c09e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-t6z79" podUID="aa931d93-a2d2-4841-8bbd-f96b1022c09e"
Jul 16 23:23:25.816214 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:25.816178    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7w8ps" podStartSLOduration=17.816165277 podStartE2EDuration="17.816165277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:23:25.814510012 +0000 UTC m=+2979.824772675" watchObservedRunningTime="2026-07-16 23:23:25.816165277 +0000 UTC m=+2979.826427940"
Jul 16 23:23:26.805944 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:26.805914    2573 scope.go:117] "RemoveContainer" containerID="a3a81db3286964ab81b32cf1a0108b2726cf7fc538484c2eb7f1e165aef60dc0"
Jul 16 23:23:26.806325 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:23:26.806111    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-t6z79_odh-ai-gateway-infra(aa931d93-a2d2-4841-8bbd-f96b1022c09e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-t6z79" podUID="aa931d93-a2d2-4841-8bbd-f96b1022c09e"
Jul 16 23:23:26.807657 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:26.807635    2573 generic.go:356] "Generic (PLEG): container finished" podID="6a303957-01da-4600-b9e3-b3699478e472" containerID="064989f514fd03283b8584f48e9880bd6b0ef01b877fb3151fea1ab90fa752e8" exitCode=6
Jul 16 23:23:26.807766 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:26.807707    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7w8ps" event={"ID":"6a303957-01da-4600-b9e3-b3699478e472","Type":"ContainerDied","Data":"064989f514fd03283b8584f48e9880bd6b0ef01b877fb3151fea1ab90fa752e8"}
Jul 16 23:23:26.807825 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:26.807764    2573 scope.go:117] "RemoveContainer" containerID="4dd3cd69fbcc4e77542873c53e067f97497d35f001192f3dd12942380eec5098"
Jul 16 23:23:26.807994 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:26.807980    2573 scope.go:117] "RemoveContainer" containerID="064989f514fd03283b8584f48e9880bd6b0ef01b877fb3151fea1ab90fa752e8"
Jul 16 23:23:26.808170 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:23:26.808148    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-7w8ps_odh-ai-gateway-infra(6a303957-01da-4600-b9e3-b3699478e472)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7w8ps" podUID="6a303957-01da-4600-b9e3-b3699478e472"
Jul 16 23:23:26.829291 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:26.829268    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7w8ps"]
Jul 16 23:23:27.943462 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:27.943440    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7w8ps"
Jul 16 23:23:28.064005 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:28.063979    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a303957-01da-4600-b9e3-b3699478e472-maas-api-service-ca\") pod \"6a303957-01da-4600-b9e3-b3699478e472\" (UID: \"6a303957-01da-4600-b9e3-b3699478e472\") "
Jul 16 23:23:28.064344 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:28.064297    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a303957-01da-4600-b9e3-b3699478e472-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6a303957-01da-4600-b9e3-b3699478e472" (UID: "6a303957-01da-4600-b9e3-b3699478e472"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:23:28.164655 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:28.164632    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a303957-01da-4600-b9e3-b3699478e472-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:23:28.819707 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:28.819644    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7w8ps"
Jul 16 23:23:28.819707 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:28.819654    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7w8ps" event={"ID":"6a303957-01da-4600-b9e3-b3699478e472","Type":"ContainerDied","Data":"eba147be2a7765bba59b5ddce6cd1f9b0291156d2bca6de505091593b9bce56e"}
Jul 16 23:23:28.819910 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:28.819712    2573 scope.go:117] "RemoveContainer" containerID="064989f514fd03283b8584f48e9880bd6b0ef01b877fb3151fea1ab90fa752e8"
Jul 16 23:23:28.836361 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:28.836335    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7w8ps"]
Jul 16 23:23:28.838280 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:28.838258    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7w8ps"]
Jul 16 23:23:29.305871 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:29.305836    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:23:29.405323 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:29.405283    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:23:30.518974 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:30.518934    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a303957-01da-4600-b9e3-b3699478e472" path="/var/lib/kubelet/pods/6a303957-01da-4600-b9e3-b3699478e472/volumes"
Jul 16 23:23:34.211174 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:34.211124    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:23:34.431448 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:34.431414    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:23:34.547162 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:34.547098    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-339cb8-r89xd"]
Jul 16 23:23:34.547446 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:34.547431    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="737600a3-de2b-4c05-8052-bd5a6eb2f482" containerName="revoke-keys"
Jul 16 23:23:34.547491 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:34.547448    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="737600a3-de2b-4c05-8052-bd5a6eb2f482" containerName="revoke-keys"
Jul 16 23:23:34.547491 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:34.547463    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a303957-01da-4600-b9e3-b3699478e472" containerName="revoke-keys"
Jul 16 23:23:34.547491 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:34.547468    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a303957-01da-4600-b9e3-b3699478e472" containerName="revoke-keys"
Jul 16 23:23:34.547491 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:34.547484    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a303957-01da-4600-b9e3-b3699478e472" containerName="revoke-keys"
Jul 16 23:23:34.547491 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:34.547490    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a303957-01da-4600-b9e3-b3699478e472" containerName="revoke-keys"
Jul 16 23:23:34.547638 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:34.547536    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a303957-01da-4600-b9e3-b3699478e472" containerName="revoke-keys"
Jul 16 23:23:34.547638 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:34.547543    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a303957-01da-4600-b9e3-b3699478e472" containerName="revoke-keys"
Jul 16 23:23:34.547638 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:34.547550    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a303957-01da-4600-b9e3-b3699478e472" containerName="revoke-keys"
Jul 16 23:23:34.551946 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:34.551930    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-339cb8-r89xd"
Jul 16 23:23:34.556271 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:34.556227    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-339cb8-r89xd"]
Jul 16 23:23:34.710200 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:34.710172    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a1638ac-5bee-4919-965b-162243aa590e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-339cb8-r89xd\" (UID: \"4a1638ac-5bee-4919-965b-162243aa590e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-339cb8-r89xd"
Jul 16 23:23:34.811373 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:34.811313    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a1638ac-5bee-4919-965b-162243aa590e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-339cb8-r89xd\" (UID: \"4a1638ac-5bee-4919-965b-162243aa590e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-339cb8-r89xd"
Jul 16 23:23:34.811860 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:34.811842    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a1638ac-5bee-4919-965b-162243aa590e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-339cb8-r89xd\" (UID: \"4a1638ac-5bee-4919-965b-162243aa590e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-339cb8-r89xd"
Jul 16 23:23:34.862945 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:34.862928    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-339cb8-r89xd"
Jul 16 23:23:35.001248 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:35.001210    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-339cb8-r89xd"]
Jul 16 23:23:35.002924 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:23:35.002897    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a1638ac_5bee_4919_965b_162243aa590e.slice/crio-e29b03e39ac73fb1166fbec08356a3cd2fc71f3ff3cce4dd86d59ddd321da09e WatchSource:0}: Error finding container e29b03e39ac73fb1166fbec08356a3cd2fc71f3ff3cce4dd86d59ddd321da09e: Status 404 returned error can't find the container with id e29b03e39ac73fb1166fbec08356a3cd2fc71f3ff3cce4dd86d59ddd321da09e
Jul 16 23:23:35.858709 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:35.858634    2573 generic.go:356] "Generic (PLEG): container finished" podID="4a1638ac-5bee-4919-965b-162243aa590e" containerID="5833b8940ea94d81e4ffec9a84e433769d3780a0db3dea1e9c367e898972fac7" exitCode=0
Jul 16 23:23:35.859046 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:35.858707    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-339cb8-r89xd" event={"ID":"4a1638ac-5bee-4919-965b-162243aa590e","Type":"ContainerDied","Data":"5833b8940ea94d81e4ffec9a84e433769d3780a0db3dea1e9c367e898972fac7"}
Jul 16 23:23:35.859046 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:35.858749    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-339cb8-r89xd" event={"ID":"4a1638ac-5bee-4919-965b-162243aa590e","Type":"ContainerStarted","Data":"e29b03e39ac73fb1166fbec08356a3cd2fc71f3ff3cce4dd86d59ddd321da09e"}
Jul 16 23:23:36.981930 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:36.981908    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-339cb8-r89xd"
Jul 16 23:23:37.132723 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:37.132657    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a1638ac-5bee-4919-965b-162243aa590e-maas-api-service-ca\") pod \"4a1638ac-5bee-4919-965b-162243aa590e\" (UID: \"4a1638ac-5bee-4919-965b-162243aa590e\") "
Jul 16 23:23:37.132962 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:37.132942    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a1638ac-5bee-4919-965b-162243aa590e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4a1638ac-5bee-4919-965b-162243aa590e" (UID: "4a1638ac-5bee-4919-965b-162243aa590e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:23:37.233060 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:37.233037    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a1638ac-5bee-4919-965b-162243aa590e-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:23:37.870080 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:37.870045    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-339cb8-r89xd"
Jul 16 23:23:37.870292 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:37.870092    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-339cb8-r89xd" event={"ID":"4a1638ac-5bee-4919-965b-162243aa590e","Type":"ContainerDied","Data":"e29b03e39ac73fb1166fbec08356a3cd2fc71f3ff3cce4dd86d59ddd321da09e"}
Jul 16 23:23:37.870292 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:37.870158    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e29b03e39ac73fb1166fbec08356a3cd2fc71f3ff3cce4dd86d59ddd321da09e"
Jul 16 23:23:39.514054 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:39.514024    2573 scope.go:117] "RemoveContainer" containerID="a3a81db3286964ab81b32cf1a0108b2726cf7fc538484c2eb7f1e165aef60dc0"
Jul 16 23:23:39.881601 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:39.881575    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-t6z79" event={"ID":"aa931d93-a2d2-4841-8bbd-f96b1022c09e","Type":"ContainerStarted","Data":"13068422ac72b07f48bc225a9aa38ac892acb4b25004dc52c9df69f9e1e43a6e"}
Jul 16 23:23:39.896498 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:39.896455    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-t6z79" podStartSLOduration=16.896440797 podStartE2EDuration="16.896440797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:23:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:23:39.895437984 +0000 UTC m=+2993.905700649" watchObservedRunningTime="2026-07-16 23:23:39.896440797 +0000 UTC m=+2993.906703459"
Jul 16 23:23:40.888100 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:40.888071    2573 generic.go:356] "Generic (PLEG): container finished" podID="aa931d93-a2d2-4841-8bbd-f96b1022c09e" containerID="13068422ac72b07f48bc225a9aa38ac892acb4b25004dc52c9df69f9e1e43a6e" exitCode=6
Jul 16 23:23:40.888491 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:40.888108    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-t6z79" event={"ID":"aa931d93-a2d2-4841-8bbd-f96b1022c09e","Type":"ContainerDied","Data":"13068422ac72b07f48bc225a9aa38ac892acb4b25004dc52c9df69f9e1e43a6e"}
Jul 16 23:23:40.888491 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:40.888159    2573 scope.go:117] "RemoveContainer" containerID="a3a81db3286964ab81b32cf1a0108b2726cf7fc538484c2eb7f1e165aef60dc0"
Jul 16 23:23:40.888491 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:40.888449    2573 scope.go:117] "RemoveContainer" containerID="13068422ac72b07f48bc225a9aa38ac892acb4b25004dc52c9df69f9e1e43a6e"
Jul 16 23:23:40.888640 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:23:40.888627    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-t6z79_odh-ai-gateway-infra(aa931d93-a2d2-4841-8bbd-f96b1022c09e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-t6z79" podUID="aa931d93-a2d2-4841-8bbd-f96b1022c09e"
Jul 16 23:23:40.911624 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:40.911593    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-t6z79"]
Jul 16 23:23:42.018865 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:42.018840    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-t6z79"
Jul 16 23:23:42.053610 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:42.053584    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-b9zp6"]
Jul 16 23:23:42.053858 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:42.053847    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa931d93-a2d2-4841-8bbd-f96b1022c09e" containerName="revoke-keys"
Jul 16 23:23:42.053903 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:42.053860    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa931d93-a2d2-4841-8bbd-f96b1022c09e" containerName="revoke-keys"
Jul 16 23:23:42.053903 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:42.053874    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a1638ac-5bee-4919-965b-162243aa590e" containerName="revoke-keys"
Jul 16 23:23:42.053903 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:42.053880    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a1638ac-5bee-4919-965b-162243aa590e" containerName="revoke-keys"
Jul 16 23:23:42.053903 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:42.053885    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a303957-01da-4600-b9e3-b3699478e472" containerName="revoke-keys"
Jul 16 23:23:42.053903 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:42.053890    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a303957-01da-4600-b9e3-b3699478e472" containerName="revoke-keys"
Jul 16 23:23:42.053903 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:42.053903    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa931d93-a2d2-4841-8bbd-f96b1022c09e" containerName="revoke-keys"
Jul 16 23:23:42.054080 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:42.053909    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa931d93-a2d2-4841-8bbd-f96b1022c09e" containerName="revoke-keys"
Jul 16 23:23:42.054080 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:42.053956    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa931d93-a2d2-4841-8bbd-f96b1022c09e" containerName="revoke-keys"
Jul 16 23:23:42.054080 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:42.053963    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a1638ac-5bee-4919-965b-162243aa590e" containerName="revoke-keys"
Jul 16 23:23:42.054080 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:42.053971    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa931d93-a2d2-4841-8bbd-f96b1022c09e" containerName="revoke-keys"
Jul 16 23:23:42.054080 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:42.053977    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa931d93-a2d2-4841-8bbd-f96b1022c09e" containerName="revoke-keys"
Jul 16 23:23:42.056585 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:42.056571    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-b9zp6"
Jul 16 23:23:42.063608 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:42.063584    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-b9zp6"]
Jul 16 23:23:42.165210 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:42.165145    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa931d93-a2d2-4841-8bbd-f96b1022c09e-maas-api-service-ca\") pod \"aa931d93-a2d2-4841-8bbd-f96b1022c09e\" (UID: \"aa931d93-a2d2-4841-8bbd-f96b1022c09e\") "
Jul 16 23:23:42.165393 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:42.165375    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1835b459-315f-47e6-968f-31efe5bdfa5f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-b9zp6\" (UID: \"1835b459-315f-47e6-968f-31efe5bdfa5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-b9zp6"
Jul 16 23:23:42.165480 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:42.165461    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa931d93-a2d2-4841-8bbd-f96b1022c09e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "aa931d93-a2d2-4841-8bbd-f96b1022c09e" (UID: "aa931d93-a2d2-4841-8bbd-f96b1022c09e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:23:42.266061 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:42.266040    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1835b459-315f-47e6-968f-31efe5bdfa5f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-b9zp6\" (UID: \"1835b459-315f-47e6-968f-31efe5bdfa5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-b9zp6"
Jul 16 23:23:42.266134 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:42.266083    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa931d93-a2d2-4841-8bbd-f96b1022c09e-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:23:42.266585 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:42.266569    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1835b459-315f-47e6-968f-31efe5bdfa5f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-b9zp6\" (UID: \"1835b459-315f-47e6-968f-31efe5bdfa5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-b9zp6"
Jul 16 23:23:42.366892 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:42.366869    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-b9zp6"
Jul 16 23:23:42.494784 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:42.491377    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-b9zp6"]
Jul 16 23:23:42.497733 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:23:42.497707    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1835b459_315f_47e6_968f_31efe5bdfa5f.slice/crio-a52fd245a930d1df55b74727282a2e0d22ee900c0df280db0e31469b5484b86e WatchSource:0}: Error finding container a52fd245a930d1df55b74727282a2e0d22ee900c0df280db0e31469b5484b86e: Status 404 returned error can't find the container with id a52fd245a930d1df55b74727282a2e0d22ee900c0df280db0e31469b5484b86e
Jul 16 23:23:42.903289 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:42.903265    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-t6z79"
Jul 16 23:23:42.903289 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:42.903271    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-t6z79" event={"ID":"aa931d93-a2d2-4841-8bbd-f96b1022c09e","Type":"ContainerDied","Data":"fc9dcba8b7b22e4ccebed60f74cf835eb0b2d6f99b96425122f2bc981a58e4e1"}
Jul 16 23:23:42.903524 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:42.903333    2573 scope.go:117] "RemoveContainer" containerID="13068422ac72b07f48bc225a9aa38ac892acb4b25004dc52c9df69f9e1e43a6e"
Jul 16 23:23:42.905087 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:42.904999    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-b9zp6" event={"ID":"1835b459-315f-47e6-968f-31efe5bdfa5f","Type":"ContainerStarted","Data":"f86d0d0410393392b1d9af1334130f7a18a5d1cb1a2d8da5417b259b6b4a52dd"}
Jul 16 23:23:42.905087 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:42.905049    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-b9zp6" event={"ID":"1835b459-315f-47e6-968f-31efe5bdfa5f","Type":"ContainerStarted","Data":"a52fd245a930d1df55b74727282a2e0d22ee900c0df280db0e31469b5484b86e"}
Jul 16 23:23:42.945189 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:42.945150    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-b9zp6" podStartSLOduration=0.945137445 podStartE2EDuration="945.137445ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:23:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:23:42.943531472 +0000 UTC m=+2996.953794135" watchObservedRunningTime="2026-07-16 23:23:42.945137445 +0000 UTC m=+2996.955400107"
Jul 16 23:23:42.960325 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:42.960282    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-t6z79"]
Jul 16 23:23:42.965909 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:42.965891    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-t6z79"]
Jul 16 23:23:43.911960 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:43.911928    2573 generic.go:356] "Generic (PLEG): container finished" podID="1835b459-315f-47e6-968f-31efe5bdfa5f" containerID="f86d0d0410393392b1d9af1334130f7a18a5d1cb1a2d8da5417b259b6b4a52dd" exitCode=6
Jul 16 23:23:43.912354 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:43.912012    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-b9zp6" event={"ID":"1835b459-315f-47e6-968f-31efe5bdfa5f","Type":"ContainerDied","Data":"f86d0d0410393392b1d9af1334130f7a18a5d1cb1a2d8da5417b259b6b4a52dd"}
Jul 16 23:23:43.912354 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:43.912126    2573 scope.go:117] "RemoveContainer" containerID="f86d0d0410393392b1d9af1334130f7a18a5d1cb1a2d8da5417b259b6b4a52dd"
Jul 16 23:23:44.518959 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:44.518932    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa931d93-a2d2-4841-8bbd-f96b1022c09e" path="/var/lib/kubelet/pods/aa931d93-a2d2-4841-8bbd-f96b1022c09e/volumes"
Jul 16 23:23:44.919218 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:44.919187    2573 generic.go:356] "Generic (PLEG): container finished" podID="1835b459-315f-47e6-968f-31efe5bdfa5f" containerID="9ffa8f166885dc20bdec211eaaa2d55d4097e72357fe709ad696eacfdb1da130" exitCode=6
Jul 16 23:23:44.919625 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:44.919277    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-b9zp6" event={"ID":"1835b459-315f-47e6-968f-31efe5bdfa5f","Type":"ContainerDied","Data":"9ffa8f166885dc20bdec211eaaa2d55d4097e72357fe709ad696eacfdb1da130"}
Jul 16 23:23:44.919625 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:44.919338    2573 scope.go:117] "RemoveContainer" containerID="f86d0d0410393392b1d9af1334130f7a18a5d1cb1a2d8da5417b259b6b4a52dd"
Jul 16 23:23:44.919625 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:44.919535    2573 scope.go:117] "RemoveContainer" containerID="9ffa8f166885dc20bdec211eaaa2d55d4097e72357fe709ad696eacfdb1da130"
Jul 16 23:23:44.919743 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:23:44.919720    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-b9zp6_odh-ai-gateway-infra(1835b459-315f-47e6-968f-31efe5bdfa5f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-b9zp6" podUID="1835b459-315f-47e6-968f-31efe5bdfa5f"
Jul 16 23:23:45.047248 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:45.047220    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:23:45.925729 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:45.925700    2573 scope.go:117] "RemoveContainer" containerID="9ffa8f166885dc20bdec211eaaa2d55d4097e72357fe709ad696eacfdb1da130"
Jul 16 23:23:45.926076 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:23:45.925851    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-b9zp6_odh-ai-gateway-infra(1835b459-315f-47e6-968f-31efe5bdfa5f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-b9zp6" podUID="1835b459-315f-47e6-968f-31efe5bdfa5f"
Jul 16 23:23:46.642675 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:46.642651    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r9jm4_506f904a-3272-4586-8528-146884e728f9/ovn-acl-logging/0.log"
Jul 16 23:23:46.651591 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:46.651572    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r9jm4_506f904a-3272-4586-8528-146884e728f9/ovn-acl-logging/0.log"
Jul 16 23:23:48.262381 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:48.262345    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:23:49.645809 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:49.645777    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:23:55.313435 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:55.313397    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:23:55.365444 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:55.365415    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:23:57.514317 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:57.514213    2573 scope.go:117] "RemoveContainer" containerID="9ffa8f166885dc20bdec211eaaa2d55d4097e72357fe709ad696eacfdb1da130"
Jul 16 23:23:57.982129 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:57.982104    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-b9zp6" event={"ID":"1835b459-315f-47e6-968f-31efe5bdfa5f","Type":"ContainerStarted","Data":"d141c807dc65e65ba7d7c2382c785db82b32d136b2d59f4061a935dcbb2af8ff"}
Jul 16 23:23:58.988525 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:58.988497    2573 generic.go:356] "Generic (PLEG): container finished" podID="1835b459-315f-47e6-968f-31efe5bdfa5f" containerID="d141c807dc65e65ba7d7c2382c785db82b32d136b2d59f4061a935dcbb2af8ff" exitCode=6
Jul 16 23:23:58.988964 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:58.988573    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-b9zp6" event={"ID":"1835b459-315f-47e6-968f-31efe5bdfa5f","Type":"ContainerDied","Data":"d141c807dc65e65ba7d7c2382c785db82b32d136b2d59f4061a935dcbb2af8ff"}
Jul 16 23:23:58.988964 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:58.988625    2573 scope.go:117] "RemoveContainer" containerID="9ffa8f166885dc20bdec211eaaa2d55d4097e72357fe709ad696eacfdb1da130"
Jul 16 23:23:58.988964 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:58.988865    2573 scope.go:117] "RemoveContainer" containerID="d141c807dc65e65ba7d7c2382c785db82b32d136b2d59f4061a935dcbb2af8ff"
Jul 16 23:23:58.989135 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:23:58.989045    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-b9zp6_odh-ai-gateway-infra(1835b459-315f-47e6-968f-31efe5bdfa5f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-b9zp6" podUID="1835b459-315f-47e6-968f-31efe5bdfa5f"
Jul 16 23:23:59.013047 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:23:59.013017    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-b9zp6"]
Jul 16 23:24:00.121994 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:00.121968    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-b9zp6"
Jul 16 23:24:00.292278 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:00.292167    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:24:00.295055 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:00.295032    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1835b459-315f-47e6-968f-31efe5bdfa5f-maas-api-service-ca\") pod \"1835b459-315f-47e6-968f-31efe5bdfa5f\" (UID: \"1835b459-315f-47e6-968f-31efe5bdfa5f\") "
Jul 16 23:24:00.295343 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:00.295324    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1835b459-315f-47e6-968f-31efe5bdfa5f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1835b459-315f-47e6-968f-31efe5bdfa5f" (UID: "1835b459-315f-47e6-968f-31efe5bdfa5f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:24:00.396046 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:00.396022    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1835b459-315f-47e6-968f-31efe5bdfa5f-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:24:00.517810 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:00.517781    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:24:00.853486 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:00.853459    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:24:01.000094 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:01.000069    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-b9zp6"
Jul 16 23:24:01.000094 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:01.000074    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-b9zp6" event={"ID":"1835b459-315f-47e6-968f-31efe5bdfa5f","Type":"ContainerDied","Data":"a52fd245a930d1df55b74727282a2e0d22ee900c0df280db0e31469b5484b86e"}
Jul 16 23:24:01.000340 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:01.000141    2573 scope.go:117] "RemoveContainer" containerID="d141c807dc65e65ba7d7c2382c785db82b32d136b2d59f4061a935dcbb2af8ff"
Jul 16 23:24:01.015012 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:01.014991    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-b9zp6"]
Jul 16 23:24:01.021462 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:01.021443    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-b9zp6"]
Jul 16 23:24:01.159064 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:01.158984    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:24:01.441720 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:01.441644    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:24:02.518754 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:02.518711    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1835b459-315f-47e6-968f-31efe5bdfa5f" path="/var/lib/kubelet/pods/1835b459-315f-47e6-968f-31efe5bdfa5f/volumes"
Jul 16 23:24:04.664116 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:04.664086    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:24:04.747386 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:04.747356    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:24:04.823849 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:04.823817    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:24:04.924083 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:04.924014    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:24:05.303980 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:05.303894    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:24:05.404409 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:05.404381    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:24:08.214360 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:08.214325    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-w4m4c"]
Jul 16 23:24:08.214735 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:08.214608    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1835b459-315f-47e6-968f-31efe5bdfa5f" containerName="revoke-keys"
Jul 16 23:24:08.214735 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:08.214621    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1835b459-315f-47e6-968f-31efe5bdfa5f" containerName="revoke-keys"
Jul 16 23:24:08.214735 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:08.214637    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa931d93-a2d2-4841-8bbd-f96b1022c09e" containerName="revoke-keys"
Jul 16 23:24:08.214735 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:08.214643    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa931d93-a2d2-4841-8bbd-f96b1022c09e" containerName="revoke-keys"
Jul 16 23:24:08.214735 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:08.214649    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1835b459-315f-47e6-968f-31efe5bdfa5f" containerName="revoke-keys"
Jul 16 23:24:08.214735 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:08.214654    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1835b459-315f-47e6-968f-31efe5bdfa5f" containerName="revoke-keys"
Jul 16 23:24:08.214735 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:08.214703    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1835b459-315f-47e6-968f-31efe5bdfa5f" containerName="revoke-keys"
Jul 16 23:24:08.214735 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:08.214710    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1835b459-315f-47e6-968f-31efe5bdfa5f" containerName="revoke-keys"
Jul 16 23:24:08.218898 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:08.218879    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-w4m4c"
Jul 16 23:24:08.221873 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:08.221850    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 23:24:08.222019 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:08.221917    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:24:08.229874 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:08.229839    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-w4m4c"]
Jul 16 23:24:08.350556 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:08.350525    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9800eaa4-d01f-4b46-b36e-fa64695a798c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-w4m4c\" (UID: \"9800eaa4-d01f-4b46-b36e-fa64695a798c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-w4m4c"
Jul 16 23:24:08.450814 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:08.450778    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9800eaa4-d01f-4b46-b36e-fa64695a798c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-w4m4c\" (UID: \"9800eaa4-d01f-4b46-b36e-fa64695a798c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-w4m4c"
Jul 16 23:24:08.451436 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:08.451416    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9800eaa4-d01f-4b46-b36e-fa64695a798c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-w4m4c\" (UID: \"9800eaa4-d01f-4b46-b36e-fa64695a798c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-w4m4c"
Jul 16 23:24:08.529337 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:08.529284    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-w4m4c"
Jul 16 23:24:08.655882 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:08.655858    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-w4m4c"]
Jul 16 23:24:08.657211 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:24:08.657184    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9800eaa4_d01f_4b46_b36e_fa64695a798c.slice/crio-238c41473a323fec9790be83e3110c21eb8e5577f03f6d79f24eabd06b7a00a8 WatchSource:0}: Error finding container 238c41473a323fec9790be83e3110c21eb8e5577f03f6d79f24eabd06b7a00a8: Status 404 returned error can't find the container with id 238c41473a323fec9790be83e3110c21eb8e5577f03f6d79f24eabd06b7a00a8
Jul 16 23:24:09.041396 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:09.041331    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-w4m4c" event={"ID":"9800eaa4-d01f-4b46-b36e-fa64695a798c","Type":"ContainerStarted","Data":"c223d7bb4f396dc9a323958b021e3ca7e379eabfe8081a9f96e4abea34012081"}
Jul 16 23:24:09.041396 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:09.041384    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-w4m4c" event={"ID":"9800eaa4-d01f-4b46-b36e-fa64695a798c","Type":"ContainerStarted","Data":"238c41473a323fec9790be83e3110c21eb8e5577f03f6d79f24eabd06b7a00a8"}
Jul 16 23:24:09.057115 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:09.057076    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-w4m4c" podStartSLOduration=1.057062421 podStartE2EDuration="1.057062421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:24:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:24:09.05569711 +0000 UTC m=+3023.065959788" watchObservedRunningTime="2026-07-16 23:24:09.057062421 +0000 UTC m=+3023.067325084"
Jul 16 23:24:09.060708 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:09.060686    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:24:09.379880 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:09.379839    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:24:09.696161 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:09.696087    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:24:09.853928 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:09.853898    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:24:10.026050 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:10.025981    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:24:10.050980 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:10.050952    2573 generic.go:356] "Generic (PLEG): container finished" podID="9800eaa4-d01f-4b46-b36e-fa64695a798c" containerID="c223d7bb4f396dc9a323958b021e3ca7e379eabfe8081a9f96e4abea34012081" exitCode=6
Jul 16 23:24:10.051107 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:10.051036    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-w4m4c" event={"ID":"9800eaa4-d01f-4b46-b36e-fa64695a798c","Type":"ContainerDied","Data":"c223d7bb4f396dc9a323958b021e3ca7e379eabfe8081a9f96e4abea34012081"}
Jul 16 23:24:10.051192 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:10.051178    2573 scope.go:117] "RemoveContainer" containerID="c223d7bb4f396dc9a323958b021e3ca7e379eabfe8081a9f96e4abea34012081"
Jul 16 23:24:10.629348 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:10.629318    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:24:11.057769 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:11.057685    2573 generic.go:356] "Generic (PLEG): container finished" podID="9800eaa4-d01f-4b46-b36e-fa64695a798c" containerID="e32dd9f651c8fcb967ed4b1408d1394020931e9e81839f5b17e8c98967d5410f" exitCode=6
Jul 16 23:24:11.057944 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:11.057768    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-w4m4c" event={"ID":"9800eaa4-d01f-4b46-b36e-fa64695a798c","Type":"ContainerDied","Data":"e32dd9f651c8fcb967ed4b1408d1394020931e9e81839f5b17e8c98967d5410f"}
Jul 16 23:24:11.057944 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:11.057830    2573 scope.go:117] "RemoveContainer" containerID="c223d7bb4f396dc9a323958b021e3ca7e379eabfe8081a9f96e4abea34012081"
Jul 16 23:24:11.058071 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:11.058030    2573 scope.go:117] "RemoveContainer" containerID="e32dd9f651c8fcb967ed4b1408d1394020931e9e81839f5b17e8c98967d5410f"
Jul 16 23:24:11.058280 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:24:11.058226    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-w4m4c_odh-ai-gateway-infra(9800eaa4-d01f-4b46-b36e-fa64695a798c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-w4m4c" podUID="9800eaa4-d01f-4b46-b36e-fa64695a798c"
Jul 16 23:24:12.064218 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:12.064185    2573 scope.go:117] "RemoveContainer" containerID="e32dd9f651c8fcb967ed4b1408d1394020931e9e81839f5b17e8c98967d5410f"
Jul 16 23:24:12.064733 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:24:12.064424    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-w4m4c_odh-ai-gateway-infra(9800eaa4-d01f-4b46-b36e-fa64695a798c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-w4m4c" podUID="9800eaa4-d01f-4b46-b36e-fa64695a798c"
Jul 16 23:24:12.633642 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:12.633578    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:24:12.710094 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:12.710067    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:24:12.786176 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:12.786151    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:24:12.949363 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:12.949289    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:24:13.266328 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:13.266252    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-kp8mp"]
Jul 16 23:24:13.266665 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:13.266568    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1835b459-315f-47e6-968f-31efe5bdfa5f" containerName="revoke-keys"
Jul 16 23:24:13.266665 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:13.266580    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1835b459-315f-47e6-968f-31efe5bdfa5f" containerName="revoke-keys"
Jul 16 23:24:13.266665 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:13.266640    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1835b459-315f-47e6-968f-31efe5bdfa5f" containerName="revoke-keys"
Jul 16 23:24:13.270668 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:13.270650    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-kp8mp"
Jul 16 23:24:13.277833 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:13.277683    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-kp8mp"]
Jul 16 23:24:13.285925 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:13.285903    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:24:13.388217 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:13.388182    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd2d779a-42ef-4f1c-b9ed-df8f95828fb9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-kp8mp\" (UID: \"bd2d779a-42ef-4f1c-b9ed-df8f95828fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-kp8mp"
Jul 16 23:24:13.388545 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:13.388523    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:24:13.488561 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:13.488531    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd2d779a-42ef-4f1c-b9ed-df8f95828fb9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-kp8mp\" (UID: \"bd2d779a-42ef-4f1c-b9ed-df8f95828fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-kp8mp"
Jul 16 23:24:13.489158 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:13.489136    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd2d779a-42ef-4f1c-b9ed-df8f95828fb9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-kp8mp\" (UID: \"bd2d779a-42ef-4f1c-b9ed-df8f95828fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-kp8mp"
Jul 16 23:24:13.581632 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:13.581608    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-kp8mp"
Jul 16 23:24:13.706462 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:13.706439    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-kp8mp"]
Jul 16 23:24:13.708486 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:24:13.708452    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd2d779a_42ef_4f1c_b9ed_df8f95828fb9.slice/crio-7fc544b69a4097eb36860e845d6e5f26443781770457d62d796180d7a3f74c4a WatchSource:0}: Error finding container 7fc544b69a4097eb36860e845d6e5f26443781770457d62d796180d7a3f74c4a: Status 404 returned error can't find the container with id 7fc544b69a4097eb36860e845d6e5f26443781770457d62d796180d7a3f74c4a
Jul 16 23:24:14.074450 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:14.074418    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-kp8mp" event={"ID":"bd2d779a-42ef-4f1c-b9ed-df8f95828fb9","Type":"ContainerStarted","Data":"90342bdc49b7c7cbbbccec13367d77ce5662bfc5ae712b5c437e457f58cbf6c0"}
Jul 16 23:24:14.074600 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:14.074479    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-kp8mp" event={"ID":"bd2d779a-42ef-4f1c-b9ed-df8f95828fb9","Type":"ContainerStarted","Data":"7fc544b69a4097eb36860e845d6e5f26443781770457d62d796180d7a3f74c4a"}
Jul 16 23:24:14.088670 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:14.088630    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-kp8mp" podStartSLOduration=1.088616121 podStartE2EDuration="1.088616121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:24:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:24:14.088304501 +0000 UTC m=+3028.098567182" watchObservedRunningTime="2026-07-16 23:24:14.088616121 +0000 UTC m=+3028.098878783"
Jul 16 23:24:15.080201 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:15.080172    2573 generic.go:356] "Generic (PLEG): container finished" podID="bd2d779a-42ef-4f1c-b9ed-df8f95828fb9" containerID="90342bdc49b7c7cbbbccec13367d77ce5662bfc5ae712b5c437e457f58cbf6c0" exitCode=6
Jul 16 23:24:15.080601 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:15.080247    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-kp8mp" event={"ID":"bd2d779a-42ef-4f1c-b9ed-df8f95828fb9","Type":"ContainerDied","Data":"90342bdc49b7c7cbbbccec13367d77ce5662bfc5ae712b5c437e457f58cbf6c0"}
Jul 16 23:24:15.080601 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:15.080432    2573 scope.go:117] "RemoveContainer" containerID="90342bdc49b7c7cbbbccec13367d77ce5662bfc5ae712b5c437e457f58cbf6c0"
Jul 16 23:24:16.088217 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:16.088185    2573 generic.go:356] "Generic (PLEG): container finished" podID="bd2d779a-42ef-4f1c-b9ed-df8f95828fb9" containerID="3f6facac10669658cf43c9f1ba73a7d661cd654aa6ff8a94b17db2eef4f0fa63" exitCode=6
Jul 16 23:24:16.088713 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:16.088279    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-kp8mp" event={"ID":"bd2d779a-42ef-4f1c-b9ed-df8f95828fb9","Type":"ContainerDied","Data":"3f6facac10669658cf43c9f1ba73a7d661cd654aa6ff8a94b17db2eef4f0fa63"}
Jul 16 23:24:16.088713 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:16.088348    2573 scope.go:117] "RemoveContainer" containerID="90342bdc49b7c7cbbbccec13367d77ce5662bfc5ae712b5c437e457f58cbf6c0"
Jul 16 23:24:16.088713 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:16.088532    2573 scope.go:117] "RemoveContainer" containerID="3f6facac10669658cf43c9f1ba73a7d661cd654aa6ff8a94b17db2eef4f0fa63"
Jul 16 23:24:16.088884 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:24:16.088749    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-kp8mp_odh-ai-gateway-infra(bd2d779a-42ef-4f1c-b9ed-df8f95828fb9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-kp8mp" podUID="bd2d779a-42ef-4f1c-b9ed-df8f95828fb9"
Jul 16 23:24:17.094514 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:17.094490    2573 scope.go:117] "RemoveContainer" containerID="3f6facac10669658cf43c9f1ba73a7d661cd654aa6ff8a94b17db2eef4f0fa63"
Jul 16 23:24:17.094901 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:24:17.094638    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-kp8mp_odh-ai-gateway-infra(bd2d779a-42ef-4f1c-b9ed-df8f95828fb9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-kp8mp" podUID="bd2d779a-42ef-4f1c-b9ed-df8f95828fb9"
Jul 16 23:24:17.107699 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:17.107651    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:24:18.851386 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:18.851350    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:24:18.870627 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:18.870595    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:24:18.947102 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:18.947076    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:24:24.513615 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:24.513581    2573 scope.go:117] "RemoveContainer" containerID="e32dd9f651c8fcb967ed4b1408d1394020931e9e81839f5b17e8c98967d5410f"
Jul 16 23:24:25.138318 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:25.138287    2573 generic.go:356] "Generic (PLEG): container finished" podID="9800eaa4-d01f-4b46-b36e-fa64695a798c" containerID="6088f4584324f3c432322cb4827369ae5c78fadb9d04ac81af390c472d90085a" exitCode=6
Jul 16 23:24:25.138456 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:25.138402    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-w4m4c" event={"ID":"9800eaa4-d01f-4b46-b36e-fa64695a798c","Type":"ContainerDied","Data":"6088f4584324f3c432322cb4827369ae5c78fadb9d04ac81af390c472d90085a"}
Jul 16 23:24:25.138534 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:25.138500    2573 scope.go:117] "RemoveContainer" containerID="e32dd9f651c8fcb967ed4b1408d1394020931e9e81839f5b17e8c98967d5410f"
Jul 16 23:24:25.138797 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:25.138775    2573 scope.go:117] "RemoveContainer" containerID="6088f4584324f3c432322cb4827369ae5c78fadb9d04ac81af390c472d90085a"
Jul 16 23:24:25.138986 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:24:25.138959    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-w4m4c_odh-ai-gateway-infra(9800eaa4-d01f-4b46-b36e-fa64695a798c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-w4m4c" podUID="9800eaa4-d01f-4b46-b36e-fa64695a798c"
Jul 16 23:24:26.165608 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:26.165520    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-w4m4c"]
Jul 16 23:24:26.294254 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:26.294217    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-w4m4c"
Jul 16 23:24:26.367927 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:26.367897    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9800eaa4-d01f-4b46-b36e-fa64695a798c-maas-api-service-ca\") pod \"9800eaa4-d01f-4b46-b36e-fa64695a798c\" (UID: \"9800eaa4-d01f-4b46-b36e-fa64695a798c\") "
Jul 16 23:24:26.368205 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:26.368185    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9800eaa4-d01f-4b46-b36e-fa64695a798c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9800eaa4-d01f-4b46-b36e-fa64695a798c" (UID: "9800eaa4-d01f-4b46-b36e-fa64695a798c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:24:26.468539 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:26.468470    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9800eaa4-d01f-4b46-b36e-fa64695a798c-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:24:27.034292 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:27.034262    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:24:27.135153 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:27.135125    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:24:27.151849 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:27.151826    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-w4m4c"
Jul 16 23:24:27.151969 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:27.151871    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-w4m4c" event={"ID":"9800eaa4-d01f-4b46-b36e-fa64695a798c","Type":"ContainerDied","Data":"238c41473a323fec9790be83e3110c21eb8e5577f03f6d79f24eabd06b7a00a8"}
Jul 16 23:24:27.151969 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:27.151932    2573 scope.go:117] "RemoveContainer" containerID="6088f4584324f3c432322cb4827369ae5c78fadb9d04ac81af390c472d90085a"
Jul 16 23:24:27.168301 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:27.168280    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-w4m4c"]
Jul 16 23:24:27.170048 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:27.170029    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-w4m4c"]
Jul 16 23:24:28.513463 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:28.513425    2573 scope.go:117] "RemoveContainer" containerID="3f6facac10669658cf43c9f1ba73a7d661cd654aa6ff8a94b17db2eef4f0fa63"
Jul 16 23:24:28.519731 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:28.519703    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9800eaa4-d01f-4b46-b36e-fa64695a798c" path="/var/lib/kubelet/pods/9800eaa4-d01f-4b46-b36e-fa64695a798c/volumes"
Jul 16 23:24:29.163981 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:29.163953    2573 generic.go:356] "Generic (PLEG): container finished" podID="bd2d779a-42ef-4f1c-b9ed-df8f95828fb9" containerID="b9f2d123703eb9c6ea489eadfcc687ec6d96b765b856b4a6d28f5cb797310d18" exitCode=6
Jul 16 23:24:29.164164 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:29.163997    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-kp8mp" event={"ID":"bd2d779a-42ef-4f1c-b9ed-df8f95828fb9","Type":"ContainerDied","Data":"b9f2d123703eb9c6ea489eadfcc687ec6d96b765b856b4a6d28f5cb797310d18"}
Jul 16 23:24:29.164164 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:29.164046    2573 scope.go:117] "RemoveContainer" containerID="3f6facac10669658cf43c9f1ba73a7d661cd654aa6ff8a94b17db2eef4f0fa63"
Jul 16 23:24:29.164375 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:29.164357    2573 scope.go:117] "RemoveContainer" containerID="b9f2d123703eb9c6ea489eadfcc687ec6d96b765b856b4a6d28f5cb797310d18"
Jul 16 23:24:29.164553 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:24:29.164533    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-kp8mp_odh-ai-gateway-infra(bd2d779a-42ef-4f1c-b9ed-df8f95828fb9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-kp8mp" podUID="bd2d779a-42ef-4f1c-b9ed-df8f95828fb9"
Jul 16 23:24:30.131075 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:30.131041    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-48992b/auth-test-model-48992b-kserve-fd54db7f-s4rwh"]
Jul 16 23:24:30.131479 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:30.131370    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9800eaa4-d01f-4b46-b36e-fa64695a798c" containerName="revoke-keys"
Jul 16 23:24:30.131479 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:30.131384    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9800eaa4-d01f-4b46-b36e-fa64695a798c" containerName="revoke-keys"
Jul 16 23:24:30.131479 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:30.131405    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9800eaa4-d01f-4b46-b36e-fa64695a798c" containerName="revoke-keys"
Jul 16 23:24:30.131479 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:30.131410    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9800eaa4-d01f-4b46-b36e-fa64695a798c" containerName="revoke-keys"
Jul 16 23:24:30.131479 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:30.131459    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9800eaa4-d01f-4b46-b36e-fa64695a798c" containerName="revoke-keys"
Jul 16 23:24:30.131479 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:30.131468    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9800eaa4-d01f-4b46-b36e-fa64695a798c" containerName="revoke-keys"
Jul 16 23:24:30.131479 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:30.131474    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9800eaa4-d01f-4b46-b36e-fa64695a798c" containerName="revoke-keys"
Jul 16 23:24:30.135604 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:30.135585    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-48992b/auth-test-model-48992b-kserve-fd54db7f-s4rwh"
Jul 16 23:24:30.138211 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:30.138189    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-48992b\"/\"kube-root-ca.crt\""
Jul 16 23:24:30.139120 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:30.139099    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-48992b\"/\"default-dockercfg-zsbp9\""
Jul 16 23:24:30.139120 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:30.139108    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-48992b\"/\"openshift-service-ca.crt\""
Jul 16 23:24:30.139298 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:30.139141    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-48992b\"/\"auth-test-model-48992b-kserve-self-signed-certs\""
Jul 16 23:24:30.143988 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:30.143962    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-48992b/auth-test-model-48992b-kserve-fd54db7f-s4rwh"]
Jul 16 23:24:30.149634 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:30.149613    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:24:30.189658 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:30.189634    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-kp8mp"]
Jul 16 23:24:30.195318 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:30.195265    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41d9d56c-aca9-4f8c-8028-cf8b08f04ed3-dshm\") pod \"auth-test-model-48992b-kserve-fd54db7f-s4rwh\" (UID: \"41d9d56c-aca9-4f8c-8028-cf8b08f04ed3\") " pod="ai-tenant-e2e-shared-a-48992b/auth-test-model-48992b-kserve-fd54db7f-s4rwh"
Jul 16 23:24:30.195430 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:30.195399    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41d9d56c-aca9-4f8c-8028-cf8b08f04ed3-home\") pod \"auth-test-model-48992b-kserve-fd54db7f-s4rwh\" (UID: \"41d9d56c-aca9-4f8c-8028-cf8b08f04ed3\") " pod="ai-tenant-e2e-shared-a-48992b/auth-test-model-48992b-kserve-fd54db7f-s4rwh"
Jul 16 23:24:30.195495 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:30.195431    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41d9d56c-aca9-4f8c-8028-cf8b08f04ed3-model-cache\") pod \"auth-test-model-48992b-kserve-fd54db7f-s4rwh\" (UID: \"41d9d56c-aca9-4f8c-8028-cf8b08f04ed3\") " pod="ai-tenant-e2e-shared-a-48992b/auth-test-model-48992b-kserve-fd54db7f-s4rwh"
Jul 16 23:24:30.195495 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:30.195470    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xvt6\" (UniqueName: \"kubernetes.io/projected/41d9d56c-aca9-4f8c-8028-cf8b08f04ed3-kube-api-access-7xvt6\") pod \"auth-test-model-48992b-kserve-fd54db7f-s4rwh\" (UID: \"41d9d56c-aca9-4f8c-8028-cf8b08f04ed3\") " pod="ai-tenant-e2e-shared-a-48992b/auth-test-model-48992b-kserve-fd54db7f-s4rwh"
Jul 16 23:24:30.195605 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:30.195540    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41d9d56c-aca9-4f8c-8028-cf8b08f04ed3-tls-certs\") pod \"auth-test-model-48992b-kserve-fd54db7f-s4rwh\" (UID: \"41d9d56c-aca9-4f8c-8028-cf8b08f04ed3\") " pod="ai-tenant-e2e-shared-a-48992b/auth-test-model-48992b-kserve-fd54db7f-s4rwh"
Jul 16 23:24:30.195605 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:30.195569    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41d9d56c-aca9-4f8c-8028-cf8b08f04ed3-tmp-dir\") pod \"auth-test-model-48992b-kserve-fd54db7f-s4rwh\" (UID: \"41d9d56c-aca9-4f8c-8028-cf8b08f04ed3\") " pod="ai-tenant-e2e-shared-a-48992b/auth-test-model-48992b-kserve-fd54db7f-s4rwh"
Jul 16 23:24:30.295929 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:30.295895    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41d9d56c-aca9-4f8c-8028-cf8b08f04ed3-tls-certs\") pod \"auth-test-model-48992b-kserve-fd54db7f-s4rwh\" (UID: \"41d9d56c-aca9-4f8c-8028-cf8b08f04ed3\") " pod="ai-tenant-e2e-shared-a-48992b/auth-test-model-48992b-kserve-fd54db7f-s4rwh"
Jul 16 23:24:30.296067 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:30.295948    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41d9d56c-aca9-4f8c-8028-cf8b08f04ed3-tmp-dir\") pod \"auth-test-model-48992b-kserve-fd54db7f-s4rwh\" (UID: \"41d9d56c-aca9-4f8c-8028-cf8b08f04ed3\") " pod="ai-tenant-e2e-shared-a-48992b/auth-test-model-48992b-kserve-fd54db7f-s4rwh"
Jul 16 23:24:30.296067 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:30.295985    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41d9d56c-aca9-4f8c-8028-cf8b08f04ed3-dshm\") pod \"auth-test-model-48992b-kserve-fd54db7f-s4rwh\" (UID: \"41d9d56c-aca9-4f8c-8028-cf8b08f04ed3\") " pod="ai-tenant-e2e-shared-a-48992b/auth-test-model-48992b-kserve-fd54db7f-s4rwh"
Jul 16 23:24:30.296067 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:30.296053    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41d9d56c-aca9-4f8c-8028-cf8b08f04ed3-home\") pod \"auth-test-model-48992b-kserve-fd54db7f-s4rwh\" (UID: \"41d9d56c-aca9-4f8c-8028-cf8b08f04ed3\") " pod="ai-tenant-e2e-shared-a-48992b/auth-test-model-48992b-kserve-fd54db7f-s4rwh"
Jul 16 23:24:30.296223 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:30.296077    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41d9d56c-aca9-4f8c-8028-cf8b08f04ed3-model-cache\") pod \"auth-test-model-48992b-kserve-fd54db7f-s4rwh\" (UID: \"41d9d56c-aca9-4f8c-8028-cf8b08f04ed3\") " pod="ai-tenant-e2e-shared-a-48992b/auth-test-model-48992b-kserve-fd54db7f-s4rwh"
Jul 16 23:24:30.296223 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:30.296113    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xvt6\" (UniqueName: \"kubernetes.io/projected/41d9d56c-aca9-4f8c-8028-cf8b08f04ed3-kube-api-access-7xvt6\") pod \"auth-test-model-48992b-kserve-fd54db7f-s4rwh\" (UID: \"41d9d56c-aca9-4f8c-8028-cf8b08f04ed3\") " pod="ai-tenant-e2e-shared-a-48992b/auth-test-model-48992b-kserve-fd54db7f-s4rwh"
Jul 16 23:24:30.296470 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:30.296448    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41d9d56c-aca9-4f8c-8028-cf8b08f04ed3-tmp-dir\") pod \"auth-test-model-48992b-kserve-fd54db7f-s4rwh\" (UID: \"41d9d56c-aca9-4f8c-8028-cf8b08f04ed3\") " pod="ai-tenant-e2e-shared-a-48992b/auth-test-model-48992b-kserve-fd54db7f-s4rwh"
Jul 16 23:24:30.296646 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:30.296502    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41d9d56c-aca9-4f8c-8028-cf8b08f04ed3-model-cache\") pod \"auth-test-model-48992b-kserve-fd54db7f-s4rwh\" (UID: \"41d9d56c-aca9-4f8c-8028-cf8b08f04ed3\") " pod="ai-tenant-e2e-shared-a-48992b/auth-test-model-48992b-kserve-fd54db7f-s4rwh"
Jul 16 23:24:30.296779 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:30.296699    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41d9d56c-aca9-4f8c-8028-cf8b08f04ed3-home\") pod \"auth-test-model-48992b-kserve-fd54db7f-s4rwh\" (UID: \"41d9d56c-aca9-4f8c-8028-cf8b08f04ed3\") " pod="ai-tenant-e2e-shared-a-48992b/auth-test-model-48992b-kserve-fd54db7f-s4rwh"
Jul 16 23:24:30.298347 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:30.298325    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41d9d56c-aca9-4f8c-8028-cf8b08f04ed3-dshm\") pod \"auth-test-model-48992b-kserve-fd54db7f-s4rwh\" (UID: \"41d9d56c-aca9-4f8c-8028-cf8b08f04ed3\") " pod="ai-tenant-e2e-shared-a-48992b/auth-test-model-48992b-kserve-fd54db7f-s4rwh"
Jul 16 23:24:30.298463 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:30.298343    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41d9d56c-aca9-4f8c-8028-cf8b08f04ed3-tls-certs\") pod \"auth-test-model-48992b-kserve-fd54db7f-s4rwh\" (UID: \"41d9d56c-aca9-4f8c-8028-cf8b08f04ed3\") " pod="ai-tenant-e2e-shared-a-48992b/auth-test-model-48992b-kserve-fd54db7f-s4rwh"
Jul 16 23:24:30.303981 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:30.303961    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xvt6\" (UniqueName: \"kubernetes.io/projected/41d9d56c-aca9-4f8c-8028-cf8b08f04ed3-kube-api-access-7xvt6\") pod \"auth-test-model-48992b-kserve-fd54db7f-s4rwh\" (UID: \"41d9d56c-aca9-4f8c-8028-cf8b08f04ed3\") " pod="ai-tenant-e2e-shared-a-48992b/auth-test-model-48992b-kserve-fd54db7f-s4rwh"
Jul 16 23:24:30.319698 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:30.319678    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-kp8mp"
Jul 16 23:24:30.396416 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:30.396364    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd2d779a-42ef-4f1c-b9ed-df8f95828fb9-maas-api-service-ca\") pod \"bd2d779a-42ef-4f1c-b9ed-df8f95828fb9\" (UID: \"bd2d779a-42ef-4f1c-b9ed-df8f95828fb9\") "
Jul 16 23:24:30.396657 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:30.396638    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd2d779a-42ef-4f1c-b9ed-df8f95828fb9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bd2d779a-42ef-4f1c-b9ed-df8f95828fb9" (UID: "bd2d779a-42ef-4f1c-b9ed-df8f95828fb9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:24:30.445399 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:30.445378    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-48992b/auth-test-model-48992b-kserve-fd54db7f-s4rwh"
Jul 16 23:24:30.497478 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:30.497448    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd2d779a-42ef-4f1c-b9ed-df8f95828fb9-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:24:30.566859 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:30.566832    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-48992b/auth-test-model-48992b-kserve-fd54db7f-s4rwh"]
Jul 16 23:24:30.568606 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:24:30.568581    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41d9d56c_aca9_4f8c_8028_cf8b08f04ed3.slice/crio-fee8d6fab4b3f2611c81c73a53a014fe642ff271f084f7605888ec058c04683d WatchSource:0}: Error finding container fee8d6fab4b3f2611c81c73a53a014fe642ff271f084f7605888ec058c04683d: Status 404 returned error can't find the container with id fee8d6fab4b3f2611c81c73a53a014fe642ff271f084f7605888ec058c04683d
Jul 16 23:24:31.176620 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:31.176585    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-48992b/auth-test-model-48992b-kserve-fd54db7f-s4rwh" event={"ID":"41d9d56c-aca9-4f8c-8028-cf8b08f04ed3","Type":"ContainerStarted","Data":"6e52aba3d97209bd083da982dadfa66932138c70202c1454fbfdd14d345f5393"}
Jul 16 23:24:31.177036 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:31.176632    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-48992b/auth-test-model-48992b-kserve-fd54db7f-s4rwh" event={"ID":"41d9d56c-aca9-4f8c-8028-cf8b08f04ed3","Type":"ContainerStarted","Data":"fee8d6fab4b3f2611c81c73a53a014fe642ff271f084f7605888ec058c04683d"}
Jul 16 23:24:31.177834 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:31.177815    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-kp8mp"
Jul 16 23:24:31.177834 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:31.177827    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-kp8mp" event={"ID":"bd2d779a-42ef-4f1c-b9ed-df8f95828fb9","Type":"ContainerDied","Data":"7fc544b69a4097eb36860e845d6e5f26443781770457d62d796180d7a3f74c4a"}
Jul 16 23:24:31.177963 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:31.177883    2573 scope.go:117] "RemoveContainer" containerID="b9f2d123703eb9c6ea489eadfcc687ec6d96b765b856b4a6d28f5cb797310d18"
Jul 16 23:24:31.197481 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:31.197436    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-48992b/auth-test-model-48992b-kserve-fd54db7f-s4rwh" podStartSLOduration=1.197421321 podStartE2EDuration="1.197421321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:24:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:24:31.196359773 +0000 UTC m=+3045.206622438" watchObservedRunningTime="2026-07-16 23:24:31.197421321 +0000 UTC m=+3045.207683984"
Jul 16 23:24:31.210964 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:31.210931    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-kp8mp"]
Jul 16 23:24:31.214249 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:31.214209    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-kp8mp"]
Jul 16 23:24:32.091930 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:32.091889    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-hkrsk"]
Jul 16 23:24:32.092205 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:32.092190    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9800eaa4-d01f-4b46-b36e-fa64695a798c" containerName="revoke-keys"
Jul 16 23:24:32.092205 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:32.092205    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9800eaa4-d01f-4b46-b36e-fa64695a798c" containerName="revoke-keys"
Jul 16 23:24:32.092326 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:32.092226    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd2d779a-42ef-4f1c-b9ed-df8f95828fb9" containerName="revoke-keys"
Jul 16 23:24:32.092326 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:32.092231    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd2d779a-42ef-4f1c-b9ed-df8f95828fb9" containerName="revoke-keys"
Jul 16 23:24:32.092326 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:32.092255    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd2d779a-42ef-4f1c-b9ed-df8f95828fb9" containerName="revoke-keys"
Jul 16 23:24:32.092326 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:32.092261    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd2d779a-42ef-4f1c-b9ed-df8f95828fb9" containerName="revoke-keys"
Jul 16 23:24:32.092326 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:32.092314    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd2d779a-42ef-4f1c-b9ed-df8f95828fb9" containerName="revoke-keys"
Jul 16 23:24:32.092326 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:32.092322    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd2d779a-42ef-4f1c-b9ed-df8f95828fb9" containerName="revoke-keys"
Jul 16 23:24:32.096612 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:32.096593    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-hkrsk"
Jul 16 23:24:32.098875 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:32.098847    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 23:24:32.099079 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:32.098855    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:24:32.102412 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:32.102387    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-hkrsk"]
Jul 16 23:24:32.108824 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:32.108798    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ec0333e-2b85-40b6-8e5d-2a234686d823-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-hkrsk\" (UID: \"5ec0333e-2b85-40b6-8e5d-2a234686d823\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-hkrsk"
Jul 16 23:24:32.209828 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:32.209804    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ec0333e-2b85-40b6-8e5d-2a234686d823-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-hkrsk\" (UID: \"5ec0333e-2b85-40b6-8e5d-2a234686d823\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-hkrsk"
Jul 16 23:24:32.210393 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:32.210368    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ec0333e-2b85-40b6-8e5d-2a234686d823-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-hkrsk\" (UID: \"5ec0333e-2b85-40b6-8e5d-2a234686d823\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-hkrsk"
Jul 16 23:24:32.409136 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:32.409077    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-hkrsk"
Jul 16 23:24:32.520155 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:32.520128    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd2d779a-42ef-4f1c-b9ed-df8f95828fb9" path="/var/lib/kubelet/pods/bd2d779a-42ef-4f1c-b9ed-df8f95828fb9/volumes"
Jul 16 23:24:32.533438 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:32.533417    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-hkrsk"]
Jul 16 23:24:32.534657 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:24:32.534630    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ec0333e_2b85_40b6_8e5d_2a234686d823.slice/crio-62679deed414e8264469f414d289c35a80a3e88353fb657f9209fbd7cbe32cf8 WatchSource:0}: Error finding container 62679deed414e8264469f414d289c35a80a3e88353fb657f9209fbd7cbe32cf8: Status 404 returned error can't find the container with id 62679deed414e8264469f414d289c35a80a3e88353fb657f9209fbd7cbe32cf8
Jul 16 23:24:33.191837 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:33.191810    2573 generic.go:356] "Generic (PLEG): container finished" podID="5ec0333e-2b85-40b6-8e5d-2a234686d823" containerID="e47576d4000e07271d630e797ff70dfd602f6b877ae326abc213a27978753d66" exitCode=6
Jul 16 23:24:33.191962 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:33.191871    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-hkrsk" event={"ID":"5ec0333e-2b85-40b6-8e5d-2a234686d823","Type":"ContainerDied","Data":"e47576d4000e07271d630e797ff70dfd602f6b877ae326abc213a27978753d66"}
Jul 16 23:24:33.191962 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:33.191909    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-hkrsk" event={"ID":"5ec0333e-2b85-40b6-8e5d-2a234686d823","Type":"ContainerStarted","Data":"62679deed414e8264469f414d289c35a80a3e88353fb657f9209fbd7cbe32cf8"}
Jul 16 23:24:33.192051 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:33.192014    2573 scope.go:117] "RemoveContainer" containerID="e47576d4000e07271d630e797ff70dfd602f6b877ae326abc213a27978753d66"
Jul 16 23:24:33.625345 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:33.625318    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:24:34.199395 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:34.199368    2573 generic.go:356] "Generic (PLEG): container finished" podID="5ec0333e-2b85-40b6-8e5d-2a234686d823" containerID="c86dc3f294d87ee9b615e0ef00bc495a8202e0a4a913ecf5e95815844ac7ba18" exitCode=6
Jul 16 23:24:34.199554 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:34.199433    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-hkrsk" event={"ID":"5ec0333e-2b85-40b6-8e5d-2a234686d823","Type":"ContainerDied","Data":"c86dc3f294d87ee9b615e0ef00bc495a8202e0a4a913ecf5e95815844ac7ba18"}
Jul 16 23:24:34.199554 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:34.199471    2573 scope.go:117] "RemoveContainer" containerID="e47576d4000e07271d630e797ff70dfd602f6b877ae326abc213a27978753d66"
Jul 16 23:24:34.199666 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:34.199648    2573 scope.go:117] "RemoveContainer" containerID="c86dc3f294d87ee9b615e0ef00bc495a8202e0a4a913ecf5e95815844ac7ba18"
Jul 16 23:24:34.199859 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:24:34.199841    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-hkrsk_odh-ai-gateway-infra(5ec0333e-2b85-40b6-8e5d-2a234686d823)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-hkrsk" podUID="5ec0333e-2b85-40b6-8e5d-2a234686d823"
Jul 16 23:24:34.886565 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:34.886530    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:24:34.985027 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:34.985000    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:24:35.206083 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:35.206018    2573 scope.go:117] "RemoveContainer" containerID="c86dc3f294d87ee9b615e0ef00bc495a8202e0a4a913ecf5e95815844ac7ba18"
Jul 16 23:24:35.206208 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:24:35.206183    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-hkrsk_odh-ai-gateway-infra(5ec0333e-2b85-40b6-8e5d-2a234686d823)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-hkrsk" podUID="5ec0333e-2b85-40b6-8e5d-2a234686d823"
Jul 16 23:24:40.446172 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:40.446090    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-48992b/auth-test-model-48992b-kserve-fd54db7f-s4rwh"
Jul 16 23:24:40.446547 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:40.446379    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-48992b/auth-test-model-48992b-kserve-fd54db7f-s4rwh"
Jul 16 23:24:40.458283 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:40.458262    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-48992b/auth-test-model-48992b-kserve-fd54db7f-s4rwh"
Jul 16 23:24:41.023757 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:41.023715    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-xhr8l"]
Jul 16 23:24:41.024079 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:41.024065    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd2d779a-42ef-4f1c-b9ed-df8f95828fb9" containerName="revoke-keys"
Jul 16 23:24:41.024140 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:41.024081    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd2d779a-42ef-4f1c-b9ed-df8f95828fb9" containerName="revoke-keys"
Jul 16 23:24:41.024177 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:41.024141    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd2d779a-42ef-4f1c-b9ed-df8f95828fb9" containerName="revoke-keys"
Jul 16 23:24:41.028440 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:41.028416    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-xhr8l"
Jul 16 23:24:41.034931 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:41.034897    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-xhr8l"]
Jul 16 23:24:41.067871 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:41.067843    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c97d1def-ea50-45bc-8e20-1f009b604dc6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-xhr8l\" (UID: \"c97d1def-ea50-45bc-8e20-1f009b604dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-xhr8l"
Jul 16 23:24:41.169048 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:41.169024    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c97d1def-ea50-45bc-8e20-1f009b604dc6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-xhr8l\" (UID: \"c97d1def-ea50-45bc-8e20-1f009b604dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-xhr8l"
Jul 16 23:24:41.169598 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:41.169579    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c97d1def-ea50-45bc-8e20-1f009b604dc6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-xhr8l\" (UID: \"c97d1def-ea50-45bc-8e20-1f009b604dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-xhr8l"
Jul 16 23:24:41.247150 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:41.247131    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-48992b/auth-test-model-48992b-kserve-fd54db7f-s4rwh"
Jul 16 23:24:41.340851 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:41.340832    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-xhr8l"
Jul 16 23:24:41.466750 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:41.466731    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-xhr8l"]
Jul 16 23:24:41.468610 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:24:41.468583    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc97d1def_ea50_45bc_8e20_1f009b604dc6.slice/crio-440e61da877ebac26d3371358b908fd0143c1c18eb2194a3b5d20acfee5c8587 WatchSource:0}: Error finding container 440e61da877ebac26d3371358b908fd0143c1c18eb2194a3b5d20acfee5c8587: Status 404 returned error can't find the container with id 440e61da877ebac26d3371358b908fd0143c1c18eb2194a3b5d20acfee5c8587
Jul 16 23:24:42.243534 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:42.243503    2573 generic.go:356] "Generic (PLEG): container finished" podID="c97d1def-ea50-45bc-8e20-1f009b604dc6" containerID="c8344c1ed5588b5f2fe54de7757a3a0c5608b9363859c0c48210e448b1e3ba7e" exitCode=6
Jul 16 23:24:42.243681 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:42.243597    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-xhr8l" event={"ID":"c97d1def-ea50-45bc-8e20-1f009b604dc6","Type":"ContainerDied","Data":"c8344c1ed5588b5f2fe54de7757a3a0c5608b9363859c0c48210e448b1e3ba7e"}
Jul 16 23:24:42.243681 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:42.243650    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-xhr8l" event={"ID":"c97d1def-ea50-45bc-8e20-1f009b604dc6","Type":"ContainerStarted","Data":"440e61da877ebac26d3371358b908fd0143c1c18eb2194a3b5d20acfee5c8587"}
Jul 16 23:24:42.243825 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:42.243806    2573 scope.go:117] "RemoveContainer" containerID="c8344c1ed5588b5f2fe54de7757a3a0c5608b9363859c0c48210e448b1e3ba7e"
Jul 16 23:24:43.251098 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:43.251066    2573 generic.go:356] "Generic (PLEG): container finished" podID="c97d1def-ea50-45bc-8e20-1f009b604dc6" containerID="699d52526632f4ee3d103532995ea837be4ddc4af5700e90b3862162146f2b5d" exitCode=6
Jul 16 23:24:43.251482 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:43.251159    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-xhr8l" event={"ID":"c97d1def-ea50-45bc-8e20-1f009b604dc6","Type":"ContainerDied","Data":"699d52526632f4ee3d103532995ea837be4ddc4af5700e90b3862162146f2b5d"}
Jul 16 23:24:43.251482 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:43.251250    2573 scope.go:117] "RemoveContainer" containerID="c8344c1ed5588b5f2fe54de7757a3a0c5608b9363859c0c48210e448b1e3ba7e"
Jul 16 23:24:43.251482 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:43.251434    2573 scope.go:117] "RemoveContainer" containerID="699d52526632f4ee3d103532995ea837be4ddc4af5700e90b3862162146f2b5d"
Jul 16 23:24:43.251645 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:24:43.251624    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-xhr8l_odh-ai-gateway-infra(c97d1def-ea50-45bc-8e20-1f009b604dc6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-xhr8l" podUID="c97d1def-ea50-45bc-8e20-1f009b604dc6"
Jul 16 23:24:44.259228 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:44.259201    2573 scope.go:117] "RemoveContainer" containerID="699d52526632f4ee3d103532995ea837be4ddc4af5700e90b3862162146f2b5d"
Jul 16 23:24:44.259600 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:24:44.259406    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-xhr8l_odh-ai-gateway-infra(c97d1def-ea50-45bc-8e20-1f009b604dc6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-xhr8l" podUID="c97d1def-ea50-45bc-8e20-1f009b604dc6"
Jul 16 23:24:46.371817 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:46.371764    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:24:47.513214 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:47.513188    2573 scope.go:117] "RemoveContainer" containerID="c86dc3f294d87ee9b615e0ef00bc495a8202e0a4a913ecf5e95815844ac7ba18"
Jul 16 23:24:48.279994 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:48.279966    2573 generic.go:356] "Generic (PLEG): container finished" podID="5ec0333e-2b85-40b6-8e5d-2a234686d823" containerID="325fa540b9a71a9c9667b306be748b95b6869af12072a7736daf6b732dd2d080" exitCode=6
Jul 16 23:24:48.280126 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:48.280033    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-hkrsk" event={"ID":"5ec0333e-2b85-40b6-8e5d-2a234686d823","Type":"ContainerDied","Data":"325fa540b9a71a9c9667b306be748b95b6869af12072a7736daf6b732dd2d080"}
Jul 16 23:24:48.280126 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:48.280094    2573 scope.go:117] "RemoveContainer" containerID="c86dc3f294d87ee9b615e0ef00bc495a8202e0a4a913ecf5e95815844ac7ba18"
Jul 16 23:24:48.280365 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:48.280348    2573 scope.go:117] "RemoveContainer" containerID="325fa540b9a71a9c9667b306be748b95b6869af12072a7736daf6b732dd2d080"
Jul 16 23:24:48.280561 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:24:48.280542    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-hkrsk_odh-ai-gateway-infra(5ec0333e-2b85-40b6-8e5d-2a234686d823)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-hkrsk" podUID="5ec0333e-2b85-40b6-8e5d-2a234686d823"
Jul 16 23:24:49.307110 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:49.307066    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-hkrsk"]
Jul 16 23:24:49.444380 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:49.444352    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-hkrsk"
Jul 16 23:24:49.531211 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:49.531183    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ec0333e-2b85-40b6-8e5d-2a234686d823-maas-api-service-ca\") pod \"5ec0333e-2b85-40b6-8e5d-2a234686d823\" (UID: \"5ec0333e-2b85-40b6-8e5d-2a234686d823\") "
Jul 16 23:24:49.531540 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:49.531521    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ec0333e-2b85-40b6-8e5d-2a234686d823-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5ec0333e-2b85-40b6-8e5d-2a234686d823" (UID: "5ec0333e-2b85-40b6-8e5d-2a234686d823"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:24:49.631687 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:49.631666    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ec0333e-2b85-40b6-8e5d-2a234686d823-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:24:50.294150 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:50.294120    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-hkrsk"
Jul 16 23:24:50.294150 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:50.294122    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-hkrsk" event={"ID":"5ec0333e-2b85-40b6-8e5d-2a234686d823","Type":"ContainerDied","Data":"62679deed414e8264469f414d289c35a80a3e88353fb657f9209fbd7cbe32cf8"}
Jul 16 23:24:50.294451 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:50.294200    2573 scope.go:117] "RemoveContainer" containerID="325fa540b9a71a9c9667b306be748b95b6869af12072a7736daf6b732dd2d080"
Jul 16 23:24:50.326112 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:50.326088    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-hkrsk"]
Jul 16 23:24:50.328067 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:50.328044    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-hkrsk"]
Jul 16 23:24:50.518904 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:50.518876    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ec0333e-2b85-40b6-8e5d-2a234686d823" path="/var/lib/kubelet/pods/5ec0333e-2b85-40b6-8e5d-2a234686d823/volumes"
Jul 16 23:24:58.274186 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:58.274155    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-chddx"]
Jul 16 23:24:58.274573 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:58.274483    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ec0333e-2b85-40b6-8e5d-2a234686d823" containerName="revoke-keys"
Jul 16 23:24:58.274573 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:58.274495    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ec0333e-2b85-40b6-8e5d-2a234686d823" containerName="revoke-keys"
Jul 16 23:24:58.274573 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:58.274507    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ec0333e-2b85-40b6-8e5d-2a234686d823" containerName="revoke-keys"
Jul 16 23:24:58.274573 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:58.274512    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ec0333e-2b85-40b6-8e5d-2a234686d823" containerName="revoke-keys"
Jul 16 23:24:58.274573 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:58.274520    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ec0333e-2b85-40b6-8e5d-2a234686d823" containerName="revoke-keys"
Jul 16 23:24:58.274573 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:58.274525    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ec0333e-2b85-40b6-8e5d-2a234686d823" containerName="revoke-keys"
Jul 16 23:24:58.274573 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:58.274576    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ec0333e-2b85-40b6-8e5d-2a234686d823" containerName="revoke-keys"
Jul 16 23:24:58.274776 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:58.274586    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ec0333e-2b85-40b6-8e5d-2a234686d823" containerName="revoke-keys"
Jul 16 23:24:58.277347 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:58.277331    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-chddx"
Jul 16 23:24:58.285043 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:58.285018    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-chddx"]
Jul 16 23:24:58.392228 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:58.392200    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94cc5af9-52e9-47f6-8bec-6297c867569f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-chddx\" (UID: \"94cc5af9-52e9-47f6-8bec-6297c867569f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-chddx"
Jul 16 23:24:58.493281 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:58.493252    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94cc5af9-52e9-47f6-8bec-6297c867569f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-chddx\" (UID: \"94cc5af9-52e9-47f6-8bec-6297c867569f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-chddx"
Jul 16 23:24:58.493826 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:58.493806    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94cc5af9-52e9-47f6-8bec-6297c867569f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-chddx\" (UID: \"94cc5af9-52e9-47f6-8bec-6297c867569f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-chddx"
Jul 16 23:24:58.587599 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:58.587574    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-chddx"
Jul 16 23:24:58.711855 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:58.711786    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-chddx"]
Jul 16 23:24:58.713931 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:24:58.713897    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94cc5af9_52e9_47f6_8bec_6297c867569f.slice/crio-7eb6b105f6d1c1f4d2ef0c35d25f1c60d9c23bf0828e5fdebd36643019cee3f2 WatchSource:0}: Error finding container 7eb6b105f6d1c1f4d2ef0c35d25f1c60d9c23bf0828e5fdebd36643019cee3f2: Status 404 returned error can't find the container with id 7eb6b105f6d1c1f4d2ef0c35d25f1c60d9c23bf0828e5fdebd36643019cee3f2
Jul 16 23:24:59.341476 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:59.341447    2573 generic.go:356] "Generic (PLEG): container finished" podID="94cc5af9-52e9-47f6-8bec-6297c867569f" containerID="cbb6188bdbd897bd564c2aada19bb9f431cdc161dcf80f2602c4304ee942cebe" exitCode=6
Jul 16 23:24:59.341766 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:59.341496    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-chddx" event={"ID":"94cc5af9-52e9-47f6-8bec-6297c867569f","Type":"ContainerDied","Data":"cbb6188bdbd897bd564c2aada19bb9f431cdc161dcf80f2602c4304ee942cebe"}
Jul 16 23:24:59.341766 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:59.341549    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-chddx" event={"ID":"94cc5af9-52e9-47f6-8bec-6297c867569f","Type":"ContainerStarted","Data":"7eb6b105f6d1c1f4d2ef0c35d25f1c60d9c23bf0828e5fdebd36643019cee3f2"}
Jul 16 23:24:59.341766 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:59.341698    2573 scope.go:117] "RemoveContainer" containerID="cbb6188bdbd897bd564c2aada19bb9f431cdc161dcf80f2602c4304ee942cebe"
Jul 16 23:24:59.513350 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:24:59.513330    2573 scope.go:117] "RemoveContainer" containerID="699d52526632f4ee3d103532995ea837be4ddc4af5700e90b3862162146f2b5d"
Jul 16 23:25:00.348834 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:00.348781    2573 generic.go:356] "Generic (PLEG): container finished" podID="c97d1def-ea50-45bc-8e20-1f009b604dc6" containerID="fed37a8f20bc3e727f1fdc8c36e1d5af07d908a72871aba3586391653db8095d" exitCode=6
Jul 16 23:25:00.349381 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:00.348855    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-xhr8l" event={"ID":"c97d1def-ea50-45bc-8e20-1f009b604dc6","Type":"ContainerDied","Data":"fed37a8f20bc3e727f1fdc8c36e1d5af07d908a72871aba3586391653db8095d"}
Jul 16 23:25:00.349381 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:00.348913    2573 scope.go:117] "RemoveContainer" containerID="699d52526632f4ee3d103532995ea837be4ddc4af5700e90b3862162146f2b5d"
Jul 16 23:25:00.349381 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:00.349186    2573 scope.go:117] "RemoveContainer" containerID="fed37a8f20bc3e727f1fdc8c36e1d5af07d908a72871aba3586391653db8095d"
Jul 16 23:25:00.349557 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:25:00.349381    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-xhr8l_odh-ai-gateway-infra(c97d1def-ea50-45bc-8e20-1f009b604dc6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-xhr8l" podUID="c97d1def-ea50-45bc-8e20-1f009b604dc6"
Jul 16 23:25:00.351019 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:00.350997    2573 generic.go:356] "Generic (PLEG): container finished" podID="94cc5af9-52e9-47f6-8bec-6297c867569f" containerID="b869bb0fc38bcef37ceaae394c414f047a89c98b4a9509e0856192a84f634ef7" exitCode=6
Jul 16 23:25:00.351119 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:00.351052    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-chddx" event={"ID":"94cc5af9-52e9-47f6-8bec-6297c867569f","Type":"ContainerDied","Data":"b869bb0fc38bcef37ceaae394c414f047a89c98b4a9509e0856192a84f634ef7"}
Jul 16 23:25:00.351269 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:00.351255    2573 scope.go:117] "RemoveContainer" containerID="b869bb0fc38bcef37ceaae394c414f047a89c98b4a9509e0856192a84f634ef7"
Jul 16 23:25:00.351405 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:25:00.351389    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-chddx_odh-ai-gateway-infra(94cc5af9-52e9-47f6-8bec-6297c867569f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-chddx" podUID="94cc5af9-52e9-47f6-8bec-6297c867569f"
Jul 16 23:25:00.362557 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:00.362532    2573 scope.go:117] "RemoveContainer" containerID="cbb6188bdbd897bd564c2aada19bb9f431cdc161dcf80f2602c4304ee942cebe"
Jul 16 23:25:01.359223 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:01.359192    2573 scope.go:117] "RemoveContainer" containerID="b869bb0fc38bcef37ceaae394c414f047a89c98b4a9509e0856192a84f634ef7"
Jul 16 23:25:01.359605 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:25:01.359387    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-chddx_odh-ai-gateway-infra(94cc5af9-52e9-47f6-8bec-6297c867569f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-chddx" podUID="94cc5af9-52e9-47f6-8bec-6297c867569f"
Jul 16 23:25:01.374970 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:01.374941    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-xhr8l"]
Jul 16 23:25:01.506349 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:01.506326    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-xhr8l"
Jul 16 23:25:01.613871 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:01.613800    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c97d1def-ea50-45bc-8e20-1f009b604dc6-maas-api-service-ca\") pod \"c97d1def-ea50-45bc-8e20-1f009b604dc6\" (UID: \"c97d1def-ea50-45bc-8e20-1f009b604dc6\") "
Jul 16 23:25:01.614149 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:01.614127    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c97d1def-ea50-45bc-8e20-1f009b604dc6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c97d1def-ea50-45bc-8e20-1f009b604dc6" (UID: "c97d1def-ea50-45bc-8e20-1f009b604dc6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:25:01.714423 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:01.714393    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c97d1def-ea50-45bc-8e20-1f009b604dc6-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:25:02.365209 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:02.365176    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-xhr8l" event={"ID":"c97d1def-ea50-45bc-8e20-1f009b604dc6","Type":"ContainerDied","Data":"440e61da877ebac26d3371358b908fd0143c1c18eb2194a3b5d20acfee5c8587"}
Jul 16 23:25:02.365620 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:02.365184    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-xhr8l"
Jul 16 23:25:02.365620 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:02.365279    2573 scope.go:117] "RemoveContainer" containerID="fed37a8f20bc3e727f1fdc8c36e1d5af07d908a72871aba3586391653db8095d"
Jul 16 23:25:02.387835 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:02.387810    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-xhr8l"]
Jul 16 23:25:02.393023 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:02.393001    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-xhr8l"]
Jul 16 23:25:02.518748 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:02.518724    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c97d1def-ea50-45bc-8e20-1f009b604dc6" path="/var/lib/kubelet/pods/c97d1def-ea50-45bc-8e20-1f009b604dc6/volumes"
Jul 16 23:25:02.646614 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:02.646543    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-b4ab52/auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq"]
Jul 16 23:25:02.646891 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:02.646875    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c97d1def-ea50-45bc-8e20-1f009b604dc6" containerName="revoke-keys"
Jul 16 23:25:02.646975 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:02.646894    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c97d1def-ea50-45bc-8e20-1f009b604dc6" containerName="revoke-keys"
Jul 16 23:25:02.646975 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:02.646909    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c97d1def-ea50-45bc-8e20-1f009b604dc6" containerName="revoke-keys"
Jul 16 23:25:02.646975 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:02.646918    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c97d1def-ea50-45bc-8e20-1f009b604dc6" containerName="revoke-keys"
Jul 16 23:25:02.646975 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:02.646932    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c97d1def-ea50-45bc-8e20-1f009b604dc6" containerName="revoke-keys"
Jul 16 23:25:02.646975 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:02.646940    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c97d1def-ea50-45bc-8e20-1f009b604dc6" containerName="revoke-keys"
Jul 16 23:25:02.647209 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:02.647013    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c97d1def-ea50-45bc-8e20-1f009b604dc6" containerName="revoke-keys"
Jul 16 23:25:02.647209 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:02.647026    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c97d1def-ea50-45bc-8e20-1f009b604dc6" containerName="revoke-keys"
Jul 16 23:25:02.647209 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:02.647034    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c97d1def-ea50-45bc-8e20-1f009b604dc6" containerName="revoke-keys"
Jul 16 23:25:02.647209 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:02.647046    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ec0333e-2b85-40b6-8e5d-2a234686d823" containerName="revoke-keys"
Jul 16 23:25:02.651419 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:02.651400    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b4ab52/auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq"
Jul 16 23:25:02.653642 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:02.653621    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b4ab52\"/\"openshift-service-ca.crt\""
Jul 16 23:25:02.654550 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:02.654506    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b4ab52\"/\"default-dockercfg-b6pn6\""
Jul 16 23:25:02.654915 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:02.654506    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b4ab52\"/\"auth-test-model-b4ab52-kserve-self-signed-certs\""
Jul 16 23:25:02.654915 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:02.654538    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b4ab52\"/\"kube-root-ca.crt\""
Jul 16 23:25:02.657460 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:02.657442    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b4ab52/auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq"]
Jul 16 23:25:02.666488 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:02.666451    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:25:02.721710 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:02.721688    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32dacbc9-8b90-4467-9cc9-807c928c8700-home\") pod \"auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq\" (UID: \"32dacbc9-8b90-4467-9cc9-807c928c8700\") " pod="ai-tenant-e2e-shared-b-b4ab52/auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq"
Jul 16 23:25:02.721802 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:02.721719    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32dacbc9-8b90-4467-9cc9-807c928c8700-tmp-dir\") pod \"auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq\" (UID: \"32dacbc9-8b90-4467-9cc9-807c928c8700\") " pod="ai-tenant-e2e-shared-b-b4ab52/auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq"
Jul 16 23:25:02.721802 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:02.721753    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32dacbc9-8b90-4467-9cc9-807c928c8700-model-cache\") pod \"auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq\" (UID: \"32dacbc9-8b90-4467-9cc9-807c928c8700\") " pod="ai-tenant-e2e-shared-b-b4ab52/auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq"
Jul 16 23:25:02.721875 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:02.721833    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32dacbc9-8b90-4467-9cc9-807c928c8700-tls-certs\") pod \"auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq\" (UID: \"32dacbc9-8b90-4467-9cc9-807c928c8700\") " pod="ai-tenant-e2e-shared-b-b4ab52/auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq"
Jul 16 23:25:02.721910 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:02.721873    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncbhg\" (UniqueName: \"kubernetes.io/projected/32dacbc9-8b90-4467-9cc9-807c928c8700-kube-api-access-ncbhg\") pod \"auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq\" (UID: \"32dacbc9-8b90-4467-9cc9-807c928c8700\") " pod="ai-tenant-e2e-shared-b-b4ab52/auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq"
Jul 16 23:25:02.721910 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:02.721901    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32dacbc9-8b90-4467-9cc9-807c928c8700-dshm\") pod \"auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq\" (UID: \"32dacbc9-8b90-4467-9cc9-807c928c8700\") " pod="ai-tenant-e2e-shared-b-b4ab52/auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq"
Jul 16 23:25:02.822832 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:02.822811    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32dacbc9-8b90-4467-9cc9-807c928c8700-dshm\") pod \"auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq\" (UID: \"32dacbc9-8b90-4467-9cc9-807c928c8700\") " pod="ai-tenant-e2e-shared-b-b4ab52/auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq"
Jul 16 23:25:02.822919 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:02.822843    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32dacbc9-8b90-4467-9cc9-807c928c8700-home\") pod \"auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq\" (UID: \"32dacbc9-8b90-4467-9cc9-807c928c8700\") " pod="ai-tenant-e2e-shared-b-b4ab52/auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq"
Jul 16 23:25:02.822919 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:02.822870    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32dacbc9-8b90-4467-9cc9-807c928c8700-tmp-dir\") pod \"auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq\" (UID: \"32dacbc9-8b90-4467-9cc9-807c928c8700\") " pod="ai-tenant-e2e-shared-b-b4ab52/auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq"
Jul 16 23:25:02.822919 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:02.822902    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32dacbc9-8b90-4467-9cc9-807c928c8700-model-cache\") pod \"auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq\" (UID: \"32dacbc9-8b90-4467-9cc9-807c928c8700\") " pod="ai-tenant-e2e-shared-b-b4ab52/auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq"
Jul 16 23:25:02.823064 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:02.822933    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32dacbc9-8b90-4467-9cc9-807c928c8700-tls-certs\") pod \"auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq\" (UID: \"32dacbc9-8b90-4467-9cc9-807c928c8700\") " pod="ai-tenant-e2e-shared-b-b4ab52/auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq"
Jul 16 23:25:02.823064 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:02.822955    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncbhg\" (UniqueName: \"kubernetes.io/projected/32dacbc9-8b90-4467-9cc9-807c928c8700-kube-api-access-ncbhg\") pod \"auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq\" (UID: \"32dacbc9-8b90-4467-9cc9-807c928c8700\") " pod="ai-tenant-e2e-shared-b-b4ab52/auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq"
Jul 16 23:25:02.823306 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:02.823276    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32dacbc9-8b90-4467-9cc9-807c928c8700-home\") pod \"auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq\" (UID: \"32dacbc9-8b90-4467-9cc9-807c928c8700\") " pod="ai-tenant-e2e-shared-b-b4ab52/auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq"
Jul 16 23:25:02.823419 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:02.823322    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32dacbc9-8b90-4467-9cc9-807c928c8700-tmp-dir\") pod \"auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq\" (UID: \"32dacbc9-8b90-4467-9cc9-807c928c8700\") " pod="ai-tenant-e2e-shared-b-b4ab52/auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq"
Jul 16 23:25:02.823419 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:02.823356    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32dacbc9-8b90-4467-9cc9-807c928c8700-model-cache\") pod \"auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq\" (UID: \"32dacbc9-8b90-4467-9cc9-807c928c8700\") " pod="ai-tenant-e2e-shared-b-b4ab52/auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq"
Jul 16 23:25:02.825023 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:02.825003    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32dacbc9-8b90-4467-9cc9-807c928c8700-dshm\") pod \"auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq\" (UID: \"32dacbc9-8b90-4467-9cc9-807c928c8700\") " pod="ai-tenant-e2e-shared-b-b4ab52/auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq"
Jul 16 23:25:02.825305 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:02.825289    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32dacbc9-8b90-4467-9cc9-807c928c8700-tls-certs\") pod \"auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq\" (UID: \"32dacbc9-8b90-4467-9cc9-807c928c8700\") " pod="ai-tenant-e2e-shared-b-b4ab52/auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq"
Jul 16 23:25:02.830212 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:02.830186    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncbhg\" (UniqueName: \"kubernetes.io/projected/32dacbc9-8b90-4467-9cc9-807c928c8700-kube-api-access-ncbhg\") pod \"auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq\" (UID: \"32dacbc9-8b90-4467-9cc9-807c928c8700\") " pod="ai-tenant-e2e-shared-b-b4ab52/auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq"
Jul 16 23:25:02.961615 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:02.961565    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b4ab52/auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq"
Jul 16 23:25:03.083517 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:03.083493    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b4ab52/auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq"]
Jul 16 23:25:03.085337 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:25:03.085313    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32dacbc9_8b90_4467_9cc9_807c928c8700.slice/crio-3ef2d699a7d9b2e0a3e6d8e120ffbf47d6d3ffeed3f207d6b7e086a77510fc2a WatchSource:0}: Error finding container 3ef2d699a7d9b2e0a3e6d8e120ffbf47d6d3ffeed3f207d6b7e086a77510fc2a: Status 404 returned error can't find the container with id 3ef2d699a7d9b2e0a3e6d8e120ffbf47d6d3ffeed3f207d6b7e086a77510fc2a
Jul 16 23:25:03.302352 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:03.302274    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-rs5j6"]
Jul 16 23:25:03.305596 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:03.305579    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-rs5j6"
Jul 16 23:25:03.313165 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:03.313140    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-rs5j6"]
Jul 16 23:25:03.327364 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:03.327329    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13298302-d07c-4423-9c47-9a41a2941063-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-rs5j6\" (UID: \"13298302-d07c-4423-9c47-9a41a2941063\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-rs5j6"
Jul 16 23:25:03.373366 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:03.373339    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b4ab52/auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq" event={"ID":"32dacbc9-8b90-4467-9cc9-807c928c8700","Type":"ContainerStarted","Data":"47fda37cda7bb9d5b9e7f87c232239648f70a827f9a3cd4c4f9d4f7d753eeaaf"}
Jul 16 23:25:03.373711 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:03.373376    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b4ab52/auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq" event={"ID":"32dacbc9-8b90-4467-9cc9-807c928c8700","Type":"ContainerStarted","Data":"3ef2d699a7d9b2e0a3e6d8e120ffbf47d6d3ffeed3f207d6b7e086a77510fc2a"}
Jul 16 23:25:03.390309 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:03.390274    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-b4ab52/auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq" podStartSLOduration=1.390260676 podStartE2EDuration="1.390260676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:25:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:25:03.389332121 +0000 UTC m=+3077.399594784" watchObservedRunningTime="2026-07-16 23:25:03.390260676 +0000 UTC m=+3077.400523330"
Jul 16 23:25:03.428369 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:03.428348    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13298302-d07c-4423-9c47-9a41a2941063-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-rs5j6\" (UID: \"13298302-d07c-4423-9c47-9a41a2941063\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-rs5j6"
Jul 16 23:25:03.428977 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:03.428959    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13298302-d07c-4423-9c47-9a41a2941063-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-rs5j6\" (UID: \"13298302-d07c-4423-9c47-9a41a2941063\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-rs5j6"
Jul 16 23:25:03.616273 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:03.616253    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-rs5j6"
Jul 16 23:25:03.735887 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:03.735858    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-rs5j6"]
Jul 16 23:25:03.737219 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:25:03.737196    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13298302_d07c_4423_9c47_9a41a2941063.slice/crio-e25e8aaa3c5ad2676e63d731d23a70038016fb87f37fa811792fcd64cf5f0930 WatchSource:0}: Error finding container e25e8aaa3c5ad2676e63d731d23a70038016fb87f37fa811792fcd64cf5f0930: Status 404 returned error can't find the container with id e25e8aaa3c5ad2676e63d731d23a70038016fb87f37fa811792fcd64cf5f0930
Jul 16 23:25:04.379702 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:04.379676    2573 generic.go:356] "Generic (PLEG): container finished" podID="13298302-d07c-4423-9c47-9a41a2941063" containerID="979f8e60e494bf35cbb357c29cdd28077ff7019d26c3960703e767ba76ccc43b" exitCode=6
Jul 16 23:25:04.380014 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:04.379763    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-rs5j6" event={"ID":"13298302-d07c-4423-9c47-9a41a2941063","Type":"ContainerDied","Data":"979f8e60e494bf35cbb357c29cdd28077ff7019d26c3960703e767ba76ccc43b"}
Jul 16 23:25:04.380014 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:04.379808    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-rs5j6" event={"ID":"13298302-d07c-4423-9c47-9a41a2941063","Type":"ContainerStarted","Data":"e25e8aaa3c5ad2676e63d731d23a70038016fb87f37fa811792fcd64cf5f0930"}
Jul 16 23:25:04.380014 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:04.379901    2573 scope.go:117] "RemoveContainer" containerID="979f8e60e494bf35cbb357c29cdd28077ff7019d26c3960703e767ba76ccc43b"
Jul 16 23:25:05.386198 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:05.386152    2573 generic.go:356] "Generic (PLEG): container finished" podID="13298302-d07c-4423-9c47-9a41a2941063" containerID="7356050dce8d8cb94a9ec8c2e4e7a694020edf3ae2d8fc85256d12a4759dce0e" exitCode=6
Jul 16 23:25:05.386583 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:05.386219    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-rs5j6" event={"ID":"13298302-d07c-4423-9c47-9a41a2941063","Type":"ContainerDied","Data":"7356050dce8d8cb94a9ec8c2e4e7a694020edf3ae2d8fc85256d12a4759dce0e"}
Jul 16 23:25:05.386583 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:05.386320    2573 scope.go:117] "RemoveContainer" containerID="979f8e60e494bf35cbb357c29cdd28077ff7019d26c3960703e767ba76ccc43b"
Jul 16 23:25:05.386583 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:05.386543    2573 scope.go:117] "RemoveContainer" containerID="7356050dce8d8cb94a9ec8c2e4e7a694020edf3ae2d8fc85256d12a4759dce0e"
Jul 16 23:25:05.386766 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:25:05.386714    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-rs5j6_odh-ai-gateway-infra(13298302-d07c-4423-9c47-9a41a2941063)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-rs5j6" podUID="13298302-d07c-4423-9c47-9a41a2941063"
Jul 16 23:25:06.392626 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:06.392596    2573 scope.go:117] "RemoveContainer" containerID="7356050dce8d8cb94a9ec8c2e4e7a694020edf3ae2d8fc85256d12a4759dce0e"
Jul 16 23:25:06.393032 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:25:06.392750    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-rs5j6_odh-ai-gateway-infra(13298302-d07c-4423-9c47-9a41a2941063)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-rs5j6" podUID="13298302-d07c-4423-9c47-9a41a2941063"
Jul 16 23:25:06.531518 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:06.531489    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:25:12.962256 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:12.962200    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-b4ab52/auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq"
Jul 16 23:25:12.962256 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:12.962263    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-b4ab52/auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq"
Jul 16 23:25:12.974748 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:12.974715    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-b4ab52/auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq"
Jul 16 23:25:13.438535 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:13.438507    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-b4ab52/auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq"
Jul 16 23:25:16.516857 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:16.516824    2573 scope.go:117] "RemoveContainer" containerID="b869bb0fc38bcef37ceaae394c414f047a89c98b4a9509e0856192a84f634ef7"
Jul 16 23:25:17.449472 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:17.449401    2573 generic.go:356] "Generic (PLEG): container finished" podID="94cc5af9-52e9-47f6-8bec-6297c867569f" containerID="aac0fd278c0a044f5f19e1b0d9f3452652567268d694c88d704586c9d3c79832" exitCode=6
Jul 16 23:25:17.449590 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:17.449479    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-chddx" event={"ID":"94cc5af9-52e9-47f6-8bec-6297c867569f","Type":"ContainerDied","Data":"aac0fd278c0a044f5f19e1b0d9f3452652567268d694c88d704586c9d3c79832"}
Jul 16 23:25:17.449590 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:17.449543    2573 scope.go:117] "RemoveContainer" containerID="b869bb0fc38bcef37ceaae394c414f047a89c98b4a9509e0856192a84f634ef7"
Jul 16 23:25:17.449782 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:17.449766    2573 scope.go:117] "RemoveContainer" containerID="aac0fd278c0a044f5f19e1b0d9f3452652567268d694c88d704586c9d3c79832"
Jul 16 23:25:17.449972 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:25:17.449954    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-chddx_odh-ai-gateway-infra(94cc5af9-52e9-47f6-8bec-6297c867569f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-chddx" podUID="94cc5af9-52e9-47f6-8bec-6297c867569f"
Jul 16 23:25:18.475114 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:18.475079    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-chddx"]
Jul 16 23:25:18.513603 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:18.513574    2573 scope.go:117] "RemoveContainer" containerID="7356050dce8d8cb94a9ec8c2e4e7a694020edf3ae2d8fc85256d12a4759dce0e"
Jul 16 23:25:18.596672 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:18.596650    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-chddx"
Jul 16 23:25:18.641650 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:18.641627    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94cc5af9-52e9-47f6-8bec-6297c867569f-maas-api-service-ca\") pod \"94cc5af9-52e9-47f6-8bec-6297c867569f\" (UID: \"94cc5af9-52e9-47f6-8bec-6297c867569f\") "
Jul 16 23:25:18.641940 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:18.641918    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94cc5af9-52e9-47f6-8bec-6297c867569f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "94cc5af9-52e9-47f6-8bec-6297c867569f" (UID: "94cc5af9-52e9-47f6-8bec-6297c867569f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:25:18.742977 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:18.742926    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94cc5af9-52e9-47f6-8bec-6297c867569f-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:25:19.466961 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:19.466888    2573 generic.go:356] "Generic (PLEG): container finished" podID="13298302-d07c-4423-9c47-9a41a2941063" containerID="7c598a112a7699c9067105bb3450790c24050883974eaffba0a2bbbb992c97cb" exitCode=6
Jul 16 23:25:19.467090 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:19.466963    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-rs5j6" event={"ID":"13298302-d07c-4423-9c47-9a41a2941063","Type":"ContainerDied","Data":"7c598a112a7699c9067105bb3450790c24050883974eaffba0a2bbbb992c97cb"}
Jul 16 23:25:19.467090 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:19.467032    2573 scope.go:117] "RemoveContainer" containerID="7356050dce8d8cb94a9ec8c2e4e7a694020edf3ae2d8fc85256d12a4759dce0e"
Jul 16 23:25:19.467377 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:19.467361    2573 scope.go:117] "RemoveContainer" containerID="7c598a112a7699c9067105bb3450790c24050883974eaffba0a2bbbb992c97cb"
Jul 16 23:25:19.467581 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:25:19.467551    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-rs5j6_odh-ai-gateway-infra(13298302-d07c-4423-9c47-9a41a2941063)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-rs5j6" podUID="13298302-d07c-4423-9c47-9a41a2941063"
Jul 16 23:25:19.468327 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:19.468311    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-chddx"
Jul 16 23:25:19.468402 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:19.468357    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-chddx" event={"ID":"94cc5af9-52e9-47f6-8bec-6297c867569f","Type":"ContainerDied","Data":"7eb6b105f6d1c1f4d2ef0c35d25f1c60d9c23bf0828e5fdebd36643019cee3f2"}
Jul 16 23:25:19.478356 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:19.478197    2573 scope.go:117] "RemoveContainer" containerID="aac0fd278c0a044f5f19e1b0d9f3452652567268d694c88d704586c9d3c79832"
Jul 16 23:25:19.496102 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:19.496082    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-chddx"]
Jul 16 23:25:19.499713 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:19.499692    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-chddx"]
Jul 16 23:25:20.492317 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:20.492284    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-rs5j6"]
Jul 16 23:25:20.519938 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:20.519901    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94cc5af9-52e9-47f6-8bec-6297c867569f" path="/var/lib/kubelet/pods/94cc5af9-52e9-47f6-8bec-6297c867569f/volumes"
Jul 16 23:25:20.614394 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:20.614366    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-rs5j6"
Jul 16 23:25:20.756479 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:20.756418    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13298302-d07c-4423-9c47-9a41a2941063-maas-api-service-ca\") pod \"13298302-d07c-4423-9c47-9a41a2941063\" (UID: \"13298302-d07c-4423-9c47-9a41a2941063\") "
Jul 16 23:25:20.756764 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:20.756738    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13298302-d07c-4423-9c47-9a41a2941063-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "13298302-d07c-4423-9c47-9a41a2941063" (UID: "13298302-d07c-4423-9c47-9a41a2941063"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:25:20.856989 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:20.856962    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13298302-d07c-4423-9c47-9a41a2941063-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:25:21.483320 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:21.483292    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-rs5j6"
Jul 16 23:25:21.483470 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:21.483327    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-rs5j6" event={"ID":"13298302-d07c-4423-9c47-9a41a2941063","Type":"ContainerDied","Data":"e25e8aaa3c5ad2676e63d731d23a70038016fb87f37fa811792fcd64cf5f0930"}
Jul 16 23:25:21.483470 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:21.483386    2573 scope.go:117] "RemoveContainer" containerID="7c598a112a7699c9067105bb3450790c24050883974eaffba0a2bbbb992c97cb"
Jul 16 23:25:21.504769 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:21.504740    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-rs5j6"]
Jul 16 23:25:21.514789 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:21.514766    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-rs5j6"]
Jul 16 23:25:22.132630 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:22.132594    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-hjj4l"]
Jul 16 23:25:22.132922 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:22.132908    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13298302-d07c-4423-9c47-9a41a2941063" containerName="revoke-keys"
Jul 16 23:25:22.132975 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:22.132923    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="13298302-d07c-4423-9c47-9a41a2941063" containerName="revoke-keys"
Jul 16 23:25:22.132975 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:22.132933    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13298302-d07c-4423-9c47-9a41a2941063" containerName="revoke-keys"
Jul 16 23:25:22.132975 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:22.132938    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="13298302-d07c-4423-9c47-9a41a2941063" containerName="revoke-keys"
Jul 16 23:25:22.132975 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:22.132946    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94cc5af9-52e9-47f6-8bec-6297c867569f" containerName="revoke-keys"
Jul 16 23:25:22.132975 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:22.132951    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="94cc5af9-52e9-47f6-8bec-6297c867569f" containerName="revoke-keys"
Jul 16 23:25:22.132975 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:22.132968    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94cc5af9-52e9-47f6-8bec-6297c867569f" containerName="revoke-keys"
Jul 16 23:25:22.132975 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:22.132974    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="94cc5af9-52e9-47f6-8bec-6297c867569f" containerName="revoke-keys"
Jul 16 23:25:22.133176 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:22.132980    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94cc5af9-52e9-47f6-8bec-6297c867569f" containerName="revoke-keys"
Jul 16 23:25:22.133176 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:22.132986    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="94cc5af9-52e9-47f6-8bec-6297c867569f" containerName="revoke-keys"
Jul 16 23:25:22.133176 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:22.133036    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="13298302-d07c-4423-9c47-9a41a2941063" containerName="revoke-keys"
Jul 16 23:25:22.133176 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:22.133046    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="13298302-d07c-4423-9c47-9a41a2941063" containerName="revoke-keys"
Jul 16 23:25:22.133176 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:22.133053    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="94cc5af9-52e9-47f6-8bec-6297c867569f" containerName="revoke-keys"
Jul 16 23:25:22.133176 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:22.133057    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="94cc5af9-52e9-47f6-8bec-6297c867569f" containerName="revoke-keys"
Jul 16 23:25:22.133176 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:22.133063    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="94cc5af9-52e9-47f6-8bec-6297c867569f" containerName="revoke-keys"
Jul 16 23:25:22.137408 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:22.137389    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-hjj4l"
Jul 16 23:25:22.139801 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:22.139774    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 23:25:22.140578 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:22.140543    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:25:22.143600 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:22.143576    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-hjj4l"]
Jul 16 23:25:22.266589 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:22.266560    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0965d941-4625-4387-a83c-2cc1802675b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-hjj4l\" (UID: \"0965d941-4625-4387-a83c-2cc1802675b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-hjj4l"
Jul 16 23:25:22.367487 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:22.367457    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0965d941-4625-4387-a83c-2cc1802675b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-hjj4l\" (UID: \"0965d941-4625-4387-a83c-2cc1802675b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-hjj4l"
Jul 16 23:25:22.368014 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:22.367995    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0965d941-4625-4387-a83c-2cc1802675b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-hjj4l\" (UID: \"0965d941-4625-4387-a83c-2cc1802675b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-hjj4l"
Jul 16 23:25:22.450640 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:22.450581    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-hjj4l"
Jul 16 23:25:22.519546 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:22.519515    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13298302-d07c-4423-9c47-9a41a2941063" path="/var/lib/kubelet/pods/13298302-d07c-4423-9c47-9a41a2941063/volumes"
Jul 16 23:25:22.582567 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:22.582534    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-hjj4l"]
Jul 16 23:25:22.585571 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:25:22.585542    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0965d941_4625_4387_a83c_2cc1802675b8.slice/crio-0e180424330a8288f52cfbceec697fc6d644e0c82b4360da7a08f51d549cf1f3 WatchSource:0}: Error finding container 0e180424330a8288f52cfbceec697fc6d644e0c82b4360da7a08f51d549cf1f3: Status 404 returned error can't find the container with id 0e180424330a8288f52cfbceec697fc6d644e0c82b4360da7a08f51d549cf1f3
Jul 16 23:25:23.499604 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:23.499520    2573 generic.go:356] "Generic (PLEG): container finished" podID="0965d941-4625-4387-a83c-2cc1802675b8" containerID="44f077eb85cb32275cf4a9905c41371d458acbd35b0327d27e14a615bf092d13" exitCode=6
Jul 16 23:25:23.499604 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:23.499574    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-hjj4l" event={"ID":"0965d941-4625-4387-a83c-2cc1802675b8","Type":"ContainerDied","Data":"44f077eb85cb32275cf4a9905c41371d458acbd35b0327d27e14a615bf092d13"}
Jul 16 23:25:23.499790 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:23.499618    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-hjj4l" event={"ID":"0965d941-4625-4387-a83c-2cc1802675b8","Type":"ContainerStarted","Data":"0e180424330a8288f52cfbceec697fc6d644e0c82b4360da7a08f51d549cf1f3"}
Jul 16 23:25:23.499790 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:23.499777    2573 scope.go:117] "RemoveContainer" containerID="44f077eb85cb32275cf4a9905c41371d458acbd35b0327d27e14a615bf092d13"
Jul 16 23:25:23.839024 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:23.839000    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:25:24.506843 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:24.506813    2573 generic.go:356] "Generic (PLEG): container finished" podID="0965d941-4625-4387-a83c-2cc1802675b8" containerID="5e0859d6ec428cdc7bbdb92828a1e3b94cbc254e837a2f28119c0d44e26c7571" exitCode=6
Jul 16 23:25:24.506965 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:24.506895    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-hjj4l" event={"ID":"0965d941-4625-4387-a83c-2cc1802675b8","Type":"ContainerDied","Data":"5e0859d6ec428cdc7bbdb92828a1e3b94cbc254e837a2f28119c0d44e26c7571"}
Jul 16 23:25:24.506965 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:24.506944    2573 scope.go:117] "RemoveContainer" containerID="44f077eb85cb32275cf4a9905c41371d458acbd35b0327d27e14a615bf092d13"
Jul 16 23:25:24.507056 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:24.507024    2573 scope.go:117] "RemoveContainer" containerID="5e0859d6ec428cdc7bbdb92828a1e3b94cbc254e837a2f28119c0d44e26c7571"
Jul 16 23:25:24.507268 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:25:24.507222    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-hjj4l_odh-ai-gateway-infra(0965d941-4625-4387-a83c-2cc1802675b8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-hjj4l" podUID="0965d941-4625-4387-a83c-2cc1802675b8"
Jul 16 23:25:25.181004 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:25.180953    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:25:25.513650 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:25.513586    2573 scope.go:117] "RemoveContainer" containerID="5e0859d6ec428cdc7bbdb92828a1e3b94cbc254e837a2f28119c0d44e26c7571"
Jul 16 23:25:25.513782 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:25:25.513720    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-hjj4l_odh-ai-gateway-infra(0965d941-4625-4387-a83c-2cc1802675b8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-hjj4l" podUID="0965d941-4625-4387-a83c-2cc1802675b8"
Jul 16 23:25:38.513775 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:38.513739    2573 scope.go:117] "RemoveContainer" containerID="5e0859d6ec428cdc7bbdb92828a1e3b94cbc254e837a2f28119c0d44e26c7571"
Jul 16 23:25:39.587319 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:39.587281    2573 generic.go:356] "Generic (PLEG): container finished" podID="0965d941-4625-4387-a83c-2cc1802675b8" containerID="8705d5f4e9dff00b82823eeeaed1e4f0fe2e54baac92e1e72304c012fcf16402" exitCode=6
Jul 16 23:25:39.587746 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:39.587332    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-hjj4l" event={"ID":"0965d941-4625-4387-a83c-2cc1802675b8","Type":"ContainerDied","Data":"8705d5f4e9dff00b82823eeeaed1e4f0fe2e54baac92e1e72304c012fcf16402"}
Jul 16 23:25:39.587746 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:39.587415    2573 scope.go:117] "RemoveContainer" containerID="5e0859d6ec428cdc7bbdb92828a1e3b94cbc254e837a2f28119c0d44e26c7571"
Jul 16 23:25:39.587746 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:39.587719    2573 scope.go:117] "RemoveContainer" containerID="8705d5f4e9dff00b82823eeeaed1e4f0fe2e54baac92e1e72304c012fcf16402"
Jul 16 23:25:39.587965 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:25:39.587944    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-hjj4l_odh-ai-gateway-infra(0965d941-4625-4387-a83c-2cc1802675b8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-hjj4l" podUID="0965d941-4625-4387-a83c-2cc1802675b8"
Jul 16 23:25:40.612480 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:40.612450    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-hjj4l"]
Jul 16 23:25:40.733823 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:40.733803    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-hjj4l"
Jul 16 23:25:40.899677 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:40.899621    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0965d941-4625-4387-a83c-2cc1802675b8-maas-api-service-ca\") pod \"0965d941-4625-4387-a83c-2cc1802675b8\" (UID: \"0965d941-4625-4387-a83c-2cc1802675b8\") "
Jul 16 23:25:40.899941 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:40.899920    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0965d941-4625-4387-a83c-2cc1802675b8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0965d941-4625-4387-a83c-2cc1802675b8" (UID: "0965d941-4625-4387-a83c-2cc1802675b8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:25:41.000999 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:41.000970    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0965d941-4625-4387-a83c-2cc1802675b8-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:25:41.599717 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:41.599689    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-hjj4l"
Jul 16 23:25:41.599898 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:41.599727    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-hjj4l" event={"ID":"0965d941-4625-4387-a83c-2cc1802675b8","Type":"ContainerDied","Data":"0e180424330a8288f52cfbceec697fc6d644e0c82b4360da7a08f51d549cf1f3"}
Jul 16 23:25:41.599898 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:41.599785    2573 scope.go:117] "RemoveContainer" containerID="8705d5f4e9dff00b82823eeeaed1e4f0fe2e54baac92e1e72304c012fcf16402"
Jul 16 23:25:41.648369 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:41.648342    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-hjj4l"]
Jul 16 23:25:41.666707 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:41.666681    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-hjj4l"]
Jul 16 23:25:42.518346 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:42.518315    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0965d941-4625-4387-a83c-2cc1802675b8" path="/var/lib/kubelet/pods/0965d941-4625-4387-a83c-2cc1802675b8/volumes"
Jul 16 23:25:43.438266 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:43.438212    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:25:59.428651 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:25:59.428614    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:26:12.175480 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:12.175395    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cmnrp"]
Jul 16 23:26:12.175926 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:12.175779    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13298302-d07c-4423-9c47-9a41a2941063" containerName="revoke-keys"
Jul 16 23:26:12.175926 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:12.175796    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="13298302-d07c-4423-9c47-9a41a2941063" containerName="revoke-keys"
Jul 16 23:26:12.175926 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:12.175814    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0965d941-4625-4387-a83c-2cc1802675b8" containerName="revoke-keys"
Jul 16 23:26:12.175926 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:12.175820    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0965d941-4625-4387-a83c-2cc1802675b8" containerName="revoke-keys"
Jul 16 23:26:12.175926 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:12.175827    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0965d941-4625-4387-a83c-2cc1802675b8" containerName="revoke-keys"
Jul 16 23:26:12.175926 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:12.175832    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0965d941-4625-4387-a83c-2cc1802675b8" containerName="revoke-keys"
Jul 16 23:26:12.175926 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:12.175841    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0965d941-4625-4387-a83c-2cc1802675b8" containerName="revoke-keys"
Jul 16 23:26:12.175926 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:12.175847    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0965d941-4625-4387-a83c-2cc1802675b8" containerName="revoke-keys"
Jul 16 23:26:12.175926 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:12.175899    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0965d941-4625-4387-a83c-2cc1802675b8" containerName="revoke-keys"
Jul 16 23:26:12.175926 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:12.175905    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0965d941-4625-4387-a83c-2cc1802675b8" containerName="revoke-keys"
Jul 16 23:26:12.175926 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:12.175912    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="13298302-d07c-4423-9c47-9a41a2941063" containerName="revoke-keys"
Jul 16 23:26:12.175926 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:12.175918    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0965d941-4625-4387-a83c-2cc1802675b8" containerName="revoke-keys"
Jul 16 23:26:12.178939 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:12.178919    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cmnrp"
Jul 16 23:26:12.181249 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:12.181207    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 23:26:12.181381 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:12.181363    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:26:12.185631 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:12.185604    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cmnrp"]
Jul 16 23:26:12.320881 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:12.320848    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42c30f2b-1ef5-4459-becd-12bb1c7bec0f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-cmnrp\" (UID: \"42c30f2b-1ef5-4459-becd-12bb1c7bec0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cmnrp"
Jul 16 23:26:12.422210 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:12.422174    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42c30f2b-1ef5-4459-becd-12bb1c7bec0f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-cmnrp\" (UID: \"42c30f2b-1ef5-4459-becd-12bb1c7bec0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cmnrp"
Jul 16 23:26:12.422806 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:12.422782    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42c30f2b-1ef5-4459-becd-12bb1c7bec0f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-cmnrp\" (UID: \"42c30f2b-1ef5-4459-becd-12bb1c7bec0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cmnrp"
Jul 16 23:26:12.490440 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:12.490380    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cmnrp"
Jul 16 23:26:12.617530 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:12.617507    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cmnrp"]
Jul 16 23:26:12.619591 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:26:12.619564    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42c30f2b_1ef5_4459_becd_12bb1c7bec0f.slice/crio-ae9daf990722bbbb4a264cbde5b93b5ef310bfc587680c98d1eb95721bfd00f1 WatchSource:0}: Error finding container ae9daf990722bbbb4a264cbde5b93b5ef310bfc587680c98d1eb95721bfd00f1: Status 404 returned error can't find the container with id ae9daf990722bbbb4a264cbde5b93b5ef310bfc587680c98d1eb95721bfd00f1
Jul 16 23:26:12.759654 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:12.759594    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cmnrp" event={"ID":"42c30f2b-1ef5-4459-becd-12bb1c7bec0f","Type":"ContainerStarted","Data":"ae9daf990722bbbb4a264cbde5b93b5ef310bfc587680c98d1eb95721bfd00f1"}
Jul 16 23:26:13.765534 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:13.765498    2573 generic.go:356] "Generic (PLEG): container finished" podID="42c30f2b-1ef5-4459-becd-12bb1c7bec0f" containerID="0000c812a6715b98e0fe99942d7ed6c4f609899fa41d715c0ed94ff0de4c66a2" exitCode=6
Jul 16 23:26:13.765965 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:13.765567    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cmnrp" event={"ID":"42c30f2b-1ef5-4459-becd-12bb1c7bec0f","Type":"ContainerDied","Data":"0000c812a6715b98e0fe99942d7ed6c4f609899fa41d715c0ed94ff0de4c66a2"}
Jul 16 23:26:13.765965 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:13.765762    2573 scope.go:117] "RemoveContainer" containerID="0000c812a6715b98e0fe99942d7ed6c4f609899fa41d715c0ed94ff0de4c66a2"
Jul 16 23:26:14.771948 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:14.771867    2573 generic.go:356] "Generic (PLEG): container finished" podID="42c30f2b-1ef5-4459-becd-12bb1c7bec0f" containerID="11f0ab26ae81704d7efc11c112ee42722f6409ea8ac1b3aa655d665011f59b17" exitCode=6
Jul 16 23:26:14.772413 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:14.771950    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cmnrp" event={"ID":"42c30f2b-1ef5-4459-becd-12bb1c7bec0f","Type":"ContainerDied","Data":"11f0ab26ae81704d7efc11c112ee42722f6409ea8ac1b3aa655d665011f59b17"}
Jul 16 23:26:14.772413 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:14.772004    2573 scope.go:117] "RemoveContainer" containerID="0000c812a6715b98e0fe99942d7ed6c4f609899fa41d715c0ed94ff0de4c66a2"
Jul 16 23:26:14.772413 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:14.772215    2573 scope.go:117] "RemoveContainer" containerID="11f0ab26ae81704d7efc11c112ee42722f6409ea8ac1b3aa655d665011f59b17"
Jul 16 23:26:14.772573 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:26:14.772428    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-cmnrp_odh-ai-gateway-infra(42c30f2b-1ef5-4459-becd-12bb1c7bec0f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cmnrp" podUID="42c30f2b-1ef5-4459-becd-12bb1c7bec0f"
Jul 16 23:26:15.778227 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:15.778199    2573 scope.go:117] "RemoveContainer" containerID="11f0ab26ae81704d7efc11c112ee42722f6409ea8ac1b3aa655d665011f59b17"
Jul 16 23:26:15.778637 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:26:15.778375    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-cmnrp_odh-ai-gateway-infra(42c30f2b-1ef5-4459-becd-12bb1c7bec0f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cmnrp" podUID="42c30f2b-1ef5-4459-becd-12bb1c7bec0f"
Jul 16 23:26:18.340823 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:18.340780    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:26:28.514001 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:28.513961    2573 scope.go:117] "RemoveContainer" containerID="11f0ab26ae81704d7efc11c112ee42722f6409ea8ac1b3aa655d665011f59b17"
Jul 16 23:26:28.840914 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:28.840887    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cmnrp" event={"ID":"42c30f2b-1ef5-4459-becd-12bb1c7bec0f","Type":"ContainerStarted","Data":"aee53e1c04e11284095756a148bea1ed0633accded7acbac3da4bedc3492756f"}
Jul 16 23:26:28.858437 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:28.858390    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cmnrp" podStartSLOduration=16.858375509 podStartE2EDuration="16.858375509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:26:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:26:28.856041001 +0000 UTC m=+3162.866303663" watchObservedRunningTime="2026-07-16 23:26:28.858375509 +0000 UTC m=+3162.868638172"
Jul 16 23:26:29.848576 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:29.848544    2573 generic.go:356] "Generic (PLEG): container finished" podID="42c30f2b-1ef5-4459-becd-12bb1c7bec0f" containerID="aee53e1c04e11284095756a148bea1ed0633accded7acbac3da4bedc3492756f" exitCode=6
Jul 16 23:26:29.849022 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:29.848618    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cmnrp" event={"ID":"42c30f2b-1ef5-4459-becd-12bb1c7bec0f","Type":"ContainerDied","Data":"aee53e1c04e11284095756a148bea1ed0633accded7acbac3da4bedc3492756f"}
Jul 16 23:26:29.849022 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:29.848677    2573 scope.go:117] "RemoveContainer" containerID="11f0ab26ae81704d7efc11c112ee42722f6409ea8ac1b3aa655d665011f59b17"
Jul 16 23:26:29.849022 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:29.848945    2573 scope.go:117] "RemoveContainer" containerID="aee53e1c04e11284095756a148bea1ed0633accded7acbac3da4bedc3492756f"
Jul 16 23:26:29.849207 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:26:29.849136    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-cmnrp_odh-ai-gateway-infra(42c30f2b-1ef5-4459-becd-12bb1c7bec0f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cmnrp" podUID="42c30f2b-1ef5-4459-becd-12bb1c7bec0f"
Jul 16 23:26:29.879706 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:29.879680    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cmnrp"]
Jul 16 23:26:30.982975 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:30.982948    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cmnrp"
Jul 16 23:26:31.054224 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:31.054200    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42c30f2b-1ef5-4459-becd-12bb1c7bec0f-maas-api-service-ca\") pod \"42c30f2b-1ef5-4459-becd-12bb1c7bec0f\" (UID: \"42c30f2b-1ef5-4459-becd-12bb1c7bec0f\") "
Jul 16 23:26:31.054516 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:31.054497    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42c30f2b-1ef5-4459-becd-12bb1c7bec0f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "42c30f2b-1ef5-4459-becd-12bb1c7bec0f" (UID: "42c30f2b-1ef5-4459-becd-12bb1c7bec0f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:26:31.155213 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:31.155156    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42c30f2b-1ef5-4459-becd-12bb1c7bec0f-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:26:31.860881 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:31.860845    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cmnrp" event={"ID":"42c30f2b-1ef5-4459-becd-12bb1c7bec0f","Type":"ContainerDied","Data":"ae9daf990722bbbb4a264cbde5b93b5ef310bfc587680c98d1eb95721bfd00f1"}
Jul 16 23:26:31.860881 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:31.860867    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cmnrp"
Jul 16 23:26:31.861130 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:31.860915    2573 scope.go:117] "RemoveContainer" containerID="aee53e1c04e11284095756a148bea1ed0633accded7acbac3da4bedc3492756f"
Jul 16 23:26:31.889883 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:31.889849    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cmnrp"]
Jul 16 23:26:31.891415 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:31.891392    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cmnrp"]
Jul 16 23:26:32.519022 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:32.518976    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42c30f2b-1ef5-4459-becd-12bb1c7bec0f" path="/var/lib/kubelet/pods/42c30f2b-1ef5-4459-becd-12bb1c7bec0f/volumes"
Jul 16 23:26:37.336646 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:37.336609    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:26:56.031071 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:56.031039    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:26:58.365641 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:58.365613    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-vl6nx"]
Jul 16 23:26:58.365999 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:58.365909    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42c30f2b-1ef5-4459-becd-12bb1c7bec0f" containerName="revoke-keys"
Jul 16 23:26:58.365999 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:58.365919    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="42c30f2b-1ef5-4459-becd-12bb1c7bec0f" containerName="revoke-keys"
Jul 16 23:26:58.365999 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:58.365941    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42c30f2b-1ef5-4459-becd-12bb1c7bec0f" containerName="revoke-keys"
Jul 16 23:26:58.365999 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:58.365947    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="42c30f2b-1ef5-4459-becd-12bb1c7bec0f" containerName="revoke-keys"
Jul 16 23:26:58.365999 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:58.365988    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="42c30f2b-1ef5-4459-becd-12bb1c7bec0f" containerName="revoke-keys"
Jul 16 23:26:58.365999 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:58.365996    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="42c30f2b-1ef5-4459-becd-12bb1c7bec0f" containerName="revoke-keys"
Jul 16 23:26:58.368616 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:58.368600    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-vl6nx"
Jul 16 23:26:58.370715 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:58.370697    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 23:26:58.370816 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:58.370702    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:26:58.376460 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:58.376439    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-vl6nx"]
Jul 16 23:26:58.447839 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:58.447817    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ae0eb69-c8cb-4aff-94d0-40cf57ad37ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-vl6nx\" (UID: \"8ae0eb69-c8cb-4aff-94d0-40cf57ad37ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-vl6nx"
Jul 16 23:26:58.548390 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:58.548367    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ae0eb69-c8cb-4aff-94d0-40cf57ad37ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-vl6nx\" (UID: \"8ae0eb69-c8cb-4aff-94d0-40cf57ad37ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-vl6nx"
Jul 16 23:26:58.548935 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:58.548915    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ae0eb69-c8cb-4aff-94d0-40cf57ad37ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-vl6nx\" (UID: \"8ae0eb69-c8cb-4aff-94d0-40cf57ad37ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-vl6nx"
Jul 16 23:26:58.679429 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:58.679381    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-vl6nx"
Jul 16 23:26:58.799266 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:58.799224    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-vl6nx"]
Jul 16 23:26:58.801019 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:26:58.800995    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ae0eb69_c8cb_4aff_94d0_40cf57ad37ad.slice/crio-7ca62443829419f806800bb10172d325acd196de6a6cfd89f18fe3cadaf8d7c2 WatchSource:0}: Error finding container 7ca62443829419f806800bb10172d325acd196de6a6cfd89f18fe3cadaf8d7c2: Status 404 returned error can't find the container with id 7ca62443829419f806800bb10172d325acd196de6a6cfd89f18fe3cadaf8d7c2
Jul 16 23:26:58.989384 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:58.989355    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-vl6nx" event={"ID":"8ae0eb69-c8cb-4aff-94d0-40cf57ad37ad","Type":"ContainerStarted","Data":"7ca62443829419f806800bb10172d325acd196de6a6cfd89f18fe3cadaf8d7c2"}
Jul 16 23:26:59.995312 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:59.995279    2573 generic.go:356] "Generic (PLEG): container finished" podID="8ae0eb69-c8cb-4aff-94d0-40cf57ad37ad" containerID="4a3aea1db07afc9f4cca841447bcd934a5b19afc4e8380da1de121d1c051f619" exitCode=6
Jul 16 23:26:59.995659 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:59.995366    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-vl6nx" event={"ID":"8ae0eb69-c8cb-4aff-94d0-40cf57ad37ad","Type":"ContainerDied","Data":"4a3aea1db07afc9f4cca841447bcd934a5b19afc4e8380da1de121d1c051f619"}
Jul 16 23:26:59.995659 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:26:59.995509    2573 scope.go:117] "RemoveContainer" containerID="4a3aea1db07afc9f4cca841447bcd934a5b19afc4e8380da1de121d1c051f619"
Jul 16 23:27:01.002513 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:01.002437    2573 generic.go:356] "Generic (PLEG): container finished" podID="8ae0eb69-c8cb-4aff-94d0-40cf57ad37ad" containerID="122d098edc2f633d0a0a766a57f0b120b3daca6e5dc02f227e4d179960c42c8b" exitCode=6
Jul 16 23:27:01.002911 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:01.002520    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-vl6nx" event={"ID":"8ae0eb69-c8cb-4aff-94d0-40cf57ad37ad","Type":"ContainerDied","Data":"122d098edc2f633d0a0a766a57f0b120b3daca6e5dc02f227e4d179960c42c8b"}
Jul 16 23:27:01.002911 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:01.002580    2573 scope.go:117] "RemoveContainer" containerID="4a3aea1db07afc9f4cca841447bcd934a5b19afc4e8380da1de121d1c051f619"
Jul 16 23:27:01.002911 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:01.002651    2573 scope.go:117] "RemoveContainer" containerID="122d098edc2f633d0a0a766a57f0b120b3daca6e5dc02f227e4d179960c42c8b"
Jul 16 23:27:01.002911 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:27:01.002809    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-vl6nx_odh-ai-gateway-infra(8ae0eb69-c8cb-4aff-94d0-40cf57ad37ad)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-vl6nx" podUID="8ae0eb69-c8cb-4aff-94d0-40cf57ad37ad"
Jul 16 23:27:02.009055 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:02.009028    2573 scope.go:117] "RemoveContainer" containerID="122d098edc2f633d0a0a766a57f0b120b3daca6e5dc02f227e4d179960c42c8b"
Jul 16 23:27:02.009450 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:27:02.009195    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-vl6nx_odh-ai-gateway-infra(8ae0eb69-c8cb-4aff-94d0-40cf57ad37ad)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-vl6nx" podUID="8ae0eb69-c8cb-4aff-94d0-40cf57ad37ad"
Jul 16 23:27:02.211335 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:02.211303    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-8xjkd"]
Jul 16 23:27:02.211667 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:02.211650    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42c30f2b-1ef5-4459-becd-12bb1c7bec0f" containerName="revoke-keys"
Jul 16 23:27:02.211749 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:02.211669    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="42c30f2b-1ef5-4459-becd-12bb1c7bec0f" containerName="revoke-keys"
Jul 16 23:27:02.211811 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:02.211756    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="42c30f2b-1ef5-4459-becd-12bb1c7bec0f" containerName="revoke-keys"
Jul 16 23:27:02.215625 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:02.215607    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-8xjkd"
Jul 16 23:27:02.220938 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:02.220920    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-8xjkd"]
Jul 16 23:27:02.276141 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:02.276085    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26594767-abbe-4679-b843-ecd1b3e56843-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-8xjkd\" (UID: \"26594767-abbe-4679-b843-ecd1b3e56843\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-8xjkd"
Jul 16 23:27:02.376743 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:02.376719    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26594767-abbe-4679-b843-ecd1b3e56843-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-8xjkd\" (UID: \"26594767-abbe-4679-b843-ecd1b3e56843\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-8xjkd"
Jul 16 23:27:02.377316 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:02.377298    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26594767-abbe-4679-b843-ecd1b3e56843-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-8xjkd\" (UID: \"26594767-abbe-4679-b843-ecd1b3e56843\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-8xjkd"
Jul 16 23:27:02.526333 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:02.526281    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-8xjkd"
Jul 16 23:27:02.650386 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:02.650364    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-8xjkd"]
Jul 16 23:27:02.652815 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:27:02.652788    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26594767_abbe_4679_b843_ecd1b3e56843.slice/crio-35fa8cdd6eddc999b914c66aeabbd5ef2d9a2aee2acc269420e4fce30b3fc2fc WatchSource:0}: Error finding container 35fa8cdd6eddc999b914c66aeabbd5ef2d9a2aee2acc269420e4fce30b3fc2fc: Status 404 returned error can't find the container with id 35fa8cdd6eddc999b914c66aeabbd5ef2d9a2aee2acc269420e4fce30b3fc2fc
Jul 16 23:27:03.015003 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:03.014974    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-8xjkd" event={"ID":"26594767-abbe-4679-b843-ecd1b3e56843","Type":"ContainerStarted","Data":"7c571597187eae571c598c0cb549e5c48eeb2b1c0f9a4dba86e31520e4af0a84"}
Jul 16 23:27:03.015360 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:03.015020    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-8xjkd" event={"ID":"26594767-abbe-4679-b843-ecd1b3e56843","Type":"ContainerStarted","Data":"35fa8cdd6eddc999b914c66aeabbd5ef2d9a2aee2acc269420e4fce30b3fc2fc"}
Jul 16 23:27:03.030325 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:03.030282    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-8xjkd" podStartSLOduration=1.030268983 podStartE2EDuration="1.030268983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:27:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:27:03.028311872 +0000 UTC m=+3197.038574557" watchObservedRunningTime="2026-07-16 23:27:03.030268983 +0000 UTC m=+3197.040531646"
Jul 16 23:27:04.020829 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:04.020796    2573 generic.go:356] "Generic (PLEG): container finished" podID="26594767-abbe-4679-b843-ecd1b3e56843" containerID="7c571597187eae571c598c0cb549e5c48eeb2b1c0f9a4dba86e31520e4af0a84" exitCode=6
Jul 16 23:27:04.021192 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:04.020880    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-8xjkd" event={"ID":"26594767-abbe-4679-b843-ecd1b3e56843","Type":"ContainerDied","Data":"7c571597187eae571c598c0cb549e5c48eeb2b1c0f9a4dba86e31520e4af0a84"}
Jul 16 23:27:04.021192 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:04.020984    2573 scope.go:117] "RemoveContainer" containerID="7c571597187eae571c598c0cb549e5c48eeb2b1c0f9a4dba86e31520e4af0a84"
Jul 16 23:27:05.027822 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:05.027740    2573 generic.go:356] "Generic (PLEG): container finished" podID="26594767-abbe-4679-b843-ecd1b3e56843" containerID="48584e537bbca8aecb3034bcfabcff91a1dfea1e7d98dfa38ed90b54a9e9e0ef" exitCode=6
Jul 16 23:27:05.028230 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:05.027828    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-8xjkd" event={"ID":"26594767-abbe-4679-b843-ecd1b3e56843","Type":"ContainerDied","Data":"48584e537bbca8aecb3034bcfabcff91a1dfea1e7d98dfa38ed90b54a9e9e0ef"}
Jul 16 23:27:05.028230 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:05.027881    2573 scope.go:117] "RemoveContainer" containerID="7c571597187eae571c598c0cb549e5c48eeb2b1c0f9a4dba86e31520e4af0a84"
Jul 16 23:27:05.028230 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:05.027962    2573 scope.go:117] "RemoveContainer" containerID="48584e537bbca8aecb3034bcfabcff91a1dfea1e7d98dfa38ed90b54a9e9e0ef"
Jul 16 23:27:05.028230 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:27:05.028118    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-8xjkd_odh-ai-gateway-infra(26594767-abbe-4679-b843-ecd1b3e56843)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-8xjkd" podUID="26594767-abbe-4679-b843-ecd1b3e56843"
Jul 16 23:27:06.034578 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:06.034553    2573 scope.go:117] "RemoveContainer" containerID="48584e537bbca8aecb3034bcfabcff91a1dfea1e7d98dfa38ed90b54a9e9e0ef"
Jul 16 23:27:06.034925 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:27:06.034687    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-8xjkd_odh-ai-gateway-infra(26594767-abbe-4679-b843-ecd1b3e56843)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-8xjkd" podUID="26594767-abbe-4679-b843-ecd1b3e56843"
Jul 16 23:27:11.748359 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:11.748316    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-48992b/auth-test-model-48992b-kserve-fd54db7f-s4rwh"]
Jul 16 23:27:11.799483 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:11.799449    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:27:12.513726 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:12.513697    2573 scope.go:117] "RemoveContainer" containerID="122d098edc2f633d0a0a766a57f0b120b3daca6e5dc02f227e4d179960c42c8b"
Jul 16 23:27:12.643319 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:12.643286    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b4ab52/auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq"]
Jul 16 23:27:12.662883 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:12.662852    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:27:13.067336 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:13.067304    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-vl6nx" event={"ID":"8ae0eb69-c8cb-4aff-94d0-40cf57ad37ad","Type":"ContainerStarted","Data":"9df44fe11fd7e2928f2b890fd4451e539349759876431fa6abf4b50a7d4aced5"}
Jul 16 23:27:13.084080 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:13.084037    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-vl6nx" podStartSLOduration=15.084023017 podStartE2EDuration="15.084023017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:26:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:27:13.081854734 +0000 UTC m=+3207.092117399" watchObservedRunningTime="2026-07-16 23:27:13.084023017 +0000 UTC m=+3207.094285678"
Jul 16 23:27:14.073539 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:14.073509    2573 generic.go:356] "Generic (PLEG): container finished" podID="8ae0eb69-c8cb-4aff-94d0-40cf57ad37ad" containerID="9df44fe11fd7e2928f2b890fd4451e539349759876431fa6abf4b50a7d4aced5" exitCode=6
Jul 16 23:27:14.073880 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:14.073561    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-vl6nx" event={"ID":"8ae0eb69-c8cb-4aff-94d0-40cf57ad37ad","Type":"ContainerDied","Data":"9df44fe11fd7e2928f2b890fd4451e539349759876431fa6abf4b50a7d4aced5"}
Jul 16 23:27:14.073880 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:14.073638    2573 scope.go:117] "RemoveContainer" containerID="122d098edc2f633d0a0a766a57f0b120b3daca6e5dc02f227e4d179960c42c8b"
Jul 16 23:27:14.073993 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:14.073886    2573 scope.go:117] "RemoveContainer" containerID="9df44fe11fd7e2928f2b890fd4451e539349759876431fa6abf4b50a7d4aced5"
Jul 16 23:27:14.074086 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:27:14.074066    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-vl6nx_odh-ai-gateway-infra(8ae0eb69-c8cb-4aff-94d0-40cf57ad37ad)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-vl6nx" podUID="8ae0eb69-c8cb-4aff-94d0-40cf57ad37ad"
Jul 16 23:27:14.098786 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:14.098762    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-vl6nx"]
Jul 16 23:27:14.529216 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:14.529149    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:27:15.211414 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:15.211392    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-vl6nx"
Jul 16 23:27:15.266199 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:15.266169    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ae0eb69-c8cb-4aff-94d0-40cf57ad37ad-maas-api-service-ca\") pod \"8ae0eb69-c8cb-4aff-94d0-40cf57ad37ad\" (UID: \"8ae0eb69-c8cb-4aff-94d0-40cf57ad37ad\") "
Jul 16 23:27:15.266505 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:15.266485    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ae0eb69-c8cb-4aff-94d0-40cf57ad37ad-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8ae0eb69-c8cb-4aff-94d0-40cf57ad37ad" (UID: "8ae0eb69-c8cb-4aff-94d0-40cf57ad37ad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:27:15.367487 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:15.367428    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ae0eb69-c8cb-4aff-94d0-40cf57ad37ad-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:27:16.086422 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:16.086381    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-vl6nx" event={"ID":"8ae0eb69-c8cb-4aff-94d0-40cf57ad37ad","Type":"ContainerDied","Data":"7ca62443829419f806800bb10172d325acd196de6a6cfd89f18fe3cadaf8d7c2"}
Jul 16 23:27:16.086422 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:16.086409    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-vl6nx"
Jul 16 23:27:16.086660 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:16.086455    2573 scope.go:117] "RemoveContainer" containerID="9df44fe11fd7e2928f2b890fd4451e539349759876431fa6abf4b50a7d4aced5"
Jul 16 23:27:16.108422 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:16.108394    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-vl6nx"]
Jul 16 23:27:16.113862 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:16.113841    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-vl6nx"]
Jul 16 23:27:16.518353 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:16.518297    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ae0eb69-c8cb-4aff-94d0-40cf57ad37ad" path="/var/lib/kubelet/pods/8ae0eb69-c8cb-4aff-94d0-40cf57ad37ad/volumes"
Jul 16 23:27:20.513784 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:20.513755    2573 scope.go:117] "RemoveContainer" containerID="48584e537bbca8aecb3034bcfabcff91a1dfea1e7d98dfa38ed90b54a9e9e0ef"
Jul 16 23:27:21.112640 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:21.112612    2573 generic.go:356] "Generic (PLEG): container finished" podID="26594767-abbe-4679-b843-ecd1b3e56843" containerID="cb8e811167edb41a8b9ecef2de8282e4ce07f49f0a13fda47943f0d6eca9eec1" exitCode=6
Jul 16 23:27:21.112760 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:21.112681    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-8xjkd" event={"ID":"26594767-abbe-4679-b843-ecd1b3e56843","Type":"ContainerDied","Data":"cb8e811167edb41a8b9ecef2de8282e4ce07f49f0a13fda47943f0d6eca9eec1"}
Jul 16 23:27:21.112760 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:21.112746    2573 scope.go:117] "RemoveContainer" containerID="48584e537bbca8aecb3034bcfabcff91a1dfea1e7d98dfa38ed90b54a9e9e0ef"
Jul 16 23:27:21.113046 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:21.113027    2573 scope.go:117] "RemoveContainer" containerID="cb8e811167edb41a8b9ecef2de8282e4ce07f49f0a13fda47943f0d6eca9eec1"
Jul 16 23:27:21.113231 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:27:21.113212    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-8xjkd_odh-ai-gateway-infra(26594767-abbe-4679-b843-ecd1b3e56843)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-8xjkd" podUID="26594767-abbe-4679-b843-ecd1b3e56843"
Jul 16 23:27:21.143609 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:21.143580    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-nwh4c"]
Jul 16 23:27:21.143882 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:21.143870    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ae0eb69-c8cb-4aff-94d0-40cf57ad37ad" containerName="revoke-keys"
Jul 16 23:27:21.143924 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:21.143883    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ae0eb69-c8cb-4aff-94d0-40cf57ad37ad" containerName="revoke-keys"
Jul 16 23:27:21.143924 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:21.143890    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ae0eb69-c8cb-4aff-94d0-40cf57ad37ad" containerName="revoke-keys"
Jul 16 23:27:21.143924 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:21.143896    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ae0eb69-c8cb-4aff-94d0-40cf57ad37ad" containerName="revoke-keys"
Jul 16 23:27:21.143924 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:21.143912    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ae0eb69-c8cb-4aff-94d0-40cf57ad37ad" containerName="revoke-keys"
Jul 16 23:27:21.143924 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:21.143917    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ae0eb69-c8cb-4aff-94d0-40cf57ad37ad" containerName="revoke-keys"
Jul 16 23:27:21.144073 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:21.143967    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ae0eb69-c8cb-4aff-94d0-40cf57ad37ad" containerName="revoke-keys"
Jul 16 23:27:21.144073 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:21.143974    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ae0eb69-c8cb-4aff-94d0-40cf57ad37ad" containerName="revoke-keys"
Jul 16 23:27:21.146704 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:21.146691    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-nwh4c"
Jul 16 23:27:21.154150 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:21.154130    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-nwh4c"]
Jul 16 23:27:21.207229 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:21.207201    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/932db21a-2c5c-4e48-9be2-e3cf473f7e34-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-nwh4c\" (UID: \"932db21a-2c5c-4e48-9be2-e3cf473f7e34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-nwh4c"
Jul 16 23:27:21.308496 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:21.308473    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/932db21a-2c5c-4e48-9be2-e3cf473f7e34-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-nwh4c\" (UID: \"932db21a-2c5c-4e48-9be2-e3cf473f7e34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-nwh4c"
Jul 16 23:27:21.309019 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:21.309001    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/932db21a-2c5c-4e48-9be2-e3cf473f7e34-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-nwh4c\" (UID: \"932db21a-2c5c-4e48-9be2-e3cf473f7e34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-nwh4c"
Jul 16 23:27:21.465416 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:21.465361    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-nwh4c"
Jul 16 23:27:21.589745 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:21.589720    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-nwh4c"]
Jul 16 23:27:21.591460 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:27:21.591430    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod932db21a_2c5c_4e48_9be2_e3cf473f7e34.slice/crio-754b57a7acf99997fa29d0637b77e45af460dd9087f362865244b8c0d9f7a486 WatchSource:0}: Error finding container 754b57a7acf99997fa29d0637b77e45af460dd9087f362865244b8c0d9f7a486: Status 404 returned error can't find the container with id 754b57a7acf99997fa29d0637b77e45af460dd9087f362865244b8c0d9f7a486
Jul 16 23:27:22.119968 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:22.119942    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-nwh4c" event={"ID":"932db21a-2c5c-4e48-9be2-e3cf473f7e34","Type":"ContainerStarted","Data":"efb650ac571a83e201e0798737c27553f1d1b67974f35a8b700889a61a2ed4c6"}
Jul 16 23:27:22.120108 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:22.119991    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-nwh4c" event={"ID":"932db21a-2c5c-4e48-9be2-e3cf473f7e34","Type":"ContainerStarted","Data":"754b57a7acf99997fa29d0637b77e45af460dd9087f362865244b8c0d9f7a486"}
Jul 16 23:27:22.137761 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:22.137734    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-8xjkd"]
Jul 16 23:27:22.139363 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:22.139328    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-nwh4c" podStartSLOduration=1.13931397 podStartE2EDuration="1.13931397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:27:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:27:22.13757536 +0000 UTC m=+3216.147838024" watchObservedRunningTime="2026-07-16 23:27:22.13931397 +0000 UTC m=+3216.149576633"
Jul 16 23:27:22.275494 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:22.275475    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-8xjkd"
Jul 16 23:27:22.416814 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:22.416756    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26594767-abbe-4679-b843-ecd1b3e56843-maas-api-service-ca\") pod \"26594767-abbe-4679-b843-ecd1b3e56843\" (UID: \"26594767-abbe-4679-b843-ecd1b3e56843\") "
Jul 16 23:27:22.417066 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:22.417046    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26594767-abbe-4679-b843-ecd1b3e56843-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "26594767-abbe-4679-b843-ecd1b3e56843" (UID: "26594767-abbe-4679-b843-ecd1b3e56843"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:27:22.517682 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:22.517661    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26594767-abbe-4679-b843-ecd1b3e56843-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:27:22.917926 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:22.917897    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-48992b/sub-test-model-48992b-kserve-7d764f4ff5-dlnl8"]
Jul 16 23:27:22.918378 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:22.918358    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26594767-abbe-4679-b843-ecd1b3e56843" containerName="revoke-keys"
Jul 16 23:27:22.918461 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:22.918383    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="26594767-abbe-4679-b843-ecd1b3e56843" containerName="revoke-keys"
Jul 16 23:27:22.918461 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:22.918407    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26594767-abbe-4679-b843-ecd1b3e56843" containerName="revoke-keys"
Jul 16 23:27:22.918461 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:22.918416    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="26594767-abbe-4679-b843-ecd1b3e56843" containerName="revoke-keys"
Jul 16 23:27:22.918624 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:22.918497    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="26594767-abbe-4679-b843-ecd1b3e56843" containerName="revoke-keys"
Jul 16 23:27:22.918624 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:22.918510    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="26594767-abbe-4679-b843-ecd1b3e56843" containerName="revoke-keys"
Jul 16 23:27:22.918624 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:22.918519    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="26594767-abbe-4679-b843-ecd1b3e56843" containerName="revoke-keys"
Jul 16 23:27:22.918624 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:22.918530    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ae0eb69-c8cb-4aff-94d0-40cf57ad37ad" containerName="revoke-keys"
Jul 16 23:27:22.921514 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:22.921496    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-48992b/sub-test-model-48992b-kserve-7d764f4ff5-dlnl8"
Jul 16 23:27:22.923918 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:22.923899    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-48992b\"/\"sub-test-model-48992b-kserve-self-signed-certs\""
Jul 16 23:27:22.932759 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:22.932737    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-48992b/sub-test-model-48992b-kserve-7d764f4ff5-dlnl8"]
Jul 16 23:27:22.950753 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:22.950722    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:27:23.021629 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:23.021606    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08491056-c0d1-4e68-8d64-d1ffd7fd5c52-home\") pod \"sub-test-model-48992b-kserve-7d764f4ff5-dlnl8\" (UID: \"08491056-c0d1-4e68-8d64-d1ffd7fd5c52\") " pod="ai-tenant-e2e-shared-a-48992b/sub-test-model-48992b-kserve-7d764f4ff5-dlnl8"
Jul 16 23:27:23.021719 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:23.021639    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flx9m\" (UniqueName: \"kubernetes.io/projected/08491056-c0d1-4e68-8d64-d1ffd7fd5c52-kube-api-access-flx9m\") pod \"sub-test-model-48992b-kserve-7d764f4ff5-dlnl8\" (UID: \"08491056-c0d1-4e68-8d64-d1ffd7fd5c52\") " pod="ai-tenant-e2e-shared-a-48992b/sub-test-model-48992b-kserve-7d764f4ff5-dlnl8"
Jul 16 23:27:23.021719 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:23.021658    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08491056-c0d1-4e68-8d64-d1ffd7fd5c52-tls-certs\") pod \"sub-test-model-48992b-kserve-7d764f4ff5-dlnl8\" (UID: \"08491056-c0d1-4e68-8d64-d1ffd7fd5c52\") " pod="ai-tenant-e2e-shared-a-48992b/sub-test-model-48992b-kserve-7d764f4ff5-dlnl8"
Jul 16 23:27:23.021798 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:23.021747    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08491056-c0d1-4e68-8d64-d1ffd7fd5c52-model-cache\") pod \"sub-test-model-48992b-kserve-7d764f4ff5-dlnl8\" (UID: \"08491056-c0d1-4e68-8d64-d1ffd7fd5c52\") " pod="ai-tenant-e2e-shared-a-48992b/sub-test-model-48992b-kserve-7d764f4ff5-dlnl8"
Jul 16 23:27:23.021798 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:23.021787    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08491056-c0d1-4e68-8d64-d1ffd7fd5c52-tmp-dir\") pod \"sub-test-model-48992b-kserve-7d764f4ff5-dlnl8\" (UID: \"08491056-c0d1-4e68-8d64-d1ffd7fd5c52\") " pod="ai-tenant-e2e-shared-a-48992b/sub-test-model-48992b-kserve-7d764f4ff5-dlnl8"
Jul 16 23:27:23.021873 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:23.021816    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08491056-c0d1-4e68-8d64-d1ffd7fd5c52-dshm\") pod \"sub-test-model-48992b-kserve-7d764f4ff5-dlnl8\" (UID: \"08491056-c0d1-4e68-8d64-d1ffd7fd5c52\") " pod="ai-tenant-e2e-shared-a-48992b/sub-test-model-48992b-kserve-7d764f4ff5-dlnl8"
Jul 16 23:27:23.122685 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:23.122665    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08491056-c0d1-4e68-8d64-d1ffd7fd5c52-model-cache\") pod \"sub-test-model-48992b-kserve-7d764f4ff5-dlnl8\" (UID: \"08491056-c0d1-4e68-8d64-d1ffd7fd5c52\") " pod="ai-tenant-e2e-shared-a-48992b/sub-test-model-48992b-kserve-7d764f4ff5-dlnl8"
Jul 16 23:27:23.122802 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:23.122700    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08491056-c0d1-4e68-8d64-d1ffd7fd5c52-tmp-dir\") pod \"sub-test-model-48992b-kserve-7d764f4ff5-dlnl8\" (UID: \"08491056-c0d1-4e68-8d64-d1ffd7fd5c52\") " pod="ai-tenant-e2e-shared-a-48992b/sub-test-model-48992b-kserve-7d764f4ff5-dlnl8"
Jul 16 23:27:23.122802 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:23.122721    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08491056-c0d1-4e68-8d64-d1ffd7fd5c52-dshm\") pod \"sub-test-model-48992b-kserve-7d764f4ff5-dlnl8\" (UID: \"08491056-c0d1-4e68-8d64-d1ffd7fd5c52\") " pod="ai-tenant-e2e-shared-a-48992b/sub-test-model-48992b-kserve-7d764f4ff5-dlnl8"
Jul 16 23:27:23.122802 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:23.122750    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08491056-c0d1-4e68-8d64-d1ffd7fd5c52-home\") pod \"sub-test-model-48992b-kserve-7d764f4ff5-dlnl8\" (UID: \"08491056-c0d1-4e68-8d64-d1ffd7fd5c52\") " pod="ai-tenant-e2e-shared-a-48992b/sub-test-model-48992b-kserve-7d764f4ff5-dlnl8"
Jul 16 23:27:23.122802 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:23.122773    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flx9m\" (UniqueName: \"kubernetes.io/projected/08491056-c0d1-4e68-8d64-d1ffd7fd5c52-kube-api-access-flx9m\") pod \"sub-test-model-48992b-kserve-7d764f4ff5-dlnl8\" (UID: \"08491056-c0d1-4e68-8d64-d1ffd7fd5c52\") " pod="ai-tenant-e2e-shared-a-48992b/sub-test-model-48992b-kserve-7d764f4ff5-dlnl8"
Jul 16 23:27:23.122802 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:23.122791    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08491056-c0d1-4e68-8d64-d1ffd7fd5c52-tls-certs\") pod \"sub-test-model-48992b-kserve-7d764f4ff5-dlnl8\" (UID: \"08491056-c0d1-4e68-8d64-d1ffd7fd5c52\") " pod="ai-tenant-e2e-shared-a-48992b/sub-test-model-48992b-kserve-7d764f4ff5-dlnl8"
Jul 16 23:27:23.123099 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:23.123075    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08491056-c0d1-4e68-8d64-d1ffd7fd5c52-model-cache\") pod \"sub-test-model-48992b-kserve-7d764f4ff5-dlnl8\" (UID: \"08491056-c0d1-4e68-8d64-d1ffd7fd5c52\") " pod="ai-tenant-e2e-shared-a-48992b/sub-test-model-48992b-kserve-7d764f4ff5-dlnl8"
Jul 16 23:27:23.123215 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:23.123155    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08491056-c0d1-4e68-8d64-d1ffd7fd5c52-tmp-dir\") pod \"sub-test-model-48992b-kserve-7d764f4ff5-dlnl8\" (UID: \"08491056-c0d1-4e68-8d64-d1ffd7fd5c52\") " pod="ai-tenant-e2e-shared-a-48992b/sub-test-model-48992b-kserve-7d764f4ff5-dlnl8"
Jul 16 23:27:23.123318 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:23.123250    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08491056-c0d1-4e68-8d64-d1ffd7fd5c52-home\") pod \"sub-test-model-48992b-kserve-7d764f4ff5-dlnl8\" (UID: \"08491056-c0d1-4e68-8d64-d1ffd7fd5c52\") " pod="ai-tenant-e2e-shared-a-48992b/sub-test-model-48992b-kserve-7d764f4ff5-dlnl8"
Jul 16 23:27:23.125068 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:23.125049    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08491056-c0d1-4e68-8d64-d1ffd7fd5c52-dshm\") pod \"sub-test-model-48992b-kserve-7d764f4ff5-dlnl8\" (UID: \"08491056-c0d1-4e68-8d64-d1ffd7fd5c52\") " pod="ai-tenant-e2e-shared-a-48992b/sub-test-model-48992b-kserve-7d764f4ff5-dlnl8"
Jul 16 23:27:23.125424 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:23.125400    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08491056-c0d1-4e68-8d64-d1ffd7fd5c52-tls-certs\") pod \"sub-test-model-48992b-kserve-7d764f4ff5-dlnl8\" (UID: \"08491056-c0d1-4e68-8d64-d1ffd7fd5c52\") " pod="ai-tenant-e2e-shared-a-48992b/sub-test-model-48992b-kserve-7d764f4ff5-dlnl8"
Jul 16 23:27:23.126665 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:23.126646    2573 generic.go:356] "Generic (PLEG): container finished" podID="932db21a-2c5c-4e48-9be2-e3cf473f7e34" containerID="efb650ac571a83e201e0798737c27553f1d1b67974f35a8b700889a61a2ed4c6" exitCode=6
Jul 16 23:27:23.126758 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:23.126714    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-nwh4c" event={"ID":"932db21a-2c5c-4e48-9be2-e3cf473f7e34","Type":"ContainerDied","Data":"efb650ac571a83e201e0798737c27553f1d1b67974f35a8b700889a61a2ed4c6"}
Jul 16 23:27:23.126876 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:23.126858    2573 scope.go:117] "RemoveContainer" containerID="efb650ac571a83e201e0798737c27553f1d1b67974f35a8b700889a61a2ed4c6"
Jul 16 23:27:23.128142 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:23.128123    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-8xjkd"
Jul 16 23:27:23.128251 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:23.128151    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-8xjkd" event={"ID":"26594767-abbe-4679-b843-ecd1b3e56843","Type":"ContainerDied","Data":"35fa8cdd6eddc999b914c66aeabbd5ef2d9a2aee2acc269420e4fce30b3fc2fc"}
Jul 16 23:27:23.128251 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:23.128197    2573 scope.go:117] "RemoveContainer" containerID="cb8e811167edb41a8b9ecef2de8282e4ce07f49f0a13fda47943f0d6eca9eec1"
Jul 16 23:27:23.136808 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:23.132580    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flx9m\" (UniqueName: \"kubernetes.io/projected/08491056-c0d1-4e68-8d64-d1ffd7fd5c52-kube-api-access-flx9m\") pod \"sub-test-model-48992b-kserve-7d764f4ff5-dlnl8\" (UID: \"08491056-c0d1-4e68-8d64-d1ffd7fd5c52\") " pod="ai-tenant-e2e-shared-a-48992b/sub-test-model-48992b-kserve-7d764f4ff5-dlnl8"
Jul 16 23:27:23.174535 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:23.174479    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-8xjkd"]
Jul 16 23:27:23.179062 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:23.179038    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-8xjkd"]
Jul 16 23:27:23.230191 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:23.230160    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-48992b/sub-test-model-48992b-kserve-7d764f4ff5-dlnl8"
Jul 16 23:27:23.357066 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:23.357041    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-48992b/sub-test-model-48992b-kserve-7d764f4ff5-dlnl8"]
Jul 16 23:27:23.358534 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:27:23.358512    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08491056_c0d1_4e68_8d64_d1ffd7fd5c52.slice/crio-211812485ca86da1307f5694adde91055ba92a5a4aa7cf2aded954ad95ba5927 WatchSource:0}: Error finding container 211812485ca86da1307f5694adde91055ba92a5a4aa7cf2aded954ad95ba5927: Status 404 returned error can't find the container with id 211812485ca86da1307f5694adde91055ba92a5a4aa7cf2aded954ad95ba5927
Jul 16 23:27:24.135724 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:24.135694    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-48992b/sub-test-model-48992b-kserve-7d764f4ff5-dlnl8" event={"ID":"08491056-c0d1-4e68-8d64-d1ffd7fd5c52","Type":"ContainerStarted","Data":"b6a0f21fc1b0a08d6a676c13041a6a1adddbbe60ae1bee336854279b85450994"}
Jul 16 23:27:24.136065 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:24.135738    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-48992b/sub-test-model-48992b-kserve-7d764f4ff5-dlnl8" event={"ID":"08491056-c0d1-4e68-8d64-d1ffd7fd5c52","Type":"ContainerStarted","Data":"211812485ca86da1307f5694adde91055ba92a5a4aa7cf2aded954ad95ba5927"}
Jul 16 23:27:24.137413 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:24.137393    2573 generic.go:356] "Generic (PLEG): container finished" podID="932db21a-2c5c-4e48-9be2-e3cf473f7e34" containerID="ca1e073d95d358be1958672d5461ac6946a488da832c68f471c8492ab783de24" exitCode=6
Jul 16 23:27:24.137513 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:24.137431    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-nwh4c" event={"ID":"932db21a-2c5c-4e48-9be2-e3cf473f7e34","Type":"ContainerDied","Data":"ca1e073d95d358be1958672d5461ac6946a488da832c68f471c8492ab783de24"}
Jul 16 23:27:24.137513 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:24.137473    2573 scope.go:117] "RemoveContainer" containerID="efb650ac571a83e201e0798737c27553f1d1b67974f35a8b700889a61a2ed4c6"
Jul 16 23:27:24.137622 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:24.137606    2573 scope.go:117] "RemoveContainer" containerID="ca1e073d95d358be1958672d5461ac6946a488da832c68f471c8492ab783de24"
Jul 16 23:27:24.137799 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:27:24.137779    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-nwh4c_odh-ai-gateway-infra(932db21a-2c5c-4e48-9be2-e3cf473f7e34)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-nwh4c" podUID="932db21a-2c5c-4e48-9be2-e3cf473f7e34"
Jul 16 23:27:24.157162 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:24.157121    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-48992b/sub-test-model-48992b-kserve-7d764f4ff5-dlnl8" podStartSLOduration=2.15710958 podStartE2EDuration="2.15710958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:27:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:27:24.155609866 +0000 UTC m=+3218.165872532" watchObservedRunningTime="2026-07-16 23:27:24.15710958 +0000 UTC m=+3218.167372268"
Jul 16 23:27:24.518542 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:24.518486    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26594767-abbe-4679-b843-ecd1b3e56843" path="/var/lib/kubelet/pods/26594767-abbe-4679-b843-ecd1b3e56843/volumes"
Jul 16 23:27:25.143947 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:25.143907    2573 scope.go:117] "RemoveContainer" containerID="ca1e073d95d358be1958672d5461ac6946a488da832c68f471c8492ab783de24"
Jul 16 23:27:25.144340 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:27:25.144070    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-nwh4c_odh-ai-gateway-infra(932db21a-2c5c-4e48-9be2-e3cf473f7e34)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-nwh4c" podUID="932db21a-2c5c-4e48-9be2-e3cf473f7e34"
Jul 16 23:27:26.767119 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:26.767058    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-48992b/auth-test-model-48992b-kserve-fd54db7f-s4rwh" podUID="41d9d56c-aca9-4f8c-8028-cf8b08f04ed3" containerName="main" containerID="cri-o://6e52aba3d97209bd083da982dadfa66932138c70202c1454fbfdd14d345f5393" gracePeriod=45
Jul 16 23:27:27.011771 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:27.011746    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-48992b/auth-test-model-48992b-kserve-fd54db7f-s4rwh"
Jul 16 23:27:27.154125 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:27.154082    2573 generic.go:356] "Generic (PLEG): container finished" podID="41d9d56c-aca9-4f8c-8028-cf8b08f04ed3" containerID="6e52aba3d97209bd083da982dadfa66932138c70202c1454fbfdd14d345f5393" exitCode=0
Jul 16 23:27:27.154125 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:27.154120    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-48992b/auth-test-model-48992b-kserve-fd54db7f-s4rwh"
Jul 16 23:27:27.154378 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:27.154121    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-48992b/auth-test-model-48992b-kserve-fd54db7f-s4rwh" event={"ID":"41d9d56c-aca9-4f8c-8028-cf8b08f04ed3","Type":"ContainerDied","Data":"6e52aba3d97209bd083da982dadfa66932138c70202c1454fbfdd14d345f5393"}
Jul 16 23:27:27.154378 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:27.154172    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-48992b/auth-test-model-48992b-kserve-fd54db7f-s4rwh" event={"ID":"41d9d56c-aca9-4f8c-8028-cf8b08f04ed3","Type":"ContainerDied","Data":"fee8d6fab4b3f2611c81c73a53a014fe642ff271f084f7605888ec058c04683d"}
Jul 16 23:27:27.154378 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:27.154192    2573 scope.go:117] "RemoveContainer" containerID="6e52aba3d97209bd083da982dadfa66932138c70202c1454fbfdd14d345f5393"
Jul 16 23:27:27.154378 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:27.154285    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41d9d56c-aca9-4f8c-8028-cf8b08f04ed3-home\") pod \"41d9d56c-aca9-4f8c-8028-cf8b08f04ed3\" (UID: \"41d9d56c-aca9-4f8c-8028-cf8b08f04ed3\") "
Jul 16 23:27:27.154378 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:27.154312    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xvt6\" (UniqueName: \"kubernetes.io/projected/41d9d56c-aca9-4f8c-8028-cf8b08f04ed3-kube-api-access-7xvt6\") pod \"41d9d56c-aca9-4f8c-8028-cf8b08f04ed3\" (UID: \"41d9d56c-aca9-4f8c-8028-cf8b08f04ed3\") "
Jul 16 23:27:27.154378 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:27.154368    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41d9d56c-aca9-4f8c-8028-cf8b08f04ed3-tls-certs\") pod \"41d9d56c-aca9-4f8c-8028-cf8b08f04ed3\" (UID: \"41d9d56c-aca9-4f8c-8028-cf8b08f04ed3\") "
Jul 16 23:27:27.154677 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:27.154431    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41d9d56c-aca9-4f8c-8028-cf8b08f04ed3-dshm\") pod \"41d9d56c-aca9-4f8c-8028-cf8b08f04ed3\" (UID: \"41d9d56c-aca9-4f8c-8028-cf8b08f04ed3\") "
Jul 16 23:27:27.154677 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:27.154472    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41d9d56c-aca9-4f8c-8028-cf8b08f04ed3-tmp-dir\") pod \"41d9d56c-aca9-4f8c-8028-cf8b08f04ed3\" (UID: \"41d9d56c-aca9-4f8c-8028-cf8b08f04ed3\") "
Jul 16 23:27:27.154677 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:27.154513    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d9d56c-aca9-4f8c-8028-cf8b08f04ed3-home" (OuterVolumeSpecName: "home") pod "41d9d56c-aca9-4f8c-8028-cf8b08f04ed3" (UID: "41d9d56c-aca9-4f8c-8028-cf8b08f04ed3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:27:27.154677 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:27.154522    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41d9d56c-aca9-4f8c-8028-cf8b08f04ed3-model-cache\") pod \"41d9d56c-aca9-4f8c-8028-cf8b08f04ed3\" (UID: \"41d9d56c-aca9-4f8c-8028-cf8b08f04ed3\") "
Jul 16 23:27:27.154868 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:27.154776    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41d9d56c-aca9-4f8c-8028-cf8b08f04ed3-home\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:27:27.154868 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:27.154773    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d9d56c-aca9-4f8c-8028-cf8b08f04ed3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "41d9d56c-aca9-4f8c-8028-cf8b08f04ed3" (UID: "41d9d56c-aca9-4f8c-8028-cf8b08f04ed3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:27:27.154868 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:27.154822    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d9d56c-aca9-4f8c-8028-cf8b08f04ed3-model-cache" (OuterVolumeSpecName: "model-cache") pod "41d9d56c-aca9-4f8c-8028-cf8b08f04ed3" (UID: "41d9d56c-aca9-4f8c-8028-cf8b08f04ed3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:27:27.156810 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:27.156777    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d9d56c-aca9-4f8c-8028-cf8b08f04ed3-dshm" (OuterVolumeSpecName: "dshm") pod "41d9d56c-aca9-4f8c-8028-cf8b08f04ed3" (UID: "41d9d56c-aca9-4f8c-8028-cf8b08f04ed3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:27:27.156976 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:27.156955    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41d9d56c-aca9-4f8c-8028-cf8b08f04ed3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "41d9d56c-aca9-4f8c-8028-cf8b08f04ed3" (UID: "41d9d56c-aca9-4f8c-8028-cf8b08f04ed3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 23:27:27.156976 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:27.156967    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41d9d56c-aca9-4f8c-8028-cf8b08f04ed3-kube-api-access-7xvt6" (OuterVolumeSpecName: "kube-api-access-7xvt6") pod "41d9d56c-aca9-4f8c-8028-cf8b08f04ed3" (UID: "41d9d56c-aca9-4f8c-8028-cf8b08f04ed3"). InnerVolumeSpecName "kube-api-access-7xvt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:27:27.171858 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:27.171841    2573 scope.go:117] "RemoveContainer" containerID="6e52aba3d97209bd083da982dadfa66932138c70202c1454fbfdd14d345f5393"
Jul 16 23:27:27.172123 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:27:27.172102    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e52aba3d97209bd083da982dadfa66932138c70202c1454fbfdd14d345f5393\": container with ID starting with 6e52aba3d97209bd083da982dadfa66932138c70202c1454fbfdd14d345f5393 not found: ID does not exist" containerID="6e52aba3d97209bd083da982dadfa66932138c70202c1454fbfdd14d345f5393"
Jul 16 23:27:27.172187 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:27.172136    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e52aba3d97209bd083da982dadfa66932138c70202c1454fbfdd14d345f5393"} err="failed to get container status \"6e52aba3d97209bd083da982dadfa66932138c70202c1454fbfdd14d345f5393\": rpc error: code = NotFound desc = could not find container \"6e52aba3d97209bd083da982dadfa66932138c70202c1454fbfdd14d345f5393\": container with ID starting with 6e52aba3d97209bd083da982dadfa66932138c70202c1454fbfdd14d345f5393 not found: ID does not exist"
Jul 16 23:27:27.255289 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:27.255255    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41d9d56c-aca9-4f8c-8028-cf8b08f04ed3-model-cache\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:27:27.255289 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:27.255283    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xvt6\" (UniqueName: \"kubernetes.io/projected/41d9d56c-aca9-4f8c-8028-cf8b08f04ed3-kube-api-access-7xvt6\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:27:27.255427 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:27.255298    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41d9d56c-aca9-4f8c-8028-cf8b08f04ed3-tls-certs\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:27:27.255427 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:27.255311    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41d9d56c-aca9-4f8c-8028-cf8b08f04ed3-dshm\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:27:27.255427 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:27.255322    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41d9d56c-aca9-4f8c-8028-cf8b08f04ed3-tmp-dir\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:27:27.474549 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:27.474515    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-48992b/auth-test-model-48992b-kserve-fd54db7f-s4rwh"]
Jul 16 23:27:27.478359 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:27.478336    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-48992b/auth-test-model-48992b-kserve-fd54db7f-s4rwh"]
Jul 16 23:27:27.664268 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:27.664226    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-b4ab52/auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq" podUID="32dacbc9-8b90-4467-9cc9-807c928c8700" containerName="main" containerID="cri-o://47fda37cda7bb9d5b9e7f87c232239648f70a827f9a3cd4c4f9d4f7d753eeaaf" gracePeriod=45
Jul 16 23:27:27.908729 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:27.908706    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b4ab52/auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq"
Jul 16 23:27:28.061707 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:28.061637    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32dacbc9-8b90-4467-9cc9-807c928c8700-tmp-dir\") pod \"32dacbc9-8b90-4467-9cc9-807c928c8700\" (UID: \"32dacbc9-8b90-4467-9cc9-807c928c8700\") "
Jul 16 23:27:28.061707 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:28.061690    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32dacbc9-8b90-4467-9cc9-807c928c8700-model-cache\") pod \"32dacbc9-8b90-4467-9cc9-807c928c8700\" (UID: \"32dacbc9-8b90-4467-9cc9-807c928c8700\") "
Jul 16 23:27:28.061897 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:28.061715    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32dacbc9-8b90-4467-9cc9-807c928c8700-tls-certs\") pod \"32dacbc9-8b90-4467-9cc9-807c928c8700\" (UID: \"32dacbc9-8b90-4467-9cc9-807c928c8700\") "
Jul 16 23:27:28.061897 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:28.061738    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ncbhg\" (UniqueName: \"kubernetes.io/projected/32dacbc9-8b90-4467-9cc9-807c928c8700-kube-api-access-ncbhg\") pod \"32dacbc9-8b90-4467-9cc9-807c928c8700\" (UID: \"32dacbc9-8b90-4467-9cc9-807c928c8700\") "
Jul 16 23:27:28.061897 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:28.061759    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32dacbc9-8b90-4467-9cc9-807c928c8700-dshm\") pod \"32dacbc9-8b90-4467-9cc9-807c928c8700\" (UID: \"32dacbc9-8b90-4467-9cc9-807c928c8700\") "
Jul 16 23:27:28.061897 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:28.061794    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32dacbc9-8b90-4467-9cc9-807c928c8700-home\") pod \"32dacbc9-8b90-4467-9cc9-807c928c8700\" (UID: \"32dacbc9-8b90-4467-9cc9-807c928c8700\") "
Jul 16 23:27:28.062080 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:28.061907    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32dacbc9-8b90-4467-9cc9-807c928c8700-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "32dacbc9-8b90-4467-9cc9-807c928c8700" (UID: "32dacbc9-8b90-4467-9cc9-807c928c8700"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:27:28.062080 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:28.061987    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32dacbc9-8b90-4467-9cc9-807c928c8700-model-cache" (OuterVolumeSpecName: "model-cache") pod "32dacbc9-8b90-4467-9cc9-807c928c8700" (UID: "32dacbc9-8b90-4467-9cc9-807c928c8700"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:27:28.062080 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:28.062015    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32dacbc9-8b90-4467-9cc9-807c928c8700-tmp-dir\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:27:28.062172 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:28.062073    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32dacbc9-8b90-4467-9cc9-807c928c8700-home" (OuterVolumeSpecName: "home") pod "32dacbc9-8b90-4467-9cc9-807c928c8700" (UID: "32dacbc9-8b90-4467-9cc9-807c928c8700"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:27:28.063857 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:28.063828    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32dacbc9-8b90-4467-9cc9-807c928c8700-dshm" (OuterVolumeSpecName: "dshm") pod "32dacbc9-8b90-4467-9cc9-807c928c8700" (UID: "32dacbc9-8b90-4467-9cc9-807c928c8700"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:27:28.063942 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:28.063899    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32dacbc9-8b90-4467-9cc9-807c928c8700-kube-api-access-ncbhg" (OuterVolumeSpecName: "kube-api-access-ncbhg") pod "32dacbc9-8b90-4467-9cc9-807c928c8700" (UID: "32dacbc9-8b90-4467-9cc9-807c928c8700"). InnerVolumeSpecName "kube-api-access-ncbhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:27:28.064621 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:28.064602    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32dacbc9-8b90-4467-9cc9-807c928c8700-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "32dacbc9-8b90-4467-9cc9-807c928c8700" (UID: "32dacbc9-8b90-4467-9cc9-807c928c8700"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 23:27:28.142526 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:28.142505    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:27:28.160394 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:28.160370    2573 generic.go:356] "Generic (PLEG): container finished" podID="32dacbc9-8b90-4467-9cc9-807c928c8700" containerID="47fda37cda7bb9d5b9e7f87c232239648f70a827f9a3cd4c4f9d4f7d753eeaaf" exitCode=0
Jul 16 23:27:28.160478 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:28.160417    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b4ab52/auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq"
Jul 16 23:27:28.160478 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:28.160456    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b4ab52/auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq" event={"ID":"32dacbc9-8b90-4467-9cc9-807c928c8700","Type":"ContainerDied","Data":"47fda37cda7bb9d5b9e7f87c232239648f70a827f9a3cd4c4f9d4f7d753eeaaf"}
Jul 16 23:27:28.160653 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:28.160498    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b4ab52/auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq" event={"ID":"32dacbc9-8b90-4467-9cc9-807c928c8700","Type":"ContainerDied","Data":"3ef2d699a7d9b2e0a3e6d8e120ffbf47d6d3ffeed3f207d6b7e086a77510fc2a"}
Jul 16 23:27:28.160653 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:28.160523    2573 scope.go:117] "RemoveContainer" containerID="47fda37cda7bb9d5b9e7f87c232239648f70a827f9a3cd4c4f9d4f7d753eeaaf"
Jul 16 23:27:28.162479 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:28.162457    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32dacbc9-8b90-4467-9cc9-807c928c8700-model-cache\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:27:28.162479 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:28.162478    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32dacbc9-8b90-4467-9cc9-807c928c8700-tls-certs\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:27:28.162638 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:28.162487    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncbhg\" (UniqueName: \"kubernetes.io/projected/32dacbc9-8b90-4467-9cc9-807c928c8700-kube-api-access-ncbhg\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:27:28.162638 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:28.162495    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32dacbc9-8b90-4467-9cc9-807c928c8700-dshm\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:27:28.162638 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:28.162504    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32dacbc9-8b90-4467-9cc9-807c928c8700-home\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:27:28.169987 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:28.169972    2573 scope.go:117] "RemoveContainer" containerID="47fda37cda7bb9d5b9e7f87c232239648f70a827f9a3cd4c4f9d4f7d753eeaaf"
Jul 16 23:27:28.170275 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:27:28.170231    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47fda37cda7bb9d5b9e7f87c232239648f70a827f9a3cd4c4f9d4f7d753eeaaf\": container with ID starting with 47fda37cda7bb9d5b9e7f87c232239648f70a827f9a3cd4c4f9d4f7d753eeaaf not found: ID does not exist" containerID="47fda37cda7bb9d5b9e7f87c232239648f70a827f9a3cd4c4f9d4f7d753eeaaf"
Jul 16 23:27:28.170338 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:28.170288    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47fda37cda7bb9d5b9e7f87c232239648f70a827f9a3cd4c4f9d4f7d753eeaaf"} err="failed to get container status \"47fda37cda7bb9d5b9e7f87c232239648f70a827f9a3cd4c4f9d4f7d753eeaaf\": rpc error: code = NotFound desc = could not find container \"47fda37cda7bb9d5b9e7f87c232239648f70a827f9a3cd4c4f9d4f7d753eeaaf\": container with ID starting with 47fda37cda7bb9d5b9e7f87c232239648f70a827f9a3cd4c4f9d4f7d753eeaaf not found: ID does not exist"
Jul 16 23:27:28.181107 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:28.181054    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b4ab52/auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq"]
Jul 16 23:27:28.183036 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:28.183018    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-b4ab52/auth-test-model-b4ab52-kserve-5c47684cf5-c5pqq"]
Jul 16 23:27:28.518578 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:28.518559    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32dacbc9-8b90-4467-9cc9-807c928c8700" path="/var/lib/kubelet/pods/32dacbc9-8b90-4467-9cc9-807c928c8700/volumes"
Jul 16 23:27:28.518903 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:28.518890    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41d9d56c-aca9-4f8c-8028-cf8b08f04ed3" path="/var/lib/kubelet/pods/41d9d56c-aca9-4f8c-8028-cf8b08f04ed3/volumes"
Jul 16 23:27:33.231153 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:33.231119    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-48992b/sub-test-model-48992b-kserve-7d764f4ff5-dlnl8"
Jul 16 23:27:33.231153 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:33.231163    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-48992b/sub-test-model-48992b-kserve-7d764f4ff5-dlnl8"
Jul 16 23:27:33.243217 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:33.243192    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-48992b/sub-test-model-48992b-kserve-7d764f4ff5-dlnl8"
Jul 16 23:27:34.201942 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:34.201910    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-48992b/sub-test-model-48992b-kserve-7d764f4ff5-dlnl8"
Jul 16 23:27:39.514173 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:39.514101    2573 scope.go:117] "RemoveContainer" containerID="ca1e073d95d358be1958672d5461ac6946a488da832c68f471c8492ab783de24"
Jul 16 23:27:40.220832 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:40.220806    2573 generic.go:356] "Generic (PLEG): container finished" podID="932db21a-2c5c-4e48-9be2-e3cf473f7e34" containerID="1f54e1eb88ccb440de4323c272b60c56445c119339476c7a5eefbd1279f6cd42" exitCode=6
Jul 16 23:27:40.220933 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:40.220878    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-nwh4c" event={"ID":"932db21a-2c5c-4e48-9be2-e3cf473f7e34","Type":"ContainerDied","Data":"1f54e1eb88ccb440de4323c272b60c56445c119339476c7a5eefbd1279f6cd42"}
Jul 16 23:27:40.220969 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:40.220935    2573 scope.go:117] "RemoveContainer" containerID="ca1e073d95d358be1958672d5461ac6946a488da832c68f471c8492ab783de24"
Jul 16 23:27:40.221188 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:40.221170    2573 scope.go:117] "RemoveContainer" containerID="1f54e1eb88ccb440de4323c272b60c56445c119339476c7a5eefbd1279f6cd42"
Jul 16 23:27:40.221380 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:27:40.221362    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-nwh4c_odh-ai-gateway-infra(932db21a-2c5c-4e48-9be2-e3cf473f7e34)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-nwh4c" podUID="932db21a-2c5c-4e48-9be2-e3cf473f7e34"
Jul 16 23:27:41.247069 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:41.247035    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-nwh4c"]
Jul 16 23:27:41.371012 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:41.370992    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-nwh4c"
Jul 16 23:27:41.461273 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:41.461226    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/932db21a-2c5c-4e48-9be2-e3cf473f7e34-maas-api-service-ca\") pod \"932db21a-2c5c-4e48-9be2-e3cf473f7e34\" (UID: \"932db21a-2c5c-4e48-9be2-e3cf473f7e34\") "
Jul 16 23:27:41.461556 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:41.461534    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/932db21a-2c5c-4e48-9be2-e3cf473f7e34-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "932db21a-2c5c-4e48-9be2-e3cf473f7e34" (UID: "932db21a-2c5c-4e48-9be2-e3cf473f7e34"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:27:41.562382 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:41.562318    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/932db21a-2c5c-4e48-9be2-e3cf473f7e34-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:27:42.233412 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:42.233384    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-nwh4c"
Jul 16 23:27:42.233580 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:42.233427    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-nwh4c" event={"ID":"932db21a-2c5c-4e48-9be2-e3cf473f7e34","Type":"ContainerDied","Data":"754b57a7acf99997fa29d0637b77e45af460dd9087f362865244b8c0d9f7a486"}
Jul 16 23:27:42.233580 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:42.233482    2573 scope.go:117] "RemoveContainer" containerID="1f54e1eb88ccb440de4323c272b60c56445c119339476c7a5eefbd1279f6cd42"
Jul 16 23:27:42.258204 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:42.258180    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-nwh4c"]
Jul 16 23:27:42.261919 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:42.261896    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-nwh4c"]
Jul 16 23:27:42.518594 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:42.518540    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="932db21a-2c5c-4e48-9be2-e3cf473f7e34" path="/var/lib/kubelet/pods/932db21a-2c5c-4e48-9be2-e3cf473f7e34/volumes"
Jul 16 23:27:48.406577 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:48.406546    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-25s5s"]
Jul 16 23:27:48.406920 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:48.406847    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41d9d56c-aca9-4f8c-8028-cf8b08f04ed3" containerName="main"
Jul 16 23:27:48.406920 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:48.406858    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="41d9d56c-aca9-4f8c-8028-cf8b08f04ed3" containerName="main"
Jul 16 23:27:48.406920 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:48.406867    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="932db21a-2c5c-4e48-9be2-e3cf473f7e34" containerName="revoke-keys"
Jul 16 23:27:48.406920 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:48.406872    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="932db21a-2c5c-4e48-9be2-e3cf473f7e34" containerName="revoke-keys"
Jul 16 23:27:48.406920 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:48.406879    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32dacbc9-8b90-4467-9cc9-807c928c8700" containerName="main"
Jul 16 23:27:48.406920 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:48.406884    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="32dacbc9-8b90-4467-9cc9-807c928c8700" containerName="main"
Jul 16 23:27:48.406920 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:48.406890    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="932db21a-2c5c-4e48-9be2-e3cf473f7e34" containerName="revoke-keys"
Jul 16 23:27:48.406920 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:48.406895    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="932db21a-2c5c-4e48-9be2-e3cf473f7e34" containerName="revoke-keys"
Jul 16 23:27:48.406920 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:48.406905    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="932db21a-2c5c-4e48-9be2-e3cf473f7e34" containerName="revoke-keys"
Jul 16 23:27:48.406920 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:48.406911    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="932db21a-2c5c-4e48-9be2-e3cf473f7e34" containerName="revoke-keys"
Jul 16 23:27:48.407216 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:48.406928    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26594767-abbe-4679-b843-ecd1b3e56843" containerName="revoke-keys"
Jul 16 23:27:48.407216 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:48.406933    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="26594767-abbe-4679-b843-ecd1b3e56843" containerName="revoke-keys"
Jul 16 23:27:48.407216 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:48.406978    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="932db21a-2c5c-4e48-9be2-e3cf473f7e34" containerName="revoke-keys"
Jul 16 23:27:48.407216 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:48.406987    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="41d9d56c-aca9-4f8c-8028-cf8b08f04ed3" containerName="main"
Jul 16 23:27:48.407216 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:48.406993    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="32dacbc9-8b90-4467-9cc9-807c928c8700" containerName="main"
Jul 16 23:27:48.407216 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:48.407002    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="932db21a-2c5c-4e48-9be2-e3cf473f7e34" containerName="revoke-keys"
Jul 16 23:27:48.411378 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:48.411358    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-25s5s"
Jul 16 23:27:48.413832 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:48.413813    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 23:27:48.414045 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:48.414027    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:27:48.417753 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:48.417700    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-25s5s"]
Jul 16 23:27:48.509411 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:48.509388    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55bad41d-1d6f-4753-8828-040d6a5cf5f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-25s5s\" (UID: \"55bad41d-1d6f-4753-8828-040d6a5cf5f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-25s5s"
Jul 16 23:27:48.610157 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:48.610133    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55bad41d-1d6f-4753-8828-040d6a5cf5f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-25s5s\" (UID: \"55bad41d-1d6f-4753-8828-040d6a5cf5f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-25s5s"
Jul 16 23:27:48.610703 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:48.610686    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55bad41d-1d6f-4753-8828-040d6a5cf5f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-25s5s\" (UID: \"55bad41d-1d6f-4753-8828-040d6a5cf5f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-25s5s"
Jul 16 23:27:48.722705 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:48.722657    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-25s5s"
Jul 16 23:27:48.843499 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:48.843472    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-25s5s"]
Jul 16 23:27:48.845074 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:27:48.845037    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55bad41d_1d6f_4753_8828_040d6a5cf5f7.slice/crio-1da921857d422fc514c71ff777b3832999cc342248d58efa357104834f6f1f14 WatchSource:0}: Error finding container 1da921857d422fc514c71ff777b3832999cc342248d58efa357104834f6f1f14: Status 404 returned error can't find the container with id 1da921857d422fc514c71ff777b3832999cc342248d58efa357104834f6f1f14
Jul 16 23:27:49.268409 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:49.268349    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-25s5s" event={"ID":"55bad41d-1d6f-4753-8828-040d6a5cf5f7","Type":"ContainerStarted","Data":"44119042e8b315596751e8931440436371cd3f450c1ac228b6900c4643db1e40"}
Jul 16 23:27:49.268409 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:49.268400    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-25s5s" event={"ID":"55bad41d-1d6f-4753-8828-040d6a5cf5f7","Type":"ContainerStarted","Data":"1da921857d422fc514c71ff777b3832999cc342248d58efa357104834f6f1f14"}
Jul 16 23:27:49.285361 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:49.285315    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-25s5s" podStartSLOduration=1.285300743 podStartE2EDuration="1.285300743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:27:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:27:49.284075438 +0000 UTC m=+3243.294338101" watchObservedRunningTime="2026-07-16 23:27:49.285300743 +0000 UTC m=+3243.295563404"
Jul 16 23:27:50.274660 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:50.274624    2573 generic.go:356] "Generic (PLEG): container finished" podID="55bad41d-1d6f-4753-8828-040d6a5cf5f7" containerID="44119042e8b315596751e8931440436371cd3f450c1ac228b6900c4643db1e40" exitCode=6
Jul 16 23:27:50.275027 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:50.274703    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-25s5s" event={"ID":"55bad41d-1d6f-4753-8828-040d6a5cf5f7","Type":"ContainerDied","Data":"44119042e8b315596751e8931440436371cd3f450c1ac228b6900c4643db1e40"}
Jul 16 23:27:50.275027 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:50.274847    2573 scope.go:117] "RemoveContainer" containerID="44119042e8b315596751e8931440436371cd3f450c1ac228b6900c4643db1e40"
Jul 16 23:27:51.282428 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:51.282350    2573 generic.go:356] "Generic (PLEG): container finished" podID="55bad41d-1d6f-4753-8828-040d6a5cf5f7" containerID="61a90360fe5157ec35280152c98cbcf2c2befb72d73abe412a49cd0c15b9bb15" exitCode=6
Jul 16 23:27:51.282428 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:51.282392    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-25s5s" event={"ID":"55bad41d-1d6f-4753-8828-040d6a5cf5f7","Type":"ContainerDied","Data":"61a90360fe5157ec35280152c98cbcf2c2befb72d73abe412a49cd0c15b9bb15"}
Jul 16 23:27:51.282913 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:51.282448    2573 scope.go:117] "RemoveContainer" containerID="44119042e8b315596751e8931440436371cd3f450c1ac228b6900c4643db1e40"
Jul 16 23:27:51.282913 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:51.282622    2573 scope.go:117] "RemoveContainer" containerID="61a90360fe5157ec35280152c98cbcf2c2befb72d73abe412a49cd0c15b9bb15"
Jul 16 23:27:51.282913 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:27:51.282808    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-25s5s_odh-ai-gateway-infra(55bad41d-1d6f-4753-8828-040d6a5cf5f7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-25s5s" podUID="55bad41d-1d6f-4753-8828-040d6a5cf5f7"
Jul 16 23:27:52.288354 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:52.288329    2573 scope.go:117] "RemoveContainer" containerID="61a90360fe5157ec35280152c98cbcf2c2befb72d73abe412a49cd0c15b9bb15"
Jul 16 23:27:52.288793 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:27:52.288495    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-25s5s_odh-ai-gateway-infra(55bad41d-1d6f-4753-8828-040d6a5cf5f7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-25s5s" podUID="55bad41d-1d6f-4753-8828-040d6a5cf5f7"
Jul 16 23:27:53.437488 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:53.437458    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s478p"]
Jul 16 23:27:53.437859 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:53.437795    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="932db21a-2c5c-4e48-9be2-e3cf473f7e34" containerName="revoke-keys"
Jul 16 23:27:53.441826 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:53.441809    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s478p"
Jul 16 23:27:53.449231 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:53.448916    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s478p"]
Jul 16 23:27:53.545788 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:53.545754    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e33fffcc-554c-41de-98bc-045d1107ed52-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-s478p\" (UID: \"e33fffcc-554c-41de-98bc-045d1107ed52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s478p"
Jul 16 23:27:53.646136 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:53.646111    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e33fffcc-554c-41de-98bc-045d1107ed52-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-s478p\" (UID: \"e33fffcc-554c-41de-98bc-045d1107ed52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s478p"
Jul 16 23:27:53.646756 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:53.646739    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e33fffcc-554c-41de-98bc-045d1107ed52-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-s478p\" (UID: \"e33fffcc-554c-41de-98bc-045d1107ed52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s478p"
Jul 16 23:27:53.752945 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:53.752897    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s478p"
Jul 16 23:27:53.873926 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:53.873896    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s478p"]
Jul 16 23:27:53.875480 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:27:53.875453    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode33fffcc_554c_41de_98bc_045d1107ed52.slice/crio-a84647a904f460b70c9d14dcfb72ac2efea24ff610e0f1df66aa1f322939ca9c WatchSource:0}: Error finding container a84647a904f460b70c9d14dcfb72ac2efea24ff610e0f1df66aa1f322939ca9c: Status 404 returned error can't find the container with id a84647a904f460b70c9d14dcfb72ac2efea24ff610e0f1df66aa1f322939ca9c
Jul 16 23:27:54.298352 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:54.298319    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s478p" event={"ID":"e33fffcc-554c-41de-98bc-045d1107ed52","Type":"ContainerStarted","Data":"712b425025318ba6d589611a724426395a7d6dc5e008986e663de7083cb98961"}
Jul 16 23:27:54.298516 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:54.298372    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s478p" event={"ID":"e33fffcc-554c-41de-98bc-045d1107ed52","Type":"ContainerStarted","Data":"a84647a904f460b70c9d14dcfb72ac2efea24ff610e0f1df66aa1f322939ca9c"}
Jul 16 23:27:54.314864 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:54.314820    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s478p" podStartSLOduration=1.314806919 podStartE2EDuration="1.314806919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:27:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:27:54.312983802 +0000 UTC m=+3248.323246491" watchObservedRunningTime="2026-07-16 23:27:54.314806919 +0000 UTC m=+3248.325069617"
Jul 16 23:27:55.304413 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:55.304385    2573 generic.go:356] "Generic (PLEG): container finished" podID="e33fffcc-554c-41de-98bc-045d1107ed52" containerID="712b425025318ba6d589611a724426395a7d6dc5e008986e663de7083cb98961" exitCode=6
Jul 16 23:27:55.304778 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:55.304471    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s478p" event={"ID":"e33fffcc-554c-41de-98bc-045d1107ed52","Type":"ContainerDied","Data":"712b425025318ba6d589611a724426395a7d6dc5e008986e663de7083cb98961"}
Jul 16 23:27:55.304778 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:55.304596    2573 scope.go:117] "RemoveContainer" containerID="712b425025318ba6d589611a724426395a7d6dc5e008986e663de7083cb98961"
Jul 16 23:27:56.310881 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:56.310799    2573 generic.go:356] "Generic (PLEG): container finished" podID="e33fffcc-554c-41de-98bc-045d1107ed52" containerID="87cd473a197402a90cc970fdf92a1f3bf7127f1c57c19e9aeb9352417125bc44" exitCode=6
Jul 16 23:27:56.310881 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:56.310843    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s478p" event={"ID":"e33fffcc-554c-41de-98bc-045d1107ed52","Type":"ContainerDied","Data":"87cd473a197402a90cc970fdf92a1f3bf7127f1c57c19e9aeb9352417125bc44"}
Jul 16 23:27:56.311403 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:56.310890    2573 scope.go:117] "RemoveContainer" containerID="712b425025318ba6d589611a724426395a7d6dc5e008986e663de7083cb98961"
Jul 16 23:27:56.311403 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:56.311075    2573 scope.go:117] "RemoveContainer" containerID="87cd473a197402a90cc970fdf92a1f3bf7127f1c57c19e9aeb9352417125bc44"
Jul 16 23:27:56.311403 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:27:56.311293    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-s478p_odh-ai-gateway-infra(e33fffcc-554c-41de-98bc-045d1107ed52)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s478p" podUID="e33fffcc-554c-41de-98bc-045d1107ed52"
Jul 16 23:27:57.317183 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:27:57.317156    2573 scope.go:117] "RemoveContainer" containerID="87cd473a197402a90cc970fdf92a1f3bf7127f1c57c19e9aeb9352417125bc44"
Jul 16 23:27:57.317553 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:27:57.317325    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-s478p_odh-ai-gateway-infra(e33fffcc-554c-41de-98bc-045d1107ed52)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s478p" podUID="e33fffcc-554c-41de-98bc-045d1107ed52"
Jul 16 23:28:00.241380 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:00.241348    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-b4ab52/sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t"]
Jul 16 23:28:00.245090 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:00.245066    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b4ab52/sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t"
Jul 16 23:28:00.247912 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:00.247888    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b4ab52\"/\"kube-root-ca.crt\""
Jul 16 23:28:00.248939 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:00.248921    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b4ab52\"/\"openshift-service-ca.crt\""
Jul 16 23:28:00.248939 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:00.248931    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b4ab52\"/\"default-dockercfg-b6pn6\""
Jul 16 23:28:00.249121 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:00.248935    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b4ab52\"/\"sub-test-model-b4ab52-kserve-self-signed-certs\""
Jul 16 23:28:00.253647 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:00.253628    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b4ab52/sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t"]
Jul 16 23:28:00.282418 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:00.282380    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:28:00.396186 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:00.396162    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72d741fb-30cc-4aa0-9721-41ebfdc5e77d-home\") pod \"sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t\" (UID: \"72d741fb-30cc-4aa0-9721-41ebfdc5e77d\") " pod="ai-tenant-e2e-shared-b-b4ab52/sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t"
Jul 16 23:28:00.396311 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:00.396191    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72d741fb-30cc-4aa0-9721-41ebfdc5e77d-model-cache\") pod \"sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t\" (UID: \"72d741fb-30cc-4aa0-9721-41ebfdc5e77d\") " pod="ai-tenant-e2e-shared-b-b4ab52/sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t"
Jul 16 23:28:00.396311 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:00.396217    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72d741fb-30cc-4aa0-9721-41ebfdc5e77d-tmp-dir\") pod \"sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t\" (UID: \"72d741fb-30cc-4aa0-9721-41ebfdc5e77d\") " pod="ai-tenant-e2e-shared-b-b4ab52/sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t"
Jul 16 23:28:00.396311 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:00.396306    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f25r\" (UniqueName: \"kubernetes.io/projected/72d741fb-30cc-4aa0-9721-41ebfdc5e77d-kube-api-access-8f25r\") pod \"sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t\" (UID: \"72d741fb-30cc-4aa0-9721-41ebfdc5e77d\") " pod="ai-tenant-e2e-shared-b-b4ab52/sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t"
Jul 16 23:28:00.396421 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:00.396326    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72d741fb-30cc-4aa0-9721-41ebfdc5e77d-dshm\") pod \"sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t\" (UID: \"72d741fb-30cc-4aa0-9721-41ebfdc5e77d\") " pod="ai-tenant-e2e-shared-b-b4ab52/sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t"
Jul 16 23:28:00.396421 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:00.396343    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72d741fb-30cc-4aa0-9721-41ebfdc5e77d-tls-certs\") pod \"sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t\" (UID: \"72d741fb-30cc-4aa0-9721-41ebfdc5e77d\") " pod="ai-tenant-e2e-shared-b-b4ab52/sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t"
Jul 16 23:28:00.497270 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:00.497190    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72d741fb-30cc-4aa0-9721-41ebfdc5e77d-home\") pod \"sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t\" (UID: \"72d741fb-30cc-4aa0-9721-41ebfdc5e77d\") " pod="ai-tenant-e2e-shared-b-b4ab52/sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t"
Jul 16 23:28:00.497270 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:00.497224    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72d741fb-30cc-4aa0-9721-41ebfdc5e77d-model-cache\") pod \"sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t\" (UID: \"72d741fb-30cc-4aa0-9721-41ebfdc5e77d\") " pod="ai-tenant-e2e-shared-b-b4ab52/sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t"
Jul 16 23:28:00.497418 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:00.497282    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72d741fb-30cc-4aa0-9721-41ebfdc5e77d-tmp-dir\") pod \"sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t\" (UID: \"72d741fb-30cc-4aa0-9721-41ebfdc5e77d\") " pod="ai-tenant-e2e-shared-b-b4ab52/sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t"
Jul 16 23:28:00.497418 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:00.497338    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f25r\" (UniqueName: \"kubernetes.io/projected/72d741fb-30cc-4aa0-9721-41ebfdc5e77d-kube-api-access-8f25r\") pod \"sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t\" (UID: \"72d741fb-30cc-4aa0-9721-41ebfdc5e77d\") " pod="ai-tenant-e2e-shared-b-b4ab52/sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t"
Jul 16 23:28:00.497527 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:00.497514    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72d741fb-30cc-4aa0-9721-41ebfdc5e77d-dshm\") pod \"sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t\" (UID: \"72d741fb-30cc-4aa0-9721-41ebfdc5e77d\") " pod="ai-tenant-e2e-shared-b-b4ab52/sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t"
Jul 16 23:28:00.497587 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:00.497540    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72d741fb-30cc-4aa0-9721-41ebfdc5e77d-tls-certs\") pod \"sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t\" (UID: \"72d741fb-30cc-4aa0-9721-41ebfdc5e77d\") " pod="ai-tenant-e2e-shared-b-b4ab52/sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t"
Jul 16 23:28:00.497646 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:00.497612    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72d741fb-30cc-4aa0-9721-41ebfdc5e77d-home\") pod \"sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t\" (UID: \"72d741fb-30cc-4aa0-9721-41ebfdc5e77d\") " pod="ai-tenant-e2e-shared-b-b4ab52/sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t"
Jul 16 23:28:00.497646 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:00.497626    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72d741fb-30cc-4aa0-9721-41ebfdc5e77d-model-cache\") pod \"sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t\" (UID: \"72d741fb-30cc-4aa0-9721-41ebfdc5e77d\") " pod="ai-tenant-e2e-shared-b-b4ab52/sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t"
Jul 16 23:28:00.497747 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:00.497692    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72d741fb-30cc-4aa0-9721-41ebfdc5e77d-tmp-dir\") pod \"sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t\" (UID: \"72d741fb-30cc-4aa0-9721-41ebfdc5e77d\") " pod="ai-tenant-e2e-shared-b-b4ab52/sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t"
Jul 16 23:28:00.499641 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:00.499618    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72d741fb-30cc-4aa0-9721-41ebfdc5e77d-dshm\") pod \"sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t\" (UID: \"72d741fb-30cc-4aa0-9721-41ebfdc5e77d\") " pod="ai-tenant-e2e-shared-b-b4ab52/sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t"
Jul 16 23:28:00.499821 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:00.499803    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72d741fb-30cc-4aa0-9721-41ebfdc5e77d-tls-certs\") pod \"sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t\" (UID: \"72d741fb-30cc-4aa0-9721-41ebfdc5e77d\") " pod="ai-tenant-e2e-shared-b-b4ab52/sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t"
Jul 16 23:28:00.505081 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:00.505063    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f25r\" (UniqueName: \"kubernetes.io/projected/72d741fb-30cc-4aa0-9721-41ebfdc5e77d-kube-api-access-8f25r\") pod \"sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t\" (UID: \"72d741fb-30cc-4aa0-9721-41ebfdc5e77d\") " pod="ai-tenant-e2e-shared-b-b4ab52/sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t"
Jul 16 23:28:00.556065 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:00.556042    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b4ab52/sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t"
Jul 16 23:28:00.680555 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:00.680528    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b4ab52/sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t"]
Jul 16 23:28:00.682019 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:28:00.681991    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72d741fb_30cc_4aa0_9721_41ebfdc5e77d.slice/crio-7167d631e1ec186d3be93e4bf9baee0d5c09aa9ae7ff51d73d3ab818958eda64 WatchSource:0}: Error finding container 7167d631e1ec186d3be93e4bf9baee0d5c09aa9ae7ff51d73d3ab818958eda64: Status 404 returned error can't find the container with id 7167d631e1ec186d3be93e4bf9baee0d5c09aa9ae7ff51d73d3ab818958eda64
Jul 16 23:28:01.337864 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:01.337828    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b4ab52/sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t" event={"ID":"72d741fb-30cc-4aa0-9721-41ebfdc5e77d","Type":"ContainerStarted","Data":"7b0666003e7247a3cb57bdf1aa243654a2f94cf268d4227903e1a8fc8c749dcc"}
Jul 16 23:28:01.337864 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:01.337871    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b4ab52/sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t" event={"ID":"72d741fb-30cc-4aa0-9721-41ebfdc5e77d","Type":"ContainerStarted","Data":"7167d631e1ec186d3be93e4bf9baee0d5c09aa9ae7ff51d73d3ab818958eda64"}
Jul 16 23:28:01.356856 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:01.356810    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-b4ab52/sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t" podStartSLOduration=1.356795809 podStartE2EDuration="1.356795809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:28:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:28:01.355141522 +0000 UTC m=+3255.365404210" watchObservedRunningTime="2026-07-16 23:28:01.356795809 +0000 UTC m=+3255.367058472"
Jul 16 23:28:02.255742 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:02.255715    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-h4489"]
Jul 16 23:28:02.259085 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:02.259069    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-h4489"
Jul 16 23:28:02.271880 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:02.271855    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-h4489"]
Jul 16 23:28:02.310732 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:02.310705    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22eceb47-1bc2-4c0a-aae3-5111b52af6c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-h4489\" (UID: \"22eceb47-1bc2-4c0a-aae3-5111b52af6c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-h4489"
Jul 16 23:28:02.411596 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:02.411568    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22eceb47-1bc2-4c0a-aae3-5111b52af6c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-h4489\" (UID: \"22eceb47-1bc2-4c0a-aae3-5111b52af6c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-h4489"
Jul 16 23:28:02.412118 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:02.412100    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22eceb47-1bc2-4c0a-aae3-5111b52af6c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-h4489\" (UID: \"22eceb47-1bc2-4c0a-aae3-5111b52af6c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-h4489"
Jul 16 23:28:02.572554 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:02.572494    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-h4489"
Jul 16 23:28:02.696135 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:02.696102    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-h4489"]
Jul 16 23:28:02.699414 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:28:02.699384    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22eceb47_1bc2_4c0a_aae3_5111b52af6c1.slice/crio-acf4f3ee2a106ffd274618c409425b997752cb60dcfae8c883c448364d23a6a8 WatchSource:0}: Error finding container acf4f3ee2a106ffd274618c409425b997752cb60dcfae8c883c448364d23a6a8: Status 404 returned error can't find the container with id acf4f3ee2a106ffd274618c409425b997752cb60dcfae8c883c448364d23a6a8
Jul 16 23:28:03.349171 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:03.349135    2573 generic.go:356] "Generic (PLEG): container finished" podID="22eceb47-1bc2-4c0a-aae3-5111b52af6c1" containerID="dd119b8086ae67ccd129a988bf9e8fb0bc6a749fd6d943afab6eea71863120f6" exitCode=6
Jul 16 23:28:03.349315 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:03.349204    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-h4489" event={"ID":"22eceb47-1bc2-4c0a-aae3-5111b52af6c1","Type":"ContainerDied","Data":"dd119b8086ae67ccd129a988bf9e8fb0bc6a749fd6d943afab6eea71863120f6"}
Jul 16 23:28:03.349315 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:03.349280    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-h4489" event={"ID":"22eceb47-1bc2-4c0a-aae3-5111b52af6c1","Type":"ContainerStarted","Data":"acf4f3ee2a106ffd274618c409425b997752cb60dcfae8c883c448364d23a6a8"}
Jul 16 23:28:03.349495 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:03.349480    2573 scope.go:117] "RemoveContainer" containerID="dd119b8086ae67ccd129a988bf9e8fb0bc6a749fd6d943afab6eea71863120f6"
Jul 16 23:28:04.356165 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:04.356137    2573 generic.go:356] "Generic (PLEG): container finished" podID="22eceb47-1bc2-4c0a-aae3-5111b52af6c1" containerID="9239d5a5d65c1ec738156a2bfd2cc5c92fa74b24c25a4b8a12cb375df211a802" exitCode=6
Jul 16 23:28:04.356581 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:04.356247    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-h4489" event={"ID":"22eceb47-1bc2-4c0a-aae3-5111b52af6c1","Type":"ContainerDied","Data":"9239d5a5d65c1ec738156a2bfd2cc5c92fa74b24c25a4b8a12cb375df211a802"}
Jul 16 23:28:04.356581 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:04.356320    2573 scope.go:117] "RemoveContainer" containerID="dd119b8086ae67ccd129a988bf9e8fb0bc6a749fd6d943afab6eea71863120f6"
Jul 16 23:28:04.356581 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:04.356400    2573 scope.go:117] "RemoveContainer" containerID="9239d5a5d65c1ec738156a2bfd2cc5c92fa74b24c25a4b8a12cb375df211a802"
Jul 16 23:28:04.356581 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:28:04.356572    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-h4489_odh-ai-gateway-infra(22eceb47-1bc2-4c0a-aae3-5111b52af6c1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-h4489" podUID="22eceb47-1bc2-4c0a-aae3-5111b52af6c1"
Jul 16 23:28:04.634651 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:04.634570    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:28:05.363952 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:05.363926    2573 scope.go:117] "RemoveContainer" containerID="9239d5a5d65c1ec738156a2bfd2cc5c92fa74b24c25a4b8a12cb375df211a802"
Jul 16 23:28:05.364362 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:28:05.364069    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-h4489_odh-ai-gateway-infra(22eceb47-1bc2-4c0a-aae3-5111b52af6c1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-h4489" podUID="22eceb47-1bc2-4c0a-aae3-5111b52af6c1"
Jul 16 23:28:07.513690 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:07.513660    2573 scope.go:117] "RemoveContainer" containerID="61a90360fe5157ec35280152c98cbcf2c2befb72d73abe412a49cd0c15b9bb15"
Jul 16 23:28:08.383316 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:08.383226    2573 generic.go:356] "Generic (PLEG): container finished" podID="55bad41d-1d6f-4753-8828-040d6a5cf5f7" containerID="9a49f1dc1ac82a09cd7d758c9a720cde66f2965991e93cfbe00bdba6d24b59e9" exitCode=6
Jul 16 23:28:08.383316 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:08.383302    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-25s5s" event={"ID":"55bad41d-1d6f-4753-8828-040d6a5cf5f7","Type":"ContainerDied","Data":"9a49f1dc1ac82a09cd7d758c9a720cde66f2965991e93cfbe00bdba6d24b59e9"}
Jul 16 23:28:08.383511 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:08.383363    2573 scope.go:117] "RemoveContainer" containerID="61a90360fe5157ec35280152c98cbcf2c2befb72d73abe412a49cd0c15b9bb15"
Jul 16 23:28:08.383631 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:08.383616    2573 scope.go:117] "RemoveContainer" containerID="9a49f1dc1ac82a09cd7d758c9a720cde66f2965991e93cfbe00bdba6d24b59e9"
Jul 16 23:28:08.383818 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:28:08.383799    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-25s5s_odh-ai-gateway-infra(55bad41d-1d6f-4753-8828-040d6a5cf5f7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-25s5s" podUID="55bad41d-1d6f-4753-8828-040d6a5cf5f7"
Jul 16 23:28:08.513692 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:08.513666    2573 scope.go:117] "RemoveContainer" containerID="87cd473a197402a90cc970fdf92a1f3bf7127f1c57c19e9aeb9352417125bc44"
Jul 16 23:28:09.391851 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:09.391782    2573 generic.go:356] "Generic (PLEG): container finished" podID="e33fffcc-554c-41de-98bc-045d1107ed52" containerID="80dcd074acb1a4f0d7faae739db693e50ba9e689901b4d27950399d3eb3a0319" exitCode=6
Jul 16 23:28:09.392165 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:09.391849    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s478p" event={"ID":"e33fffcc-554c-41de-98bc-045d1107ed52","Type":"ContainerDied","Data":"80dcd074acb1a4f0d7faae739db693e50ba9e689901b4d27950399d3eb3a0319"}
Jul 16 23:28:09.392165 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:09.391902    2573 scope.go:117] "RemoveContainer" containerID="87cd473a197402a90cc970fdf92a1f3bf7127f1c57c19e9aeb9352417125bc44"
Jul 16 23:28:09.392165 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:09.392136    2573 scope.go:117] "RemoveContainer" containerID="80dcd074acb1a4f0d7faae739db693e50ba9e689901b4d27950399d3eb3a0319"
Jul 16 23:28:09.392356 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:28:09.392337    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-s478p_odh-ai-gateway-infra(e33fffcc-554c-41de-98bc-045d1107ed52)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s478p" podUID="e33fffcc-554c-41de-98bc-045d1107ed52"
Jul 16 23:28:09.407916 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:09.407896    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-25s5s"]
Jul 16 23:28:09.530821 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:09.530801    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-25s5s"
Jul 16 23:28:09.559124 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:09.559091    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55bad41d-1d6f-4753-8828-040d6a5cf5f7-maas-api-service-ca\") pod \"55bad41d-1d6f-4753-8828-040d6a5cf5f7\" (UID: \"55bad41d-1d6f-4753-8828-040d6a5cf5f7\") "
Jul 16 23:28:09.559400 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:09.559377    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55bad41d-1d6f-4753-8828-040d6a5cf5f7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "55bad41d-1d6f-4753-8828-040d6a5cf5f7" (UID: "55bad41d-1d6f-4753-8828-040d6a5cf5f7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:28:09.659855 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:09.659797    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55bad41d-1d6f-4753-8828-040d6a5cf5f7-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:28:10.399189 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:10.399155    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-25s5s"
Jul 16 23:28:10.399603 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:10.399156    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-25s5s" event={"ID":"55bad41d-1d6f-4753-8828-040d6a5cf5f7","Type":"ContainerDied","Data":"1da921857d422fc514c71ff777b3832999cc342248d58efa357104834f6f1f14"}
Jul 16 23:28:10.399603 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:10.399321    2573 scope.go:117] "RemoveContainer" containerID="9a49f1dc1ac82a09cd7d758c9a720cde66f2965991e93cfbe00bdba6d24b59e9"
Jul 16 23:28:10.421367 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:10.421344    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-25s5s"]
Jul 16 23:28:10.424321 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:10.424300    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s478p"]
Jul 16 23:28:10.429887 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:10.429867    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-25s5s"]
Jul 16 23:28:10.519873 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:10.519844    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55bad41d-1d6f-4753-8828-040d6a5cf5f7" path="/var/lib/kubelet/pods/55bad41d-1d6f-4753-8828-040d6a5cf5f7/volumes"
Jul 16 23:28:10.549210 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:10.549188    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s478p"
Jul 16 23:28:10.556771 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:10.556753    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-b4ab52/sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t"
Jul 16 23:28:10.556867 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:10.556786    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-b4ab52/sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t"
Jul 16 23:28:10.568074 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:10.568055    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e33fffcc-554c-41de-98bc-045d1107ed52-maas-api-service-ca\") pod \"e33fffcc-554c-41de-98bc-045d1107ed52\" (UID: \"e33fffcc-554c-41de-98bc-045d1107ed52\") "
Jul 16 23:28:10.568481 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:10.568456    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e33fffcc-554c-41de-98bc-045d1107ed52-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e33fffcc-554c-41de-98bc-045d1107ed52" (UID: "e33fffcc-554c-41de-98bc-045d1107ed52"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:28:10.570767 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:10.570743    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-b4ab52/sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t"
Jul 16 23:28:10.668956 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:10.668887    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e33fffcc-554c-41de-98bc-045d1107ed52-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:28:11.409438 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:11.409412    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s478p"
Jul 16 23:28:11.409843 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:11.409453    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s478p" event={"ID":"e33fffcc-554c-41de-98bc-045d1107ed52","Type":"ContainerDied","Data":"a84647a904f460b70c9d14dcfb72ac2efea24ff610e0f1df66aa1f322939ca9c"}
Jul 16 23:28:11.409843 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:11.409514    2573 scope.go:117] "RemoveContainer" containerID="80dcd074acb1a4f0d7faae739db693e50ba9e689901b4d27950399d3eb3a0319"
Jul 16 23:28:11.423789 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:11.423764    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-b4ab52/sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t"
Jul 16 23:28:11.435116 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:11.435092    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s478p"]
Jul 16 23:28:11.440323 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:11.440302    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s478p"]
Jul 16 23:28:12.517893 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:12.517864    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e33fffcc-554c-41de-98bc-045d1107ed52" path="/var/lib/kubelet/pods/e33fffcc-554c-41de-98bc-045d1107ed52/volumes"
Jul 16 23:28:18.513864 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:18.513837    2573 scope.go:117] "RemoveContainer" containerID="9239d5a5d65c1ec738156a2bfd2cc5c92fa74b24c25a4b8a12cb375df211a802"
Jul 16 23:28:19.451062 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:19.450993    2573 generic.go:356] "Generic (PLEG): container finished" podID="22eceb47-1bc2-4c0a-aae3-5111b52af6c1" containerID="56301817f48447ee5213e605d75fa23140add891937eaa304cbc955281e98764" exitCode=6
Jul 16 23:28:19.451206 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:19.451059    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-h4489" event={"ID":"22eceb47-1bc2-4c0a-aae3-5111b52af6c1","Type":"ContainerDied","Data":"56301817f48447ee5213e605d75fa23140add891937eaa304cbc955281e98764"}
Jul 16 23:28:19.451206 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:19.451131    2573 scope.go:117] "RemoveContainer" containerID="9239d5a5d65c1ec738156a2bfd2cc5c92fa74b24c25a4b8a12cb375df211a802"
Jul 16 23:28:19.451462 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:19.451442    2573 scope.go:117] "RemoveContainer" containerID="56301817f48447ee5213e605d75fa23140add891937eaa304cbc955281e98764"
Jul 16 23:28:19.451653 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:28:19.451634    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-h4489_odh-ai-gateway-infra(22eceb47-1bc2-4c0a-aae3-5111b52af6c1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-h4489" podUID="22eceb47-1bc2-4c0a-aae3-5111b52af6c1"
Jul 16 23:28:20.475969 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:20.475931    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-h4489"]
Jul 16 23:28:20.603931 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:20.603908    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-h4489"
Jul 16 23:28:20.636878 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:20.636836    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:28:20.641543 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:20.641524    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22eceb47-1bc2-4c0a-aae3-5111b52af6c1-maas-api-service-ca\") pod \"22eceb47-1bc2-4c0a-aae3-5111b52af6c1\" (UID: \"22eceb47-1bc2-4c0a-aae3-5111b52af6c1\") "
Jul 16 23:28:20.641829 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:20.641805    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22eceb47-1bc2-4c0a-aae3-5111b52af6c1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "22eceb47-1bc2-4c0a-aae3-5111b52af6c1" (UID: "22eceb47-1bc2-4c0a-aae3-5111b52af6c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:28:20.742761 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:20.742708    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22eceb47-1bc2-4c0a-aae3-5111b52af6c1-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:28:21.462140 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:21.462111    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-h4489" event={"ID":"22eceb47-1bc2-4c0a-aae3-5111b52af6c1","Type":"ContainerDied","Data":"acf4f3ee2a106ffd274618c409425b997752cb60dcfae8c883c448364d23a6a8"}
Jul 16 23:28:21.462140 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:21.462127    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-h4489"
Jul 16 23:28:21.462437 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:21.462166    2573 scope.go:117] "RemoveContainer" containerID="56301817f48447ee5213e605d75fa23140add891937eaa304cbc955281e98764"
Jul 16 23:28:21.482901 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:21.482877    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-h4489"]
Jul 16 23:28:21.488018 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:21.487999    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-h4489"]
Jul 16 23:28:22.518544 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:22.518509    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22eceb47-1bc2-4c0a-aae3-5111b52af6c1" path="/var/lib/kubelet/pods/22eceb47-1bc2-4c0a-aae3-5111b52af6c1/volumes"
Jul 16 23:28:32.462169 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:32.462138    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:28:36.116760 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:36.116725    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-48992b/sub-test-model-48992b-kserve-7d764f4ff5-dlnl8"]
Jul 16 23:28:36.147780 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:36.147736    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:28:37.023724 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:37.023687    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b4ab52/sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t"]
Jul 16 23:28:37.042509 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:37.042485    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:28:40.633967 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.633933    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-48992b/rate-test-model-48992b-kserve-69c46b5fbc-r7vtz"]
Jul 16 23:28:40.634372 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.634209    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e33fffcc-554c-41de-98bc-045d1107ed52" containerName="revoke-keys"
Jul 16 23:28:40.634372 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.634219    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e33fffcc-554c-41de-98bc-045d1107ed52" containerName="revoke-keys"
Jul 16 23:28:40.634372 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.634227    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22eceb47-1bc2-4c0a-aae3-5111b52af6c1" containerName="revoke-keys"
Jul 16 23:28:40.634372 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.634245    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="22eceb47-1bc2-4c0a-aae3-5111b52af6c1" containerName="revoke-keys"
Jul 16 23:28:40.634372 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.634257    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e33fffcc-554c-41de-98bc-045d1107ed52" containerName="revoke-keys"
Jul 16 23:28:40.634372 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.634266    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e33fffcc-554c-41de-98bc-045d1107ed52" containerName="revoke-keys"
Jul 16 23:28:40.634372 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.634279    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22eceb47-1bc2-4c0a-aae3-5111b52af6c1" containerName="revoke-keys"
Jul 16 23:28:40.634372 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.634284    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="22eceb47-1bc2-4c0a-aae3-5111b52af6c1" containerName="revoke-keys"
Jul 16 23:28:40.634372 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.634291    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55bad41d-1d6f-4753-8828-040d6a5cf5f7" containerName="revoke-keys"
Jul 16 23:28:40.634372 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.634296    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="55bad41d-1d6f-4753-8828-040d6a5cf5f7" containerName="revoke-keys"
Jul 16 23:28:40.634372 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.634307    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55bad41d-1d6f-4753-8828-040d6a5cf5f7" containerName="revoke-keys"
Jul 16 23:28:40.634372 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.634312    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="55bad41d-1d6f-4753-8828-040d6a5cf5f7" containerName="revoke-keys"
Jul 16 23:28:40.634372 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.634320    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55bad41d-1d6f-4753-8828-040d6a5cf5f7" containerName="revoke-keys"
Jul 16 23:28:40.634372 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.634324    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="55bad41d-1d6f-4753-8828-040d6a5cf5f7" containerName="revoke-keys"
Jul 16 23:28:40.634372 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.634368    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="55bad41d-1d6f-4753-8828-040d6a5cf5f7" containerName="revoke-keys"
Jul 16 23:28:40.634372 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.634376    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="22eceb47-1bc2-4c0a-aae3-5111b52af6c1" containerName="revoke-keys"
Jul 16 23:28:40.634372 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.634382    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="22eceb47-1bc2-4c0a-aae3-5111b52af6c1" containerName="revoke-keys"
Jul 16 23:28:40.634963 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.634387    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e33fffcc-554c-41de-98bc-045d1107ed52" containerName="revoke-keys"
Jul 16 23:28:40.634963 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.634393    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e33fffcc-554c-41de-98bc-045d1107ed52" containerName="revoke-keys"
Jul 16 23:28:40.634963 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.634398    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="22eceb47-1bc2-4c0a-aae3-5111b52af6c1" containerName="revoke-keys"
Jul 16 23:28:40.634963 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.634402    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e33fffcc-554c-41de-98bc-045d1107ed52" containerName="revoke-keys"
Jul 16 23:28:40.634963 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.634408    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="55bad41d-1d6f-4753-8828-040d6a5cf5f7" containerName="revoke-keys"
Jul 16 23:28:40.637124 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.637107    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-48992b/rate-test-model-48992b-kserve-69c46b5fbc-r7vtz"
Jul 16 23:28:40.639324 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.639305    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-48992b\"/\"rate-test-model-48992b-kserve-self-signed-certs\""
Jul 16 23:28:40.644899 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.644873    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-48992b/rate-test-model-48992b-kserve-69c46b5fbc-r7vtz"]
Jul 16 23:28:40.687551 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.687525    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1ed9736-96af-4bb9-b829-3bdb1c0ef10d-tls-certs\") pod \"rate-test-model-48992b-kserve-69c46b5fbc-r7vtz\" (UID: \"e1ed9736-96af-4bb9-b829-3bdb1c0ef10d\") " pod="ai-tenant-e2e-shared-a-48992b/rate-test-model-48992b-kserve-69c46b5fbc-r7vtz"
Jul 16 23:28:40.687665 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.687563    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1ed9736-96af-4bb9-b829-3bdb1c0ef10d-tmp-dir\") pod \"rate-test-model-48992b-kserve-69c46b5fbc-r7vtz\" (UID: \"e1ed9736-96af-4bb9-b829-3bdb1c0ef10d\") " pod="ai-tenant-e2e-shared-a-48992b/rate-test-model-48992b-kserve-69c46b5fbc-r7vtz"
Jul 16 23:28:40.687665 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.687586    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1ed9736-96af-4bb9-b829-3bdb1c0ef10d-home\") pod \"rate-test-model-48992b-kserve-69c46b5fbc-r7vtz\" (UID: \"e1ed9736-96af-4bb9-b829-3bdb1c0ef10d\") " pod="ai-tenant-e2e-shared-a-48992b/rate-test-model-48992b-kserve-69c46b5fbc-r7vtz"
Jul 16 23:28:40.687749 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.687668    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1ed9736-96af-4bb9-b829-3bdb1c0ef10d-dshm\") pod \"rate-test-model-48992b-kserve-69c46b5fbc-r7vtz\" (UID: \"e1ed9736-96af-4bb9-b829-3bdb1c0ef10d\") " pod="ai-tenant-e2e-shared-a-48992b/rate-test-model-48992b-kserve-69c46b5fbc-r7vtz"
Jul 16 23:28:40.687749 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.687704    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1ed9736-96af-4bb9-b829-3bdb1c0ef10d-model-cache\") pod \"rate-test-model-48992b-kserve-69c46b5fbc-r7vtz\" (UID: \"e1ed9736-96af-4bb9-b829-3bdb1c0ef10d\") " pod="ai-tenant-e2e-shared-a-48992b/rate-test-model-48992b-kserve-69c46b5fbc-r7vtz"
Jul 16 23:28:40.687749 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.687736    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzm5h\" (UniqueName: \"kubernetes.io/projected/e1ed9736-96af-4bb9-b829-3bdb1c0ef10d-kube-api-access-qzm5h\") pod \"rate-test-model-48992b-kserve-69c46b5fbc-r7vtz\" (UID: \"e1ed9736-96af-4bb9-b829-3bdb1c0ef10d\") " pod="ai-tenant-e2e-shared-a-48992b/rate-test-model-48992b-kserve-69c46b5fbc-r7vtz"
Jul 16 23:28:40.727913 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.727891    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:28:40.788982 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.788955    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1ed9736-96af-4bb9-b829-3bdb1c0ef10d-home\") pod \"rate-test-model-48992b-kserve-69c46b5fbc-r7vtz\" (UID: \"e1ed9736-96af-4bb9-b829-3bdb1c0ef10d\") " pod="ai-tenant-e2e-shared-a-48992b/rate-test-model-48992b-kserve-69c46b5fbc-r7vtz"
Jul 16 23:28:40.789099 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.788998    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1ed9736-96af-4bb9-b829-3bdb1c0ef10d-dshm\") pod \"rate-test-model-48992b-kserve-69c46b5fbc-r7vtz\" (UID: \"e1ed9736-96af-4bb9-b829-3bdb1c0ef10d\") " pod="ai-tenant-e2e-shared-a-48992b/rate-test-model-48992b-kserve-69c46b5fbc-r7vtz"
Jul 16 23:28:40.789099 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.789013    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1ed9736-96af-4bb9-b829-3bdb1c0ef10d-model-cache\") pod \"rate-test-model-48992b-kserve-69c46b5fbc-r7vtz\" (UID: \"e1ed9736-96af-4bb9-b829-3bdb1c0ef10d\") " pod="ai-tenant-e2e-shared-a-48992b/rate-test-model-48992b-kserve-69c46b5fbc-r7vtz"
Jul 16 23:28:40.789206 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.789102    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzm5h\" (UniqueName: \"kubernetes.io/projected/e1ed9736-96af-4bb9-b829-3bdb1c0ef10d-kube-api-access-qzm5h\") pod \"rate-test-model-48992b-kserve-69c46b5fbc-r7vtz\" (UID: \"e1ed9736-96af-4bb9-b829-3bdb1c0ef10d\") " pod="ai-tenant-e2e-shared-a-48992b/rate-test-model-48992b-kserve-69c46b5fbc-r7vtz"
Jul 16 23:28:40.789296 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.789215    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1ed9736-96af-4bb9-b829-3bdb1c0ef10d-tls-certs\") pod \"rate-test-model-48992b-kserve-69c46b5fbc-r7vtz\" (UID: \"e1ed9736-96af-4bb9-b829-3bdb1c0ef10d\") " pod="ai-tenant-e2e-shared-a-48992b/rate-test-model-48992b-kserve-69c46b5fbc-r7vtz"
Jul 16 23:28:40.789365 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.789317    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1ed9736-96af-4bb9-b829-3bdb1c0ef10d-tmp-dir\") pod \"rate-test-model-48992b-kserve-69c46b5fbc-r7vtz\" (UID: \"e1ed9736-96af-4bb9-b829-3bdb1c0ef10d\") " pod="ai-tenant-e2e-shared-a-48992b/rate-test-model-48992b-kserve-69c46b5fbc-r7vtz"
Jul 16 23:28:40.789473 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.789454    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1ed9736-96af-4bb9-b829-3bdb1c0ef10d-home\") pod \"rate-test-model-48992b-kserve-69c46b5fbc-r7vtz\" (UID: \"e1ed9736-96af-4bb9-b829-3bdb1c0ef10d\") " pod="ai-tenant-e2e-shared-a-48992b/rate-test-model-48992b-kserve-69c46b5fbc-r7vtz"
Jul 16 23:28:40.789527 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.789484    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1ed9736-96af-4bb9-b829-3bdb1c0ef10d-model-cache\") pod \"rate-test-model-48992b-kserve-69c46b5fbc-r7vtz\" (UID: \"e1ed9736-96af-4bb9-b829-3bdb1c0ef10d\") " pod="ai-tenant-e2e-shared-a-48992b/rate-test-model-48992b-kserve-69c46b5fbc-r7vtz"
Jul 16 23:28:40.789582 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.789563    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1ed9736-96af-4bb9-b829-3bdb1c0ef10d-tmp-dir\") pod \"rate-test-model-48992b-kserve-69c46b5fbc-r7vtz\" (UID: \"e1ed9736-96af-4bb9-b829-3bdb1c0ef10d\") " pod="ai-tenant-e2e-shared-a-48992b/rate-test-model-48992b-kserve-69c46b5fbc-r7vtz"
Jul 16 23:28:40.791122 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.791097    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1ed9736-96af-4bb9-b829-3bdb1c0ef10d-dshm\") pod \"rate-test-model-48992b-kserve-69c46b5fbc-r7vtz\" (UID: \"e1ed9736-96af-4bb9-b829-3bdb1c0ef10d\") " pod="ai-tenant-e2e-shared-a-48992b/rate-test-model-48992b-kserve-69c46b5fbc-r7vtz"
Jul 16 23:28:40.791519 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.791503    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1ed9736-96af-4bb9-b829-3bdb1c0ef10d-tls-certs\") pod \"rate-test-model-48992b-kserve-69c46b5fbc-r7vtz\" (UID: \"e1ed9736-96af-4bb9-b829-3bdb1c0ef10d\") " pod="ai-tenant-e2e-shared-a-48992b/rate-test-model-48992b-kserve-69c46b5fbc-r7vtz"
Jul 16 23:28:40.796799 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.796781    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzm5h\" (UniqueName: \"kubernetes.io/projected/e1ed9736-96af-4bb9-b829-3bdb1c0ef10d-kube-api-access-qzm5h\") pod \"rate-test-model-48992b-kserve-69c46b5fbc-r7vtz\" (UID: \"e1ed9736-96af-4bb9-b829-3bdb1c0ef10d\") " pod="ai-tenant-e2e-shared-a-48992b/rate-test-model-48992b-kserve-69c46b5fbc-r7vtz"
Jul 16 23:28:40.946303 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:40.946232    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-48992b/rate-test-model-48992b-kserve-69c46b5fbc-r7vtz"
Jul 16 23:28:41.075053 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:41.075026    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-48992b/rate-test-model-48992b-kserve-69c46b5fbc-r7vtz"]
Jul 16 23:28:41.076763 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:28:41.076732    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1ed9736_96af_4bb9_b829_3bdb1c0ef10d.slice/crio-dc0bd288d68c231ebff28820b27d1b67a7eefd2fac8063c735aa97eb3d1185fc WatchSource:0}: Error finding container dc0bd288d68c231ebff28820b27d1b67a7eefd2fac8063c735aa97eb3d1185fc: Status 404 returned error can't find the container with id dc0bd288d68c231ebff28820b27d1b67a7eefd2fac8063c735aa97eb3d1185fc
Jul 16 23:28:41.560505 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:41.560469    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-48992b/rate-test-model-48992b-kserve-69c46b5fbc-r7vtz" event={"ID":"e1ed9736-96af-4bb9-b829-3bdb1c0ef10d","Type":"ContainerStarted","Data":"208d6d2cee51f18ec67960f1e7e39377cf61c34a6647b89fbac93b365978cef3"}
Jul 16 23:28:41.560667 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:41.560517    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-48992b/rate-test-model-48992b-kserve-69c46b5fbc-r7vtz" event={"ID":"e1ed9736-96af-4bb9-b829-3bdb1c0ef10d","Type":"ContainerStarted","Data":"dc0bd288d68c231ebff28820b27d1b67a7eefd2fac8063c735aa97eb3d1185fc"}
Jul 16 23:28:42.029921 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:42.029869    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-48992b/rate-test-model-48992b-kserve-69c46b5fbc-r7vtz" podStartSLOduration=2.029852102 podStartE2EDuration="2.029852102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:28:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:28:41.576675978 +0000 UTC m=+3295.586938663" watchObservedRunningTime="2026-07-16 23:28:42.029852102 +0000 UTC m=+3296.040114764"
Jul 16 23:28:42.030590 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:42.030572    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:28:43.114643 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:43.114608    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:28:43.215764 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:43.215735    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:28:46.673728 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:46.673699    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r9jm4_506f904a-3272-4586-8528-146884e728f9/ovn-acl-logging/0.log"
Jul 16 23:28:46.682688 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:46.682667    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r9jm4_506f904a-3272-4586-8528-146884e728f9/ovn-acl-logging/0.log"
Jul 16 23:28:50.947149 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:50.947113    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-48992b/rate-test-model-48992b-kserve-69c46b5fbc-r7vtz"
Jul 16 23:28:50.947149 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:50.947158    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-48992b/rate-test-model-48992b-kserve-69c46b5fbc-r7vtz"
Jul 16 23:28:50.959029 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:50.959007    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-48992b/rate-test-model-48992b-kserve-69c46b5fbc-r7vtz"
Jul 16 23:28:51.135814 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:51.135772    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-48992b/sub-test-model-48992b-kserve-7d764f4ff5-dlnl8" podUID="08491056-c0d1-4e68-8d64-d1ffd7fd5c52" containerName="main" containerID="cri-o://b6a0f21fc1b0a08d6a676c13041a6a1adddbbe60ae1bee336854279b85450994" gracePeriod=45
Jul 16 23:28:51.383350 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:51.383320    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-48992b/sub-test-model-48992b-kserve-7d764f4ff5-dlnl8"
Jul 16 23:28:51.466641 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:51.466601    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08491056-c0d1-4e68-8d64-d1ffd7fd5c52-tmp-dir\") pod \"08491056-c0d1-4e68-8d64-d1ffd7fd5c52\" (UID: \"08491056-c0d1-4e68-8d64-d1ffd7fd5c52\") "
Jul 16 23:28:51.466641 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:51.466649    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08491056-c0d1-4e68-8d64-d1ffd7fd5c52-dshm\") pod \"08491056-c0d1-4e68-8d64-d1ffd7fd5c52\" (UID: \"08491056-c0d1-4e68-8d64-d1ffd7fd5c52\") "
Jul 16 23:28:51.466935 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:51.466678    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08491056-c0d1-4e68-8d64-d1ffd7fd5c52-home\") pod \"08491056-c0d1-4e68-8d64-d1ffd7fd5c52\" (UID: \"08491056-c0d1-4e68-8d64-d1ffd7fd5c52\") "
Jul 16 23:28:51.466935 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:51.466750    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08491056-c0d1-4e68-8d64-d1ffd7fd5c52-model-cache\") pod \"08491056-c0d1-4e68-8d64-d1ffd7fd5c52\" (UID: \"08491056-c0d1-4e68-8d64-d1ffd7fd5c52\") "
Jul 16 23:28:51.466935 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:51.466790    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08491056-c0d1-4e68-8d64-d1ffd7fd5c52-tls-certs\") pod \"08491056-c0d1-4e68-8d64-d1ffd7fd5c52\" (UID: \"08491056-c0d1-4e68-8d64-d1ffd7fd5c52\") "
Jul 16 23:28:51.466935 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:51.466815    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-flx9m\" (UniqueName: \"kubernetes.io/projected/08491056-c0d1-4e68-8d64-d1ffd7fd5c52-kube-api-access-flx9m\") pod \"08491056-c0d1-4e68-8d64-d1ffd7fd5c52\" (UID: \"08491056-c0d1-4e68-8d64-d1ffd7fd5c52\") "
Jul 16 23:28:51.467149 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:51.466975    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08491056-c0d1-4e68-8d64-d1ffd7fd5c52-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "08491056-c0d1-4e68-8d64-d1ffd7fd5c52" (UID: "08491056-c0d1-4e68-8d64-d1ffd7fd5c52"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:28:51.467149 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:51.466998    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08491056-c0d1-4e68-8d64-d1ffd7fd5c52-home" (OuterVolumeSpecName: "home") pod "08491056-c0d1-4e68-8d64-d1ffd7fd5c52" (UID: "08491056-c0d1-4e68-8d64-d1ffd7fd5c52"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:28:51.467149 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:51.467070    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08491056-c0d1-4e68-8d64-d1ffd7fd5c52-tmp-dir\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:28:51.467149 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:51.467079    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08491056-c0d1-4e68-8d64-d1ffd7fd5c52-model-cache" (OuterVolumeSpecName: "model-cache") pod "08491056-c0d1-4e68-8d64-d1ffd7fd5c52" (UID: "08491056-c0d1-4e68-8d64-d1ffd7fd5c52"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:28:51.467149 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:51.467095    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08491056-c0d1-4e68-8d64-d1ffd7fd5c52-home\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:28:51.468718 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:51.468692    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08491056-c0d1-4e68-8d64-d1ffd7fd5c52-dshm" (OuterVolumeSpecName: "dshm") pod "08491056-c0d1-4e68-8d64-d1ffd7fd5c52" (UID: "08491056-c0d1-4e68-8d64-d1ffd7fd5c52"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:28:51.468900 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:51.468880    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08491056-c0d1-4e68-8d64-d1ffd7fd5c52-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "08491056-c0d1-4e68-8d64-d1ffd7fd5c52" (UID: "08491056-c0d1-4e68-8d64-d1ffd7fd5c52"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 23:28:51.468981 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:51.468908    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08491056-c0d1-4e68-8d64-d1ffd7fd5c52-kube-api-access-flx9m" (OuterVolumeSpecName: "kube-api-access-flx9m") pod "08491056-c0d1-4e68-8d64-d1ffd7fd5c52" (UID: "08491056-c0d1-4e68-8d64-d1ffd7fd5c52"). InnerVolumeSpecName "kube-api-access-flx9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:28:51.568046 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:51.567957    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08491056-c0d1-4e68-8d64-d1ffd7fd5c52-model-cache\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:28:51.568046 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:51.567987    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08491056-c0d1-4e68-8d64-d1ffd7fd5c52-tls-certs\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:28:51.568046 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:51.567995    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flx9m\" (UniqueName: \"kubernetes.io/projected/08491056-c0d1-4e68-8d64-d1ffd7fd5c52-kube-api-access-flx9m\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:28:51.568046 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:51.568003    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08491056-c0d1-4e68-8d64-d1ffd7fd5c52-dshm\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:28:51.608779 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:51.608742    2573 generic.go:356] "Generic (PLEG): container finished" podID="08491056-c0d1-4e68-8d64-d1ffd7fd5c52" containerID="b6a0f21fc1b0a08d6a676c13041a6a1adddbbe60ae1bee336854279b85450994" exitCode=0
Jul 16 23:28:51.608779 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:51.608783    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-48992b/sub-test-model-48992b-kserve-7d764f4ff5-dlnl8"
Jul 16 23:28:51.609007 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:51.608830    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-48992b/sub-test-model-48992b-kserve-7d764f4ff5-dlnl8" event={"ID":"08491056-c0d1-4e68-8d64-d1ffd7fd5c52","Type":"ContainerDied","Data":"b6a0f21fc1b0a08d6a676c13041a6a1adddbbe60ae1bee336854279b85450994"}
Jul 16 23:28:51.609007 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:51.608874    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-48992b/sub-test-model-48992b-kserve-7d764f4ff5-dlnl8" event={"ID":"08491056-c0d1-4e68-8d64-d1ffd7fd5c52","Type":"ContainerDied","Data":"211812485ca86da1307f5694adde91055ba92a5a4aa7cf2aded954ad95ba5927"}
Jul 16 23:28:51.609007 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:51.608893    2573 scope.go:117] "RemoveContainer" containerID="b6a0f21fc1b0a08d6a676c13041a6a1adddbbe60ae1bee336854279b85450994"
Jul 16 23:28:51.620558 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:51.620508    2573 scope.go:117] "RemoveContainer" containerID="b6a0f21fc1b0a08d6a676c13041a6a1adddbbe60ae1bee336854279b85450994"
Jul 16 23:28:51.620936 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:28:51.620917    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6a0f21fc1b0a08d6a676c13041a6a1adddbbe60ae1bee336854279b85450994\": container with ID starting with b6a0f21fc1b0a08d6a676c13041a6a1adddbbe60ae1bee336854279b85450994 not found: ID does not exist" containerID="b6a0f21fc1b0a08d6a676c13041a6a1adddbbe60ae1bee336854279b85450994"
Jul 16 23:28:51.621035 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:51.620947    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6a0f21fc1b0a08d6a676c13041a6a1adddbbe60ae1bee336854279b85450994"} err="failed to get container status \"b6a0f21fc1b0a08d6a676c13041a6a1adddbbe60ae1bee336854279b85450994\": rpc error: code = NotFound desc = could not find container \"b6a0f21fc1b0a08d6a676c13041a6a1adddbbe60ae1bee336854279b85450994\": container with ID starting with b6a0f21fc1b0a08d6a676c13041a6a1adddbbe60ae1bee336854279b85450994 not found: ID does not exist"
Jul 16 23:28:51.622787 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:51.622769    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-48992b/rate-test-model-48992b-kserve-69c46b5fbc-r7vtz"
Jul 16 23:28:51.630319 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:51.630296    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-48992b/sub-test-model-48992b-kserve-7d764f4ff5-dlnl8"]
Jul 16 23:28:51.633838 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:51.633817    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-48992b/sub-test-model-48992b-kserve-7d764f4ff5-dlnl8"]
Jul 16 23:28:52.043118 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:52.043077    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-b4ab52/sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t" podUID="72d741fb-30cc-4aa0-9721-41ebfdc5e77d" containerName="main" containerID="cri-o://7b0666003e7247a3cb57bdf1aa243654a2f94cf268d4227903e1a8fc8c749dcc" gracePeriod=45
Jul 16 23:28:52.315983 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:52.315961    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b4ab52/sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t"
Jul 16 23:28:52.373854 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:52.373831    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72d741fb-30cc-4aa0-9721-41ebfdc5e77d-home\") pod \"72d741fb-30cc-4aa0-9721-41ebfdc5e77d\" (UID: \"72d741fb-30cc-4aa0-9721-41ebfdc5e77d\") "
Jul 16 23:28:52.373966 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:52.373861    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72d741fb-30cc-4aa0-9721-41ebfdc5e77d-dshm\") pod \"72d741fb-30cc-4aa0-9721-41ebfdc5e77d\" (UID: \"72d741fb-30cc-4aa0-9721-41ebfdc5e77d\") "
Jul 16 23:28:52.373966 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:52.373890    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72d741fb-30cc-4aa0-9721-41ebfdc5e77d-model-cache\") pod \"72d741fb-30cc-4aa0-9721-41ebfdc5e77d\" (UID: \"72d741fb-30cc-4aa0-9721-41ebfdc5e77d\") "
Jul 16 23:28:52.373966 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:52.373942    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72d741fb-30cc-4aa0-9721-41ebfdc5e77d-tmp-dir\") pod \"72d741fb-30cc-4aa0-9721-41ebfdc5e77d\" (UID: \"72d741fb-30cc-4aa0-9721-41ebfdc5e77d\") "
Jul 16 23:28:52.374089 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:52.373966    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72d741fb-30cc-4aa0-9721-41ebfdc5e77d-tls-certs\") pod \"72d741fb-30cc-4aa0-9721-41ebfdc5e77d\" (UID: \"72d741fb-30cc-4aa0-9721-41ebfdc5e77d\") "
Jul 16 23:28:52.374089 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:52.373995    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8f25r\" (UniqueName: \"kubernetes.io/projected/72d741fb-30cc-4aa0-9721-41ebfdc5e77d-kube-api-access-8f25r\") pod \"72d741fb-30cc-4aa0-9721-41ebfdc5e77d\" (UID: \"72d741fb-30cc-4aa0-9721-41ebfdc5e77d\") "
Jul 16 23:28:52.374188 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:52.374120    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72d741fb-30cc-4aa0-9721-41ebfdc5e77d-home" (OuterVolumeSpecName: "home") pod "72d741fb-30cc-4aa0-9721-41ebfdc5e77d" (UID: "72d741fb-30cc-4aa0-9721-41ebfdc5e77d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:28:52.374265 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:52.374178    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72d741fb-30cc-4aa0-9721-41ebfdc5e77d-model-cache" (OuterVolumeSpecName: "model-cache") pod "72d741fb-30cc-4aa0-9721-41ebfdc5e77d" (UID: "72d741fb-30cc-4aa0-9721-41ebfdc5e77d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:28:52.374265 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:52.374228    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72d741fb-30cc-4aa0-9721-41ebfdc5e77d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "72d741fb-30cc-4aa0-9721-41ebfdc5e77d" (UID: "72d741fb-30cc-4aa0-9721-41ebfdc5e77d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:28:52.374375 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:52.374298    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72d741fb-30cc-4aa0-9721-41ebfdc5e77d-home\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:28:52.374375 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:52.374311    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72d741fb-30cc-4aa0-9721-41ebfdc5e77d-model-cache\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:28:52.375828 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:52.375806    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72d741fb-30cc-4aa0-9721-41ebfdc5e77d-dshm" (OuterVolumeSpecName: "dshm") pod "72d741fb-30cc-4aa0-9721-41ebfdc5e77d" (UID: "72d741fb-30cc-4aa0-9721-41ebfdc5e77d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:28:52.375988 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:52.375933    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72d741fb-30cc-4aa0-9721-41ebfdc5e77d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "72d741fb-30cc-4aa0-9721-41ebfdc5e77d" (UID: "72d741fb-30cc-4aa0-9721-41ebfdc5e77d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 23:28:52.375988 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:52.375960    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72d741fb-30cc-4aa0-9721-41ebfdc5e77d-kube-api-access-8f25r" (OuterVolumeSpecName: "kube-api-access-8f25r") pod "72d741fb-30cc-4aa0-9721-41ebfdc5e77d" (UID: "72d741fb-30cc-4aa0-9721-41ebfdc5e77d"). InnerVolumeSpecName "kube-api-access-8f25r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:28:52.475639 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:52.475617    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72d741fb-30cc-4aa0-9721-41ebfdc5e77d-dshm\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:28:52.475639 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:52.475636    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72d741fb-30cc-4aa0-9721-41ebfdc5e77d-tmp-dir\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:28:52.475766 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:52.475646    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72d741fb-30cc-4aa0-9721-41ebfdc5e77d-tls-certs\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:28:52.475766 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:52.475654    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f25r\" (UniqueName: \"kubernetes.io/projected/72d741fb-30cc-4aa0-9721-41ebfdc5e77d-kube-api-access-8f25r\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:28:52.519525 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:52.519497    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08491056-c0d1-4e68-8d64-d1ffd7fd5c52" path="/var/lib/kubelet/pods/08491056-c0d1-4e68-8d64-d1ffd7fd5c52/volumes"
Jul 16 23:28:52.615405 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:52.615382    2573 generic.go:356] "Generic (PLEG): container finished" podID="72d741fb-30cc-4aa0-9721-41ebfdc5e77d" containerID="7b0666003e7247a3cb57bdf1aa243654a2f94cf268d4227903e1a8fc8c749dcc" exitCode=0
Jul 16 23:28:52.615510 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:52.615423    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b4ab52/sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t"
Jul 16 23:28:52.615510 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:52.615465    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b4ab52/sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t" event={"ID":"72d741fb-30cc-4aa0-9721-41ebfdc5e77d","Type":"ContainerDied","Data":"7b0666003e7247a3cb57bdf1aa243654a2f94cf268d4227903e1a8fc8c749dcc"}
Jul 16 23:28:52.615510 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:52.615503    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b4ab52/sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t" event={"ID":"72d741fb-30cc-4aa0-9721-41ebfdc5e77d","Type":"ContainerDied","Data":"7167d631e1ec186d3be93e4bf9baee0d5c09aa9ae7ff51d73d3ab818958eda64"}
Jul 16 23:28:52.615631 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:52.615526    2573 scope.go:117] "RemoveContainer" containerID="7b0666003e7247a3cb57bdf1aa243654a2f94cf268d4227903e1a8fc8c749dcc"
Jul 16 23:28:52.623467 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:52.623450    2573 scope.go:117] "RemoveContainer" containerID="7b0666003e7247a3cb57bdf1aa243654a2f94cf268d4227903e1a8fc8c749dcc"
Jul 16 23:28:52.623684 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:28:52.623665    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b0666003e7247a3cb57bdf1aa243654a2f94cf268d4227903e1a8fc8c749dcc\": container with ID starting with 7b0666003e7247a3cb57bdf1aa243654a2f94cf268d4227903e1a8fc8c749dcc not found: ID does not exist" containerID="7b0666003e7247a3cb57bdf1aa243654a2f94cf268d4227903e1a8fc8c749dcc"
Jul 16 23:28:52.623739 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:52.623690    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b0666003e7247a3cb57bdf1aa243654a2f94cf268d4227903e1a8fc8c749dcc"} err="failed to get container status \"7b0666003e7247a3cb57bdf1aa243654a2f94cf268d4227903e1a8fc8c749dcc\": rpc error: code = NotFound desc = could not find container \"7b0666003e7247a3cb57bdf1aa243654a2f94cf268d4227903e1a8fc8c749dcc\": container with ID starting with 7b0666003e7247a3cb57bdf1aa243654a2f94cf268d4227903e1a8fc8c749dcc not found: ID does not exist"
Jul 16 23:28:52.633738 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:52.633716    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b4ab52/sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t"]
Jul 16 23:28:52.637139 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:52.637121    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-b4ab52/sub-test-model-b4ab52-kserve-ff6c49d4c-4mc2t"]
Jul 16 23:28:54.518760 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:54.518723    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72d741fb-30cc-4aa0-9721-41ebfdc5e77d" path="/var/lib/kubelet/pods/72d741fb-30cc-4aa0-9721-41ebfdc5e77d/volumes"
Jul 16 23:28:55.131164 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:28:55.131129    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:29:04.085209 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:04.085176    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-b4ab52/rate-test-model-b4ab52-kserve-8865d75f-49kw6"]
Jul 16 23:29:04.085651 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:04.085470    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72d741fb-30cc-4aa0-9721-41ebfdc5e77d" containerName="main"
Jul 16 23:29:04.085651 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:04.085481    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="72d741fb-30cc-4aa0-9721-41ebfdc5e77d" containerName="main"
Jul 16 23:29:04.085651 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:04.085492    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e33fffcc-554c-41de-98bc-045d1107ed52" containerName="revoke-keys"
Jul 16 23:29:04.085651 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:04.085497    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e33fffcc-554c-41de-98bc-045d1107ed52" containerName="revoke-keys"
Jul 16 23:29:04.085651 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:04.085507    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22eceb47-1bc2-4c0a-aae3-5111b52af6c1" containerName="revoke-keys"
Jul 16 23:29:04.085651 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:04.085514    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="22eceb47-1bc2-4c0a-aae3-5111b52af6c1" containerName="revoke-keys"
Jul 16 23:29:04.085651 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:04.085526    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08491056-c0d1-4e68-8d64-d1ffd7fd5c52" containerName="main"
Jul 16 23:29:04.085651 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:04.085531    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="08491056-c0d1-4e68-8d64-d1ffd7fd5c52" containerName="main"
Jul 16 23:29:04.085651 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:04.085584    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="55bad41d-1d6f-4753-8828-040d6a5cf5f7" containerName="revoke-keys"
Jul 16 23:29:04.085651 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:04.085591    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="72d741fb-30cc-4aa0-9721-41ebfdc5e77d" containerName="main"
Jul 16 23:29:04.085651 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:04.085597    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="08491056-c0d1-4e68-8d64-d1ffd7fd5c52" containerName="main"
Jul 16 23:29:04.088411 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:04.088395    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b4ab52/rate-test-model-b4ab52-kserve-8865d75f-49kw6"
Jul 16 23:29:04.091600 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:04.091575    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b4ab52\"/\"default-dockercfg-b6pn6\""
Jul 16 23:29:04.091600 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:04.091592    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b4ab52\"/\"rate-test-model-b4ab52-kserve-self-signed-certs\""
Jul 16 23:29:04.091804 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:04.091592    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b4ab52\"/\"kube-root-ca.crt\""
Jul 16 23:29:04.091804 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:04.091656    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b4ab52\"/\"openshift-service-ca.crt\""
Jul 16 23:29:04.097400 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:04.097379    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b4ab52/rate-test-model-b4ab52-kserve-8865d75f-49kw6"]
Jul 16 23:29:04.157519 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:04.157489    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d81ed59-ec2a-4db9-b82e-0905ad43b31b-home\") pod \"rate-test-model-b4ab52-kserve-8865d75f-49kw6\" (UID: \"0d81ed59-ec2a-4db9-b82e-0905ad43b31b\") " pod="ai-tenant-e2e-shared-b-b4ab52/rate-test-model-b4ab52-kserve-8865d75f-49kw6"
Jul 16 23:29:04.157641 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:04.157532    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d81ed59-ec2a-4db9-b82e-0905ad43b31b-dshm\") pod \"rate-test-model-b4ab52-kserve-8865d75f-49kw6\" (UID: \"0d81ed59-ec2a-4db9-b82e-0905ad43b31b\") " pod="ai-tenant-e2e-shared-b-b4ab52/rate-test-model-b4ab52-kserve-8865d75f-49kw6"
Jul 16 23:29:04.157641 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:04.157563    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d81ed59-ec2a-4db9-b82e-0905ad43b31b-tmp-dir\") pod \"rate-test-model-b4ab52-kserve-8865d75f-49kw6\" (UID: \"0d81ed59-ec2a-4db9-b82e-0905ad43b31b\") " pod="ai-tenant-e2e-shared-b-b4ab52/rate-test-model-b4ab52-kserve-8865d75f-49kw6"
Jul 16 23:29:04.157641 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:04.157587    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d81ed59-ec2a-4db9-b82e-0905ad43b31b-tls-certs\") pod \"rate-test-model-b4ab52-kserve-8865d75f-49kw6\" (UID: \"0d81ed59-ec2a-4db9-b82e-0905ad43b31b\") " pod="ai-tenant-e2e-shared-b-b4ab52/rate-test-model-b4ab52-kserve-8865d75f-49kw6"
Jul 16 23:29:04.157641 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:04.157613    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8hvr\" (UniqueName: \"kubernetes.io/projected/0d81ed59-ec2a-4db9-b82e-0905ad43b31b-kube-api-access-n8hvr\") pod \"rate-test-model-b4ab52-kserve-8865d75f-49kw6\" (UID: \"0d81ed59-ec2a-4db9-b82e-0905ad43b31b\") " pod="ai-tenant-e2e-shared-b-b4ab52/rate-test-model-b4ab52-kserve-8865d75f-49kw6"
Jul 16 23:29:04.157831 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:04.157704    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d81ed59-ec2a-4db9-b82e-0905ad43b31b-model-cache\") pod \"rate-test-model-b4ab52-kserve-8865d75f-49kw6\" (UID: \"0d81ed59-ec2a-4db9-b82e-0905ad43b31b\") " pod="ai-tenant-e2e-shared-b-b4ab52/rate-test-model-b4ab52-kserve-8865d75f-49kw6"
Jul 16 23:29:04.258263 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:04.258224    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d81ed59-ec2a-4db9-b82e-0905ad43b31b-model-cache\") pod \"rate-test-model-b4ab52-kserve-8865d75f-49kw6\" (UID: \"0d81ed59-ec2a-4db9-b82e-0905ad43b31b\") " pod="ai-tenant-e2e-shared-b-b4ab52/rate-test-model-b4ab52-kserve-8865d75f-49kw6"
Jul 16 23:29:04.258368 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:04.258289    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d81ed59-ec2a-4db9-b82e-0905ad43b31b-home\") pod \"rate-test-model-b4ab52-kserve-8865d75f-49kw6\" (UID: \"0d81ed59-ec2a-4db9-b82e-0905ad43b31b\") " pod="ai-tenant-e2e-shared-b-b4ab52/rate-test-model-b4ab52-kserve-8865d75f-49kw6"
Jul 16 23:29:04.258368 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:04.258315    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d81ed59-ec2a-4db9-b82e-0905ad43b31b-dshm\") pod \"rate-test-model-b4ab52-kserve-8865d75f-49kw6\" (UID: \"0d81ed59-ec2a-4db9-b82e-0905ad43b31b\") " pod="ai-tenant-e2e-shared-b-b4ab52/rate-test-model-b4ab52-kserve-8865d75f-49kw6"
Jul 16 23:29:04.258368 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:04.258333    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d81ed59-ec2a-4db9-b82e-0905ad43b31b-tmp-dir\") pod \"rate-test-model-b4ab52-kserve-8865d75f-49kw6\" (UID: \"0d81ed59-ec2a-4db9-b82e-0905ad43b31b\") " pod="ai-tenant-e2e-shared-b-b4ab52/rate-test-model-b4ab52-kserve-8865d75f-49kw6"
Jul 16 23:29:04.258368 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:04.258346    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d81ed59-ec2a-4db9-b82e-0905ad43b31b-tls-certs\") pod \"rate-test-model-b4ab52-kserve-8865d75f-49kw6\" (UID: \"0d81ed59-ec2a-4db9-b82e-0905ad43b31b\") " pod="ai-tenant-e2e-shared-b-b4ab52/rate-test-model-b4ab52-kserve-8865d75f-49kw6"
Jul 16 23:29:04.258368 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:04.258362    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8hvr\" (UniqueName: \"kubernetes.io/projected/0d81ed59-ec2a-4db9-b82e-0905ad43b31b-kube-api-access-n8hvr\") pod \"rate-test-model-b4ab52-kserve-8865d75f-49kw6\" (UID: \"0d81ed59-ec2a-4db9-b82e-0905ad43b31b\") " pod="ai-tenant-e2e-shared-b-b4ab52/rate-test-model-b4ab52-kserve-8865d75f-49kw6"
Jul 16 23:29:04.258682 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:04.258660    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d81ed59-ec2a-4db9-b82e-0905ad43b31b-home\") pod \"rate-test-model-b4ab52-kserve-8865d75f-49kw6\" (UID: \"0d81ed59-ec2a-4db9-b82e-0905ad43b31b\") " pod="ai-tenant-e2e-shared-b-b4ab52/rate-test-model-b4ab52-kserve-8865d75f-49kw6"
Jul 16 23:29:04.258748 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:04.258704    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d81ed59-ec2a-4db9-b82e-0905ad43b31b-model-cache\") pod \"rate-test-model-b4ab52-kserve-8865d75f-49kw6\" (UID: \"0d81ed59-ec2a-4db9-b82e-0905ad43b31b\") " pod="ai-tenant-e2e-shared-b-b4ab52/rate-test-model-b4ab52-kserve-8865d75f-49kw6"
Jul 16 23:29:04.258748 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:04.258739    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d81ed59-ec2a-4db9-b82e-0905ad43b31b-tmp-dir\") pod \"rate-test-model-b4ab52-kserve-8865d75f-49kw6\" (UID: \"0d81ed59-ec2a-4db9-b82e-0905ad43b31b\") " pod="ai-tenant-e2e-shared-b-b4ab52/rate-test-model-b4ab52-kserve-8865d75f-49kw6"
Jul 16 23:29:04.260434 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:04.260418    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d81ed59-ec2a-4db9-b82e-0905ad43b31b-dshm\") pod \"rate-test-model-b4ab52-kserve-8865d75f-49kw6\" (UID: \"0d81ed59-ec2a-4db9-b82e-0905ad43b31b\") " pod="ai-tenant-e2e-shared-b-b4ab52/rate-test-model-b4ab52-kserve-8865d75f-49kw6"
Jul 16 23:29:04.260749 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:04.260733    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d81ed59-ec2a-4db9-b82e-0905ad43b31b-tls-certs\") pod \"rate-test-model-b4ab52-kserve-8865d75f-49kw6\" (UID: \"0d81ed59-ec2a-4db9-b82e-0905ad43b31b\") " pod="ai-tenant-e2e-shared-b-b4ab52/rate-test-model-b4ab52-kserve-8865d75f-49kw6"
Jul 16 23:29:04.267631 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:04.267609    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8hvr\" (UniqueName: \"kubernetes.io/projected/0d81ed59-ec2a-4db9-b82e-0905ad43b31b-kube-api-access-n8hvr\") pod \"rate-test-model-b4ab52-kserve-8865d75f-49kw6\" (UID: \"0d81ed59-ec2a-4db9-b82e-0905ad43b31b\") " pod="ai-tenant-e2e-shared-b-b4ab52/rate-test-model-b4ab52-kserve-8865d75f-49kw6"
Jul 16 23:29:04.397966 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:04.397912    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b4ab52/rate-test-model-b4ab52-kserve-8865d75f-49kw6"
Jul 16 23:29:04.520132 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:04.520107    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b4ab52/rate-test-model-b4ab52-kserve-8865d75f-49kw6"]
Jul 16 23:29:04.521248 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:29:04.521213    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d81ed59_ec2a_4db9_b82e_0905ad43b31b.slice/crio-6362051dc594baad13a22c376de8fabfabf531a7262039f8ea285dadae6d866c WatchSource:0}: Error finding container 6362051dc594baad13a22c376de8fabfabf531a7262039f8ea285dadae6d866c: Status 404 returned error can't find the container with id 6362051dc594baad13a22c376de8fabfabf531a7262039f8ea285dadae6d866c
Jul 16 23:29:04.671768 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:04.671691    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b4ab52/rate-test-model-b4ab52-kserve-8865d75f-49kw6" event={"ID":"0d81ed59-ec2a-4db9-b82e-0905ad43b31b","Type":"ContainerStarted","Data":"cc5c41f7ca7931ad75f4f2113c189388e2ada23764a4944d54ac477bb269bb05"}
Jul 16 23:29:04.671768 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:04.671735    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b4ab52/rate-test-model-b4ab52-kserve-8865d75f-49kw6" event={"ID":"0d81ed59-ec2a-4db9-b82e-0905ad43b31b","Type":"ContainerStarted","Data":"6362051dc594baad13a22c376de8fabfabf531a7262039f8ea285dadae6d866c"}
Jul 16 23:29:04.691024 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:04.690976    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-b4ab52/rate-test-model-b4ab52-kserve-8865d75f-49kw6" podStartSLOduration=0.690960381 podStartE2EDuration="690.960381ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:29:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:29:04.688710891 +0000 UTC m=+3318.698973575" watchObservedRunningTime="2026-07-16 23:29:04.690960381 +0000 UTC m=+3318.701223043"
Jul 16 23:29:04.970354 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:04.970265    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:29:10.339536 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:10.339454    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:29:14.398462 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:14.398431    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-b4ab52/rate-test-model-b4ab52-kserve-8865d75f-49kw6"
Jul 16 23:29:14.398831 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:14.398475    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-b4ab52/rate-test-model-b4ab52-kserve-8865d75f-49kw6"
Jul 16 23:29:14.410564 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:14.410543    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-b4ab52/rate-test-model-b4ab52-kserve-8865d75f-49kw6"
Jul 16 23:29:14.730499 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:14.730438    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-b4ab52/rate-test-model-b4ab52-kserve-8865d75f-49kw6"
Jul 16 23:29:19.814174 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:19.814128    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:29:19.914029 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:19.914002    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:29:21.229210 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:21.229177    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-lngs7"]
Jul 16 23:29:21.232449 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:21.232433    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-lngs7"
Jul 16 23:29:21.234673 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:21.234652    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:29:21.234749 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:21.234692    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 23:29:21.238995 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:21.238968    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-lngs7"]
Jul 16 23:29:21.390866 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:21.390838    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fcf89c1-4ac5-42cd-911d-dff2542d0db5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-lngs7\" (UID: \"7fcf89c1-4ac5-42cd-911d-dff2542d0db5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-lngs7"
Jul 16 23:29:21.491664 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:21.491596    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fcf89c1-4ac5-42cd-911d-dff2542d0db5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-lngs7\" (UID: \"7fcf89c1-4ac5-42cd-911d-dff2542d0db5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-lngs7"
Jul 16 23:29:21.492131 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:21.492113    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fcf89c1-4ac5-42cd-911d-dff2542d0db5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-lngs7\" (UID: \"7fcf89c1-4ac5-42cd-911d-dff2542d0db5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-lngs7"
Jul 16 23:29:21.544621 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:21.544602    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-lngs7"
Jul 16 23:29:21.670562 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:21.670539    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-lngs7"]
Jul 16 23:29:21.672423 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:29:21.672390    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fcf89c1_4ac5_42cd_911d_dff2542d0db5.slice/crio-66c7e7933d92918549fab0bd133b30784e823ef938f8466ded551809d5562a42 WatchSource:0}: Error finding container 66c7e7933d92918549fab0bd133b30784e823ef938f8466ded551809d5562a42: Status 404 returned error can't find the container with id 66c7e7933d92918549fab0bd133b30784e823ef938f8466ded551809d5562a42
Jul 16 23:29:21.752346 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:21.752282    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-lngs7" event={"ID":"7fcf89c1-4ac5-42cd-911d-dff2542d0db5","Type":"ContainerStarted","Data":"66c7e7933d92918549fab0bd133b30784e823ef938f8466ded551809d5562a42"}
Jul 16 23:29:22.758185 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:22.758085    2573 generic.go:356] "Generic (PLEG): container finished" podID="7fcf89c1-4ac5-42cd-911d-dff2542d0db5" containerID="572116245b2b67b65abe3827e9b040c7b6206b65888b20a24ed28299191e84a4" exitCode=6
Jul 16 23:29:22.758641 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:22.758179    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-lngs7" event={"ID":"7fcf89c1-4ac5-42cd-911d-dff2542d0db5","Type":"ContainerDied","Data":"572116245b2b67b65abe3827e9b040c7b6206b65888b20a24ed28299191e84a4"}
Jul 16 23:29:22.758641 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:22.758354    2573 scope.go:117] "RemoveContainer" containerID="572116245b2b67b65abe3827e9b040c7b6206b65888b20a24ed28299191e84a4"
Jul 16 23:29:23.765230 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:23.765142    2573 generic.go:356] "Generic (PLEG): container finished" podID="7fcf89c1-4ac5-42cd-911d-dff2542d0db5" containerID="cdd178f00a6cdc19b2f50cf4575e05aeb62d468aa48266214593e80177c9958c" exitCode=6
Jul 16 23:29:23.765628 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:23.765221    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-lngs7" event={"ID":"7fcf89c1-4ac5-42cd-911d-dff2542d0db5","Type":"ContainerDied","Data":"cdd178f00a6cdc19b2f50cf4575e05aeb62d468aa48266214593e80177c9958c"}
Jul 16 23:29:23.765628 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:23.765298    2573 scope.go:117] "RemoveContainer" containerID="572116245b2b67b65abe3827e9b040c7b6206b65888b20a24ed28299191e84a4"
Jul 16 23:29:23.765628 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:23.765491    2573 scope.go:117] "RemoveContainer" containerID="cdd178f00a6cdc19b2f50cf4575e05aeb62d468aa48266214593e80177c9958c"
Jul 16 23:29:23.765776 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:29:23.765707    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-lngs7_odh-ai-gateway-infra(7fcf89c1-4ac5-42cd-911d-dff2542d0db5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-lngs7" podUID="7fcf89c1-4ac5-42cd-911d-dff2542d0db5"
Jul 16 23:29:24.771686 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:24.771662    2573 scope.go:117] "RemoveContainer" containerID="cdd178f00a6cdc19b2f50cf4575e05aeb62d468aa48266214593e80177c9958c"
Jul 16 23:29:24.772028 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:29:24.771827    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-lngs7_odh-ai-gateway-infra(7fcf89c1-4ac5-42cd-911d-dff2542d0db5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-lngs7" podUID="7fcf89c1-4ac5-42cd-911d-dff2542d0db5"
Jul 16 23:29:25.139994 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:25.139960    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:29:28.119119 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:28.119086    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:29:28.228841 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:28.228815    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:29:36.182327 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:36.182289    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:29:36.283189 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:36.283158    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:29:38.489638 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:38.489610    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-69hcm"]
Jul 16 23:29:38.492874 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:38.492857    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-69hcm"
Jul 16 23:29:38.502761 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:38.502739    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-69hcm"]
Jul 16 23:29:38.619345 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:38.619309    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebf02700-9f96-48db-8f69-94e9f2bcb7ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-69hcm\" (UID: \"ebf02700-9f96-48db-8f69-94e9f2bcb7ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-69hcm"
Jul 16 23:29:38.720399 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:38.720367    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebf02700-9f96-48db-8f69-94e9f2bcb7ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-69hcm\" (UID: \"ebf02700-9f96-48db-8f69-94e9f2bcb7ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-69hcm"
Jul 16 23:29:38.720958 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:38.720940    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebf02700-9f96-48db-8f69-94e9f2bcb7ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-69hcm\" (UID: \"ebf02700-9f96-48db-8f69-94e9f2bcb7ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-69hcm"
Jul 16 23:29:38.802904 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:38.802850    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-69hcm"
Jul 16 23:29:38.922862 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:38.922838    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-69hcm"]
Jul 16 23:29:38.924584 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:29:38.924557    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebf02700_9f96_48db_8f69_94e9f2bcb7ac.slice/crio-946015b9c710e27faf4576fde06083784b4f528c658b92d95c8419a463664db9 WatchSource:0}: Error finding container 946015b9c710e27faf4576fde06083784b4f528c658b92d95c8419a463664db9: Status 404 returned error can't find the container with id 946015b9c710e27faf4576fde06083784b4f528c658b92d95c8419a463664db9
Jul 16 23:29:39.513490 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:39.513470    2573 scope.go:117] "RemoveContainer" containerID="cdd178f00a6cdc19b2f50cf4575e05aeb62d468aa48266214593e80177c9958c"
Jul 16 23:29:39.844306 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:39.844274    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-lngs7" event={"ID":"7fcf89c1-4ac5-42cd-911d-dff2542d0db5","Type":"ContainerStarted","Data":"86b146e3ac751c27656172e184f7d2c2e1ccbb4ab29f25cbca995b9dce9b5b8a"}
Jul 16 23:29:39.845591 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:39.845571    2573 generic.go:356] "Generic (PLEG): container finished" podID="ebf02700-9f96-48db-8f69-94e9f2bcb7ac" containerID="c4a8b2a76ad41f3fac2331a38934a468fd1dae449706c56e321138439dcaee6d" exitCode=6
Jul 16 23:29:39.845688 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:39.845638    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-69hcm" event={"ID":"ebf02700-9f96-48db-8f69-94e9f2bcb7ac","Type":"ContainerDied","Data":"c4a8b2a76ad41f3fac2331a38934a468fd1dae449706c56e321138439dcaee6d"}
Jul 16 23:29:39.845688 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:39.845667    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-69hcm" event={"ID":"ebf02700-9f96-48db-8f69-94e9f2bcb7ac","Type":"ContainerStarted","Data":"946015b9c710e27faf4576fde06083784b4f528c658b92d95c8419a463664db9"}
Jul 16 23:29:39.845765 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:39.845743    2573 scope.go:117] "RemoveContainer" containerID="c4a8b2a76ad41f3fac2331a38934a468fd1dae449706c56e321138439dcaee6d"
Jul 16 23:29:39.859342 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:39.859297    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-lngs7" podStartSLOduration=18.859279958 podStartE2EDuration="18.859279958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:29:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:29:39.857711797 +0000 UTC m=+3353.867974460" watchObservedRunningTime="2026-07-16 23:29:39.859279958 +0000 UTC m=+3353.869542621"
Jul 16 23:29:40.852150 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:40.852121    2573 generic.go:356] "Generic (PLEG): container finished" podID="ebf02700-9f96-48db-8f69-94e9f2bcb7ac" containerID="4576cdb88df2f6c118e575075be2e5ceae8a2e98ebfeda2bed53aaf8d38bf95e" exitCode=6
Jul 16 23:29:40.852558 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:40.852182    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-69hcm" event={"ID":"ebf02700-9f96-48db-8f69-94e9f2bcb7ac","Type":"ContainerDied","Data":"4576cdb88df2f6c118e575075be2e5ceae8a2e98ebfeda2bed53aaf8d38bf95e"}
Jul 16 23:29:40.852558 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:40.852264    2573 scope.go:117] "RemoveContainer" containerID="c4a8b2a76ad41f3fac2331a38934a468fd1dae449706c56e321138439dcaee6d"
Jul 16 23:29:40.852558 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:40.852437    2573 scope.go:117] "RemoveContainer" containerID="4576cdb88df2f6c118e575075be2e5ceae8a2e98ebfeda2bed53aaf8d38bf95e"
Jul 16 23:29:40.852717 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:29:40.852611    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-69hcm_odh-ai-gateway-infra(ebf02700-9f96-48db-8f69-94e9f2bcb7ac)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-69hcm" podUID="ebf02700-9f96-48db-8f69-94e9f2bcb7ac"
Jul 16 23:29:40.854186 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:40.854169    2573 generic.go:356] "Generic (PLEG): container finished" podID="7fcf89c1-4ac5-42cd-911d-dff2542d0db5" containerID="86b146e3ac751c27656172e184f7d2c2e1ccbb4ab29f25cbca995b9dce9b5b8a" exitCode=6
Jul 16 23:29:40.854299 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:40.854228    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-lngs7" event={"ID":"7fcf89c1-4ac5-42cd-911d-dff2542d0db5","Type":"ContainerDied","Data":"86b146e3ac751c27656172e184f7d2c2e1ccbb4ab29f25cbca995b9dce9b5b8a"}
Jul 16 23:29:40.854467 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:40.854455    2573 scope.go:117] "RemoveContainer" containerID="86b146e3ac751c27656172e184f7d2c2e1ccbb4ab29f25cbca995b9dce9b5b8a"
Jul 16 23:29:40.854618 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:29:40.854602    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-lngs7_odh-ai-gateway-infra(7fcf89c1-4ac5-42cd-911d-dff2542d0db5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-lngs7" podUID="7fcf89c1-4ac5-42cd-911d-dff2542d0db5"
Jul 16 23:29:40.863178 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:40.863151    2573 scope.go:117] "RemoveContainer" containerID="cdd178f00a6cdc19b2f50cf4575e05aeb62d468aa48266214593e80177c9958c"
Jul 16 23:29:40.867742 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:40.867722    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-lngs7"]
Jul 16 23:29:41.240790 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:41.240723    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:29:41.860425 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:41.860401    2573 scope.go:117] "RemoveContainer" containerID="4576cdb88df2f6c118e575075be2e5ceae8a2e98ebfeda2bed53aaf8d38bf95e"
Jul 16 23:29:41.860811 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:29:41.860562    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-69hcm_odh-ai-gateway-infra(ebf02700-9f96-48db-8f69-94e9f2bcb7ac)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-69hcm" podUID="ebf02700-9f96-48db-8f69-94e9f2bcb7ac"
Jul 16 23:29:41.990761 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:41.990735    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-lngs7"
Jul 16 23:29:42.145542 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:42.145482    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fcf89c1-4ac5-42cd-911d-dff2542d0db5-maas-api-service-ca\") pod \"7fcf89c1-4ac5-42cd-911d-dff2542d0db5\" (UID: \"7fcf89c1-4ac5-42cd-911d-dff2542d0db5\") "
Jul 16 23:29:42.145832 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:42.145813    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fcf89c1-4ac5-42cd-911d-dff2542d0db5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7fcf89c1-4ac5-42cd-911d-dff2542d0db5" (UID: "7fcf89c1-4ac5-42cd-911d-dff2542d0db5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:29:42.246601 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:42.246576    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fcf89c1-4ac5-42cd-911d-dff2542d0db5-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:29:42.867822 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:42.867794    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-lngs7"
Jul 16 23:29:42.868286 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:42.867796    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-lngs7" event={"ID":"7fcf89c1-4ac5-42cd-911d-dff2542d0db5","Type":"ContainerDied","Data":"66c7e7933d92918549fab0bd133b30784e823ef938f8466ded551809d5562a42"}
Jul 16 23:29:42.868286 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:42.867916    2573 scope.go:117] "RemoveContainer" containerID="86b146e3ac751c27656172e184f7d2c2e1ccbb4ab29f25cbca995b9dce9b5b8a"
Jul 16 23:29:42.902198 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:42.902174    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-lngs7"]
Jul 16 23:29:42.907366 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:42.907345    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-lngs7"]
Jul 16 23:29:44.518316 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:44.518279    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fcf89c1-4ac5-42cd-911d-dff2542d0db5" path="/var/lib/kubelet/pods/7fcf89c1-4ac5-42cd-911d-dff2542d0db5/volumes"
Jul 16 23:29:47.720787 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:47.720751    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-48992b/rate-test-model-48992b-kserve-69c46b5fbc-r7vtz"]
Jul 16 23:29:47.747940 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:47.747910    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:29:48.725730 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:48.725696    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b4ab52/rate-test-model-b4ab52-kserve-8865d75f-49kw6"]
Jul 16 23:29:49.144146 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:49.144112    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:29:53.058349 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:53.058317    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7299-139721934934960"]
Jul 16 23:29:53.058729 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:53.058615    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fcf89c1-4ac5-42cd-911d-dff2542d0db5" containerName="revoke-keys"
Jul 16 23:29:53.058729 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:53.058628    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fcf89c1-4ac5-42cd-911d-dff2542d0db5" containerName="revoke-keys"
Jul 16 23:29:53.058729 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:53.058648    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fcf89c1-4ac5-42cd-911d-dff2542d0db5" containerName="revoke-keys"
Jul 16 23:29:53.058729 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:53.058653    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fcf89c1-4ac5-42cd-911d-dff2542d0db5" containerName="revoke-keys"
Jul 16 23:29:53.058729 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:53.058659    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fcf89c1-4ac5-42cd-911d-dff2542d0db5" containerName="revoke-keys"
Jul 16 23:29:53.058729 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:53.058665    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fcf89c1-4ac5-42cd-911d-dff2542d0db5" containerName="revoke-keys"
Jul 16 23:29:53.058729 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:53.058724    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fcf89c1-4ac5-42cd-911d-dff2542d0db5" containerName="revoke-keys"
Jul 16 23:29:53.058729 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:53.058733    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fcf89c1-4ac5-42cd-911d-dff2542d0db5" containerName="revoke-keys"
Jul 16 23:29:53.063391 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:53.063367    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7299-139721934934960"
Jul 16 23:29:53.065611 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:53.065589    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zn5bl\""
Jul 16 23:29:53.066088 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:53.066059    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7299-139721934934960"]
Jul 16 23:29:53.127425 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:53.127402    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxkj4\" (UniqueName: \"kubernetes.io/projected/48080eab-168a-4240-848d-320961298455-kube-api-access-qxkj4\") pod \"test-curl-7299-139721934934960\" (UID: \"48080eab-168a-4240-848d-320961298455\") " pod="opendatahub/test-curl-7299-139721934934960"
Jul 16 23:29:53.228445 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:53.228422    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxkj4\" (UniqueName: \"kubernetes.io/projected/48080eab-168a-4240-848d-320961298455-kube-api-access-qxkj4\") pod \"test-curl-7299-139721934934960\" (UID: \"48080eab-168a-4240-848d-320961298455\") " pod="opendatahub/test-curl-7299-139721934934960"
Jul 16 23:29:53.236975 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:53.236954    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxkj4\" (UniqueName: \"kubernetes.io/projected/48080eab-168a-4240-848d-320961298455-kube-api-access-qxkj4\") pod \"test-curl-7299-139721934934960\" (UID: \"48080eab-168a-4240-848d-320961298455\") " pod="opendatahub/test-curl-7299-139721934934960"
Jul 16 23:29:53.373973 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:53.373945    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7299-139721934934960"
Jul 16 23:29:53.494920 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:53.494894    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7299-139721934934960"]
Jul 16 23:29:53.496997 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:29:53.496973    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48080eab_168a_4240_848d_320961298455.slice/crio-4a75165fd6425af30e2c0f6daf61df5abbf8f789e7d7f1844ac7e8746dd18ad5 WatchSource:0}: Error finding container 4a75165fd6425af30e2c0f6daf61df5abbf8f789e7d7f1844ac7e8746dd18ad5: Status 404 returned error can't find the container with id 4a75165fd6425af30e2c0f6daf61df5abbf8f789e7d7f1844ac7e8746dd18ad5
Jul 16 23:29:53.498296 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:53.498274    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 23:29:53.922895 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:53.922859    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7299-139721934934960" event={"ID":"48080eab-168a-4240-848d-320961298455","Type":"ContainerStarted","Data":"4a75165fd6425af30e2c0f6daf61df5abbf8f789e7d7f1844ac7e8746dd18ad5"}
Jul 16 23:29:54.929155 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:54.929125    2573 generic.go:356] "Generic (PLEG): container finished" podID="48080eab-168a-4240-848d-320961298455" containerID="e7cbedba4c87b528107c1f0b200b87c1ab2097ff2d2eb8c99509a0f587a6f72c" exitCode=0
Jul 16 23:29:54.929527 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:54.929188    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7299-139721934934960" event={"ID":"48080eab-168a-4240-848d-320961298455","Type":"ContainerDied","Data":"e7cbedba4c87b528107c1f0b200b87c1ab2097ff2d2eb8c99509a0f587a6f72c"}
Jul 16 23:29:55.513995 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:55.513969    2573 scope.go:117] "RemoveContainer" containerID="4576cdb88df2f6c118e575075be2e5ceae8a2e98ebfeda2bed53aaf8d38bf95e"
Jul 16 23:29:55.936051 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:55.936017    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-69hcm" event={"ID":"ebf02700-9f96-48db-8f69-94e9f2bcb7ac","Type":"ContainerStarted","Data":"b0cf8aa959d031668e2bf34dd6cfd0a8ffb65a417b932cb445fa46357300da2a"}
Jul 16 23:29:55.952741 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:55.952693    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-69hcm" podStartSLOduration=17.95267512 podStartE2EDuration="17.95267512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:29:55.951452423 +0000 UTC m=+3369.961715123" watchObservedRunningTime="2026-07-16 23:29:55.95267512 +0000 UTC m=+3369.962937782"
Jul 16 23:29:56.060657 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:56.060632    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7299-139721934934960"
Jul 16 23:29:56.079549 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:56.079526    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7299-139721934934960_48080eab-168a-4240-848d-320961298455/test-curl-7299-139721934934960/0.log"
Jul 16 23:29:56.099751 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:56.099722    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7299-139721934934960"]
Jul 16 23:29:56.103164 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:56.103140    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7299-139721934934960"]
Jul 16 23:29:56.151903 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:56.151873    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxkj4\" (UniqueName: \"kubernetes.io/projected/48080eab-168a-4240-848d-320961298455-kube-api-access-qxkj4\") pod \"48080eab-168a-4240-848d-320961298455\" (UID: \"48080eab-168a-4240-848d-320961298455\") "
Jul 16 23:29:56.153928 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:56.153907    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48080eab-168a-4240-848d-320961298455-kube-api-access-qxkj4" (OuterVolumeSpecName: "kube-api-access-qxkj4") pod "48080eab-168a-4240-848d-320961298455" (UID: "48080eab-168a-4240-848d-320961298455"). InnerVolumeSpecName "kube-api-access-qxkj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:29:56.231185 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:56.231151    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:29:56.253014 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:56.252989    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxkj4\" (UniqueName: \"kubernetes.io/projected/48080eab-168a-4240-848d-320961298455-kube-api-access-qxkj4\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:29:56.524042 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:56.524011    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48080eab-168a-4240-848d-320961298455" path="/var/lib/kubelet/pods/48080eab-168a-4240-848d-320961298455/volumes"
Jul 16 23:29:56.942972 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:56.942943    2573 generic.go:356] "Generic (PLEG): container finished" podID="ebf02700-9f96-48db-8f69-94e9f2bcb7ac" containerID="b0cf8aa959d031668e2bf34dd6cfd0a8ffb65a417b932cb445fa46357300da2a" exitCode=6
Jul 16 23:29:56.943398 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:56.943010    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-69hcm" event={"ID":"ebf02700-9f96-48db-8f69-94e9f2bcb7ac","Type":"ContainerDied","Data":"b0cf8aa959d031668e2bf34dd6cfd0a8ffb65a417b932cb445fa46357300da2a"}
Jul 16 23:29:56.943398 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:56.943060    2573 scope.go:117] "RemoveContainer" containerID="4576cdb88df2f6c118e575075be2e5ceae8a2e98ebfeda2bed53aaf8d38bf95e"
Jul 16 23:29:56.943398 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:56.943356    2573 scope.go:117] "RemoveContainer" containerID="b0cf8aa959d031668e2bf34dd6cfd0a8ffb65a417b932cb445fa46357300da2a"
Jul 16 23:29:56.943568 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:29:56.943547    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-69hcm_odh-ai-gateway-infra(ebf02700-9f96-48db-8f69-94e9f2bcb7ac)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-69hcm" podUID="ebf02700-9f96-48db-8f69-94e9f2bcb7ac"
Jul 16 23:29:56.944431 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:56.944410    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7299-139721934934960"
Jul 16 23:29:56.953977 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:56.953961    2573 scope.go:117] "RemoveContainer" containerID="e7cbedba4c87b528107c1f0b200b87c1ab2097ff2d2eb8c99509a0f587a6f72c"
Jul 16 23:29:56.967645 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:56.967625    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-69hcm"]
Jul 16 23:29:58.070330 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:58.070309    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-69hcm"
Jul 16 23:29:58.168010 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:58.167979    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebf02700-9f96-48db-8f69-94e9f2bcb7ac-maas-api-service-ca\") pod \"ebf02700-9f96-48db-8f69-94e9f2bcb7ac\" (UID: \"ebf02700-9f96-48db-8f69-94e9f2bcb7ac\") "
Jul 16 23:29:58.168318 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:58.168295    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebf02700-9f96-48db-8f69-94e9f2bcb7ac-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ebf02700-9f96-48db-8f69-94e9f2bcb7ac" (UID: "ebf02700-9f96-48db-8f69-94e9f2bcb7ac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:29:58.268982 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:58.268928    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebf02700-9f96-48db-8f69-94e9f2bcb7ac-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:29:58.957796 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:58.957776    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-69hcm"
Jul 16 23:29:58.957961 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:58.957814    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-69hcm" event={"ID":"ebf02700-9f96-48db-8f69-94e9f2bcb7ac","Type":"ContainerDied","Data":"946015b9c710e27faf4576fde06083784b4f528c658b92d95c8419a463664db9"}
Jul 16 23:29:58.957961 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:58.957879    2573 scope.go:117] "RemoveContainer" containerID="b0cf8aa959d031668e2bf34dd6cfd0a8ffb65a417b932cb445fa46357300da2a"
Jul 16 23:29:58.974923 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:58.974900    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-69hcm"]
Jul 16 23:29:58.978801 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:58.978780    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-69hcm"]
Jul 16 23:29:59.105220 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:59.105199    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7299-139721935029568"]
Jul 16 23:29:59.105501 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:59.105490    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48080eab-168a-4240-848d-320961298455" containerName="test-curl-7299-139721934934960"
Jul 16 23:29:59.105501 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:59.105502    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="48080eab-168a-4240-848d-320961298455" containerName="test-curl-7299-139721934934960"
Jul 16 23:29:59.105576 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:59.105512    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebf02700-9f96-48db-8f69-94e9f2bcb7ac" containerName="revoke-keys"
Jul 16 23:29:59.105576 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:59.105517    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebf02700-9f96-48db-8f69-94e9f2bcb7ac" containerName="revoke-keys"
Jul 16 23:29:59.105576 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:59.105524    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebf02700-9f96-48db-8f69-94e9f2bcb7ac" containerName="revoke-keys"
Jul 16 23:29:59.105576 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:59.105529    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebf02700-9f96-48db-8f69-94e9f2bcb7ac" containerName="revoke-keys"
Jul 16 23:29:59.105576 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:59.105536    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebf02700-9f96-48db-8f69-94e9f2bcb7ac" containerName="revoke-keys"
Jul 16 23:29:59.105576 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:59.105540    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebf02700-9f96-48db-8f69-94e9f2bcb7ac" containerName="revoke-keys"
Jul 16 23:29:59.105748 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:59.105589    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fcf89c1-4ac5-42cd-911d-dff2542d0db5" containerName="revoke-keys"
Jul 16 23:29:59.105748 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:59.105597    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebf02700-9f96-48db-8f69-94e9f2bcb7ac" containerName="revoke-keys"
Jul 16 23:29:59.105748 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:59.105603    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="48080eab-168a-4240-848d-320961298455" containerName="test-curl-7299-139721934934960"
Jul 16 23:29:59.105748 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:59.105609    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebf02700-9f96-48db-8f69-94e9f2bcb7ac" containerName="revoke-keys"
Jul 16 23:29:59.109706 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:59.109690    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7299-139721935029568"
Jul 16 23:29:59.112202 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:59.112078    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zn5bl\""
Jul 16 23:29:59.114607 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:59.114585    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7299-139721935029568"]
Jul 16 23:29:59.175598 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:59.175577    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9c7p\" (UniqueName: \"kubernetes.io/projected/8baec746-84e4-4791-8c52-f04de33af4a2-kube-api-access-m9c7p\") pod \"test-curl-7299-139721935029568\" (UID: \"8baec746-84e4-4791-8c52-f04de33af4a2\") " pod="opendatahub/test-curl-7299-139721935029568"
Jul 16 23:29:59.276283 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:59.276212    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9c7p\" (UniqueName: \"kubernetes.io/projected/8baec746-84e4-4791-8c52-f04de33af4a2-kube-api-access-m9c7p\") pod \"test-curl-7299-139721935029568\" (UID: \"8baec746-84e4-4791-8c52-f04de33af4a2\") " pod="opendatahub/test-curl-7299-139721935029568"
Jul 16 23:29:59.283944 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:59.283925    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9c7p\" (UniqueName: \"kubernetes.io/projected/8baec746-84e4-4791-8c52-f04de33af4a2-kube-api-access-m9c7p\") pod \"test-curl-7299-139721935029568\" (UID: \"8baec746-84e4-4791-8c52-f04de33af4a2\") " pod="opendatahub/test-curl-7299-139721935029568"
Jul 16 23:29:59.419610 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:59.419590    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7299-139721935029568"
Jul 16 23:29:59.540469 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:59.540443    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7299-139721935029568"]
Jul 16 23:29:59.542857 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:29:59.542826    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8baec746_84e4_4791_8c52_f04de33af4a2.slice/crio-c181262f3a9c545fd0480d17693d1bbbb24d18be5f316e08caf01caf168b12d9 WatchSource:0}: Error finding container c181262f3a9c545fd0480d17693d1bbbb24d18be5f316e08caf01caf168b12d9: Status 404 returned error can't find the container with id c181262f3a9c545fd0480d17693d1bbbb24d18be5f316e08caf01caf168b12d9
Jul 16 23:29:59.964822 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:59.964792    2573 generic.go:356] "Generic (PLEG): container finished" podID="8baec746-84e4-4791-8c52-f04de33af4a2" containerID="7b98cecaffd539a419a63c560543eda2cac0690aee67b165711d2b810709c733" exitCode=0
Jul 16 23:29:59.964937 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:59.964828    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7299-139721935029568" event={"ID":"8baec746-84e4-4791-8c52-f04de33af4a2","Type":"ContainerDied","Data":"7b98cecaffd539a419a63c560543eda2cac0690aee67b165711d2b810709c733"}
Jul 16 23:29:59.964937 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:29:59.964858    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7299-139721935029568" event={"ID":"8baec746-84e4-4791-8c52-f04de33af4a2","Type":"ContainerStarted","Data":"c181262f3a9c545fd0480d17693d1bbbb24d18be5f316e08caf01caf168b12d9"}
Jul 16 23:30:00.522104 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:00.522067    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebf02700-9f96-48db-8f69-94e9f2bcb7ac" path="/var/lib/kubelet/pods/ebf02700-9f96-48db-8f69-94e9f2bcb7ac/volumes"
Jul 16 23:30:01.087663 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:01.087645    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7299-139721935029568"
Jul 16 23:30:01.106369 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:01.106342    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7299-139721935029568_8baec746-84e4-4791-8c52-f04de33af4a2/test-curl-7299-139721935029568/0.log"
Jul 16 23:30:01.126920 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:01.126898    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7299-139721935029568"]
Jul 16 23:30:01.130262 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:01.130210    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7299-139721935029568"]
Jul 16 23:30:01.190172 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:01.190147    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9c7p\" (UniqueName: \"kubernetes.io/projected/8baec746-84e4-4791-8c52-f04de33af4a2-kube-api-access-m9c7p\") pod \"8baec746-84e4-4791-8c52-f04de33af4a2\" (UID: \"8baec746-84e4-4791-8c52-f04de33af4a2\") "
Jul 16 23:30:01.192190 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:01.192167    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8baec746-84e4-4791-8c52-f04de33af4a2-kube-api-access-m9c7p" (OuterVolumeSpecName: "kube-api-access-m9c7p") pod "8baec746-84e4-4791-8c52-f04de33af4a2" (UID: "8baec746-84e4-4791-8c52-f04de33af4a2"). InnerVolumeSpecName "kube-api-access-m9c7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:30:01.256025 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:01.255959    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7299-139721935030576"]
Jul 16 23:30:01.256312 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:01.256295    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8baec746-84e4-4791-8c52-f04de33af4a2" containerName="test-curl-7299-139721935029568"
Jul 16 23:30:01.256380 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:01.256315    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8baec746-84e4-4791-8c52-f04de33af4a2" containerName="test-curl-7299-139721935029568"
Jul 16 23:30:01.256431 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:01.256419    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebf02700-9f96-48db-8f69-94e9f2bcb7ac" containerName="revoke-keys"
Jul 16 23:30:01.256494 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:01.256432    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8baec746-84e4-4791-8c52-f04de33af4a2" containerName="test-curl-7299-139721935029568"
Jul 16 23:30:01.259293 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:01.259276    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7299-139721935030576"
Jul 16 23:30:01.268103 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:01.268076    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7299-139721935030576"]
Jul 16 23:30:01.291177 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:01.291148    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nt8h\" (UniqueName: \"kubernetes.io/projected/85ed6421-3f70-4978-a934-e987ccd57bea-kube-api-access-2nt8h\") pod \"test-curl-7299-139721935030576\" (UID: \"85ed6421-3f70-4978-a934-e987ccd57bea\") " pod="opendatahub/test-curl-7299-139721935030576"
Jul 16 23:30:01.291307 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:01.291260    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9c7p\" (UniqueName: \"kubernetes.io/projected/8baec746-84e4-4791-8c52-f04de33af4a2-kube-api-access-m9c7p\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:30:01.392153 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:01.392126    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nt8h\" (UniqueName: \"kubernetes.io/projected/85ed6421-3f70-4978-a934-e987ccd57bea-kube-api-access-2nt8h\") pod \"test-curl-7299-139721935030576\" (UID: \"85ed6421-3f70-4978-a934-e987ccd57bea\") " pod="opendatahub/test-curl-7299-139721935030576"
Jul 16 23:30:01.399986 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:01.399959    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nt8h\" (UniqueName: \"kubernetes.io/projected/85ed6421-3f70-4978-a934-e987ccd57bea-kube-api-access-2nt8h\") pod \"test-curl-7299-139721935030576\" (UID: \"85ed6421-3f70-4978-a934-e987ccd57bea\") " pod="opendatahub/test-curl-7299-139721935030576"
Jul 16 23:30:01.569481 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:01.569411    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7299-139721935030576"
Jul 16 23:30:01.692175 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:01.692136    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7299-139721935030576"]
Jul 16 23:30:01.694363 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:30:01.694337    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85ed6421_3f70_4978_a934_e987ccd57bea.slice/crio-0282eec39f8349ca902ac744228aff50894a438141638bcf2cb08930e9c2b36e WatchSource:0}: Error finding container 0282eec39f8349ca902ac744228aff50894a438141638bcf2cb08930e9c2b36e: Status 404 returned error can't find the container with id 0282eec39f8349ca902ac744228aff50894a438141638bcf2cb08930e9c2b36e
Jul 16 23:30:01.975178 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:01.975143    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7299-139721935030576" event={"ID":"85ed6421-3f70-4978-a934-e987ccd57bea","Type":"ContainerStarted","Data":"0282eec39f8349ca902ac744228aff50894a438141638bcf2cb08930e9c2b36e"}
Jul 16 23:30:01.976303 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:01.976287    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7299-139721935029568"
Jul 16 23:30:01.976378 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:01.976343    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c181262f3a9c545fd0480d17693d1bbbb24d18be5f316e08caf01caf168b12d9"
Jul 16 23:30:01.985953 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:30:01.985932    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7299-139721935029568\" is forbidden: User \"system:node:ip-10-0-140-146.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-146.ec2.internal' and this object" podUID="8baec746-84e4-4791-8c52-f04de33af4a2" pod="opendatahub/test-curl-7299-139721935029568"
Jul 16 23:30:02.518507 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:02.518453    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8baec746-84e4-4791-8c52-f04de33af4a2" path="/var/lib/kubelet/pods/8baec746-84e4-4791-8c52-f04de33af4a2/volumes"
Jul 16 23:30:02.739508 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:02.739480    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-48992b/rate-test-model-48992b-kserve-69c46b5fbc-r7vtz" podUID="e1ed9736-96af-4bb9-b829-3bdb1c0ef10d" containerName="main" containerID="cri-o://208d6d2cee51f18ec67960f1e7e39377cf61c34a6647b89fbac93b365978cef3" gracePeriod=45
Jul 16 23:30:02.982049 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:02.982028    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-48992b/rate-test-model-48992b-kserve-69c46b5fbc-r7vtz"
Jul 16 23:30:02.983021 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:02.982999    2573 generic.go:356] "Generic (PLEG): container finished" podID="e1ed9736-96af-4bb9-b829-3bdb1c0ef10d" containerID="208d6d2cee51f18ec67960f1e7e39377cf61c34a6647b89fbac93b365978cef3" exitCode=0
Jul 16 23:30:02.983130 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:02.983088    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-48992b/rate-test-model-48992b-kserve-69c46b5fbc-r7vtz" event={"ID":"e1ed9736-96af-4bb9-b829-3bdb1c0ef10d","Type":"ContainerDied","Data":"208d6d2cee51f18ec67960f1e7e39377cf61c34a6647b89fbac93b365978cef3"}
Jul 16 23:30:02.983183 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:02.983138    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-48992b/rate-test-model-48992b-kserve-69c46b5fbc-r7vtz" event={"ID":"e1ed9736-96af-4bb9-b829-3bdb1c0ef10d","Type":"ContainerDied","Data":"dc0bd288d68c231ebff28820b27d1b67a7eefd2fac8063c735aa97eb3d1185fc"}
Jul 16 23:30:02.983183 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:02.983161    2573 scope.go:117] "RemoveContainer" containerID="208d6d2cee51f18ec67960f1e7e39377cf61c34a6647b89fbac93b365978cef3"
Jul 16 23:30:02.984521 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:02.984501    2573 generic.go:356] "Generic (PLEG): container finished" podID="85ed6421-3f70-4978-a934-e987ccd57bea" containerID="c1f0619b84367bd804014fb304e22095e4427fc022740b79b7dc3bec922dc66a" exitCode=0
Jul 16 23:30:02.984620 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:02.984561    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7299-139721935030576" event={"ID":"85ed6421-3f70-4978-a934-e987ccd57bea","Type":"ContainerDied","Data":"c1f0619b84367bd804014fb304e22095e4427fc022740b79b7dc3bec922dc66a"}
Jul 16 23:30:02.991169 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:02.991153    2573 scope.go:117] "RemoveContainer" containerID="208d6d2cee51f18ec67960f1e7e39377cf61c34a6647b89fbac93b365978cef3"
Jul 16 23:30:02.991429 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:30:02.991409    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"208d6d2cee51f18ec67960f1e7e39377cf61c34a6647b89fbac93b365978cef3\": container with ID starting with 208d6d2cee51f18ec67960f1e7e39377cf61c34a6647b89fbac93b365978cef3 not found: ID does not exist" containerID="208d6d2cee51f18ec67960f1e7e39377cf61c34a6647b89fbac93b365978cef3"
Jul 16 23:30:02.991483 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:02.991437    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"208d6d2cee51f18ec67960f1e7e39377cf61c34a6647b89fbac93b365978cef3"} err="failed to get container status \"208d6d2cee51f18ec67960f1e7e39377cf61c34a6647b89fbac93b365978cef3\": rpc error: code = NotFound desc = could not find container \"208d6d2cee51f18ec67960f1e7e39377cf61c34a6647b89fbac93b365978cef3\": container with ID starting with 208d6d2cee51f18ec67960f1e7e39377cf61c34a6647b89fbac93b365978cef3 not found: ID does not exist"
Jul 16 23:30:03.107281 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:03.107183    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1ed9736-96af-4bb9-b829-3bdb1c0ef10d-dshm\") pod \"e1ed9736-96af-4bb9-b829-3bdb1c0ef10d\" (UID: \"e1ed9736-96af-4bb9-b829-3bdb1c0ef10d\") "
Jul 16 23:30:03.107281 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:03.107216    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1ed9736-96af-4bb9-b829-3bdb1c0ef10d-model-cache\") pod \"e1ed9736-96af-4bb9-b829-3bdb1c0ef10d\" (UID: \"e1ed9736-96af-4bb9-b829-3bdb1c0ef10d\") "
Jul 16 23:30:03.107281 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:03.107267    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1ed9736-96af-4bb9-b829-3bdb1c0ef10d-tmp-dir\") pod \"e1ed9736-96af-4bb9-b829-3bdb1c0ef10d\" (UID: \"e1ed9736-96af-4bb9-b829-3bdb1c0ef10d\") "
Jul 16 23:30:03.107509 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:03.107287    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1ed9736-96af-4bb9-b829-3bdb1c0ef10d-tls-certs\") pod \"e1ed9736-96af-4bb9-b829-3bdb1c0ef10d\" (UID: \"e1ed9736-96af-4bb9-b829-3bdb1c0ef10d\") "
Jul 16 23:30:03.107509 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:03.107312    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1ed9736-96af-4bb9-b829-3bdb1c0ef10d-home\") pod \"e1ed9736-96af-4bb9-b829-3bdb1c0ef10d\" (UID: \"e1ed9736-96af-4bb9-b829-3bdb1c0ef10d\") "
Jul 16 23:30:03.107509 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:03.107330    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qzm5h\" (UniqueName: \"kubernetes.io/projected/e1ed9736-96af-4bb9-b829-3bdb1c0ef10d-kube-api-access-qzm5h\") pod \"e1ed9736-96af-4bb9-b829-3bdb1c0ef10d\" (UID: \"e1ed9736-96af-4bb9-b829-3bdb1c0ef10d\") "
Jul 16 23:30:03.107681 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:03.107573    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1ed9736-96af-4bb9-b829-3bdb1c0ef10d-home" (OuterVolumeSpecName: "home") pod "e1ed9736-96af-4bb9-b829-3bdb1c0ef10d" (UID: "e1ed9736-96af-4bb9-b829-3bdb1c0ef10d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:30:03.107681 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:03.107581    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1ed9736-96af-4bb9-b829-3bdb1c0ef10d-model-cache" (OuterVolumeSpecName: "model-cache") pod "e1ed9736-96af-4bb9-b829-3bdb1c0ef10d" (UID: "e1ed9736-96af-4bb9-b829-3bdb1c0ef10d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:30:03.107681 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:03.107592    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1ed9736-96af-4bb9-b829-3bdb1c0ef10d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e1ed9736-96af-4bb9-b829-3bdb1c0ef10d" (UID: "e1ed9736-96af-4bb9-b829-3bdb1c0ef10d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:30:03.109345 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:03.109317    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1ed9736-96af-4bb9-b829-3bdb1c0ef10d-dshm" (OuterVolumeSpecName: "dshm") pod "e1ed9736-96af-4bb9-b829-3bdb1c0ef10d" (UID: "e1ed9736-96af-4bb9-b829-3bdb1c0ef10d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:30:03.109456 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:03.109357    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1ed9736-96af-4bb9-b829-3bdb1c0ef10d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e1ed9736-96af-4bb9-b829-3bdb1c0ef10d" (UID: "e1ed9736-96af-4bb9-b829-3bdb1c0ef10d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 23:30:03.109456 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:03.109361    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1ed9736-96af-4bb9-b829-3bdb1c0ef10d-kube-api-access-qzm5h" (OuterVolumeSpecName: "kube-api-access-qzm5h") pod "e1ed9736-96af-4bb9-b829-3bdb1c0ef10d" (UID: "e1ed9736-96af-4bb9-b829-3bdb1c0ef10d"). InnerVolumeSpecName "kube-api-access-qzm5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:30:03.208854 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:03.208825    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1ed9736-96af-4bb9-b829-3bdb1c0ef10d-tmp-dir\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:30:03.208854 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:03.208849    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1ed9736-96af-4bb9-b829-3bdb1c0ef10d-tls-certs\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:30:03.208854 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:03.208857    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1ed9736-96af-4bb9-b829-3bdb1c0ef10d-home\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:30:03.209015 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:03.208866    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzm5h\" (UniqueName: \"kubernetes.io/projected/e1ed9736-96af-4bb9-b829-3bdb1c0ef10d-kube-api-access-qzm5h\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:30:03.209015 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:03.208875    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1ed9736-96af-4bb9-b829-3bdb1c0ef10d-dshm\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:30:03.209015 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:03.208882    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1ed9736-96af-4bb9-b829-3bdb1c0ef10d-model-cache\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:30:03.744443 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:03.744399    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-b4ab52/rate-test-model-b4ab52-kserve-8865d75f-49kw6" podUID="0d81ed59-ec2a-4db9-b82e-0905ad43b31b" containerName="main" containerID="cri-o://cc5c41f7ca7931ad75f4f2113c189388e2ada23764a4944d54ac477bb269bb05" gracePeriod=45
Jul 16 23:30:03.980586 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:03.980560    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b4ab52/rate-test-model-b4ab52-kserve-8865d75f-49kw6"
Jul 16 23:30:03.990112 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:03.990085    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-48992b/rate-test-model-48992b-kserve-69c46b5fbc-r7vtz"
Jul 16 23:30:03.997610 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:03.997536    2573 generic.go:356] "Generic (PLEG): container finished" podID="0d81ed59-ec2a-4db9-b82e-0905ad43b31b" containerID="cc5c41f7ca7931ad75f4f2113c189388e2ada23764a4944d54ac477bb269bb05" exitCode=0
Jul 16 23:30:03.998750 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:03.997678    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b4ab52/rate-test-model-b4ab52-kserve-8865d75f-49kw6"
Jul 16 23:30:03.998975 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:03.997707    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b4ab52/rate-test-model-b4ab52-kserve-8865d75f-49kw6" event={"ID":"0d81ed59-ec2a-4db9-b82e-0905ad43b31b","Type":"ContainerDied","Data":"cc5c41f7ca7931ad75f4f2113c189388e2ada23764a4944d54ac477bb269bb05"}
Jul 16 23:30:03.999095 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:03.999006    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b4ab52/rate-test-model-b4ab52-kserve-8865d75f-49kw6" event={"ID":"0d81ed59-ec2a-4db9-b82e-0905ad43b31b","Type":"ContainerDied","Data":"6362051dc594baad13a22c376de8fabfabf531a7262039f8ea285dadae6d866c"}
Jul 16 23:30:03.999095 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:03.999036    2573 scope.go:117] "RemoveContainer" containerID="cc5c41f7ca7931ad75f4f2113c189388e2ada23764a4944d54ac477bb269bb05"
Jul 16 23:30:04.011743 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.011301    2573 scope.go:117] "RemoveContainer" containerID="cc5c41f7ca7931ad75f4f2113c189388e2ada23764a4944d54ac477bb269bb05"
Jul 16 23:30:04.011743 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:30:04.011603    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc5c41f7ca7931ad75f4f2113c189388e2ada23764a4944d54ac477bb269bb05\": container with ID starting with cc5c41f7ca7931ad75f4f2113c189388e2ada23764a4944d54ac477bb269bb05 not found: ID does not exist" containerID="cc5c41f7ca7931ad75f4f2113c189388e2ada23764a4944d54ac477bb269bb05"
Jul 16 23:30:04.011743 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.011629    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc5c41f7ca7931ad75f4f2113c189388e2ada23764a4944d54ac477bb269bb05"} err="failed to get container status \"cc5c41f7ca7931ad75f4f2113c189388e2ada23764a4944d54ac477bb269bb05\": rpc error: code = NotFound desc = could not find container \"cc5c41f7ca7931ad75f4f2113c189388e2ada23764a4944d54ac477bb269bb05\": container with ID starting with cc5c41f7ca7931ad75f4f2113c189388e2ada23764a4944d54ac477bb269bb05 not found: ID does not exist"
Jul 16 23:30:04.013982 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.013957    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d81ed59-ec2a-4db9-b82e-0905ad43b31b-home\") pod \"0d81ed59-ec2a-4db9-b82e-0905ad43b31b\" (UID: \"0d81ed59-ec2a-4db9-b82e-0905ad43b31b\") "
Jul 16 23:30:04.014078 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.014044    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d81ed59-ec2a-4db9-b82e-0905ad43b31b-tls-certs\") pod \"0d81ed59-ec2a-4db9-b82e-0905ad43b31b\" (UID: \"0d81ed59-ec2a-4db9-b82e-0905ad43b31b\") "
Jul 16 23:30:04.014150 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.014091    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d81ed59-ec2a-4db9-b82e-0905ad43b31b-tmp-dir\") pod \"0d81ed59-ec2a-4db9-b82e-0905ad43b31b\" (UID: \"0d81ed59-ec2a-4db9-b82e-0905ad43b31b\") "
Jul 16 23:30:04.014150 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.014135    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d81ed59-ec2a-4db9-b82e-0905ad43b31b-dshm\") pod \"0d81ed59-ec2a-4db9-b82e-0905ad43b31b\" (UID: \"0d81ed59-ec2a-4db9-b82e-0905ad43b31b\") "
Jul 16 23:30:04.014282 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.014158    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d81ed59-ec2a-4db9-b82e-0905ad43b31b-model-cache\") pod \"0d81ed59-ec2a-4db9-b82e-0905ad43b31b\" (UID: \"0d81ed59-ec2a-4db9-b82e-0905ad43b31b\") "
Jul 16 23:30:04.014282 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.014188    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8hvr\" (UniqueName: \"kubernetes.io/projected/0d81ed59-ec2a-4db9-b82e-0905ad43b31b-kube-api-access-n8hvr\") pod \"0d81ed59-ec2a-4db9-b82e-0905ad43b31b\" (UID: \"0d81ed59-ec2a-4db9-b82e-0905ad43b31b\") "
Jul 16 23:30:04.014282 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.014208    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d81ed59-ec2a-4db9-b82e-0905ad43b31b-home" (OuterVolumeSpecName: "home") pod "0d81ed59-ec2a-4db9-b82e-0905ad43b31b" (UID: "0d81ed59-ec2a-4db9-b82e-0905ad43b31b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:30:04.014456 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.014382    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d81ed59-ec2a-4db9-b82e-0905ad43b31b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0d81ed59-ec2a-4db9-b82e-0905ad43b31b" (UID: "0d81ed59-ec2a-4db9-b82e-0905ad43b31b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:30:04.014520 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.014477    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d81ed59-ec2a-4db9-b82e-0905ad43b31b-model-cache" (OuterVolumeSpecName: "model-cache") pod "0d81ed59-ec2a-4db9-b82e-0905ad43b31b" (UID: "0d81ed59-ec2a-4db9-b82e-0905ad43b31b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:30:04.014520 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.014487    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d81ed59-ec2a-4db9-b82e-0905ad43b31b-tmp-dir\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:30:04.014520 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.014505    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d81ed59-ec2a-4db9-b82e-0905ad43b31b-home\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:30:04.016618 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.016596    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d81ed59-ec2a-4db9-b82e-0905ad43b31b-kube-api-access-n8hvr" (OuterVolumeSpecName: "kube-api-access-n8hvr") pod "0d81ed59-ec2a-4db9-b82e-0905ad43b31b" (UID: "0d81ed59-ec2a-4db9-b82e-0905ad43b31b"). InnerVolumeSpecName "kube-api-access-n8hvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:30:04.016753 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.016728    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d81ed59-ec2a-4db9-b82e-0905ad43b31b-dshm" (OuterVolumeSpecName: "dshm") pod "0d81ed59-ec2a-4db9-b82e-0905ad43b31b" (UID: "0d81ed59-ec2a-4db9-b82e-0905ad43b31b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:30:04.016855 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.016838    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d81ed59-ec2a-4db9-b82e-0905ad43b31b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0d81ed59-ec2a-4db9-b82e-0905ad43b31b" (UID: "0d81ed59-ec2a-4db9-b82e-0905ad43b31b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 23:30:04.018021 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.018001    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-48992b/rate-test-model-48992b-kserve-69c46b5fbc-r7vtz"]
Jul 16 23:30:04.024183 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.024163    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-48992b/rate-test-model-48992b-kserve-69c46b5fbc-r7vtz"]
Jul 16 23:30:04.115698 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.115668    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d81ed59-ec2a-4db9-b82e-0905ad43b31b-tls-certs\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:30:04.115698 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.115696    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d81ed59-ec2a-4db9-b82e-0905ad43b31b-dshm\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:30:04.115820 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.115710    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d81ed59-ec2a-4db9-b82e-0905ad43b31b-model-cache\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:30:04.115820 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.115725    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8hvr\" (UniqueName: \"kubernetes.io/projected/0d81ed59-ec2a-4db9-b82e-0905ad43b31b-kube-api-access-n8hvr\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:30:04.121724 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.121704    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7299-139721935030576"
Jul 16 23:30:04.140433 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.140413    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7299-139721935030576_85ed6421-3f70-4978-a934-e987ccd57bea/test-curl-7299-139721935030576/0.log"
Jul 16 23:30:04.163781 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.163756    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7299-139721935030576"]
Jul 16 23:30:04.167286 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.167266    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7299-139721935030576"]
Jul 16 23:30:04.216279 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.216257    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2nt8h\" (UniqueName: \"kubernetes.io/projected/85ed6421-3f70-4978-a934-e987ccd57bea-kube-api-access-2nt8h\") pod \"85ed6421-3f70-4978-a934-e987ccd57bea\" (UID: \"85ed6421-3f70-4978-a934-e987ccd57bea\") "
Jul 16 23:30:04.218172 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.218144    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85ed6421-3f70-4978-a934-e987ccd57bea-kube-api-access-2nt8h" (OuterVolumeSpecName: "kube-api-access-2nt8h") pod "85ed6421-3f70-4978-a934-e987ccd57bea" (UID: "85ed6421-3f70-4978-a934-e987ccd57bea"). InnerVolumeSpecName "kube-api-access-2nt8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:30:04.283486 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.283431    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7299-139721935030576"]
Jul 16 23:30:04.283719 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.283707    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d81ed59-ec2a-4db9-b82e-0905ad43b31b" containerName="main"
Jul 16 23:30:04.283763 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.283720    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d81ed59-ec2a-4db9-b82e-0905ad43b31b" containerName="main"
Jul 16 23:30:04.283763 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.283731    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1ed9736-96af-4bb9-b829-3bdb1c0ef10d" containerName="main"
Jul 16 23:30:04.283763 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.283736    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1ed9736-96af-4bb9-b829-3bdb1c0ef10d" containerName="main"
Jul 16 23:30:04.283763 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.283747    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85ed6421-3f70-4978-a934-e987ccd57bea" containerName="test-curl-7299-139721935030576"
Jul 16 23:30:04.283763 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.283753    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="85ed6421-3f70-4978-a934-e987ccd57bea" containerName="test-curl-7299-139721935030576"
Jul 16 23:30:04.283899 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.283803    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d81ed59-ec2a-4db9-b82e-0905ad43b31b" containerName="main"
Jul 16 23:30:04.283899 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.283810    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="85ed6421-3f70-4978-a934-e987ccd57bea" containerName="test-curl-7299-139721935030576"
Jul 16 23:30:04.283899 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.283819    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1ed9736-96af-4bb9-b829-3bdb1c0ef10d" containerName="main"
Jul 16 23:30:04.286602 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.286588    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7299-139721935030576"
Jul 16 23:30:04.291867 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.291848    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7299-139721935030576"]
Jul 16 23:30:04.317349 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.317326    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzncf\" (UniqueName: \"kubernetes.io/projected/c0b142e2-2afb-435f-b076-1dc1b1dbc0ae-kube-api-access-wzncf\") pod \"test-curl-7299-139721935030576\" (UID: \"c0b142e2-2afb-435f-b076-1dc1b1dbc0ae\") " pod="opendatahub/test-curl-7299-139721935030576"
Jul 16 23:30:04.317467 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.317394    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nt8h\" (UniqueName: \"kubernetes.io/projected/85ed6421-3f70-4978-a934-e987ccd57bea-kube-api-access-2nt8h\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:30:04.320211 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.320187    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b4ab52/rate-test-model-b4ab52-kserve-8865d75f-49kw6"]
Jul 16 23:30:04.325230 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.325206    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-b4ab52/rate-test-model-b4ab52-kserve-8865d75f-49kw6"]
Jul 16 23:30:04.417717 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.417694    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzncf\" (UniqueName: \"kubernetes.io/projected/c0b142e2-2afb-435f-b076-1dc1b1dbc0ae-kube-api-access-wzncf\") pod \"test-curl-7299-139721935030576\" (UID: \"c0b142e2-2afb-435f-b076-1dc1b1dbc0ae\") " pod="opendatahub/test-curl-7299-139721935030576"
Jul 16 23:30:04.425962 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.425942    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzncf\" (UniqueName: \"kubernetes.io/projected/c0b142e2-2afb-435f-b076-1dc1b1dbc0ae-kube-api-access-wzncf\") pod \"test-curl-7299-139721935030576\" (UID: \"c0b142e2-2afb-435f-b076-1dc1b1dbc0ae\") " pod="opendatahub/test-curl-7299-139721935030576"
Jul 16 23:30:04.518399 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.518374    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d81ed59-ec2a-4db9-b82e-0905ad43b31b" path="/var/lib/kubelet/pods/0d81ed59-ec2a-4db9-b82e-0905ad43b31b/volumes"
Jul 16 23:30:04.518691 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.518678    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85ed6421-3f70-4978-a934-e987ccd57bea" path="/var/lib/kubelet/pods/85ed6421-3f70-4978-a934-e987ccd57bea/volumes"
Jul 16 23:30:04.519004 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.518992    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1ed9736-96af-4bb9-b829-3bdb1c0ef10d" path="/var/lib/kubelet/pods/e1ed9736-96af-4bb9-b829-3bdb1c0ef10d/volumes"
Jul 16 23:30:04.595840 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.595821    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7299-139721935030576"
Jul 16 23:30:04.722127 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:04.722105    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7299-139721935030576"]
Jul 16 23:30:04.724082 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:30:04.724056    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0b142e2_2afb_435f_b076_1dc1b1dbc0ae.slice/crio-b095b76862cbdcfb5791e8f3639dbda4dc891c149265103d240a1c2163c08747 WatchSource:0}: Error finding container b095b76862cbdcfb5791e8f3639dbda4dc891c149265103d240a1c2163c08747: Status 404 returned error can't find the container with id b095b76862cbdcfb5791e8f3639dbda4dc891c149265103d240a1c2163c08747
Jul 16 23:30:05.005519 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:05.005493    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7299-139721935030576"
Jul 16 23:30:05.005867 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:05.005520    2573 scope.go:117] "RemoveContainer" containerID="c1f0619b84367bd804014fb304e22095e4427fc022740b79b7dc3bec922dc66a"
Jul 16 23:30:05.006668 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:05.006645    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7299-139721935030576" event={"ID":"c0b142e2-2afb-435f-b076-1dc1b1dbc0ae","Type":"ContainerStarted","Data":"b095b76862cbdcfb5791e8f3639dbda4dc891c149265103d240a1c2163c08747"}
Jul 16 23:30:06.013787 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:06.013755    2573 generic.go:356] "Generic (PLEG): container finished" podID="c0b142e2-2afb-435f-b076-1dc1b1dbc0ae" containerID="f4d433e0318724e8bf8801feaa798becf27cfbd522bbb9d23b925d4fbcd72550" exitCode=0
Jul 16 23:30:06.014141 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:06.013841    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7299-139721935030576" event={"ID":"c0b142e2-2afb-435f-b076-1dc1b1dbc0ae","Type":"ContainerDied","Data":"f4d433e0318724e8bf8801feaa798becf27cfbd522bbb9d23b925d4fbcd72550"}
Jul 16 23:30:07.141147 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:07.141118    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7299-139721935030576"
Jul 16 23:30:07.160203 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:07.160177    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7299-139721935030576_c0b142e2-2afb-435f-b076-1dc1b1dbc0ae/test-curl-7299-139721935030576/0.log"
Jul 16 23:30:07.181788 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:07.181763    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7299-139721935030576"]
Jul 16 23:30:07.187701 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:07.187679    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7299-139721935030576"]
Jul 16 23:30:07.237152 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:07.237131    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzncf\" (UniqueName: \"kubernetes.io/projected/c0b142e2-2afb-435f-b076-1dc1b1dbc0ae-kube-api-access-wzncf\") pod \"c0b142e2-2afb-435f-b076-1dc1b1dbc0ae\" (UID: \"c0b142e2-2afb-435f-b076-1dc1b1dbc0ae\") "
Jul 16 23:30:07.239049 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:07.239022    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0b142e2-2afb-435f-b076-1dc1b1dbc0ae-kube-api-access-wzncf" (OuterVolumeSpecName: "kube-api-access-wzncf") pod "c0b142e2-2afb-435f-b076-1dc1b1dbc0ae" (UID: "c0b142e2-2afb-435f-b076-1dc1b1dbc0ae"). InnerVolumeSpecName "kube-api-access-wzncf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:30:07.338181 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:07.338154    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzncf\" (UniqueName: \"kubernetes.io/projected/c0b142e2-2afb-435f-b076-1dc1b1dbc0ae-kube-api-access-wzncf\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:30:07.364865 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:07.364838    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-48992b-gn4wr"]
Jul 16 23:30:07.365174 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:07.365162    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0b142e2-2afb-435f-b076-1dc1b1dbc0ae" containerName="test-curl-7299-139721935030576"
Jul 16 23:30:07.365217 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:07.365176    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0b142e2-2afb-435f-b076-1dc1b1dbc0ae" containerName="test-curl-7299-139721935030576"
Jul 16 23:30:07.365272 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:07.365227    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0b142e2-2afb-435f-b076-1dc1b1dbc0ae" containerName="test-curl-7299-139721935030576"
Jul 16 23:30:07.367978 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:07.367964    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-48992b-gn4wr"
Jul 16 23:30:07.370260 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:07.370231    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 23:30:07.370320 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:07.370295    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:30:07.375302 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:07.375280    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-48992b-gn4wr"]
Jul 16 23:30:07.439311 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:07.439280    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9cc179e-c014-407c-8ce0-e60cd54fc10c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-48992b-gn4wr\" (UID: \"e9cc179e-c014-407c-8ce0-e60cd54fc10c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-48992b-gn4wr"
Jul 16 23:30:07.540177 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:07.540153    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9cc179e-c014-407c-8ce0-e60cd54fc10c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-48992b-gn4wr\" (UID: \"e9cc179e-c014-407c-8ce0-e60cd54fc10c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-48992b-gn4wr"
Jul 16 23:30:07.540720 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:07.540702    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9cc179e-c014-407c-8ce0-e60cd54fc10c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-48992b-gn4wr\" (UID: \"e9cc179e-c014-407c-8ce0-e60cd54fc10c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-48992b-gn4wr"
Jul 16 23:30:07.678824 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:07.678766    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-48992b-gn4wr"
Jul 16 23:30:07.801275 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:07.801249    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-48992b-gn4wr"]
Jul 16 23:30:07.802958 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:30:07.802933    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9cc179e_c014_407c_8ce0_e60cd54fc10c.slice/crio-7ba2f2f089eb771d4d153672bff64d2ab9326a48e35477af845c66330372a6bd WatchSource:0}: Error finding container 7ba2f2f089eb771d4d153672bff64d2ab9326a48e35477af845c66330372a6bd: Status 404 returned error can't find the container with id 7ba2f2f089eb771d4d153672bff64d2ab9326a48e35477af845c66330372a6bd
Jul 16 23:30:07.859351 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:07.859324    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:30:07.959886 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:07.959826    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:30:08.023788 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:08.023762    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-48992b-gn4wr" event={"ID":"e9cc179e-c014-407c-8ce0-e60cd54fc10c","Type":"ContainerStarted","Data":"762c3bcb7491e3d6633035b48744213c21715d4d2577663aa549ee82670a42e3"}
Jul 16 23:30:08.023902 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:08.023816    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-48992b-gn4wr" event={"ID":"e9cc179e-c014-407c-8ce0-e60cd54fc10c","Type":"ContainerStarted","Data":"7ba2f2f089eb771d4d153672bff64d2ab9326a48e35477af845c66330372a6bd"}
Jul 16 23:30:08.025066 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:08.025049    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7299-139721935030576"
Jul 16 23:30:08.025145 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:08.025099    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b095b76862cbdcfb5791e8f3639dbda4dc891c149265103d240a1c2163c08747"
Jul 16 23:30:08.037904 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:08.037867    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-48992b-gn4wr" podStartSLOduration=1.037855869 podStartE2EDuration="1.037855869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:30:08.037533998 +0000 UTC m=+3382.047796682" watchObservedRunningTime="2026-07-16 23:30:08.037855869 +0000 UTC m=+3382.048118563"
Jul 16 23:30:08.039375 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:30:08.039351    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7299-139721935030576\" is forbidden: User \"system:node:ip-10-0-140-146.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-146.ec2.internal' and this object" podUID="c0b142e2-2afb-435f-b076-1dc1b1dbc0ae" pod="opendatahub/test-curl-7299-139721935030576"
Jul 16 23:30:08.518582 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:08.518558    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0b142e2-2afb-435f-b076-1dc1b1dbc0ae" path="/var/lib/kubelet/pods/c0b142e2-2afb-435f-b076-1dc1b1dbc0ae/volumes"
Jul 16 23:30:09.032522 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:09.032491    2573 generic.go:356] "Generic (PLEG): container finished" podID="e9cc179e-c014-407c-8ce0-e60cd54fc10c" containerID="762c3bcb7491e3d6633035b48744213c21715d4d2577663aa549ee82670a42e3" exitCode=0
Jul 16 23:30:09.032676 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:09.032579    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-48992b-gn4wr" event={"ID":"e9cc179e-c014-407c-8ce0-e60cd54fc10c","Type":"ContainerDied","Data":"762c3bcb7491e3d6633035b48744213c21715d4d2577663aa549ee82670a42e3"}
Jul 16 23:30:10.161683 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:10.161662    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-48992b-gn4wr"
Jul 16 23:30:10.259602 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:10.259560    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9cc179e-c014-407c-8ce0-e60cd54fc10c-maas-api-service-ca\") pod \"e9cc179e-c014-407c-8ce0-e60cd54fc10c\" (UID: \"e9cc179e-c014-407c-8ce0-e60cd54fc10c\") "
Jul 16 23:30:10.259883 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:10.259860    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9cc179e-c014-407c-8ce0-e60cd54fc10c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e9cc179e-c014-407c-8ce0-e60cd54fc10c" (UID: "e9cc179e-c014-407c-8ce0-e60cd54fc10c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:30:10.360783 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:10.360750    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9cc179e-c014-407c-8ce0-e60cd54fc10c-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:30:11.043166 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:11.043136    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-48992b-gn4wr"
Jul 16 23:30:11.043316 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:11.043183    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-48992b-gn4wr" event={"ID":"e9cc179e-c014-407c-8ce0-e60cd54fc10c","Type":"ContainerDied","Data":"7ba2f2f089eb771d4d153672bff64d2ab9326a48e35477af845c66330372a6bd"}
Jul 16 23:30:11.043316 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:11.043227    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ba2f2f089eb771d4d153672bff64d2ab9326a48e35477af845c66330372a6bd"
Jul 16 23:30:17.756563 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:17.756519    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:30:20.932435 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:20.932402    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:30:21.269773 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:21.269702    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-qbmcx"]
Jul 16 23:30:21.269980 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:21.269969    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9cc179e-c014-407c-8ce0-e60cd54fc10c" containerName="revoke-keys"
Jul 16 23:30:21.270025 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:21.269981    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9cc179e-c014-407c-8ce0-e60cd54fc10c" containerName="revoke-keys"
Jul 16 23:30:21.270057 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:21.270037    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9cc179e-c014-407c-8ce0-e60cd54fc10c" containerName="revoke-keys"
Jul 16 23:30:21.272792 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:21.272777    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-qbmcx"
Jul 16 23:30:21.274967 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:21.274950    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 23:30:21.275089 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:21.275015    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:30:21.280170 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:21.280147    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-qbmcx"]
Jul 16 23:30:21.340228 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:21.340201    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c21f443a-786b-4f12-aa77-653957dc4f4a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-qbmcx\" (UID: \"c21f443a-786b-4f12-aa77-653957dc4f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-qbmcx"
Jul 16 23:30:21.441509 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:21.441482    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c21f443a-786b-4f12-aa77-653957dc4f4a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-qbmcx\" (UID: \"c21f443a-786b-4f12-aa77-653957dc4f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-qbmcx"
Jul 16 23:30:21.442033 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:21.442014    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c21f443a-786b-4f12-aa77-653957dc4f4a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-qbmcx\" (UID: \"c21f443a-786b-4f12-aa77-653957dc4f4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-qbmcx"
Jul 16 23:30:21.584063 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:21.584043    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-qbmcx"
Jul 16 23:30:21.703231 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:21.703184    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-qbmcx"]
Jul 16 23:30:21.704985 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:30:21.704964    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc21f443a_786b_4f12_aa77_653957dc4f4a.slice/crio-32dc4a3c78b921ed4229d2e6df5657c0b2a4ec1f30d07327f9e177d5d9df0309 WatchSource:0}: Error finding container 32dc4a3c78b921ed4229d2e6df5657c0b2a4ec1f30d07327f9e177d5d9df0309: Status 404 returned error can't find the container with id 32dc4a3c78b921ed4229d2e6df5657c0b2a4ec1f30d07327f9e177d5d9df0309
Jul 16 23:30:22.097308 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:22.097280    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-qbmcx" event={"ID":"c21f443a-786b-4f12-aa77-653957dc4f4a","Type":"ContainerStarted","Data":"3ef679670ac71b3a1fd7201ad5eac7f04064ef535a94d47e916e775def1f42d8"}
Jul 16 23:30:22.097591 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:22.097329    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-qbmcx" event={"ID":"c21f443a-786b-4f12-aa77-653957dc4f4a","Type":"ContainerStarted","Data":"32dc4a3c78b921ed4229d2e6df5657c0b2a4ec1f30d07327f9e177d5d9df0309"}
Jul 16 23:30:22.112328 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:22.112289    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-qbmcx" podStartSLOduration=1.112275887 podStartE2EDuration="1.112275887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:30:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:30:22.110309782 +0000 UTC m=+3396.120572446" watchObservedRunningTime="2026-07-16 23:30:22.112275887 +0000 UTC m=+3396.122538550"
Jul 16 23:30:22.456279 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:22.456250    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:30:23.103705 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:23.103676    2573 generic.go:356] "Generic (PLEG): container finished" podID="c21f443a-786b-4f12-aa77-653957dc4f4a" containerID="3ef679670ac71b3a1fd7201ad5eac7f04064ef535a94d47e916e775def1f42d8" exitCode=6
Jul 16 23:30:23.104080 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:23.103759    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-qbmcx" event={"ID":"c21f443a-786b-4f12-aa77-653957dc4f4a","Type":"ContainerDied","Data":"3ef679670ac71b3a1fd7201ad5eac7f04064ef535a94d47e916e775def1f42d8"}
Jul 16 23:30:23.104080 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:23.103873    2573 scope.go:117] "RemoveContainer" containerID="3ef679670ac71b3a1fd7201ad5eac7f04064ef535a94d47e916e775def1f42d8"
Jul 16 23:30:24.110819 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:24.110787    2573 generic.go:356] "Generic (PLEG): container finished" podID="c21f443a-786b-4f12-aa77-653957dc4f4a" containerID="4f80a1d71aa3672b1c76f861a8e645763e6f3648d64b6f4b22385dfc109fdccd" exitCode=6
Jul 16 23:30:24.111213 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:24.110870    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-qbmcx" event={"ID":"c21f443a-786b-4f12-aa77-653957dc4f4a","Type":"ContainerDied","Data":"4f80a1d71aa3672b1c76f861a8e645763e6f3648d64b6f4b22385dfc109fdccd"}
Jul 16 23:30:24.111213 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:24.110938    2573 scope.go:117] "RemoveContainer" containerID="3ef679670ac71b3a1fd7201ad5eac7f04064ef535a94d47e916e775def1f42d8"
Jul 16 23:30:24.111213 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:24.111126    2573 scope.go:117] "RemoveContainer" containerID="4f80a1d71aa3672b1c76f861a8e645763e6f3648d64b6f4b22385dfc109fdccd"
Jul 16 23:30:24.111377 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:30:24.111286    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-qbmcx_odh-ai-gateway-infra(c21f443a-786b-4f12-aa77-653957dc4f4a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-qbmcx" podUID="c21f443a-786b-4f12-aa77-653957dc4f4a"
Jul 16 23:30:25.118019 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:25.117993    2573 scope.go:117] "RemoveContainer" containerID="4f80a1d71aa3672b1c76f861a8e645763e6f3648d64b6f4b22385dfc109fdccd"
Jul 16 23:30:25.118506 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:30:25.118179    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-qbmcx_odh-ai-gateway-infra(c21f443a-786b-4f12-aa77-653957dc4f4a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-qbmcx" podUID="c21f443a-786b-4f12-aa77-653957dc4f4a"
Jul 16 23:30:28.281871 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:28.281835    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:30:28.485869 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:28.485823    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:30:33.147677 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:33.147642    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:30:33.385695 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:33.385660    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:30:33.505929 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:33.505867    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b4ab52-qb88d"]
Jul 16 23:30:33.514917 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:33.514894    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b4ab52-qb88d"
Jul 16 23:30:33.517285 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:33.517266    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b4ab52-qb88d"]
Jul 16 23:30:33.614023 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:33.613999    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-l88w5"]
Jul 16 23:30:33.616993 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:33.616979    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-l88w5"
Jul 16 23:30:33.621470 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:33.621445    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43ca2602-2cb2-4db6-846d-23a922d2309b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-b4ab52-qb88d\" (UID: \"43ca2602-2cb2-4db6-846d-23a922d2309b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b4ab52-qb88d"
Jul 16 23:30:33.624283 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:33.624264    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-l88w5"]
Jul 16 23:30:33.722092 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:33.722052    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b31e1acb-7c79-400c-85c7-29521056ab2b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-l88w5\" (UID: \"b31e1acb-7c79-400c-85c7-29521056ab2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-l88w5"
Jul 16 23:30:33.722270 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:33.722198    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43ca2602-2cb2-4db6-846d-23a922d2309b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-b4ab52-qb88d\" (UID: \"43ca2602-2cb2-4db6-846d-23a922d2309b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b4ab52-qb88d"
Jul 16 23:30:33.722824 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:33.722800    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43ca2602-2cb2-4db6-846d-23a922d2309b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-b4ab52-qb88d\" (UID: \"43ca2602-2cb2-4db6-846d-23a922d2309b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b4ab52-qb88d"
Jul 16 23:30:33.822743 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:33.822667    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b31e1acb-7c79-400c-85c7-29521056ab2b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-l88w5\" (UID: \"b31e1acb-7c79-400c-85c7-29521056ab2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-l88w5"
Jul 16 23:30:33.823219 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:33.823202    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b31e1acb-7c79-400c-85c7-29521056ab2b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-l88w5\" (UID: \"b31e1acb-7c79-400c-85c7-29521056ab2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-l88w5"
Jul 16 23:30:33.825584 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:33.825565    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b4ab52-qb88d"
Jul 16 23:30:33.927636 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:33.927606    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-l88w5"
Jul 16 23:30:33.950535 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:33.950505    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b4ab52-qb88d"]
Jul 16 23:30:33.951227 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:30:33.951198    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ca2602_2cb2_4db6_846d_23a922d2309b.slice/crio-0bfbe8ebf65345e3773bd24880eec8124ceb7a47f19235404e482037731211bc WatchSource:0}: Error finding container 0bfbe8ebf65345e3773bd24880eec8124ceb7a47f19235404e482037731211bc: Status 404 returned error can't find the container with id 0bfbe8ebf65345e3773bd24880eec8124ceb7a47f19235404e482037731211bc
Jul 16 23:30:34.052739 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:34.052713    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-l88w5"]
Jul 16 23:30:34.054018 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:30:34.053993    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb31e1acb_7c79_400c_85c7_29521056ab2b.slice/crio-a435eb2a712039ca02388d5a26048bb4196c1c741cf97c05fa35c2dda3fae7fc WatchSource:0}: Error finding container a435eb2a712039ca02388d5a26048bb4196c1c741cf97c05fa35c2dda3fae7fc: Status 404 returned error can't find the container with id a435eb2a712039ca02388d5a26048bb4196c1c741cf97c05fa35c2dda3fae7fc
Jul 16 23:30:34.158804 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:34.158767    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:30:34.162421 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:34.162390    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b4ab52-qb88d" event={"ID":"43ca2602-2cb2-4db6-846d-23a922d2309b","Type":"ContainerStarted","Data":"0bfbe8ebf65345e3773bd24880eec8124ceb7a47f19235404e482037731211bc"}
Jul 16 23:30:34.163540 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:34.163520    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-l88w5" event={"ID":"b31e1acb-7c79-400c-85c7-29521056ab2b","Type":"ContainerStarted","Data":"a435eb2a712039ca02388d5a26048bb4196c1c741cf97c05fa35c2dda3fae7fc"}
Jul 16 23:30:34.257567 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:34.257546    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:30:35.170867 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:35.170841    2573 generic.go:356] "Generic (PLEG): container finished" podID="43ca2602-2cb2-4db6-846d-23a922d2309b" containerID="fcd5630c0ac0d1fc806da7945ed8c2e468a5ea386716453c5596c8de63c8f6ce" exitCode=0
Jul 16 23:30:35.171205 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:35.170925    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b4ab52-qb88d" event={"ID":"43ca2602-2cb2-4db6-846d-23a922d2309b","Type":"ContainerDied","Data":"fcd5630c0ac0d1fc806da7945ed8c2e468a5ea386716453c5596c8de63c8f6ce"}
Jul 16 23:30:35.172428 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:35.172409    2573 generic.go:356] "Generic (PLEG): container finished" podID="b31e1acb-7c79-400c-85c7-29521056ab2b" containerID="12eec6300b0b2b6803e52f2de1267cb482de8c67abb78a2d14e73412c2c93de2" exitCode=6
Jul 16 23:30:35.172489 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:35.172449    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-l88w5" event={"ID":"b31e1acb-7c79-400c-85c7-29521056ab2b","Type":"ContainerDied","Data":"12eec6300b0b2b6803e52f2de1267cb482de8c67abb78a2d14e73412c2c93de2"}
Jul 16 23:30:35.172618 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:35.172606    2573 scope.go:117] "RemoveContainer" containerID="12eec6300b0b2b6803e52f2de1267cb482de8c67abb78a2d14e73412c2c93de2"
Jul 16 23:30:36.179127 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:36.179089    2573 generic.go:356] "Generic (PLEG): container finished" podID="b31e1acb-7c79-400c-85c7-29521056ab2b" containerID="6b5e47f7aa9beec4197da12963f3f91105ddfc93afdce96f9104218125ca8c54" exitCode=6
Jul 16 23:30:36.179569 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:36.179175    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-l88w5" event={"ID":"b31e1acb-7c79-400c-85c7-29521056ab2b","Type":"ContainerDied","Data":"6b5e47f7aa9beec4197da12963f3f91105ddfc93afdce96f9104218125ca8c54"}
Jul 16 23:30:36.179569 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:36.179270    2573 scope.go:117] "RemoveContainer" containerID="12eec6300b0b2b6803e52f2de1267cb482de8c67abb78a2d14e73412c2c93de2"
Jul 16 23:30:36.179569 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:36.179545    2573 scope.go:117] "RemoveContainer" containerID="6b5e47f7aa9beec4197da12963f3f91105ddfc93afdce96f9104218125ca8c54"
Jul 16 23:30:36.180119 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:30:36.179722    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-l88w5_odh-ai-gateway-infra(b31e1acb-7c79-400c-85c7-29521056ab2b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-l88w5" podUID="b31e1acb-7c79-400c-85c7-29521056ab2b"
Jul 16 23:30:36.306310 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:36.306287    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b4ab52-qb88d"
Jul 16 23:30:36.442843 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:36.442783    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43ca2602-2cb2-4db6-846d-23a922d2309b-maas-api-service-ca\") pod \"43ca2602-2cb2-4db6-846d-23a922d2309b\" (UID: \"43ca2602-2cb2-4db6-846d-23a922d2309b\") "
Jul 16 23:30:36.443090 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:36.443070    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43ca2602-2cb2-4db6-846d-23a922d2309b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "43ca2602-2cb2-4db6-846d-23a922d2309b" (UID: "43ca2602-2cb2-4db6-846d-23a922d2309b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:30:36.543709 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:36.543684    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43ca2602-2cb2-4db6-846d-23a922d2309b-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:30:37.190880 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:37.190853    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b4ab52-qb88d"
Jul 16 23:30:37.191351 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:37.190899    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b4ab52-qb88d" event={"ID":"43ca2602-2cb2-4db6-846d-23a922d2309b","Type":"ContainerDied","Data":"0bfbe8ebf65345e3773bd24880eec8124ceb7a47f19235404e482037731211bc"}
Jul 16 23:30:37.191351 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:37.190984    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bfbe8ebf65345e3773bd24880eec8124ceb7a47f19235404e482037731211bc"
Jul 16 23:30:37.192935 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:37.192882    2573 scope.go:117] "RemoveContainer" containerID="6b5e47f7aa9beec4197da12963f3f91105ddfc93afdce96f9104218125ca8c54"
Jul 16 23:30:37.193065 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:30:37.193046    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-l88w5_odh-ai-gateway-infra(b31e1acb-7c79-400c-85c7-29521056ab2b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-l88w5" podUID="b31e1acb-7c79-400c-85c7-29521056ab2b"
Jul 16 23:30:38.532309 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:38.532278    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7l6p8"]
Jul 16 23:30:38.532739 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:38.532554    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43ca2602-2cb2-4db6-846d-23a922d2309b" containerName="revoke-keys"
Jul 16 23:30:38.532739 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:38.532564    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="43ca2602-2cb2-4db6-846d-23a922d2309b" containerName="revoke-keys"
Jul 16 23:30:38.532739 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:38.532623    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="43ca2602-2cb2-4db6-846d-23a922d2309b" containerName="revoke-keys"
Jul 16 23:30:38.535262 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:38.535229    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7l6p8"
Jul 16 23:30:38.544939 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:38.544915    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7l6p8"]
Jul 16 23:30:38.660303 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:38.660233    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f347b84c-1c9b-41bb-8a7f-3ea0376134a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-7l6p8\" (UID: \"f347b84c-1c9b-41bb-8a7f-3ea0376134a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7l6p8"
Jul 16 23:30:38.760820 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:38.760795    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f347b84c-1c9b-41bb-8a7f-3ea0376134a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-7l6p8\" (UID: \"f347b84c-1c9b-41bb-8a7f-3ea0376134a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7l6p8"
Jul 16 23:30:38.761348 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:38.761332    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f347b84c-1c9b-41bb-8a7f-3ea0376134a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-7l6p8\" (UID: \"f347b84c-1c9b-41bb-8a7f-3ea0376134a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7l6p8"
Jul 16 23:30:38.846285 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:38.846263    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7l6p8"
Jul 16 23:30:38.973778 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:38.973750    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7l6p8"]
Jul 16 23:30:38.975034 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:30:38.975005    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf347b84c_1c9b_41bb_8a7f_3ea0376134a3.slice/crio-8b3d589e7c9650740185e5ba31e50641a433a4333833b1f85ee875a0482402b5 WatchSource:0}: Error finding container 8b3d589e7c9650740185e5ba31e50641a433a4333833b1f85ee875a0482402b5: Status 404 returned error can't find the container with id 8b3d589e7c9650740185e5ba31e50641a433a4333833b1f85ee875a0482402b5
Jul 16 23:30:39.204999 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:39.204973    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7l6p8" event={"ID":"f347b84c-1c9b-41bb-8a7f-3ea0376134a3","Type":"ContainerStarted","Data":"a7490eece656964d878e1f8cec0e46069c96ec7eb29dfae2b4ecd8d6390ae8dd"}
Jul 16 23:30:39.205088 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:39.205020    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7l6p8" event={"ID":"f347b84c-1c9b-41bb-8a7f-3ea0376134a3","Type":"ContainerStarted","Data":"8b3d589e7c9650740185e5ba31e50641a433a4333833b1f85ee875a0482402b5"}
Jul 16 23:30:39.220399 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:39.220360    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7l6p8" podStartSLOduration=1.220346316 podStartE2EDuration="1.220346316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:30:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:30:39.218646952 +0000 UTC m=+3413.228909628" watchObservedRunningTime="2026-07-16 23:30:39.220346316 +0000 UTC m=+3413.230608977"
Jul 16 23:30:40.211584 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:40.211551    2573 generic.go:356] "Generic (PLEG): container finished" podID="f347b84c-1c9b-41bb-8a7f-3ea0376134a3" containerID="a7490eece656964d878e1f8cec0e46069c96ec7eb29dfae2b4ecd8d6390ae8dd" exitCode=6
Jul 16 23:30:40.211967 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:40.211586    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7l6p8" event={"ID":"f347b84c-1c9b-41bb-8a7f-3ea0376134a3","Type":"ContainerDied","Data":"a7490eece656964d878e1f8cec0e46069c96ec7eb29dfae2b4ecd8d6390ae8dd"}
Jul 16 23:30:40.211967 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:40.211819    2573 scope.go:117] "RemoveContainer" containerID="a7490eece656964d878e1f8cec0e46069c96ec7eb29dfae2b4ecd8d6390ae8dd"
Jul 16 23:30:40.513315 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:40.513224    2573 scope.go:117] "RemoveContainer" containerID="4f80a1d71aa3672b1c76f861a8e645763e6f3648d64b6f4b22385dfc109fdccd"
Jul 16 23:30:41.218426 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:41.218388    2573 generic.go:356] "Generic (PLEG): container finished" podID="f347b84c-1c9b-41bb-8a7f-3ea0376134a3" containerID="7c6993439ad4582caf71e4a1ecfed1e6852e636fd41e209935872fc1ecca383f" exitCode=6
Jul 16 23:30:41.218775 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:41.218465    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7l6p8" event={"ID":"f347b84c-1c9b-41bb-8a7f-3ea0376134a3","Type":"ContainerDied","Data":"7c6993439ad4582caf71e4a1ecfed1e6852e636fd41e209935872fc1ecca383f"}
Jul 16 23:30:41.218775 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:41.218522    2573 scope.go:117] "RemoveContainer" containerID="a7490eece656964d878e1f8cec0e46069c96ec7eb29dfae2b4ecd8d6390ae8dd"
Jul 16 23:30:41.218775 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:41.218682    2573 scope.go:117] "RemoveContainer" containerID="7c6993439ad4582caf71e4a1ecfed1e6852e636fd41e209935872fc1ecca383f"
Jul 16 23:30:41.218948 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:30:41.218839    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-7l6p8_odh-ai-gateway-infra(f347b84c-1c9b-41bb-8a7f-3ea0376134a3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7l6p8" podUID="f347b84c-1c9b-41bb-8a7f-3ea0376134a3"
Jul 16 23:30:41.220479 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:41.220454    2573 generic.go:356] "Generic (PLEG): container finished" podID="c21f443a-786b-4f12-aa77-653957dc4f4a" containerID="336a5cd7019a089a63d7b448c090796e8befcd2b85aed229013abe32359bdf08" exitCode=6
Jul 16 23:30:41.220566 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:41.220516    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-qbmcx" event={"ID":"c21f443a-786b-4f12-aa77-653957dc4f4a","Type":"ContainerDied","Data":"336a5cd7019a089a63d7b448c090796e8befcd2b85aed229013abe32359bdf08"}
Jul 16 23:30:41.220756 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:41.220744    2573 scope.go:117] "RemoveContainer" containerID="336a5cd7019a089a63d7b448c090796e8befcd2b85aed229013abe32359bdf08"
Jul 16 23:30:41.220884 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:30:41.220870    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-qbmcx_odh-ai-gateway-infra(c21f443a-786b-4f12-aa77-653957dc4f4a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-qbmcx" podUID="c21f443a-786b-4f12-aa77-653957dc4f4a"
Jul 16 23:30:41.233075 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:41.233048    2573 scope.go:117] "RemoveContainer" containerID="4f80a1d71aa3672b1c76f861a8e645763e6f3648d64b6f4b22385dfc109fdccd"
Jul 16 23:30:42.140160 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:42.140129    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:30:42.227072 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:42.227050    2573 scope.go:117] "RemoveContainer" containerID="7c6993439ad4582caf71e4a1ecfed1e6852e636fd41e209935872fc1ecca383f"
Jul 16 23:30:42.227482 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:30:42.227262    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-7l6p8_odh-ai-gateway-infra(f347b84c-1c9b-41bb-8a7f-3ea0376134a3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7l6p8" podUID="f347b84c-1c9b-41bb-8a7f-3ea0376134a3"
Jul 16 23:30:42.259537 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:42.259510    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-qbmcx"]
Jul 16 23:30:42.383966 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:42.383945    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-qbmcx"
Jul 16 23:30:42.414248 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:42.414177    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lw5jb"]
Jul 16 23:30:42.414511 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:42.414497    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c21f443a-786b-4f12-aa77-653957dc4f4a" containerName="revoke-keys"
Jul 16 23:30:42.414556 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:42.414512    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c21f443a-786b-4f12-aa77-653957dc4f4a" containerName="revoke-keys"
Jul 16 23:30:42.414556 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:42.414524    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c21f443a-786b-4f12-aa77-653957dc4f4a" containerName="revoke-keys"
Jul 16 23:30:42.414556 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:42.414529    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c21f443a-786b-4f12-aa77-653957dc4f4a" containerName="revoke-keys"
Jul 16 23:30:42.414653 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:42.414589    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c21f443a-786b-4f12-aa77-653957dc4f4a" containerName="revoke-keys"
Jul 16 23:30:42.414653 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:42.414597    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c21f443a-786b-4f12-aa77-653957dc4f4a" containerName="revoke-keys"
Jul 16 23:30:42.414653 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:42.414603    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c21f443a-786b-4f12-aa77-653957dc4f4a" containerName="revoke-keys"
Jul 16 23:30:42.417418 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:42.417403    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lw5jb"
Jul 16 23:30:42.434210 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:42.434181    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lw5jb"]
Jul 16 23:30:42.491919 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:42.491886    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c21f443a-786b-4f12-aa77-653957dc4f4a-maas-api-service-ca\") pod \"c21f443a-786b-4f12-aa77-653957dc4f4a\" (UID: \"c21f443a-786b-4f12-aa77-653957dc4f4a\") "
Jul 16 23:30:42.492119 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:42.492052    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e288fd8-8da3-4ace-87d6-8f17e8a1b3ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-lw5jb\" (UID: \"1e288fd8-8da3-4ace-87d6-8f17e8a1b3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lw5jb"
Jul 16 23:30:42.492347 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:42.492322    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c21f443a-786b-4f12-aa77-653957dc4f4a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c21f443a-786b-4f12-aa77-653957dc4f4a" (UID: "c21f443a-786b-4f12-aa77-653957dc4f4a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:30:42.592464 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:42.592425    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e288fd8-8da3-4ace-87d6-8f17e8a1b3ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-lw5jb\" (UID: \"1e288fd8-8da3-4ace-87d6-8f17e8a1b3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lw5jb"
Jul 16 23:30:42.592669 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:42.592523    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c21f443a-786b-4f12-aa77-653957dc4f4a-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:30:42.593103 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:42.593075    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e288fd8-8da3-4ace-87d6-8f17e8a1b3ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-lw5jb\" (UID: \"1e288fd8-8da3-4ace-87d6-8f17e8a1b3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lw5jb"
Jul 16 23:30:42.727872 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:42.727776    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lw5jb"
Jul 16 23:30:42.858890 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:42.858868    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lw5jb"]
Jul 16 23:30:42.860938 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:30:42.860909    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e288fd8_8da3_4ace_87d6_8f17e8a1b3ff.slice/crio-1b13f421585a4c69f7856ab2d4b29bf3e228846154f4abd9aae734a98a6a2ee9 WatchSource:0}: Error finding container 1b13f421585a4c69f7856ab2d4b29bf3e228846154f4abd9aae734a98a6a2ee9: Status 404 returned error can't find the container with id 1b13f421585a4c69f7856ab2d4b29bf3e228846154f4abd9aae734a98a6a2ee9
Jul 16 23:30:43.234415 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:43.234389    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-qbmcx"
Jul 16 23:30:43.234774 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:43.234435    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-qbmcx" event={"ID":"c21f443a-786b-4f12-aa77-653957dc4f4a","Type":"ContainerDied","Data":"32dc4a3c78b921ed4229d2e6df5657c0b2a4ec1f30d07327f9e177d5d9df0309"}
Jul 16 23:30:43.234774 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:43.234496    2573 scope.go:117] "RemoveContainer" containerID="336a5cd7019a089a63d7b448c090796e8befcd2b85aed229013abe32359bdf08"
Jul 16 23:30:43.236004 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:43.235979    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lw5jb" event={"ID":"1e288fd8-8da3-4ace-87d6-8f17e8a1b3ff","Type":"ContainerStarted","Data":"45e622bcf31a85b9c1d5ccb83603addd7a19c2ca6a968f3a5bcc63cd262134ca"}
Jul 16 23:30:43.236106 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:43.236031    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lw5jb" event={"ID":"1e288fd8-8da3-4ace-87d6-8f17e8a1b3ff","Type":"ContainerStarted","Data":"1b13f421585a4c69f7856ab2d4b29bf3e228846154f4abd9aae734a98a6a2ee9"}
Jul 16 23:30:43.250645 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:43.250624    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-qbmcx"]
Jul 16 23:30:43.252551 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:43.252532    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-qbmcx"]
Jul 16 23:30:43.264786 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:43.264748    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lw5jb" podStartSLOduration=1.2647365019999999 podStartE2EDuration="1.264736502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:30:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:30:43.263874401 +0000 UTC m=+3417.274137064" watchObservedRunningTime="2026-07-16 23:30:43.264736502 +0000 UTC m=+3417.274999165"
Jul 16 23:30:43.899300 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:43.899271    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:30:44.242526 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:44.242455    2573 generic.go:356] "Generic (PLEG): container finished" podID="1e288fd8-8da3-4ace-87d6-8f17e8a1b3ff" containerID="45e622bcf31a85b9c1d5ccb83603addd7a19c2ca6a968f3a5bcc63cd262134ca" exitCode=6
Jul 16 23:30:44.242939 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:44.242541    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lw5jb" event={"ID":"1e288fd8-8da3-4ace-87d6-8f17e8a1b3ff","Type":"ContainerDied","Data":"45e622bcf31a85b9c1d5ccb83603addd7a19c2ca6a968f3a5bcc63cd262134ca"}
Jul 16 23:30:44.242939 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:44.242709    2573 scope.go:117] "RemoveContainer" containerID="45e622bcf31a85b9c1d5ccb83603addd7a19c2ca6a968f3a5bcc63cd262134ca"
Jul 16 23:30:44.518546 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:44.518489    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c21f443a-786b-4f12-aa77-653957dc4f4a" path="/var/lib/kubelet/pods/c21f443a-786b-4f12-aa77-653957dc4f4a/volumes"
Jul 16 23:30:45.250749 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:45.250671    2573 generic.go:356] "Generic (PLEG): container finished" podID="1e288fd8-8da3-4ace-87d6-8f17e8a1b3ff" containerID="0ff30fe855b57333453349fbf4eee5cb00cac90d0a70ed7baef80b592207befa" exitCode=6
Jul 16 23:30:45.251121 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:45.250756    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lw5jb" event={"ID":"1e288fd8-8da3-4ace-87d6-8f17e8a1b3ff","Type":"ContainerDied","Data":"0ff30fe855b57333453349fbf4eee5cb00cac90d0a70ed7baef80b592207befa"}
Jul 16 23:30:45.251121 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:45.250814    2573 scope.go:117] "RemoveContainer" containerID="45e622bcf31a85b9c1d5ccb83603addd7a19c2ca6a968f3a5bcc63cd262134ca"
Jul 16 23:30:45.251121 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:45.250982    2573 scope.go:117] "RemoveContainer" containerID="0ff30fe855b57333453349fbf4eee5cb00cac90d0a70ed7baef80b592207befa"
Jul 16 23:30:45.251310 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:30:45.251161    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-lw5jb_odh-ai-gateway-infra(1e288fd8-8da3-4ace-87d6-8f17e8a1b3ff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lw5jb" podUID="1e288fd8-8da3-4ace-87d6-8f17e8a1b3ff"
Jul 16 23:30:46.258375 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:46.258344    2573 scope.go:117] "RemoveContainer" containerID="0ff30fe855b57333453349fbf4eee5cb00cac90d0a70ed7baef80b592207befa"
Jul 16 23:30:46.258733 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:30:46.258527    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-lw5jb_odh-ai-gateway-infra(1e288fd8-8da3-4ace-87d6-8f17e8a1b3ff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lw5jb" podUID="1e288fd8-8da3-4ace-87d6-8f17e8a1b3ff"
Jul 16 23:30:46.731780 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:46.731750    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-48ptc"]
Jul 16 23:30:48.587292 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:48.587258    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:30:50.513637 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:50.513611    2573 scope.go:117] "RemoveContainer" containerID="6b5e47f7aa9beec4197da12963f3f91105ddfc93afdce96f9104218125ca8c54"
Jul 16 23:30:51.284596 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:51.284561    2573 generic.go:356] "Generic (PLEG): container finished" podID="b31e1acb-7c79-400c-85c7-29521056ab2b" containerID="d5fca419dab13d7c0703d91152d495d52f646f32342a24ed55b61102ccbe3695" exitCode=6
Jul 16 23:30:51.284772 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:51.284637    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-l88w5" event={"ID":"b31e1acb-7c79-400c-85c7-29521056ab2b","Type":"ContainerDied","Data":"d5fca419dab13d7c0703d91152d495d52f646f32342a24ed55b61102ccbe3695"}
Jul 16 23:30:51.284772 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:51.284714    2573 scope.go:117] "RemoveContainer" containerID="6b5e47f7aa9beec4197da12963f3f91105ddfc93afdce96f9104218125ca8c54"
Jul 16 23:30:51.285008 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:51.284989    2573 scope.go:117] "RemoveContainer" containerID="d5fca419dab13d7c0703d91152d495d52f646f32342a24ed55b61102ccbe3695"
Jul 16 23:30:51.285190 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:30:51.285170    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-l88w5_odh-ai-gateway-infra(b31e1acb-7c79-400c-85c7-29521056ab2b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-l88w5" podUID="b31e1acb-7c79-400c-85c7-29521056ab2b"
Jul 16 23:30:52.309933 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:52.309902    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-l88w5"]
Jul 16 23:30:52.432367 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:52.432346    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-l88w5"
Jul 16 23:30:52.566159 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:52.566099    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b31e1acb-7c79-400c-85c7-29521056ab2b-maas-api-service-ca\") pod \"b31e1acb-7c79-400c-85c7-29521056ab2b\" (UID: \"b31e1acb-7c79-400c-85c7-29521056ab2b\") "
Jul 16 23:30:52.566430 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:52.566403    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b31e1acb-7c79-400c-85c7-29521056ab2b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b31e1acb-7c79-400c-85c7-29521056ab2b" (UID: "b31e1acb-7c79-400c-85c7-29521056ab2b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:30:52.667287 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:52.667260    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b31e1acb-7c79-400c-85c7-29521056ab2b-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:30:53.297383 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:53.297348    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-l88w5" event={"ID":"b31e1acb-7c79-400c-85c7-29521056ab2b","Type":"ContainerDied","Data":"a435eb2a712039ca02388d5a26048bb4196c1c741cf97c05fa35c2dda3fae7fc"}
Jul 16 23:30:53.297383 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:53.297385    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-l88w5"
Jul 16 23:30:53.297638 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:53.297411    2573 scope.go:117] "RemoveContainer" containerID="d5fca419dab13d7c0703d91152d495d52f646f32342a24ed55b61102ccbe3695"
Jul 16 23:30:53.319064 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:53.319035    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-l88w5"]
Jul 16 23:30:53.327058 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:53.327032    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-l88w5"]
Jul 16 23:30:54.518393 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:54.518365    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b31e1acb-7c79-400c-85c7-29521056ab2b" path="/var/lib/kubelet/pods/b31e1acb-7c79-400c-85c7-29521056ab2b/volumes"
Jul 16 23:30:54.532761 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:54.532738    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:30:54.643822 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:54.643799    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:30:55.514003 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:55.513974    2573 scope.go:117] "RemoveContainer" containerID="7c6993439ad4582caf71e4a1ecfed1e6852e636fd41e209935872fc1ecca383f"
Jul 16 23:30:56.314250 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:56.314202    2573 generic.go:356] "Generic (PLEG): container finished" podID="f347b84c-1c9b-41bb-8a7f-3ea0376134a3" containerID="0fbbe79ecd5db7b3f9aedac6bad1d19d63df1af90ddba99d4ab117003469ac36" exitCode=6
Jul 16 23:30:56.314639 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:56.314278    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7l6p8" event={"ID":"f347b84c-1c9b-41bb-8a7f-3ea0376134a3","Type":"ContainerDied","Data":"0fbbe79ecd5db7b3f9aedac6bad1d19d63df1af90ddba99d4ab117003469ac36"}
Jul 16 23:30:56.314639 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:56.314388    2573 scope.go:117] "RemoveContainer" containerID="7c6993439ad4582caf71e4a1ecfed1e6852e636fd41e209935872fc1ecca383f"
Jul 16 23:30:56.314721 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:56.314668    2573 scope.go:117] "RemoveContainer" containerID="0fbbe79ecd5db7b3f9aedac6bad1d19d63df1af90ddba99d4ab117003469ac36"
Jul 16 23:30:56.314881 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:30:56.314861    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-7l6p8_odh-ai-gateway-infra(f347b84c-1c9b-41bb-8a7f-3ea0376134a3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7l6p8" podUID="f347b84c-1c9b-41bb-8a7f-3ea0376134a3"
Jul 16 23:30:57.339926 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:57.339893    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7l6p8"]
Jul 16 23:30:57.466740 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:57.466717    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7l6p8"
Jul 16 23:30:57.602581 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:57.602527    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f347b84c-1c9b-41bb-8a7f-3ea0376134a3-maas-api-service-ca\") pod \"f347b84c-1c9b-41bb-8a7f-3ea0376134a3\" (UID: \"f347b84c-1c9b-41bb-8a7f-3ea0376134a3\") "
Jul 16 23:30:57.602854 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:57.602830    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f347b84c-1c9b-41bb-8a7f-3ea0376134a3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f347b84c-1c9b-41bb-8a7f-3ea0376134a3" (UID: "f347b84c-1c9b-41bb-8a7f-3ea0376134a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:30:57.703350 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:57.703318    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f347b84c-1c9b-41bb-8a7f-3ea0376134a3-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:30:58.326608 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:58.326581    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7l6p8"
Jul 16 23:30:58.326778 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:58.326622    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7l6p8" event={"ID":"f347b84c-1c9b-41bb-8a7f-3ea0376134a3","Type":"ContainerDied","Data":"8b3d589e7c9650740185e5ba31e50641a433a4333833b1f85ee875a0482402b5"}
Jul 16 23:30:58.326778 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:58.326679    2573 scope.go:117] "RemoveContainer" containerID="0fbbe79ecd5db7b3f9aedac6bad1d19d63df1af90ddba99d4ab117003469ac36"
Jul 16 23:30:58.348346 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:58.348315    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7l6p8"]
Jul 16 23:30:58.353468 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:58.353448    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-7l6p8"]
Jul 16 23:30:58.514069 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:58.514040    2573 scope.go:117] "RemoveContainer" containerID="0ff30fe855b57333453349fbf4eee5cb00cac90d0a70ed7baef80b592207befa"
Jul 16 23:30:58.518985 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:58.518964    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f347b84c-1c9b-41bb-8a7f-3ea0376134a3" path="/var/lib/kubelet/pods/f347b84c-1c9b-41bb-8a7f-3ea0376134a3/volumes"
Jul 16 23:30:59.238204 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:59.238167    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:30:59.333803 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:59.333730    2573 generic.go:356] "Generic (PLEG): container finished" podID="1e288fd8-8da3-4ace-87d6-8f17e8a1b3ff" containerID="08ec2399399d0994a58dba56f726ef564be556784833ed1e211a46cb64020d69" exitCode=6
Jul 16 23:30:59.333949 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:59.333803    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lw5jb" event={"ID":"1e288fd8-8da3-4ace-87d6-8f17e8a1b3ff","Type":"ContainerDied","Data":"08ec2399399d0994a58dba56f726ef564be556784833ed1e211a46cb64020d69"}
Jul 16 23:30:59.333949 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:59.333878    2573 scope.go:117] "RemoveContainer" containerID="0ff30fe855b57333453349fbf4eee5cb00cac90d0a70ed7baef80b592207befa"
Jul 16 23:30:59.334140 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:59.334126    2573 scope.go:117] "RemoveContainer" containerID="08ec2399399d0994a58dba56f726ef564be556784833ed1e211a46cb64020d69"
Jul 16 23:30:59.334347 ip-10-0-140-146 kubenswrapper[2573]: E0716 23:30:59.334325    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-lw5jb_odh-ai-gateway-infra(1e288fd8-8da3-4ace-87d6-8f17e8a1b3ff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lw5jb" podUID="1e288fd8-8da3-4ace-87d6-8f17e8a1b3ff"
Jul 16 23:30:59.466681 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:30:59.466651    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:31:00.358555 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:00.358527    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lw5jb"]
Jul 16 23:31:00.483488 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:00.483467    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lw5jb"
Jul 16 23:31:00.623842 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:00.623766    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e288fd8-8da3-4ace-87d6-8f17e8a1b3ff-maas-api-service-ca\") pod \"1e288fd8-8da3-4ace-87d6-8f17e8a1b3ff\" (UID: \"1e288fd8-8da3-4ace-87d6-8f17e8a1b3ff\") "
Jul 16 23:31:00.624110 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:00.624087    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e288fd8-8da3-4ace-87d6-8f17e8a1b3ff-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1e288fd8-8da3-4ace-87d6-8f17e8a1b3ff" (UID: "1e288fd8-8da3-4ace-87d6-8f17e8a1b3ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:31:00.724332 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:00.724306    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e288fd8-8da3-4ace-87d6-8f17e8a1b3ff-maas-api-service-ca\") on node \"ip-10-0-140-146.ec2.internal\" DevicePath \"\""
Jul 16 23:31:01.347033 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:01.347001    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lw5jb" event={"ID":"1e288fd8-8da3-4ace-87d6-8f17e8a1b3ff","Type":"ContainerDied","Data":"1b13f421585a4c69f7856ab2d4b29bf3e228846154f4abd9aae734a98a6a2ee9"}
Jul 16 23:31:01.347202 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:01.347036    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lw5jb"
Jul 16 23:31:01.347202 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:01.347059    2573 scope.go:117] "RemoveContainer" containerID="08ec2399399d0994a58dba56f726ef564be556784833ed1e211a46cb64020d69"
Jul 16 23:31:01.370036 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:01.370016    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lw5jb"]
Jul 16 23:31:01.376969 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:01.376948    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-lw5jb"]
Jul 16 23:31:02.517903 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:02.517870    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e288fd8-8da3-4ace-87d6-8f17e8a1b3ff" path="/var/lib/kubelet/pods/1e288fd8-8da3-4ace-87d6-8f17e8a1b3ff/volumes"
Jul 16 23:31:05.363649 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:05.363612    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-tnzvx_0e390942-75d4-4f6c-8e43-287304f8b838/server/0.log"
Jul 16 23:31:05.478996 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:05.478972    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-9bdlb_752d23b3-773c-484b-8e97-1c511173fdc8/manager/0.log"
Jul 16 23:31:05.852293 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:05.852273    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bb6b77ff4-crfwm_64962d67-8646-4f77-8e5b-b2accdb71a35/copy-manifests/0.log"
Jul 16 23:31:05.858707 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:05.858645    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bb6b77ff4-crfwm_64962d67-8646-4f77-8e5b-b2accdb71a35/manager/0.log"
Jul 16 23:31:06.419537 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:06.419511    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29737395-kg7ct_59453ec6-6eac-47fe-add4-63aa8b128063/cleanup/0.log"
Jul 16 23:31:06.629416 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:06.629388    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-d302e4-klvz4_bba0259b-9239-4f63-b8e6-4cce88092ac0/revoke-keys/0.log"
Jul 16 23:31:06.733880 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:06.733807    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-339cb8-r89xd_4a1638ac-5bee-4919-965b-162243aa590e/revoke-keys/0.log"
Jul 16 23:31:06.942256 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:06.942211    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-4d775e-nn8l8_818d5380-3cca-4741-bc74-de88a5a5d701/revoke-keys/0.log"
Jul 16 23:31:07.046336 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:07.046230    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-4b61afbb-dcvjr_960f3ef4-671f-4f86-8a81-8ac8994b5695/revoke-keys/0.log"
Jul 16 23:31:07.257231 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:07.257210    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-48992b-gn4wr_e9cc179e-c014-407c-8ce0-e60cd54fc10c/revoke-keys/0.log"
Jul 16 23:31:07.360577 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:07.360557    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-b4ab52-qb88d_43ca2602-2cb2-4db6-846d-23a922d2309b/revoke-keys/0.log"
Jul 16 23:31:08.239015 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:08.238988    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwrc8_2c500107-02bb-469f-8b28-5eebf1ed90b8/pull/0.log"
Jul 16 23:31:08.244470 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:08.244444    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwrc8_2c500107-02bb-469f-8b28-5eebf1ed90b8/extract/0.log"
Jul 16 23:31:08.249995 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:08.249978    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwrc8_2c500107-02bb-469f-8b28-5eebf1ed90b8/util/0.log"
Jul 16 23:31:08.353758 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:08.353738    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qlxnc_c2a323d2-7e88-4e00-9c09-918cffae5ae2/pull/0.log"
Jul 16 23:31:08.359987 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:08.359970    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qlxnc_c2a323d2-7e88-4e00-9c09-918cffae5ae2/extract/0.log"
Jul 16 23:31:08.365834 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:08.365819    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qlxnc_c2a323d2-7e88-4e00-9c09-918cffae5ae2/util/0.log"
Jul 16 23:31:08.467775 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:08.467755    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hqb5q_83bf8282-8854-4c8b-967f-09b40c128e93/extract/0.log"
Jul 16 23:31:08.473068 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:08.473048    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hqb5q_83bf8282-8854-4c8b-967f-09b40c128e93/util/0.log"
Jul 16 23:31:08.478836 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:08.478814    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hqb5q_83bf8282-8854-4c8b-967f-09b40c128e93/pull/0.log"
Jul 16 23:31:08.584451 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:08.584429    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1k4f4l_3aa4994d-8f92-42a9-9150-fbd6f9c164fb/pull/0.log"
Jul 16 23:31:08.590154 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:08.590139    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1k4f4l_3aa4994d-8f92-42a9-9150-fbd6f9c164fb/extract/0.log"
Jul 16 23:31:08.595702 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:08.595676    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1k4f4l_3aa4994d-8f92-42a9-9150-fbd6f9c164fb/util/0.log"
Jul 16 23:31:09.374326 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:09.374301    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-48ptc_5380f474-4078-4ad2-a64f-15523278be37/limitador/0.log"
Jul 16 23:31:10.045380 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:10.045352    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b54d476c5-9kgs2_81eb7d79-86f5-4007-87df-a70530de357a/kube-auth-proxy/0.log"
Jul 16 23:31:10.612180 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:10.612138    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6669f9958b-wkhdn_4a28ac1e-8676-4819-aae2-68bf9cd4ea37/router/0.log"
Jul 16 23:31:11.495024 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:11.494995    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-prbjs_eeec56ff-611a-40e0-bda4-006ef5149278/main/0.log"
Jul 16 23:31:16.676054 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:16.676015    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rvb4k/must-gather-mtvbf"]
Jul 16 23:31:16.676537 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:16.676432    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e288fd8-8da3-4ace-87d6-8f17e8a1b3ff" containerName="revoke-keys"
Jul 16 23:31:16.676537 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:16.676451    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e288fd8-8da3-4ace-87d6-8f17e8a1b3ff" containerName="revoke-keys"
Jul 16 23:31:16.676537 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:16.676462    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f347b84c-1c9b-41bb-8a7f-3ea0376134a3" containerName="revoke-keys"
Jul 16 23:31:16.676537 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:16.676470    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f347b84c-1c9b-41bb-8a7f-3ea0376134a3" containerName="revoke-keys"
Jul 16 23:31:16.676537 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:16.676479    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f347b84c-1c9b-41bb-8a7f-3ea0376134a3" containerName="revoke-keys"
Jul 16 23:31:16.676537 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:16.676486    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f347b84c-1c9b-41bb-8a7f-3ea0376134a3" containerName="revoke-keys"
Jul 16 23:31:16.676537 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:16.676496    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e288fd8-8da3-4ace-87d6-8f17e8a1b3ff" containerName="revoke-keys"
Jul 16 23:31:16.676537 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:16.676504    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e288fd8-8da3-4ace-87d6-8f17e8a1b3ff" containerName="revoke-keys"
Jul 16 23:31:16.676537 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:16.676516    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b31e1acb-7c79-400c-85c7-29521056ab2b" containerName="revoke-keys"
Jul 16 23:31:16.676537 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:16.676523    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b31e1acb-7c79-400c-85c7-29521056ab2b" containerName="revoke-keys"
Jul 16 23:31:16.676537 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:16.676541    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e288fd8-8da3-4ace-87d6-8f17e8a1b3ff" containerName="revoke-keys"
Jul 16 23:31:16.677088 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:16.676549    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e288fd8-8da3-4ace-87d6-8f17e8a1b3ff" containerName="revoke-keys"
Jul 16 23:31:16.677088 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:16.676569    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c21f443a-786b-4f12-aa77-653957dc4f4a" containerName="revoke-keys"
Jul 16 23:31:16.677088 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:16.676578    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c21f443a-786b-4f12-aa77-653957dc4f4a" containerName="revoke-keys"
Jul 16 23:31:16.677088 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:16.676588    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b31e1acb-7c79-400c-85c7-29521056ab2b" containerName="revoke-keys"
Jul 16 23:31:16.677088 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:16.676597    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b31e1acb-7c79-400c-85c7-29521056ab2b" containerName="revoke-keys"
Jul 16 23:31:16.677088 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:16.676615    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b31e1acb-7c79-400c-85c7-29521056ab2b" containerName="revoke-keys"
Jul 16 23:31:16.677088 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:16.676622    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b31e1acb-7c79-400c-85c7-29521056ab2b" containerName="revoke-keys"
Jul 16 23:31:16.677088 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:16.676702    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f347b84c-1c9b-41bb-8a7f-3ea0376134a3" containerName="revoke-keys"
Jul 16 23:31:16.677088 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:16.676713    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e288fd8-8da3-4ace-87d6-8f17e8a1b3ff" containerName="revoke-keys"
Jul 16 23:31:16.677088 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:16.676724    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e288fd8-8da3-4ace-87d6-8f17e8a1b3ff" containerName="revoke-keys"
Jul 16 23:31:16.677088 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:16.676733    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b31e1acb-7c79-400c-85c7-29521056ab2b" containerName="revoke-keys"
Jul 16 23:31:16.677088 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:16.676742    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b31e1acb-7c79-400c-85c7-29521056ab2b" containerName="revoke-keys"
Jul 16 23:31:16.677088 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:16.676751    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f347b84c-1c9b-41bb-8a7f-3ea0376134a3" containerName="revoke-keys"
Jul 16 23:31:16.677088 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:16.676760    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b31e1acb-7c79-400c-85c7-29521056ab2b" containerName="revoke-keys"
Jul 16 23:31:16.677088 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:16.676830    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f347b84c-1c9b-41bb-8a7f-3ea0376134a3" containerName="revoke-keys"
Jul 16 23:31:16.677088 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:16.676839    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f347b84c-1c9b-41bb-8a7f-3ea0376134a3" containerName="revoke-keys"
Jul 16 23:31:16.677088 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:16.676925    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f347b84c-1c9b-41bb-8a7f-3ea0376134a3" containerName="revoke-keys"
Jul 16 23:31:16.677088 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:16.676934    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e288fd8-8da3-4ace-87d6-8f17e8a1b3ff" containerName="revoke-keys"
Jul 16 23:31:16.681190 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:16.681152    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:31:16.681566 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:16.681548    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rvb4k/must-gather-mtvbf"
Jul 16 23:31:16.683981 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:16.683955    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rvb4k\"/\"openshift-service-ca.crt\""
Jul 16 23:31:16.684124 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:16.684077    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rvb4k\"/\"kube-root-ca.crt\""
Jul 16 23:31:16.684778 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:16.684758    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rvb4k\"/\"default-dockercfg-8wz9g\""
Jul 16 23:31:16.690640 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:16.690615    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rvb4k/must-gather-mtvbf"]
Jul 16 23:31:16.779938 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:16.779915    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:31:16.842975 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:16.842950    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g6tk\" (UniqueName: \"kubernetes.io/projected/b54d4afc-37a5-4e64-bef7-79757a0907ca-kube-api-access-8g6tk\") pod \"must-gather-mtvbf\" (UID: \"b54d4afc-37a5-4e64-bef7-79757a0907ca\") " pod="openshift-must-gather-rvb4k/must-gather-mtvbf"
Jul 16 23:31:16.843077 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:16.842994    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b54d4afc-37a5-4e64-bef7-79757a0907ca-must-gather-output\") pod \"must-gather-mtvbf\" (UID: \"b54d4afc-37a5-4e64-bef7-79757a0907ca\") " pod="openshift-must-gather-rvb4k/must-gather-mtvbf"
Jul 16 23:31:16.943864 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:16.943804    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b54d4afc-37a5-4e64-bef7-79757a0907ca-must-gather-output\") pod \"must-gather-mtvbf\" (UID: \"b54d4afc-37a5-4e64-bef7-79757a0907ca\") " pod="openshift-must-gather-rvb4k/must-gather-mtvbf"
Jul 16 23:31:16.943864 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:16.943856    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g6tk\" (UniqueName: \"kubernetes.io/projected/b54d4afc-37a5-4e64-bef7-79757a0907ca-kube-api-access-8g6tk\") pod \"must-gather-mtvbf\" (UID: \"b54d4afc-37a5-4e64-bef7-79757a0907ca\") " pod="openshift-must-gather-rvb4k/must-gather-mtvbf"
Jul 16 23:31:16.944138 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:16.944118    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b54d4afc-37a5-4e64-bef7-79757a0907ca-must-gather-output\") pod \"must-gather-mtvbf\" (UID: \"b54d4afc-37a5-4e64-bef7-79757a0907ca\") " pod="openshift-must-gather-rvb4k/must-gather-mtvbf"
Jul 16 23:31:16.952839 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:16.952811    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g6tk\" (UniqueName: \"kubernetes.io/projected/b54d4afc-37a5-4e64-bef7-79757a0907ca-kube-api-access-8g6tk\") pod \"must-gather-mtvbf\" (UID: \"b54d4afc-37a5-4e64-bef7-79757a0907ca\") " pod="openshift-must-gather-rvb4k/must-gather-mtvbf"
Jul 16 23:31:16.991869 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:16.991845    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rvb4k/must-gather-mtvbf"
Jul 16 23:31:17.128929 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:17.128901    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rvb4k/must-gather-mtvbf"]
Jul 16 23:31:17.130618 ip-10-0-140-146 kubenswrapper[2573]: W0716 23:31:17.130586    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb54d4afc_37a5_4e64_bef7_79757a0907ca.slice/crio-14e00ece22ed1e1b91b8d657514806829c63d30609c3eddf8399a1c17fb3763f WatchSource:0}: Error finding container 14e00ece22ed1e1b91b8d657514806829c63d30609c3eddf8399a1c17fb3763f: Status 404 returned error can't find the container with id 14e00ece22ed1e1b91b8d657514806829c63d30609c3eddf8399a1c17fb3763f
Jul 16 23:31:17.426485 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:17.426448    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rvb4k/must-gather-mtvbf" event={"ID":"b54d4afc-37a5-4e64-bef7-79757a0907ca","Type":"ContainerStarted","Data":"14e00ece22ed1e1b91b8d657514806829c63d30609c3eddf8399a1c17fb3763f"}
Jul 16 23:31:18.437704 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:18.436860    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rvb4k/must-gather-mtvbf" event={"ID":"b54d4afc-37a5-4e64-bef7-79757a0907ca","Type":"ContainerStarted","Data":"d81838031424a5a95b350a8e594835f83d458c8d30b5f4e4658b7aa8b0935c78"}
Jul 16 23:31:18.437704 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:18.436915    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rvb4k/must-gather-mtvbf" event={"ID":"b54d4afc-37a5-4e64-bef7-79757a0907ca","Type":"ContainerStarted","Data":"537eb666a84e59b37f4757f665f32620974a519e29f8095094789fe8dd8a1246"}
Jul 16 23:31:18.456650 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:18.454659    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rvb4k/must-gather-mtvbf" podStartSLOduration=1.707047336 podStartE2EDuration="2.454641788s" totalImagesPullingTime="747.594452ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:31:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:31:18.454255756 +0000 UTC m=+3452.464518417" watchObservedRunningTime="2026-07-16 23:31:18.454641788 +0000 UTC m=+3452.464904452"
Jul 16 23:31:19.444340 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:19.444311    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kxpfx_c8d410fb-20ac-4175-a4b4-38d5b44636d6/global-pull-secret-syncer/0.log"
Jul 16 23:31:19.597355 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:19.597327    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log"
Jul 16 23:31:19.752527 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:19.752447    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-146.ec2.internal_f66c4d021c76588d605e3a72206506df/haproxy/0.log"
Jul 16 23:31:20.878829 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:20.878779    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-466rj_7144d772-5219-4c76-b0a6-950c4e94a235/konnectivity-agent/0.log" limit=1024
Jul 16 23:31:23.998263 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:23.998150    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwrc8_2c500107-02bb-469f-8b28-5eebf1ed90b8/extract/0.log"
Jul 16 23:31:24.024225 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:24.024200    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwrc8_2c500107-02bb-469f-8b28-5eebf1ed90b8/util/0.log"
Jul 16 23:31:24.059406 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:24.059379    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwrc8_2c500107-02bb-469f-8b28-5eebf1ed90b8/pull/0.log"
Jul 16 23:31:24.091324 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:24.091284    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qlxnc_c2a323d2-7e88-4e00-9c09-918cffae5ae2/extract/0.log"
Jul 16 23:31:24.120491 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:24.120464    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qlxnc_c2a323d2-7e88-4e00-9c09-918cffae5ae2/util/0.log"
Jul 16 23:31:24.150379 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:24.150355    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qlxnc_c2a323d2-7e88-4e00-9c09-918cffae5ae2/pull/0.log"
Jul 16 23:31:24.191220 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:24.191177    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hqb5q_83bf8282-8854-4c8b-967f-09b40c128e93/extract/0.log"
Jul 16 23:31:24.221623 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:24.221596    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hqb5q_83bf8282-8854-4c8b-967f-09b40c128e93/util/0.log"
Jul 16 23:31:24.255351 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:24.255219    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hqb5q_83bf8282-8854-4c8b-967f-09b40c128e93/pull/0.log"
Jul 16 23:31:24.286058 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:24.286032    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1k4f4l_3aa4994d-8f92-42a9-9150-fbd6f9c164fb/extract/0.log"
Jul 16 23:31:24.356997 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:24.356891    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1k4f4l_3aa4994d-8f92-42a9-9150-fbd6f9c164fb/util/0.log"
Jul 16 23:31:24.408297 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:24.408255    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1k4f4l_3aa4994d-8f92-42a9-9150-fbd6f9c164fb/pull/0.log"
Jul 16 23:31:25.324663 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:25.324638    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-48ptc_5380f474-4078-4ad2-a64f-15523278be37/limitador/0.log"
Jul 16 23:31:27.351263 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:27.351207    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fr7wp_1b4e74e3-c3b1-40c8-9f0b-34f84325f497/node-exporter/0.log"
Jul 16 23:31:27.369214 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:27.369180    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fr7wp_1b4e74e3-c3b1-40c8-9f0b-34f84325f497/kube-rbac-proxy/0.log"
Jul 16 23:31:27.388584 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:27.388558    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fr7wp_1b4e74e3-c3b1-40c8-9f0b-34f84325f497/init-textfile/0.log"
Jul 16 23:31:28.755201 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:28.755163    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rvb4k/perf-node-gather-daemonset-gzwqr"]
Jul 16 23:31:28.760196 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:28.760170    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rvb4k/perf-node-gather-daemonset-gzwqr"
Jul 16 23:31:28.771482 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:28.771447    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rvb4k/perf-node-gather-daemonset-gzwqr"]
Jul 16 23:31:28.849007 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:28.848949    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/de0b09ba-ecf0-46c5-bf99-4bb041079af1-proc\") pod \"perf-node-gather-daemonset-gzwqr\" (UID: \"de0b09ba-ecf0-46c5-bf99-4bb041079af1\") " pod="openshift-must-gather-rvb4k/perf-node-gather-daemonset-gzwqr"
Jul 16 23:31:28.849230 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:28.849012    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/de0b09ba-ecf0-46c5-bf99-4bb041079af1-lib-modules\") pod \"perf-node-gather-daemonset-gzwqr\" (UID: \"de0b09ba-ecf0-46c5-bf99-4bb041079af1\") " pod="openshift-must-gather-rvb4k/perf-node-gather-daemonset-gzwqr"
Jul 16 23:31:28.849230 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:28.849091    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tn85\" (UniqueName: \"kubernetes.io/projected/de0b09ba-ecf0-46c5-bf99-4bb041079af1-kube-api-access-8tn85\") pod \"perf-node-gather-daemonset-gzwqr\" (UID: \"de0b09ba-ecf0-46c5-bf99-4bb041079af1\") " pod="openshift-must-gather-rvb4k/perf-node-gather-daemonset-gzwqr"
Jul 16 23:31:28.849230 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:28.849169    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de0b09ba-ecf0-46c5-bf99-4bb041079af1-sys\") pod \"perf-node-gather-daemonset-gzwqr\" (UID: \"de0b09ba-ecf0-46c5-bf99-4bb041079af1\") " pod="openshift-must-gather-rvb4k/perf-node-gather-daemonset-gzwqr"
Jul 16 23:31:28.849230 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:28.849199    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/de0b09ba-ecf0-46c5-bf99-4bb041079af1-podres\") pod \"perf-node-gather-daemonset-gzwqr\" (UID: \"de0b09ba-ecf0-46c5-bf99-4bb041079af1\") " pod="openshift-must-gather-rvb4k/perf-node-gather-daemonset-gzwqr"
Jul 16 23:31:28.949771 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:28.949733    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de0b09ba-ecf0-46c5-bf99-4bb041079af1-sys\") pod \"perf-node-gather-daemonset-gzwqr\" (UID: \"de0b09ba-ecf0-46c5-bf99-4bb041079af1\") " pod="openshift-must-gather-rvb4k/perf-node-gather-daemonset-gzwqr"
Jul 16 23:31:28.949956 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:28.949786    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/de0b09ba-ecf0-46c5-bf99-4bb041079af1-podres\") pod \"perf-node-gather-daemonset-gzwqr\" (UID: \"de0b09ba-ecf0-46c5-bf99-4bb041079af1\") " pod="openshift-must-gather-rvb4k/perf-node-gather-daemonset-gzwqr"
Jul 16 23:31:28.949956 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:28.949825    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/de0b09ba-ecf0-46c5-bf99-4bb041079af1-proc\") pod \"perf-node-gather-daemonset-gzwqr\" (UID: \"de0b09ba-ecf0-46c5-bf99-4bb041079af1\") " pod="openshift-must-gather-rvb4k/perf-node-gather-daemonset-gzwqr"
Jul 16 23:31:28.949956 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:28.949853    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/de0b09ba-ecf0-46c5-bf99-4bb041079af1-lib-modules\") pod \"perf-node-gather-daemonset-gzwqr\" (UID: \"de0b09ba-ecf0-46c5-bf99-4bb041079af1\") " pod="openshift-must-gather-rvb4k/perf-node-gather-daemonset-gzwqr"
Jul 16 23:31:28.949956 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:28.949852    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de0b09ba-ecf0-46c5-bf99-4bb041079af1-sys\") pod \"perf-node-gather-daemonset-gzwqr\" (UID: \"de0b09ba-ecf0-46c5-bf99-4bb041079af1\") " pod="openshift-must-gather-rvb4k/perf-node-gather-daemonset-gzwqr"
Jul 16 23:31:28.949956 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:28.949926    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/de0b09ba-ecf0-46c5-bf99-4bb041079af1-proc\") pod \"perf-node-gather-daemonset-gzwqr\" (UID: \"de0b09ba-ecf0-46c5-bf99-4bb041079af1\") " pod="openshift-must-gather-rvb4k/perf-node-gather-daemonset-gzwqr"
Jul 16 23:31:28.950219 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:28.949973    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/de0b09ba-ecf0-46c5-bf99-4bb041079af1-lib-modules\") pod \"perf-node-gather-daemonset-gzwqr\" (UID: \"de0b09ba-ecf0-46c5-bf99-4bb041079af1\") " pod="openshift-must-gather-rvb4k/perf-node-gather-daemonset-gzwqr"
Jul 16 23:31:28.950219 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:28.949982    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/de0b09ba-ecf0-46c5-bf99-4bb041079af1-podres\") pod \"perf-node-gather-daemonset-gzwqr\" (UID: \"de0b09ba-ecf0-46c5-bf99-4bb041079af1\") " pod="openshift-must-gather-rvb4k/perf-node-gather-daemonset-gzwqr"
Jul 16 23:31:28.950219 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:28.949987    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tn85\" (UniqueName: \"kubernetes.io/projected/de0b09ba-ecf0-46c5-bf99-4bb041079af1-kube-api-access-8tn85\") pod \"perf-node-gather-daemonset-gzwqr\" (UID: \"de0b09ba-ecf0-46c5-bf99-4bb041079af1\") " pod="openshift-must-gather-rvb4k/perf-node-gather-daemonset-gzwqr"
Jul 16 23:31:28.958127 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:28.958102    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tn85\" (UniqueName: \"kubernetes.io/projected/de0b09ba-ecf0-46c5-bf99-4bb041079af1-kube-api-access-8tn85\") pod \"perf-node-gather-daemonset-gzwqr\" (UID: \"de0b09ba-ecf0-46c5-bf99-4bb041079af1\") " pod="openshift-must-gather-rvb4k/perf-node-gather-daemonset-gzwqr"
Jul 16 23:31:29.075145 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:29.075110    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rvb4k/perf-node-gather-daemonset-gzwqr"
Jul 16 23:31:29.257532 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:29.255893    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rvb4k/perf-node-gather-daemonset-gzwqr"]
Jul 16 23:31:29.523194 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:29.523163    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rvb4k/perf-node-gather-daemonset-gzwqr" event={"ID":"de0b09ba-ecf0-46c5-bf99-4bb041079af1","Type":"ContainerStarted","Data":"5566b25e703f43759d2515ed5588de7a5293c7c7d34be43e59746f703794201d"}
Jul 16 23:31:29.523386 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:29.523218    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rvb4k/perf-node-gather-daemonset-gzwqr" event={"ID":"de0b09ba-ecf0-46c5-bf99-4bb041079af1","Type":"ContainerStarted","Data":"b97269502b182e2fda6bc663e61b1ee9506a7b9ac81c2d7db9c5c8524814774e"}
Jul 16 23:31:29.523386 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:29.523268    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rvb4k/perf-node-gather-daemonset-gzwqr"
Jul 16 23:31:29.539904 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:29.539853    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rvb4k/perf-node-gather-daemonset-gzwqr" podStartSLOduration=1.539837873 podStartE2EDuration="1.539837873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 23:31:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:31:29.539150913 +0000 UTC m=+3463.549413576" watchObservedRunningTime="2026-07-16 23:31:29.539837873 +0000 UTC m=+3463.550100535"
Jul 16 23:31:30.806190 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:30.806162    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-mtdx8_03851daf-9b83-4d3a-8282-8466d1d4b574/volume-data-source-validator/0.log"
Jul 16 23:31:31.893668 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:31.893642    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nbc9n_53ae108c-2ec1-4c24-b756-4a899fbcf19d/dns/0.log"
Jul 16 23:31:31.923657 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:31.923623    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nbc9n_53ae108c-2ec1-4c24-b756-4a899fbcf19d/kube-rbac-proxy/0.log"
Jul 16 23:31:31.945854 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:31.945827    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8bst9_268b78b0-10f6-44ce-a822-a124771df729/dns-node-resolver/0.log"
Jul 16 23:31:32.620873 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:32.620843    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-x4th8_1d026b73-92ba-465b-bc31-5843395ae3e4/node-ca/0.log"
Jul 16 23:31:33.684313 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:33.684284    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b54d476c5-9kgs2_81eb7d79-86f5-4007-87df-a70530de357a/kube-auth-proxy/0.log"
Jul 16 23:31:33.857426 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:33.857399    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6669f9958b-wkhdn_4a28ac1e-8676-4819-aae2-68bf9cd4ea37/router/0.log"
Jul 16 23:31:34.484626 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:34.484570    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-btltg_93773e2d-fdc2-4715-825e-ea777c6e6916/serve-healthcheck-canary/0.log"
Jul 16 23:31:35.111041 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:35.111017    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7qrwd_420b20df-d1c4-4c01-afc1-bee33dc7ba03/kube-rbac-proxy/0.log"
Jul 16 23:31:35.137339 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:35.137314    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7qrwd_420b20df-d1c4-4c01-afc1-bee33dc7ba03/exporter/0.log"
Jul 16 23:31:35.158702 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:35.158682    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7qrwd_420b20df-d1c4-4c01-afc1-bee33dc7ba03/extractor/0.log"
Jul 16 23:31:35.543683 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:35.543601    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rvb4k/perf-node-gather-daemonset-gzwqr"
Jul 16 23:31:38.003466 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:38.003441    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-tnzvx_0e390942-75d4-4f6c-8e43-287304f8b838/server/0.log"
Jul 16 23:31:38.103389 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:38.103310    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-9bdlb_752d23b3-773c-484b-8e97-1c511173fdc8/manager/0.log"
Jul 16 23:31:38.289381 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:38.289356    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bb6b77ff4-crfwm_64962d67-8646-4f77-8e5b-b2accdb71a35/manager/0.log"
Jul 16 23:31:38.313877 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:38.313853    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bb6b77ff4-crfwm_64962d67-8646-4f77-8e5b-b2accdb71a35/copy-manifests/0.log"
Jul 16 23:31:38.817465 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:38.817436    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b54d476c5-9kgs2_81eb7d79-86f5-4007-87df-a70530de357a/kube-auth-proxy/0.log"
Jul 16 23:31:38.975812 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:38.975780    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6669f9958b-wkhdn_4a28ac1e-8676-4819-aae2-68bf9cd4ea37/router/0.log"
Jul 16 23:31:39.596205 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:39.596179    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-84c74b69b8-xzh5k_6c62a9d3-c22d-4f21-af46-c3f713816d18/manager/0.log"
Jul 16 23:31:40.263303 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:40.263271    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b54d476c5-9kgs2_81eb7d79-86f5-4007-87df-a70530de357a/kube-auth-proxy/0.log"
Jul 16 23:31:40.427592 ip-10-0-140-146 kubenswrapper[2573]: I0716 23:31:40.427555    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6669f9958b-wkhdn_4a28ac1e-8676-4819-aae2-68bf9cd4ea37/router/0.log"