Jul 22 01:47:27.026661 ip-10-0-135-230 systemd[1]: Starting Kubernetes Kubelet...
Jul 22 01:47:27.477086 ip-10-0-135-230 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 22 01:47:27.477086 ip-10-0-135-230 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 22 01:47:27.477086 ip-10-0-135-230 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 22 01:47:27.477086 ip-10-0-135-230 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 22 01:47:27.477086 ip-10-0-135-230 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 22 01:47:27.480119 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.480035    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 22 01:47:27.486601 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486583    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 22 01:47:27.486601 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486598    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 22 01:47:27.486601 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486601    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 22 01:47:27.486689 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486605    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 22 01:47:27.486689 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486610    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 22 01:47:27.486689 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486615    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 22 01:47:27.486689 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486619    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 22 01:47:27.486689 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486622    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 22 01:47:27.486689 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486625    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 22 01:47:27.486689 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486629    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 22 01:47:27.486689 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486632    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 22 01:47:27.486689 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486635    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 22 01:47:27.486689 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486638    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 22 01:47:27.486689 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486642    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 22 01:47:27.486689 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486645    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 22 01:47:27.486689 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486647    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 22 01:47:27.486689 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486650    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 22 01:47:27.486689 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486653    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 22 01:47:27.486689 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486656    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 22 01:47:27.486689 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486658    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 22 01:47:27.486689 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486661    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 22 01:47:27.486689 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486663    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 22 01:47:27.487186 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486666    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 22 01:47:27.487186 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486669    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 22 01:47:27.487186 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486671    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 22 01:47:27.487186 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486674    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 22 01:47:27.487186 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486676    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 22 01:47:27.487186 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486679    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 22 01:47:27.487186 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486681    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 22 01:47:27.487186 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486684    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 22 01:47:27.487186 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486687    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 22 01:47:27.487186 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486689    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 22 01:47:27.487186 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486692    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 22 01:47:27.487186 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486695    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 22 01:47:27.487186 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486697    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 22 01:47:27.487186 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486700    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 22 01:47:27.487186 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486703    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 22 01:47:27.487186 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486706    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 22 01:47:27.487186 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486708    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 22 01:47:27.487186 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486711    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 22 01:47:27.487186 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486713    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 22 01:47:27.487982 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486716    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 22 01:47:27.487982 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486719    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 22 01:47:27.487982 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486722    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 22 01:47:27.487982 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486726    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 22 01:47:27.487982 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486730    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 22 01:47:27.487982 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486733    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 22 01:47:27.487982 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486735    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 22 01:47:27.487982 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486738    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 22 01:47:27.487982 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486740    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 22 01:47:27.487982 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486743    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 22 01:47:27.487982 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486745    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 22 01:47:27.487982 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486747    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 22 01:47:27.487982 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486750    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 22 01:47:27.487982 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486753    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 22 01:47:27.487982 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486756    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 22 01:47:27.487982 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486759    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 22 01:47:27.487982 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486762    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 22 01:47:27.487982 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486765    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 22 01:47:27.487982 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486768    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 22 01:47:27.487982 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486771    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 22 01:47:27.488545 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486773    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 22 01:47:27.488545 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486776    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 22 01:47:27.488545 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486778    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 22 01:47:27.488545 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486781    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 22 01:47:27.488545 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486784    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 22 01:47:27.488545 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486786    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 22 01:47:27.488545 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486788    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 22 01:47:27.488545 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486791    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 22 01:47:27.488545 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486793    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 22 01:47:27.488545 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486796    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 22 01:47:27.488545 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486798    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 22 01:47:27.488545 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486801    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 22 01:47:27.488545 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486805    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 22 01:47:27.488545 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486807    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 22 01:47:27.488545 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486810    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 22 01:47:27.488545 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486813    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 22 01:47:27.488545 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486815    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 22 01:47:27.488545 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486818    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 22 01:47:27.488545 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486821    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 22 01:47:27.488545 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486824    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 22 01:47:27.489040 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486827    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 22 01:47:27.489040 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486829    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 22 01:47:27.489040 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486832    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 22 01:47:27.489040 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486835    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 22 01:47:27.489040 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486837    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 22 01:47:27.489040 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486840    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 22 01:47:27.489040 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486843    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 22 01:47:27.489040 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486845    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 22 01:47:27.489040 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486848    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 22 01:47:27.489040 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486850    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 22 01:47:27.489040 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486853    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 22 01:47:27.489040 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486855    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 22 01:47:27.489040 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486858    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 22 01:47:27.489040 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486861    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 22 01:47:27.489040 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486863    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 22 01:47:27.489040 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486866    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 22 01:47:27.489040 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486869    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 22 01:47:27.489040 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.486871    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 22 01:47:27.489497 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489235    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 22 01:47:27.489497 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489246    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 22 01:47:27.489497 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489251    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 22 01:47:27.489497 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489254    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 22 01:47:27.489497 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489257    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 22 01:47:27.489497 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489261    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 22 01:47:27.489497 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489263    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 22 01:47:27.489497 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489266    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 22 01:47:27.489497 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489268    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 22 01:47:27.489497 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489271    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 22 01:47:27.489497 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489274    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 22 01:47:27.489497 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489277    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 22 01:47:27.489497 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489279    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 22 01:47:27.489497 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489282    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 22 01:47:27.489497 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489285    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 22 01:47:27.489497 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489287    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 22 01:47:27.489497 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489290    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 22 01:47:27.489497 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489293    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 22 01:47:27.489497 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489295    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 22 01:47:27.489497 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489299    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 22 01:47:27.490022 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489302    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 22 01:47:27.490022 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489304    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 22 01:47:27.490022 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489307    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 22 01:47:27.490022 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489309    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 22 01:47:27.490022 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489312    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 22 01:47:27.490022 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489315    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 22 01:47:27.490022 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489317    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 22 01:47:27.490022 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489321    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 22 01:47:27.490022 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489323    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 22 01:47:27.490022 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489326    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 22 01:47:27.490022 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489328    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 22 01:47:27.490022 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489332    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 22 01:47:27.490022 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489335    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 22 01:47:27.490022 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489339    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 22 01:47:27.490022 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489341    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 22 01:47:27.490022 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489344    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 22 01:47:27.490022 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489347    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 22 01:47:27.490022 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489349    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 22 01:47:27.490022 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489352    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 22 01:47:27.490022 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489355    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 22 01:47:27.490535 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489358    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 22 01:47:27.490535 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489360    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 22 01:47:27.490535 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489363    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 22 01:47:27.490535 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489366    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 22 01:47:27.490535 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489368    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 22 01:47:27.490535 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489371    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 22 01:47:27.490535 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489395    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 22 01:47:27.490535 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489402    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 22 01:47:27.490535 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489406    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 22 01:47:27.490535 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489410    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 22 01:47:27.490535 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489413    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 22 01:47:27.490535 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489415    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 22 01:47:27.490535 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489418    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 22 01:47:27.490535 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489420    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 22 01:47:27.490535 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489424    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 22 01:47:27.490535 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489429    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 22 01:47:27.490535 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489433    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 22 01:47:27.490535 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489442    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 22 01:47:27.490535 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489446    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 22 01:47:27.490992 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489450    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 22 01:47:27.490992 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489453    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 22 01:47:27.490992 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489456    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 22 01:47:27.490992 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489459    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 22 01:47:27.490992 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489461    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 22 01:47:27.490992 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489464    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 22 01:47:27.490992 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489468    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 22 01:47:27.490992 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489470    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 22 01:47:27.490992 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489473    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 22 01:47:27.490992 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489475    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 22 01:47:27.490992 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489478    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 22 01:47:27.490992 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489481    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 22 01:47:27.490992 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489483    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 22 01:47:27.490992 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489486    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 22 01:47:27.490992 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489488    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 22 01:47:27.490992 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489491    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 22 01:47:27.490992 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489493    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 22 01:47:27.490992 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489496    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 22 01:47:27.490992 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489498    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 22 01:47:27.490992 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489501    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 22 01:47:27.491480 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489504    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 22 01:47:27.491480 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489507    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 22 01:47:27.491480 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489510    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 22 01:47:27.491480 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489512    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 22 01:47:27.491480 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489516    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 22 01:47:27.491480 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489520    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 22 01:47:27.491480 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489523    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 22 01:47:27.491480 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489526    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 22 01:47:27.491480 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489529    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 22 01:47:27.491480 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489531    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 22 01:47:27.491480 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489534    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 22 01:47:27.491480 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489536    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 22 01:47:27.491480 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489539    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 22 01:47:27.491480 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489542    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 22 01:47:27.491480 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489545    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 22 01:47:27.491480 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489547    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 22 01:47:27.491480 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489550    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 22 01:47:27.491480 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489552    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 22 01:47:27.491480 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489555    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 22 01:47:27.491480 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489558    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 22 01:47:27.491964 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489961    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 22 01:47:27.491964 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489967    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 22 01:47:27.491964 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489970    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 22 01:47:27.491964 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489973    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 22 01:47:27.491964 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489975    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 22 01:47:27.491964 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489978    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 22 01:47:27.491964 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489981    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 22 01:47:27.491964 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489983    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 22 01:47:27.491964 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489986    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 22 01:47:27.491964 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489989    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 22 01:47:27.491964 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489991    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 22 01:47:27.491964 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.489998    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 22 01:47:27.491964 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490001    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 22 01:47:27.491964 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490005    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 22 01:47:27.491964 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490008    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 22 01:47:27.491964 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490010    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 22 01:47:27.491964 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490014    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 22 01:47:27.491964 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490017    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 22 01:47:27.491964 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490019    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 22 01:47:27.491964 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490022    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 22 01:47:27.491964 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490024    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 22 01:47:27.492555 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490027    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 22 01:47:27.492555 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490030    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 22 01:47:27.492555 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490032    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 22 01:47:27.492555 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490035    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 22 01:47:27.492555 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490038    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 22 01:47:27.492555 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490041    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 22 01:47:27.492555 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490044    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 22 01:47:27.492555 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490047    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 22 01:47:27.492555 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490049    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 22 01:47:27.492555 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490052    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 22 01:47:27.492555 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490055    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 22 01:47:27.492555 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490057    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 22 01:47:27.492555 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490060    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 22 01:47:27.492555 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490062    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 22 01:47:27.492555 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490065    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 22 01:47:27.492555 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490067    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 22 01:47:27.492555 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490070    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 22 01:47:27.492555 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490073    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 22 01:47:27.492555 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490075    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 22 01:47:27.492555 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490077    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 22 01:47:27.493029 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490080    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 22 01:47:27.493029 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490082    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 22 01:47:27.493029 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490085    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 22 01:47:27.493029 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490088    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 22 01:47:27.493029 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490091    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 22 01:47:27.493029 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490093    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 22 01:47:27.493029 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490096    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 22 01:47:27.493029 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490099    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 22 01:47:27.493029 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490102    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 22 01:47:27.493029 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490104    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 22 01:47:27.493029 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490107    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 22 01:47:27.493029 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490109    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 22 01:47:27.493029 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490112    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 22 01:47:27.493029 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490114    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 22 01:47:27.493029 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490117    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 22 01:47:27.493029 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490120    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 22 01:47:27.493029 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490122    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 22 01:47:27.493029 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490125    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 22 01:47:27.493029 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490128    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 22 01:47:27.493029 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490130    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 22 01:47:27.493543 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490132    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 22 01:47:27.493543 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490135    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 22 01:47:27.493543 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490137    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 22 01:47:27.493543 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490140    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 22 01:47:27.493543 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490143    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 22 01:47:27.493543 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490146    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 22 01:47:27.493543 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490148    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 22 01:47:27.493543 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490151    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 22 01:47:27.493543 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490154    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 22 01:47:27.493543 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490156    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 22 01:47:27.493543 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490159    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 22 01:47:27.493543 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490162    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 22 01:47:27.493543 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490166    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 22 01:47:27.493543 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490169    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 22 01:47:27.493543 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490173    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 22 01:47:27.493543 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490177    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 22 01:47:27.493543 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490180    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 22 01:47:27.493543 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490183    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 22 01:47:27.493543 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490186    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 22 01:47:27.494004 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490188    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 22 01:47:27.494004 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490191    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 22 01:47:27.494004 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490194    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 22 01:47:27.494004 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490196    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 22 01:47:27.494004 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490199    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 22 01:47:27.494004 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490202    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 22 01:47:27.494004 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490204    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 22 01:47:27.494004 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490207    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 22 01:47:27.494004 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490209    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 22 01:47:27.494004 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490211    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 22 01:47:27.494004 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490218    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 22 01:47:27.494004 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490220    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 22 01:47:27.494004 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490223    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 22 01:47:27.494004 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490225    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 22 01:47:27.494004 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490228    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 22 01:47:27.494004 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490230    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 22 01:47:27.494004 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490233    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 22 01:47:27.494004 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490236    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 22 01:47:27.494004 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490239    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 22 01:47:27.494478 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490312    2563 flags.go:64] FLAG: --address="0.0.0.0"
Jul 22 01:47:27.494478 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490319    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 22 01:47:27.494478 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490326    2563 flags.go:64] FLAG: --anonymous-auth="true"
Jul 22 01:47:27.494478 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490331    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 22 01:47:27.494478 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490335    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 22 01:47:27.494478 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490338    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 22 01:47:27.494478 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490342    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 22 01:47:27.494478 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490346    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 22 01:47:27.494478 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490349    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 22 01:47:27.494478 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490352    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 22 01:47:27.494478 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490356    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 22 01:47:27.494478 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490359    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 22 01:47:27.494478 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490362    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 22 01:47:27.494478 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490365    2563 flags.go:64] FLAG: --cgroup-root=""
Jul 22 01:47:27.494478 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490367    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 22 01:47:27.494478 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490370    2563 flags.go:64] FLAG: --client-ca-file=""
Jul 22 01:47:27.494478 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490389    2563 flags.go:64] FLAG: --cloud-provider="external"
Jul 22 01:47:27.494478 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490395    2563 flags.go:64] FLAG: --cluster-dns="[]"
Jul 22 01:47:27.494478 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490399    2563 flags.go:64] FLAG: --cluster-domain=""
Jul 22 01:47:27.494478 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490402    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 22 01:47:27.494478 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490405    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 22 01:47:27.494478 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490408    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 22 01:47:27.494478 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490412    2563 flags.go:64] FLAG: --container-log-max-files="5"
Jul 22 01:47:27.494478 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490417    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 22 01:47:27.495048 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490420    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 22 01:47:27.495048 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490423    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 22 01:47:27.495048 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490426    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 22 01:47:27.495048 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490429    2563 flags.go:64] FLAG: --contention-profiling="false"
Jul 22 01:47:27.495048 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490432    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 22 01:47:27.495048 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490435    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 22 01:47:27.495048 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490438    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 22 01:47:27.495048 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490441    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 22 01:47:27.495048 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490445    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 22 01:47:27.495048 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490449    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 22 01:47:27.495048 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490452    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 22 01:47:27.495048 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490455    2563 flags.go:64] FLAG: --enable-load-reader="false"
Jul 22 01:47:27.495048 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490458    2563 flags.go:64] FLAG: --enable-server="true"
Jul 22 01:47:27.495048 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490461    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 22 01:47:27.495048 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490466    2563 flags.go:64] FLAG: --event-burst="100"
Jul 22 01:47:27.495048 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490469    2563 flags.go:64] FLAG: --event-qps="50"
Jul 22 01:47:27.495048 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490472    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 22 01:47:27.495048 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490475    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 22 01:47:27.495048 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490478    2563 flags.go:64] FLAG: --eviction-hard=""
Jul 22 01:47:27.495048 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490482    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 22 01:47:27.495048 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490485    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 22 01:47:27.495048 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490488    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 22 01:47:27.495048 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490490    2563 flags.go:64] FLAG: --eviction-soft=""
Jul 22 01:47:27.495048 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490493    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 22 01:47:27.495048 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490496    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 22 01:47:27.495708 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490501    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 22 01:47:27.495708 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490504    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 22 01:47:27.495708 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490507    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 22 01:47:27.495708 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490509    2563 flags.go:64] FLAG: --fail-swap-on="true"
Jul 22 01:47:27.495708 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490512    2563 flags.go:64] FLAG: --feature-gates=""
Jul 22 01:47:27.495708 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490517    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 22 01:47:27.495708 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490521    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 22 01:47:27.495708 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490524    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 22 01:47:27.495708 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490527    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 22 01:47:27.495708 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490530    2563 flags.go:64] FLAG: --healthz-port="10248"
Jul 22 01:47:27.495708 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490533    2563 flags.go:64] FLAG: --help="false"
Jul 22 01:47:27.495708 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490536    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-135-230.ec2.internal"
Jul 22 01:47:27.495708 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490539    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 22 01:47:27.495708 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490542    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 22 01:47:27.495708 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490545    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 22 01:47:27.495708 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490549    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 22 01:47:27.495708 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490553    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 22 01:47:27.495708 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490556    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 22 01:47:27.495708 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490559    2563 flags.go:64] FLAG: --image-service-endpoint=""
Jul 22 01:47:27.495708 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490562    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 22 01:47:27.495708 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490564    2563 flags.go:64] FLAG: --kube-api-burst="100"
Jul 22 01:47:27.495708 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490567    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 22 01:47:27.495708 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490570    2563 flags.go:64] FLAG: --kube-api-qps="50"
Jul 22 01:47:27.495708 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490573    2563 flags.go:64] FLAG: --kube-reserved=""
Jul 22 01:47:27.496299 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490576    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 22 01:47:27.496299 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490579    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 22 01:47:27.496299 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490582    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 22 01:47:27.496299 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490584    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 22 01:47:27.496299 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490587    2563 flags.go:64] FLAG: --lock-file=""
Jul 22 01:47:27.496299 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490590    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 22 01:47:27.496299 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490593    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 22 01:47:27.496299 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490596    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 22 01:47:27.496299 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490601    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 22 01:47:27.496299 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490605    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 22 01:47:27.496299 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490607    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 22 01:47:27.496299 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490610    2563 flags.go:64] FLAG: --logging-format="text"
Jul 22 01:47:27.496299 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490613    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 22 01:47:27.496299 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490616    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 22 01:47:27.496299 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490620    2563 flags.go:64] FLAG: --manifest-url=""
Jul 22 01:47:27.496299 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490623    2563 flags.go:64] FLAG: --manifest-url-header=""
Jul 22 01:47:27.496299 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490628    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 22 01:47:27.496299 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490630    2563 flags.go:64] FLAG: --max-open-files="1000000"
Jul 22 01:47:27.496299 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490634    2563 flags.go:64] FLAG: --max-pods="110"
Jul 22 01:47:27.496299 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490637    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 22 01:47:27.496299 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490640    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 22 01:47:27.496299 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490643    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 22 01:47:27.496299 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490646    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 22 01:47:27.496299 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490650    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 22 01:47:27.496299 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490653    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 22 01:47:27.496926 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490656    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 22 01:47:27.496926 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490663    2563 flags.go:64] FLAG: --node-status-max-images="50"
Jul 22 01:47:27.496926 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490666    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 22 01:47:27.496926 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490668    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 22 01:47:27.496926 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490671    2563 flags.go:64] FLAG: --pod-cidr=""
Jul 22 01:47:27.496926 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490674    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 22 01:47:27.496926 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490680    2563 flags.go:64] FLAG: --pod-manifest-path=""
Jul 22 01:47:27.496926 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490682    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 22 01:47:27.496926 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490685    2563 flags.go:64] FLAG: --pods-per-core="0"
Jul 22 01:47:27.496926 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490688    2563 flags.go:64] FLAG: --port="10250"
Jul 22 01:47:27.496926 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490692    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 22 01:47:27.496926 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490694    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0871496704610b716"
Jul 22 01:47:27.496926 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490697    2563 flags.go:64] FLAG: --qos-reserved=""
Jul 22 01:47:27.496926 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490700    2563 flags.go:64] FLAG: --read-only-port="10255"
Jul 22 01:47:27.496926 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490703    2563 flags.go:64] FLAG: --register-node="true"
Jul 22 01:47:27.496926 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490706    2563 flags.go:64] FLAG: --register-with-taints=""
Jul 22 01:47:27.496926 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490710    2563 flags.go:64] FLAG: --registry-burst="10"
Jul 22 01:47:27.496926 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490713    2563 flags.go:64] FLAG: --registry-qps="5"
Jul 22 01:47:27.496926 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490716    2563 flags.go:64] FLAG: --reserved-cpus=""
Jul 22 01:47:27.496926 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490719    2563 flags.go:64] FLAG: --reserved-memory=""
Jul 22 01:47:27.496926 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490723    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 22 01:47:27.496926 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490726    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 22 01:47:27.496926 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490730    2563 flags.go:64] FLAG: --rotate-certificates="false"
Jul 22 01:47:27.496926 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490733    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 22 01:47:27.496926 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490735    2563 flags.go:64] FLAG: --runonce="false"
Jul 22 01:47:27.497566 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490738    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 22 01:47:27.497566 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490741    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 22 01:47:27.497566 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490744    2563 flags.go:64] FLAG: --seccomp-default="false"
Jul 22 01:47:27.497566 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490747    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 22 01:47:27.497566 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490750    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 22 01:47:27.497566 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490753    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 22 01:47:27.497566 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490756    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 22 01:47:27.497566 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490759    2563 flags.go:64] FLAG: --storage-driver-password="root"
Jul 22 01:47:27.497566 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490762    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 22 01:47:27.497566 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490765    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 22 01:47:27.497566 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490767    2563 flags.go:64] FLAG: --storage-driver-user="root"
Jul 22 01:47:27.497566 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490770    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 22 01:47:27.497566 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490773    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 22 01:47:27.497566 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490776    2563 flags.go:64] FLAG: --system-cgroups=""
Jul 22 01:47:27.497566 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490779    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 22 01:47:27.497566 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490784    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 22 01:47:27.497566 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490787    2563 flags.go:64] FLAG: --tls-cert-file=""
Jul 22 01:47:27.497566 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490789    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 22 01:47:27.497566 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490793    2563 flags.go:64] FLAG: --tls-min-version=""
Jul 22 01:47:27.497566 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490796    2563 flags.go:64] FLAG: --tls-private-key-file=""
Jul 22 01:47:27.497566 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490799    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 22 01:47:27.497566 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490802    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 22 01:47:27.497566 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490804    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 22 01:47:27.497566 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490807    2563 flags.go:64] FLAG: --v="2"
Jul 22 01:47:27.497566 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490812    2563 flags.go:64] FLAG: --version="false"
Jul 22 01:47:27.498163 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490816    2563 flags.go:64] FLAG: --vmodule=""
Jul 22 01:47:27.498163 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490819    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 22 01:47:27.498163 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.490823    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 22 01:47:27.498163 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490939    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 22 01:47:27.498163 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490945    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 22 01:47:27.498163 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490948    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 22 01:47:27.498163 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490951    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 22 01:47:27.498163 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490954    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 22 01:47:27.498163 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490956    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 22 01:47:27.498163 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490959    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 22 01:47:27.498163 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490963    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 22 01:47:27.498163 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490966    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 22 01:47:27.498163 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490969    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 22 01:47:27.498163 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490972    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 22 01:47:27.498163 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490974    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 22 01:47:27.498163 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490978    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 22 01:47:27.498163 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490980    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 22 01:47:27.498163 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490983    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 22 01:47:27.498163 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490987    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 22 01:47:27.498163 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490990    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 22 01:47:27.498929 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490994    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 22 01:47:27.498929 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.490997    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 22 01:47:27.498929 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491000    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 22 01:47:27.498929 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491002    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 22 01:47:27.498929 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491005    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 22 01:47:27.498929 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491008    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 22 01:47:27.498929 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491010    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 22 01:47:27.498929 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491013    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 22 01:47:27.498929 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491016    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 22 01:47:27.498929 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491018    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 22 01:47:27.498929 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491021    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 22 01:47:27.498929 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491024    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 22 01:47:27.498929 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491027    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 22 01:47:27.498929 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491030    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 22 01:47:27.498929 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491032    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 22 01:47:27.498929 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491035    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 22 01:47:27.498929 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491039    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 22 01:47:27.498929 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491041    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 22 01:47:27.498929 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491044    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 22 01:47:27.499774 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491046    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 22 01:47:27.499774 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491049    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 22 01:47:27.499774 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491051    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 22 01:47:27.499774 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491054    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 22 01:47:27.499774 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491057    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 22 01:47:27.499774 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491059    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 22 01:47:27.499774 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491061    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 22 01:47:27.499774 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491064    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 22 01:47:27.499774 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491067    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 22 01:47:27.499774 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491069    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 22 01:47:27.499774 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491072    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 22 01:47:27.499774 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491075    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 22 01:47:27.499774 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491077    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 22 01:47:27.499774 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491080    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 22 01:47:27.499774 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491082    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 22 01:47:27.499774 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491085    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 22 01:47:27.499774 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491088    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 22 01:47:27.499774 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491090    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 22 01:47:27.499774 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491092    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 22 01:47:27.499774 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491095    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 22 01:47:27.500631 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491098    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 22 01:47:27.500631 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491104    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 22 01:47:27.500631 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491106    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 22 01:47:27.500631 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491109    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 22 01:47:27.500631 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491112    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 22 01:47:27.500631 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491114    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 22 01:47:27.500631 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491117    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 22 01:47:27.500631 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491121    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 22 01:47:27.500631 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491123    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 22 01:47:27.500631 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491127    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 22 01:47:27.500631 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491129    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 22 01:47:27.500631 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491132    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 22 01:47:27.500631 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491134    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 22 01:47:27.500631 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491137    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 22 01:47:27.500631 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491140    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 22 01:47:27.500631 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491142    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 22 01:47:27.500631 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491145    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 22 01:47:27.500631 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491147    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 22 01:47:27.500631 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491150    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 22 01:47:27.500631 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491152    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 22 01:47:27.501534 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491155    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 22 01:47:27.501534 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491159    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 22 01:47:27.501534 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491162    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 22 01:47:27.501534 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491165    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 22 01:47:27.501534 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491168    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 22 01:47:27.501534 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491172    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 22 01:47:27.501534 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491175    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 22 01:47:27.501534 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491177    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 22 01:47:27.501534 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491180    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 22 01:47:27.501534 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491183    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 22 01:47:27.501534 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491186    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 22 01:47:27.501534 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491189    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 22 01:47:27.501534 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491192    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 22 01:47:27.501534 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491196    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 22 01:47:27.501534 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491199    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 22 01:47:27.501534 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491201    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 22 01:47:27.501534 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491204    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 22 01:47:27.501534 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491207    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 22 01:47:27.501534 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491209    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 22 01:47:27.501534 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491212    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 22 01:47:27.502262 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491215    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 22 01:47:27.502262 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491219    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 22 01:47:27.502262 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.491221    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 22 01:47:27.502262 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.491999    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 22 01:47:27.502262 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.500438    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 22 01:47:27.502262 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.500459    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 22 01:47:27.502262 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.500485    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 22 01:47:27.502262 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.500494    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 22 01:47:27.502262 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500585    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 22 01:47:27.502262 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500593    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 22 01:47:27.502262 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500600    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 22 01:47:27.502262 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500605    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 22 01:47:27.502262 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500610    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 22 01:47:27.502262 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500615    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 22 01:47:27.502639 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500620    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 22 01:47:27.502639 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500624    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 22 01:47:27.502639 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500629    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 22 01:47:27.502639 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500633    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 22 01:47:27.502639 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500636    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 22 01:47:27.502639 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500643    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 22 01:47:27.502639 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500648    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 22 01:47:27.502639 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500652    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 22 01:47:27.502639 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500657    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 22 01:47:27.502639 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500661    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 22 01:47:27.502639 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500667    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 22 01:47:27.502639 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500671    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 22 01:47:27.502639 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500676    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 22 01:47:27.502639 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500680    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 22 01:47:27.502639 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500685    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 22 01:47:27.502639 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500690    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 22 01:47:27.502639 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500694    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 22 01:47:27.502639 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500698    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 22 01:47:27.502639 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500702    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 22 01:47:27.503355 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500707    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 22 01:47:27.503355 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500712    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 22 01:47:27.503355 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500716    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 22 01:47:27.503355 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500721    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 22 01:47:27.503355 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500732    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 22 01:47:27.503355 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500736    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 22 01:47:27.503355 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500741    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 22 01:47:27.503355 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500745    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 22 01:47:27.503355 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500749    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 22 01:47:27.503355 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500754    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 22 01:47:27.503355 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500758    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 22 01:47:27.503355 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500762    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 22 01:47:27.503355 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500767    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 22 01:47:27.503355 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500771    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 22 01:47:27.503355 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500775    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 22 01:47:27.503355 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500779    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 22 01:47:27.503355 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500783    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 22 01:47:27.503355 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500787    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 22 01:47:27.503355 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500791    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 22 01:47:27.503355 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500795    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 22 01:47:27.504219 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500800    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 22 01:47:27.504219 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500804    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 22 01:47:27.504219 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500809    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 22 01:47:27.504219 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500813    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 22 01:47:27.504219 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500817    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 22 01:47:27.504219 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500821    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 22 01:47:27.504219 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500825    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 22 01:47:27.504219 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500831    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 22 01:47:27.504219 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500835    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 22 01:47:27.504219 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500839    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 22 01:47:27.504219 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500843    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 22 01:47:27.504219 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500848    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 22 01:47:27.504219 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500853    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 22 01:47:27.504219 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500858    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 22 01:47:27.504219 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500862    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 22 01:47:27.504219 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500870    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 22 01:47:27.504219 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500876    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 22 01:47:27.504219 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500881    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 22 01:47:27.504219 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500886    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 22 01:47:27.504796 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500891    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 22 01:47:27.504796 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500896    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 22 01:47:27.504796 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500901    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 22 01:47:27.504796 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500905    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 22 01:47:27.504796 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500910    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 22 01:47:27.504796 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500914    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 22 01:47:27.504796 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500918    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 22 01:47:27.504796 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500923    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 22 01:47:27.504796 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500927    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 22 01:47:27.504796 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500931    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 22 01:47:27.504796 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500936    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 22 01:47:27.504796 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500940    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 22 01:47:27.504796 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500944    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 22 01:47:27.504796 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500948    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 22 01:47:27.504796 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500953    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 22 01:47:27.504796 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500957    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 22 01:47:27.504796 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500961    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 22 01:47:27.504796 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500965    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 22 01:47:27.504796 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500969    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 22 01:47:27.504796 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500973    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 22 01:47:27.505335 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500977    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 22 01:47:27.505335 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500984    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 22 01:47:27.505335 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500990    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 22 01:47:27.505335 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500994    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 22 01:47:27.505335 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.500998    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 22 01:47:27.505335 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501003    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 22 01:47:27.505335 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501007    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 22 01:47:27.505335 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501011    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 22 01:47:27.505335 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501015    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 22 01:47:27.505335 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501020    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 22 01:47:27.505335 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501025    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 22 01:47:27.505335 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501037    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 22 01:47:27.505335 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501042    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 22 01:47:27.505335 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501046    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 22 01:47:27.505335 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501050    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 22 01:47:27.505740 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.501058    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 22 01:47:27.505740 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501266    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 22 01:47:27.505740 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501274    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 22 01:47:27.505740 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501279    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 22 01:47:27.505740 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501284    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 22 01:47:27.505740 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501288    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 22 01:47:27.505740 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501293    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 22 01:47:27.505740 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501298    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 22 01:47:27.505740 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501305    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 22 01:47:27.505740 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501312    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 22 01:47:27.505740 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501317    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 22 01:47:27.505740 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501322    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 22 01:47:27.505740 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501326    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 22 01:47:27.505740 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501330    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 22 01:47:27.505740 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501334    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 22 01:47:27.506192 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501339    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 22 01:47:27.506192 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501343    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 22 01:47:27.506192 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501348    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 22 01:47:27.506192 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501352    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 22 01:47:27.506192 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501356    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 22 01:47:27.506192 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501361    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 22 01:47:27.506192 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501365    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 22 01:47:27.506192 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501369    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 22 01:47:27.506192 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501393    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 22 01:47:27.506192 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501398    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 22 01:47:27.506192 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501402    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 22 01:47:27.506192 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501406    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 22 01:47:27.506192 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501413    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 22 01:47:27.506192 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501419    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 22 01:47:27.506192 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501429    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 22 01:47:27.506192 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501433    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 22 01:47:27.506192 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501444    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 22 01:47:27.506192 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501449    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 22 01:47:27.506192 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501453    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 22 01:47:27.506714 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501458    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 22 01:47:27.506714 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501463    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 22 01:47:27.506714 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501468    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 22 01:47:27.506714 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501472    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 22 01:47:27.506714 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501476    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 22 01:47:27.506714 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501481    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 22 01:47:27.506714 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501485    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 22 01:47:27.506714 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501490    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 22 01:47:27.506714 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501495    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 22 01:47:27.506714 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501500    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 22 01:47:27.506714 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501504    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 22 01:47:27.506714 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501509    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 22 01:47:27.506714 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501513    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 22 01:47:27.506714 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501517    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 22 01:47:27.506714 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501521    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 22 01:47:27.506714 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501526    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 22 01:47:27.506714 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501530    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 22 01:47:27.506714 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501534    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 22 01:47:27.506714 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501538    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 22 01:47:27.506714 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501542    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 22 01:47:27.507224 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501545    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 22 01:47:27.507224 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501549    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 22 01:47:27.507224 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501553    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 22 01:47:27.507224 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501557    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 22 01:47:27.507224 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501561    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 22 01:47:27.507224 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501566    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 22 01:47:27.507224 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501570    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 22 01:47:27.507224 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501575    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 22 01:47:27.507224 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501581    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 22 01:47:27.507224 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501585    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 22 01:47:27.507224 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501589    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 22 01:47:27.507224 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501593    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 22 01:47:27.507224 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501597    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 22 01:47:27.507224 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501601    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 22 01:47:27.507224 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501605    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 22 01:47:27.507224 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501610    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 22 01:47:27.507224 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501614    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 22 01:47:27.507224 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501618    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 22 01:47:27.507224 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501622    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 22 01:47:27.507224 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501626    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 22 01:47:27.507723 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501630    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 22 01:47:27.507723 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501634    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 22 01:47:27.507723 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501638    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 22 01:47:27.507723 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501642    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 22 01:47:27.507723 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501647    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 22 01:47:27.507723 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501651    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 22 01:47:27.507723 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501656    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 22 01:47:27.507723 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501661    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 22 01:47:27.507723 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501665    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 22 01:47:27.507723 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501670    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 22 01:47:27.507723 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501674    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 22 01:47:27.507723 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501678    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 22 01:47:27.507723 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501682    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 22 01:47:27.507723 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501686    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 22 01:47:27.507723 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501690    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 22 01:47:27.507723 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501694    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 22 01:47:27.507723 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501699    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 22 01:47:27.507723 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501703    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 22 01:47:27.507723 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501709    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 22 01:47:27.508242 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501714    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 22 01:47:27.508242 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501719    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 22 01:47:27.508242 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501732    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 22 01:47:27.508242 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501737    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 22 01:47:27.508242 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501742    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 22 01:47:27.508242 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501746    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 22 01:47:27.508242 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:27.501751    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 22 01:47:27.508242 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.501759    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 22 01:47:27.508242 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.501939    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 22 01:47:27.508242 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.506083    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 22 01:47:27.508242 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.507812    2563 server.go:1019] "Starting client certificate rotation"
Jul 22 01:47:27.508242 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.507918    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 22 01:47:27.508711 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.508620    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 22 01:47:27.511618 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.511578    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 22 01:47:27.512930 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.512908    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 22 01:47:27.526027 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.526007    2563 log.go:25] "Validated CRI v1 runtime API"
Jul 22 01:47:27.529712 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.529688    2563 log.go:25] "Validated CRI v1 image API"
Jul 22 01:47:27.532434 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.532408    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 22 01:47:27.536853 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.536830    2563 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 81b26788-721c-4c87-b7e3-64723b0e74e7:/dev/nvme0n1p3 dde6147f-d045-461a-bb0f-ebaf865b05c7:/dev/nvme0n1p4]
Jul 22 01:47:27.536950 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.536851    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 22 01:47:27.539856 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.539840    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 22 01:47:27.541995 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.541875    2563 manager.go:216] Machine: {Timestamp:2026-07-22 01:47:27.541402831 +0000 UTC m=+0.385231953 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098038 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27e89cbc641f4fc516d67f7a19417a SystemUUID:ec27e89c-bc64-1f4f-c516-d67f7a19417a BootID:2f13987c-6738-4db0-925d-6465ad51c8cb Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ef:c4:16:0a:0b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ef:c4:16:0a:0b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:da:47:fe:c5:21:8d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 22 01:47:27.541995 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.541987    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 22 01:47:27.542114 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.542091    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 22 01:47:27.545768 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.545736    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 22 01:47:27.545946 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.545770    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-230.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 22 01:47:27.547404 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.547391    2563 topology_manager.go:138] "Creating topology manager with none policy"
Jul 22 01:47:27.547461 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.547408    2563 container_manager_linux.go:309] "Creating device plugin manager"
Jul 22 01:47:27.547461 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.547418    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 22 01:47:27.548249 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.548237    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 22 01:47:27.550222 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.550209    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 22 01:47:27.555403 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.555388    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 22 01:47:27.555543 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.555531    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 22 01:47:27.555606 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.555570    2563 kubelet.go:486] "Attempting to sync node with API server"
Jul 22 01:47:27.555606 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.555586    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 22 01:47:27.555606 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.555605    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 22 01:47:27.555729 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.555619    2563 kubelet.go:398] "Adding apiserver pod source"
Jul 22 01:47:27.555729 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.555632    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 22 01:47:27.555998 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.555984    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 22 01:47:27.556059 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.556007    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 22 01:47:27.558441 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.558426    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 22 01:47:27.560883 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.560868    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 22 01:47:27.560954 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.560899    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 22 01:47:27.562341 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.562329    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 22 01:47:27.562392 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.562345    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 22 01:47:27.562392 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.562351    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 22 01:47:27.562392 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.562357    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 22 01:47:27.562392 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.562365    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 22 01:47:27.562392 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.562372    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 22 01:47:27.562392 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.562392    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 22 01:47:27.562544 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.562397    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 22 01:47:27.562544 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.562405    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 22 01:47:27.562544 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.562411    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 22 01:47:27.562544 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.562424    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 22 01:47:27.562544 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.562430    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 22 01:47:27.563428 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.563419    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 22 01:47:27.563466 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.563428    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 22 01:47:27.565995 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.565957    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-230.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 22 01:47:27.566069 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:27.565975    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 22 01:47:27.566069 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:27.566007    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-230.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 22 01:47:27.566623 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.566609    2563 server.go:1268] "Started kubelet"
Jul 22 01:47:27.566716 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.566677    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 22 01:47:27.567407 ip-10-0-135-230 systemd[1]: Started Kubernetes Kubelet.
Jul 22 01:47:27.567909 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.567412    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 22 01:47:27.567909 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.567457    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 22 01:47:27.568651 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.568631    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9t8sh"
Jul 22 01:47:27.569325 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.569310    2563 server.go:310] "Adding debug handlers to kubelet server"
Jul 22 01:47:27.569661 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.569630    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 22 01:47:27.571447 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:27.571427    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 22 01:47:27.574504 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.574489    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9t8sh"
Jul 22 01:47:27.577228 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:27.576084    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-230.ec2.internal.18c47969c7399cce  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-230.ec2.internal,UID:ip-10-0-135-230.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-230.ec2.internal,},FirstTimestamp:2026-07-22 01:47:27.566527694 +0000 UTC m=+0.410356816,LastTimestamp:2026-07-22 01:47:27.566527694 +0000 UTC m=+0.410356816,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-230.ec2.internal,}"
Jul 22 01:47:27.580928 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.580906    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 22 01:47:27.580928 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.580915    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 22 01:47:27.581189 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:27.581046    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-230.ec2.internal\" not found"
Jul 22 01:47:27.582910 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.582628    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 22 01:47:27.582910 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:27.582812    2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-230.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 22 01:47:27.583119 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.583101    2563 factory.go:55] Registering systemd factory
Jul 22 01:47:27.583195 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.583125    2563 factory.go:223] Registration of the systemd container factory successfully
Jul 22 01:47:27.583249 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:27.583223    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 22 01:47:27.583298 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.583263    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 22 01:47:27.583298 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.583275    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 22 01:47:27.583449 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.583433    2563 reconstruct.go:97] "Volume reconstruction finished"
Jul 22 01:47:27.583449 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.583449    2563 reconciler.go:29] "Reconciler: start to sync state"
Jul 22 01:47:27.583608 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.583522    2563 factory.go:153] Registering CRI-O factory
Jul 22 01:47:27.583608 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.583535    2563 factory.go:223] Registration of the crio container factory successfully
Jul 22 01:47:27.583608 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.583578    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 22 01:47:27.583608 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.583600    2563 factory.go:103] Registering Raw factory
Jul 22 01:47:27.583774 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.583613    2563 manager.go:1196] Started watching for new ooms in manager
Jul 22 01:47:27.584001 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.583988    2563 manager.go:349] Starting recovery of all containers
Jul 22 01:47:27.593553 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.593538    2563 manager.go:354] Recovery completed
Jul 22 01:47:27.598789 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.598777    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 22 01:47:27.604635 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.604619    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-230.ec2.internal" event="NodeHasSufficientMemory"
Jul 22 01:47:27.604704 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.604648    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-230.ec2.internal" event="NodeHasNoDiskPressure"
Jul 22 01:47:27.604704 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.604658    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-230.ec2.internal" event="NodeHasSufficientPID"
Jul 22 01:47:27.605213 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.605196    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 22 01:47:27.605213 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.605211    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 22 01:47:27.605356 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.605217    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 22 01:47:27.605356 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.605229    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 22 01:47:27.607305 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.607294    2563 policy_none.go:49] "None policy: Start"
Jul 22 01:47:27.607353 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.607308    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 22 01:47:27.607353 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.607318    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 22 01:47:27.607863 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.607848    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-230.ec2.internal" event="NodeHasSufficientMemory"
Jul 22 01:47:27.607920 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.607875    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-230.ec2.internal" event="NodeHasNoDiskPressure"
Jul 22 01:47:27.607920 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.607886    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-230.ec2.internal" event="NodeHasSufficientPID"
Jul 22 01:47:27.609361 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.609349    2563 policy_none.go:47] "Start"
Jul 22 01:47:27.649110 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.648952    2563 manager.go:337] "Starting Device Plugin manager"
Jul 22 01:47:27.665799 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:27.649142    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 22 01:47:27.665799 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.649165    2563 server.go:87] "Starting device plugin registration server"
Jul 22 01:47:27.665799 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.649741    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 22 01:47:27.665799 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.649759    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 22 01:47:27.665799 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.650001    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 22 01:47:27.665799 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.650086    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 22 01:47:27.665799 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.650095    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 22 01:47:27.665799 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:27.650609    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 22 01:47:27.665799 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:27.650669    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-230.ec2.internal\" not found"
Jul 22 01:47:27.665799 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.655589    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 22 01:47:27.665799 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.656828    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 22 01:47:27.665799 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.656859    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 22 01:47:27.665799 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.656877    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 22 01:47:27.665799 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:27.656906    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 22 01:47:27.665799 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.659340    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 22 01:47:27.750793 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.750773    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 22 01:47:27.751718 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.751700    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-230.ec2.internal" event="NodeHasSufficientMemory"
Jul 22 01:47:27.751852 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.751729    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-230.ec2.internal" event="NodeHasNoDiskPressure"
Jul 22 01:47:27.751852 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.751741    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-230.ec2.internal" event="NodeHasSufficientPID"
Jul 22 01:47:27.751852 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.751766    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-230.ec2.internal"
Jul 22 01:47:27.757586 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.757562    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-230.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-230.ec2.internal"]
Jul 22 01:47:27.757679 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.757636    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 22 01:47:27.759362 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.759348    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-230.ec2.internal" event="NodeHasSufficientMemory"
Jul 22 01:47:27.759456 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.759376    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-230.ec2.internal" event="NodeHasNoDiskPressure"
Jul 22 01:47:27.759456 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.759405    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-230.ec2.internal" event="NodeHasSufficientPID"
Jul 22 01:47:27.761526 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.761512    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-230.ec2.internal"
Jul 22 01:47:27.761567 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:27.761532    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-230.ec2.internal\": node \"ip-10-0-135-230.ec2.internal\" not found"
Jul 22 01:47:27.761889 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.761876    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 22 01:47:27.762055 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.762037    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-230.ec2.internal"
Jul 22 01:47:27.762120 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.762077    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 22 01:47:27.762746 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.762729    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-230.ec2.internal" event="NodeHasSufficientMemory"
Jul 22 01:47:27.762746 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.762743    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-230.ec2.internal" event="NodeHasSufficientMemory"
Jul 22 01:47:27.762850 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.762759    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-230.ec2.internal" event="NodeHasNoDiskPressure"
Jul 22 01:47:27.762850 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.762763    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-230.ec2.internal" event="NodeHasNoDiskPressure"
Jul 22 01:47:27.762850 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.762772    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-230.ec2.internal" event="NodeHasSufficientPID"
Jul 22 01:47:27.762850 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.762775    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-230.ec2.internal" event="NodeHasSufficientPID"
Jul 22 01:47:27.765214 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.765196    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-230.ec2.internal"
Jul 22 01:47:27.765297 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.765231    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 22 01:47:27.766234 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.766217    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-230.ec2.internal" event="NodeHasSufficientMemory"
Jul 22 01:47:27.766313 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.766245    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-230.ec2.internal" event="NodeHasNoDiskPressure"
Jul 22 01:47:27.766313 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.766255    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-230.ec2.internal" event="NodeHasSufficientPID"
Jul 22 01:47:27.775657 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:27.775641    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-230.ec2.internal\" not found"
Jul 22 01:47:27.784388 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.784368    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f96cb24b354bdc0f39f6acdd48c651dc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-230.ec2.internal\" (UID: \"f96cb24b354bdc0f39f6acdd48c651dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-230.ec2.internal"
Jul 22 01:47:27.784455 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.784406    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a67e292d73c90fc17a80248cb369ca7d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-230.ec2.internal\" (UID: \"a67e292d73c90fc17a80248cb369ca7d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-230.ec2.internal"
Jul 22 01:47:27.784455 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.784431    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f96cb24b354bdc0f39f6acdd48c651dc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-230.ec2.internal\" (UID: \"f96cb24b354bdc0f39f6acdd48c651dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-230.ec2.internal"
Jul 22 01:47:27.788259 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:27.788240    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-230.ec2.internal\" not found" node="ip-10-0-135-230.ec2.internal"
Jul 22 01:47:27.792932 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:27.792915    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-230.ec2.internal\" not found" node="ip-10-0-135-230.ec2.internal"
Jul 22 01:47:27.876110 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:27.876043    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-230.ec2.internal\" not found"
Jul 22 01:47:27.884785 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.884749    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f96cb24b354bdc0f39f6acdd48c651dc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-230.ec2.internal\" (UID: \"f96cb24b354bdc0f39f6acdd48c651dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-230.ec2.internal"
Jul 22 01:47:27.884906 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.884787    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f96cb24b354bdc0f39f6acdd48c651dc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-230.ec2.internal\" (UID: \"f96cb24b354bdc0f39f6acdd48c651dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-230.ec2.internal"
Jul 22 01:47:27.884906 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.884812    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a67e292d73c90fc17a80248cb369ca7d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-230.ec2.internal\" (UID: \"a67e292d73c90fc17a80248cb369ca7d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-230.ec2.internal"
Jul 22 01:47:27.884906 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.884831    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f96cb24b354bdc0f39f6acdd48c651dc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-230.ec2.internal\" (UID: \"f96cb24b354bdc0f39f6acdd48c651dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-230.ec2.internal"
Jul 22 01:47:27.884906 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.884873    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a67e292d73c90fc17a80248cb369ca7d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-230.ec2.internal\" (UID: \"a67e292d73c90fc17a80248cb369ca7d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-230.ec2.internal"
Jul 22 01:47:27.884906 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:27.884897    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f96cb24b354bdc0f39f6acdd48c651dc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-230.ec2.internal\" (UID: \"f96cb24b354bdc0f39f6acdd48c651dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-230.ec2.internal"
Jul 22 01:47:27.976936 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:27.976906    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-230.ec2.internal\" not found"
Jul 22 01:47:28.078003 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:28.077959    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-230.ec2.internal\" not found"
Jul 22 01:47:28.092580 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:28.092552    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-230.ec2.internal"
Jul 22 01:47:28.097020 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:28.096999    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-230.ec2.internal"
Jul 22 01:47:28.178149 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:28.178055    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-230.ec2.internal\" not found"
Jul 22 01:47:28.285888 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:28.285854    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-230.ec2.internal\" not found"
Jul 22 01:47:28.386905 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:28.386877    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-230.ec2.internal\" not found"
Jul 22 01:47:28.488098 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:28.488033    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-230.ec2.internal\" not found"
Jul 22 01:47:28.507654 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:28.507629    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 22 01:47:28.507802 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:28.507785    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 22 01:47:28.584000 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:28.583969    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-21 01:42:28 +0000 UTC" deadline="2028-01-10 11:49:42.951744758 +0000 UTC"
Jul 22 01:47:28.584000 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:28.583996    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12898h2m14.36775244s"
Jul 22 01:47:28.588548 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:28.588530    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-230.ec2.internal\" not found"
Jul 22 01:47:28.688862 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:28.688829    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-230.ec2.internal\" not found"
Jul 22 01:47:28.724694 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:28.724672    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 22 01:47:28.732053 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:28.732028    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf96cb24b354bdc0f39f6acdd48c651dc.slice/crio-3b113b66604f2dc93861baa276f26ea8617ee0f34c47ee27cbc85e929c51a542 WatchSource:0}: Error finding container 3b113b66604f2dc93861baa276f26ea8617ee0f34c47ee27cbc85e929c51a542: Status 404 returned error can't find the container with id 3b113b66604f2dc93861baa276f26ea8617ee0f34c47ee27cbc85e929c51a542
Jul 22 01:47:28.738350 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:28.738302    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 22 01:47:28.757903 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:28.757882    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda67e292d73c90fc17a80248cb369ca7d.slice/crio-3d9c5cbba759c6e09acf3b7c1449063e5a2c5e8ba0b1a3e393b79f8e3b56f177 WatchSource:0}: Error finding container 3d9c5cbba759c6e09acf3b7c1449063e5a2c5e8ba0b1a3e393b79f8e3b56f177: Status 404 returned error can't find the container with id 3d9c5cbba759c6e09acf3b7c1449063e5a2c5e8ba0b1a3e393b79f8e3b56f177
Jul 22 01:47:28.781576 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:28.781550    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-230.ec2.internal"
Jul 22 01:47:28.792565 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:28.792551    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 22 01:47:28.794772 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:28.794759    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-230.ec2.internal"
Jul 22 01:47:28.803868 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:28.803852    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 22 01:47:28.914731 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:28.914699    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 22 01:47:29.007082 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.007007    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 22 01:47:29.557106 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.557073    2563 apiserver.go:52] "Watching apiserver"
Jul 22 01:47:29.565006 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.564978    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 22 01:47:29.566786 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.566754    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-mjkvg","kube-system/kube-apiserver-proxy-ip-10-0-135-230.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzqpt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-230.ec2.internal","openshift-multus/multus-additional-cni-plugins-pbf4b","openshift-multus/network-metrics-daemon-t7sb9","openshift-network-diagnostics/network-check-target-fltbj","openshift-network-operator/iptables-alerter-pmrf8","kube-system/konnectivity-agent-rd82t","openshift-cluster-node-tuning-operator/tuned-sq599","openshift-dns/node-resolver-4xkbt","openshift-image-registry/node-ca-drsr9","openshift-multus/multus-78zsq","openshift-ovn-kubernetes/ovnkube-node-bpxz8"]
Jul 22 01:47:29.569562 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.569537    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sq599"
Jul 22 01:47:29.572157 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.572133    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 22 01:47:29.572330 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.572308    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 22 01:47:29.572853 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.572411    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zqlx4\""
Jul 22 01:47:29.572853 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.572628    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.575549 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.575522    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 22 01:47:29.576940 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.576907    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzqpt"
Jul 22 01:47:29.577156 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.577123    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 22 01:47:29.577282 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.577212    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rt8gk\""
Jul 22 01:47:29.577548 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.577349    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 22 01:47:29.577693 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.577673    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 22 01:47:29.579824 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.579804    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.582967 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.581865    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 22 01:47:29.582967 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.581893    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 22 01:47:29.582967 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.582098    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 22 01:47:29.582967 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.582197    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pgbfd\""
Jul 22 01:47:29.582967 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.582342    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 22 01:47:29.584696 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.584252    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 22 01:47:29.584696 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.584337    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 22 01:47:29.584696 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.584568    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 22 01:47:29.584891 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.584788    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 22 01:47:29.585254 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.584981    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 22 01:47:29.585254 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.585102    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pm2n6\""
Jul 22 01:47:29.585254 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.585250    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 22 01:47:29.585477 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.585320    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pbf4b"
Jul 22 01:47:29.588409 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.587104    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t7sb9"
Jul 22 01:47:29.588409 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:29.587171    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t7sb9" podUID="22905b02-8ed1-4c26-92ae-7282baf885cd"
Jul 22 01:47:29.588409 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.587662    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 22 01:47:29.588409 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.587712    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-w7kvf\""
Jul 22 01:47:29.588838 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.588733    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 22 01:47:29.589853 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.589836    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rd82t"
Jul 22 01:47:29.592502 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.592435    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fltbj"
Jul 22 01:47:29.592609 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:29.592506    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fltbj" podUID="e6cd3d7c-b979-4d69-aad7-68ab39aed83d"
Jul 22 01:47:29.593239 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.592788    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 22 01:47:29.593239 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.592899    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2f8g8\""
Jul 22 01:47:29.593239 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.593009    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 22 01:47:29.593239 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.593106    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2553bf24-562e-4e3b-b5d3-d89687dd807f-hostroot\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.593239 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.593147    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/00689b5e-a714-4b23-8b13-b16b3c05c817-socket-dir\") pod \"aws-ebs-csi-driver-node-zzqpt\" (UID: \"00689b5e-a714-4b23-8b13-b16b3c05c817\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzqpt"
Jul 22 01:47:29.593239 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.593172    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-host-cni-bin\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.593239 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.593198    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8ctp\" (UniqueName: \"kubernetes.io/projected/91595658-5e3d-40eb-9763-67b1f193589d-kube-api-access-f8ctp\") pod \"tuned-sq599\" (UID: \"91595658-5e3d-40eb-9763-67b1f193589d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq599"
Jul 22 01:47:29.593239 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.593222    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-node-log\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.593732 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.593249    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4290331-2538-4205-b3d0-58ab1a1126b6-system-cni-dir\") pod \"multus-additional-cni-plugins-pbf4b\" (UID: \"c4290331-2538-4205-b3d0-58ab1a1126b6\") " pod="openshift-multus/multus-additional-cni-plugins-pbf4b"
Jul 22 01:47:29.593732 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.593284    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c4290331-2538-4205-b3d0-58ab1a1126b6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pbf4b\" (UID: \"c4290331-2538-4205-b3d0-58ab1a1126b6\") " pod="openshift-multus/multus-additional-cni-plugins-pbf4b"
Jul 22 01:47:29.593732 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.593309    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/91595658-5e3d-40eb-9763-67b1f193589d-etc-sysconfig\") pod \"tuned-sq599\" (UID: \"91595658-5e3d-40eb-9763-67b1f193589d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq599"
Jul 22 01:47:29.593732 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.593336    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2553bf24-562e-4e3b-b5d3-d89687dd807f-multus-cni-dir\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.593732 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.593423    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2553bf24-562e-4e3b-b5d3-d89687dd807f-etc-kubernetes\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.593732 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.593451    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4smjw\" (UniqueName: \"kubernetes.io/projected/2553bf24-562e-4e3b-b5d3-d89687dd807f-kube-api-access-4smjw\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.593732 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.593479    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62fx2\" (UniqueName: \"kubernetes.io/projected/00689b5e-a714-4b23-8b13-b16b3c05c817-kube-api-access-62fx2\") pod \"aws-ebs-csi-driver-node-zzqpt\" (UID: \"00689b5e-a714-4b23-8b13-b16b3c05c817\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzqpt"
Jul 22 01:47:29.593732 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.593506    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-run-ovn\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.593732 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.593534    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-var-lib-openvswitch\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.593732 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.593559    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2553bf24-562e-4e3b-b5d3-d89687dd807f-host-run-k8s-cni-cncf-io\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.593732 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.593618    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91595658-5e3d-40eb-9763-67b1f193589d-lib-modules\") pod \"tuned-sq599\" (UID: \"91595658-5e3d-40eb-9763-67b1f193589d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq599"
Jul 22 01:47:29.593732 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.593649    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2553bf24-562e-4e3b-b5d3-d89687dd807f-cnibin\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.593732 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.593674    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt77g\" (UniqueName: \"kubernetes.io/projected/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-kube-api-access-tt77g\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.593732 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.593711    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2553bf24-562e-4e3b-b5d3-d89687dd807f-host-var-lib-kubelet\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.593732 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.593738    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/91595658-5e3d-40eb-9763-67b1f193589d-etc-modprobe-d\") pod \"tuned-sq599\" (UID: \"91595658-5e3d-40eb-9763-67b1f193589d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq599"
Jul 22 01:47:29.594502 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.593762    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91595658-5e3d-40eb-9763-67b1f193589d-etc-kubernetes\") pod \"tuned-sq599\" (UID: \"91595658-5e3d-40eb-9763-67b1f193589d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq599"
Jul 22 01:47:29.594502 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.593800    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/91595658-5e3d-40eb-9763-67b1f193589d-etc-sysctl-d\") pod \"tuned-sq599\" (UID: \"91595658-5e3d-40eb-9763-67b1f193589d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq599"
Jul 22 01:47:29.594502 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.593816    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91595658-5e3d-40eb-9763-67b1f193589d-sys\") pod \"tuned-sq599\" (UID: \"91595658-5e3d-40eb-9763-67b1f193589d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq599"
Jul 22 01:47:29.594502 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.593849    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91595658-5e3d-40eb-9763-67b1f193589d-var-lib-kubelet\") pod \"tuned-sq599\" (UID: \"91595658-5e3d-40eb-9763-67b1f193589d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq599"
Jul 22 01:47:29.594502 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.593866    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/00689b5e-a714-4b23-8b13-b16b3c05c817-registration-dir\") pod \"aws-ebs-csi-driver-node-zzqpt\" (UID: \"00689b5e-a714-4b23-8b13-b16b3c05c817\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzqpt"
Jul 22 01:47:29.594502 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.593894    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91595658-5e3d-40eb-9763-67b1f193589d-tmp\") pod \"tuned-sq599\" (UID: \"91595658-5e3d-40eb-9763-67b1f193589d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq599"
Jul 22 01:47:29.594502 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.593941    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2553bf24-562e-4e3b-b5d3-d89687dd807f-multus-daemon-config\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.594502 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.593971    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-host-kubelet\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.594502 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.593998    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-host-run-ovn-kubernetes\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.594502 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.594026    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.594502 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.594054    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2553bf24-562e-4e3b-b5d3-d89687dd807f-host-var-lib-cni-bin\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.594502 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.594077    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/00689b5e-a714-4b23-8b13-b16b3c05c817-device-dir\") pod \"aws-ebs-csi-driver-node-zzqpt\" (UID: \"00689b5e-a714-4b23-8b13-b16b3c05c817\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzqpt"
Jul 22 01:47:29.594502 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.594103    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/00689b5e-a714-4b23-8b13-b16b3c05c817-etc-selinux\") pod \"aws-ebs-csi-driver-node-zzqpt\" (UID: \"00689b5e-a714-4b23-8b13-b16b3c05c817\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzqpt"
Jul 22 01:47:29.594502 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.594128    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-etc-openvswitch\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.594502 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.594152    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c4290331-2538-4205-b3d0-58ab1a1126b6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pbf4b\" (UID: \"c4290331-2538-4205-b3d0-58ab1a1126b6\") " pod="openshift-multus/multus-additional-cni-plugins-pbf4b"
Jul 22 01:47:29.594502 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.594170    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2553bf24-562e-4e3b-b5d3-d89687dd807f-multus-socket-dir-parent\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.595030 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.594193    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2553bf24-562e-4e3b-b5d3-d89687dd807f-host-run-multus-certs\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.595030 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.594215    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-host-cni-netd\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.595030 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.594240    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2553bf24-562e-4e3b-b5d3-d89687dd807f-os-release\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.595030 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.594266    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/00689b5e-a714-4b23-8b13-b16b3c05c817-sys-fs\") pod \"aws-ebs-csi-driver-node-zzqpt\" (UID: \"00689b5e-a714-4b23-8b13-b16b3c05c817\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzqpt"
Jul 22 01:47:29.595030 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.594313    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-host-run-netns\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.595030 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.594338    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-run-openvswitch\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.595030 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.594361    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-env-overrides\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.595030 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.594387    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c4290331-2538-4205-b3d0-58ab1a1126b6-cnibin\") pod \"multus-additional-cni-plugins-pbf4b\" (UID: \"c4290331-2538-4205-b3d0-58ab1a1126b6\") " pod="openshift-multus/multus-additional-cni-plugins-pbf4b"
Jul 22 01:47:29.595030 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.594412    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c4290331-2538-4205-b3d0-58ab1a1126b6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pbf4b\" (UID: \"c4290331-2538-4205-b3d0-58ab1a1126b6\") " pod="openshift-multus/multus-additional-cni-plugins-pbf4b"
Jul 22 01:47:29.595030 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.594457    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/00689b5e-a714-4b23-8b13-b16b3c05c817-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zzqpt\" (UID: \"00689b5e-a714-4b23-8b13-b16b3c05c817\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzqpt"
Jul 22 01:47:29.595030 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.594496    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g6gd\" (UniqueName: \"kubernetes.io/projected/c4290331-2538-4205-b3d0-58ab1a1126b6-kube-api-access-8g6gd\") pod \"multus-additional-cni-plugins-pbf4b\" (UID: \"c4290331-2538-4205-b3d0-58ab1a1126b6\") " pod="openshift-multus/multus-additional-cni-plugins-pbf4b"
Jul 22 01:47:29.595030 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.594515    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2553bf24-562e-4e3b-b5d3-d89687dd807f-system-cni-dir\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.595030 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.594535    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-host-slash\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.595030 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.594549    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-ovnkube-config\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.595030 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.594564    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c4290331-2538-4205-b3d0-58ab1a1126b6-os-release\") pod \"multus-additional-cni-plugins-pbf4b\" (UID: \"c4290331-2538-4205-b3d0-58ab1a1126b6\") " pod="openshift-multus/multus-additional-cni-plugins-pbf4b"
Jul 22 01:47:29.595030 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.594585    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2553bf24-562e-4e3b-b5d3-d89687dd807f-host-run-netns\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.595747 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.594620    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/91595658-5e3d-40eb-9763-67b1f193589d-run\") pod \"tuned-sq599\" (UID: \"91595658-5e3d-40eb-9763-67b1f193589d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq599"
Jul 22 01:47:29.595747 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.594643    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91595658-5e3d-40eb-9763-67b1f193589d-host\") pod \"tuned-sq599\" (UID: \"91595658-5e3d-40eb-9763-67b1f193589d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq599"
Jul 22 01:47:29.595747 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.594683    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2553bf24-562e-4e3b-b5d3-d89687dd807f-cni-binary-copy\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.595747 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.594700    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2553bf24-562e-4e3b-b5d3-d89687dd807f-multus-conf-dir\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.595747 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.594718    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-systemd-units\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.595747 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.594734    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c4290331-2538-4205-b3d0-58ab1a1126b6-cni-binary-copy\") pod \"multus-additional-cni-plugins-pbf4b\" (UID: \"c4290331-2538-4205-b3d0-58ab1a1126b6\") " pod="openshift-multus/multus-additional-cni-plugins-pbf4b"
Jul 22 01:47:29.595747 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.594750    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/91595658-5e3d-40eb-9763-67b1f193589d-etc-tuned\") pod \"tuned-sq599\" (UID: \"91595658-5e3d-40eb-9763-67b1f193589d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq599"
Jul 22 01:47:29.595747 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.594763    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-run-systemd\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.595747 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.594779    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-ovnkube-script-lib\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.595747 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.594795    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-log-socket\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.595747 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.594813    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/91595658-5e3d-40eb-9763-67b1f193589d-etc-sysctl-conf\") pod \"tuned-sq599\" (UID: \"91595658-5e3d-40eb-9763-67b1f193589d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq599"
Jul 22 01:47:29.595747 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.594841    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-ovn-node-metrics-cert\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.595747 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.594867    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2553bf24-562e-4e3b-b5d3-d89687dd807f-host-var-lib-cni-multus\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.595747 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.594890    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/91595658-5e3d-40eb-9763-67b1f193589d-etc-systemd\") pod \"tuned-sq599\" (UID: \"91595658-5e3d-40eb-9763-67b1f193589d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq599"
Jul 22 01:47:29.595747 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.595624    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pmrf8"
Jul 22 01:47:29.598342 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.598319    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mjkvg"
Jul 22 01:47:29.598452 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:29.598411    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mjkvg" podUID="2f67b074-15ce-4e39-afdc-5c30c8addc97"
Jul 22 01:47:29.600575 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.600546    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4xkbt"
Jul 22 01:47:29.602091 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.602065    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 22 01:47:29.602417 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.602392    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 22 01:47:29.603153 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.602787    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 22 01:47:29.603153 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.602930    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sfnlg\""
Jul 22 01:47:29.603288 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.603227    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-drsr9"
Jul 22 01:47:29.603661 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.603642    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 22 01:47:29.603972 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.603950    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 22 01:47:29.604064 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.604011    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wnsk7\""
Jul 22 01:47:29.604120 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.603962    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 22 01:47:29.606529 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.606507    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gg4rm\""
Jul 22 01:47:29.606612 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.606566    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 22 01:47:29.606775 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.606754    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 22 01:47:29.606922 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.606808    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 22 01:47:29.619472 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.619433    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xh2tr"
Jul 22 01:47:29.628175 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.628152    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xh2tr"
Jul 22 01:47:29.662603 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.662554    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-230.ec2.internal" event={"ID":"a67e292d73c90fc17a80248cb369ca7d","Type":"ContainerStarted","Data":"3d9c5cbba759c6e09acf3b7c1449063e5a2c5e8ba0b1a3e393b79f8e3b56f177"}
Jul 22 01:47:29.663789 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.663762    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-230.ec2.internal" event={"ID":"f96cb24b354bdc0f39f6acdd48c651dc","Type":"ContainerStarted","Data":"3b113b66604f2dc93861baa276f26ea8617ee0f34c47ee27cbc85e929c51a542"}
Jul 22 01:47:29.684654 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.684625    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 22 01:47:29.695644 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.695599    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/91595658-5e3d-40eb-9763-67b1f193589d-etc-systemd\") pod \"tuned-sq599\" (UID: \"91595658-5e3d-40eb-9763-67b1f193589d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq599"
Jul 22 01:47:29.695783 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.695646    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2553bf24-562e-4e3b-b5d3-d89687dd807f-hostroot\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.695783 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.695676    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/00689b5e-a714-4b23-8b13-b16b3c05c817-socket-dir\") pod \"aws-ebs-csi-driver-node-zzqpt\" (UID: \"00689b5e-a714-4b23-8b13-b16b3c05c817\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzqpt"
Jul 22 01:47:29.695783 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.695706    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-host-cni-bin\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.695783 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.695760    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2553bf24-562e-4e3b-b5d3-d89687dd807f-hostroot\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.695783 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.695779    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce13bc5e-08ec-4b2b-a9fb-63ac3c1a34f1-host-slash\") pod \"iptables-alerter-pmrf8\" (UID: \"ce13bc5e-08ec-4b2b-a9fb-63ac3c1a34f1\") " pod="openshift-network-operator/iptables-alerter-pmrf8"
Jul 22 01:47:29.696053 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.695790    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/91595658-5e3d-40eb-9763-67b1f193589d-etc-systemd\") pod \"tuned-sq599\" (UID: \"91595658-5e3d-40eb-9763-67b1f193589d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq599"
Jul 22 01:47:29.696053 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.695805    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8ctp\" (UniqueName: \"kubernetes.io/projected/91595658-5e3d-40eb-9763-67b1f193589d-kube-api-access-f8ctp\") pod \"tuned-sq599\" (UID: \"91595658-5e3d-40eb-9763-67b1f193589d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq599"
Jul 22 01:47:29.696053 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.695829    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-node-log\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.696053 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.695854    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4290331-2538-4205-b3d0-58ab1a1126b6-system-cni-dir\") pod \"multus-additional-cni-plugins-pbf4b\" (UID: \"c4290331-2538-4205-b3d0-58ab1a1126b6\") " pod="openshift-multus/multus-additional-cni-plugins-pbf4b"
Jul 22 01:47:29.696053 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.695876    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-host-cni-bin\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.696053 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.695879    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c4290331-2538-4205-b3d0-58ab1a1126b6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pbf4b\" (UID: \"c4290331-2538-4205-b3d0-58ab1a1126b6\") " pod="openshift-multus/multus-additional-cni-plugins-pbf4b"
Jul 22 01:47:29.696053 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.695912    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/00689b5e-a714-4b23-8b13-b16b3c05c817-socket-dir\") pod \"aws-ebs-csi-driver-node-zzqpt\" (UID: \"00689b5e-a714-4b23-8b13-b16b3c05c817\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzqpt"
Jul 22 01:47:29.696053 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.695921    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/91595658-5e3d-40eb-9763-67b1f193589d-etc-sysconfig\") pod \"tuned-sq599\" (UID: \"91595658-5e3d-40eb-9763-67b1f193589d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq599"
Jul 22 01:47:29.696053 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.695951    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2553bf24-562e-4e3b-b5d3-d89687dd807f-multus-cni-dir\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.696053 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.695976    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2553bf24-562e-4e3b-b5d3-d89687dd807f-etc-kubernetes\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.696053 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.696001    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4smjw\" (UniqueName: \"kubernetes.io/projected/2553bf24-562e-4e3b-b5d3-d89687dd807f-kube-api-access-4smjw\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.696053 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.696027    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62fx2\" (UniqueName: \"kubernetes.io/projected/00689b5e-a714-4b23-8b13-b16b3c05c817-kube-api-access-62fx2\") pod \"aws-ebs-csi-driver-node-zzqpt\" (UID: \"00689b5e-a714-4b23-8b13-b16b3c05c817\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzqpt"
Jul 22 01:47:29.696053 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.696052    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-run-ovn\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.696744 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.696078    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-var-lib-openvswitch\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.696744 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.696120    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2393693b-9aef-40c6-ab3f-df34a7500ed8-serviceca\") pod \"node-ca-drsr9\" (UID: \"2393693b-9aef-40c6-ab3f-df34a7500ed8\") " pod="openshift-image-registry/node-ca-drsr9"
Jul 22 01:47:29.696744 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.696147    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2553bf24-562e-4e3b-b5d3-d89687dd807f-host-run-k8s-cni-cncf-io\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.696744 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.696175    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91595658-5e3d-40eb-9763-67b1f193589d-lib-modules\") pod \"tuned-sq599\" (UID: \"91595658-5e3d-40eb-9763-67b1f193589d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq599"
Jul 22 01:47:29.696744 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.696199    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2553bf24-562e-4e3b-b5d3-d89687dd807f-cnibin\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.696744 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.696231    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt77g\" (UniqueName: \"kubernetes.io/projected/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-kube-api-access-tt77g\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.696744 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.696260    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08637867-465c-4df3-b707-ef1e33b64ed2-tmp-dir\") pod \"node-resolver-4xkbt\" (UID: \"08637867-465c-4df3-b707-ef1e33b64ed2\") " pod="openshift-dns/node-resolver-4xkbt"
Jul 22 01:47:29.696744 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.696288    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7gqd\" (UniqueName: \"kubernetes.io/projected/e6cd3d7c-b979-4d69-aad7-68ab39aed83d-kube-api-access-z7gqd\") pod \"network-check-target-fltbj\" (UID: \"e6cd3d7c-b979-4d69-aad7-68ab39aed83d\") " pod="openshift-network-diagnostics/network-check-target-fltbj"
Jul 22 01:47:29.696744 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.696297    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4290331-2538-4205-b3d0-58ab1a1126b6-system-cni-dir\") pod \"multus-additional-cni-plugins-pbf4b\" (UID: \"c4290331-2538-4205-b3d0-58ab1a1126b6\") " pod="openshift-multus/multus-additional-cni-plugins-pbf4b"
Jul 22 01:47:29.696744 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.696314    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2553bf24-562e-4e3b-b5d3-d89687dd807f-host-var-lib-kubelet\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.696744 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.696342    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/91595658-5e3d-40eb-9763-67b1f193589d-etc-modprobe-d\") pod \"tuned-sq599\" (UID: \"91595658-5e3d-40eb-9763-67b1f193589d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq599"
Jul 22 01:47:29.696744 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.696526    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c4290331-2538-4205-b3d0-58ab1a1126b6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pbf4b\" (UID: \"c4290331-2538-4205-b3d0-58ab1a1126b6\") " pod="openshift-multus/multus-additional-cni-plugins-pbf4b"
Jul 22 01:47:29.696744 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.696594    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/91595658-5e3d-40eb-9763-67b1f193589d-etc-modprobe-d\") pod \"tuned-sq599\" (UID: \"91595658-5e3d-40eb-9763-67b1f193589d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq599"
Jul 22 01:47:29.696744 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.696600    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2553bf24-562e-4e3b-b5d3-d89687dd807f-multus-cni-dir\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.696744 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.696632    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91595658-5e3d-40eb-9763-67b1f193589d-etc-kubernetes\") pod \"tuned-sq599\" (UID: \"91595658-5e3d-40eb-9763-67b1f193589d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq599"
Jul 22 01:47:29.696744 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.696668    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/91595658-5e3d-40eb-9763-67b1f193589d-etc-sysctl-d\") pod \"tuned-sq599\" (UID: \"91595658-5e3d-40eb-9763-67b1f193589d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq599"
Jul 22 01:47:29.696744 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.696692    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-var-lib-openvswitch\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.697557 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.696695    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91595658-5e3d-40eb-9763-67b1f193589d-lib-modules\") pod \"tuned-sq599\" (UID: \"91595658-5e3d-40eb-9763-67b1f193589d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq599"
Jul 22 01:47:29.697557 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.696700    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91595658-5e3d-40eb-9763-67b1f193589d-sys\") pod \"tuned-sq599\" (UID: \"91595658-5e3d-40eb-9763-67b1f193589d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq599"
Jul 22 01:47:29.697557 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.696730    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2553bf24-562e-4e3b-b5d3-d89687dd807f-host-run-k8s-cni-cncf-io\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.697557 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.696777    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2553bf24-562e-4e3b-b5d3-d89687dd807f-etc-kubernetes\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.697557 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.696787    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91595658-5e3d-40eb-9763-67b1f193589d-etc-kubernetes\") pod \"tuned-sq599\" (UID: \"91595658-5e3d-40eb-9763-67b1f193589d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq599"
Jul 22 01:47:29.697557 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.696805    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2553bf24-562e-4e3b-b5d3-d89687dd807f-cnibin\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.697557 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.696835    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91595658-5e3d-40eb-9763-67b1f193589d-sys\") pod \"tuned-sq599\" (UID: \"91595658-5e3d-40eb-9763-67b1f193589d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq599"
Jul 22 01:47:29.697557 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.696849    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-run-ovn\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.697557 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.696880    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2553bf24-562e-4e3b-b5d3-d89687dd807f-host-var-lib-kubelet\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.697557 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.696895    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/91595658-5e3d-40eb-9763-67b1f193589d-etc-sysctl-d\") pod \"tuned-sq599\" (UID: \"91595658-5e3d-40eb-9763-67b1f193589d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq599"
Jul 22 01:47:29.697557 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.696910    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/91595658-5e3d-40eb-9763-67b1f193589d-etc-sysconfig\") pod \"tuned-sq599\" (UID: \"91595658-5e3d-40eb-9763-67b1f193589d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq599"
Jul 22 01:47:29.697557 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.696943    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91595658-5e3d-40eb-9763-67b1f193589d-var-lib-kubelet\") pod \"tuned-sq599\" (UID: \"91595658-5e3d-40eb-9763-67b1f193589d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq599"
Jul 22 01:47:29.697557 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.696954    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-node-log\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.697557 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.696999    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/00689b5e-a714-4b23-8b13-b16b3c05c817-registration-dir\") pod \"aws-ebs-csi-driver-node-zzqpt\" (UID: \"00689b5e-a714-4b23-8b13-b16b3c05c817\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzqpt"
Jul 22 01:47:29.697557 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.697060    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91595658-5e3d-40eb-9763-67b1f193589d-tmp\") pod \"tuned-sq599\" (UID: \"91595658-5e3d-40eb-9763-67b1f193589d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq599"
Jul 22 01:47:29.697557 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.697063    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91595658-5e3d-40eb-9763-67b1f193589d-var-lib-kubelet\") pod \"tuned-sq599\" (UID: \"91595658-5e3d-40eb-9763-67b1f193589d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq599"
Jul 22 01:47:29.697557 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.697136    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2553bf24-562e-4e3b-b5d3-d89687dd807f-multus-daemon-config\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.697557 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.697138    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/00689b5e-a714-4b23-8b13-b16b3c05c817-registration-dir\") pod \"aws-ebs-csi-driver-node-zzqpt\" (UID: \"00689b5e-a714-4b23-8b13-b16b3c05c817\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzqpt"
Jul 22 01:47:29.698218 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.697208    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-host-kubelet\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.698218 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.697234    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-host-run-ovn-kubernetes\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.698218 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.697273    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.698218 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.697304    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2553bf24-562e-4e3b-b5d3-d89687dd807f-host-var-lib-cni-bin\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.698218 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.697328    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/00689b5e-a714-4b23-8b13-b16b3c05c817-device-dir\") pod \"aws-ebs-csi-driver-node-zzqpt\" (UID: \"00689b5e-a714-4b23-8b13-b16b3c05c817\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzqpt"
Jul 22 01:47:29.698218 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.697354    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/00689b5e-a714-4b23-8b13-b16b3c05c817-etc-selinux\") pod \"aws-ebs-csi-driver-node-zzqpt\" (UID: \"00689b5e-a714-4b23-8b13-b16b3c05c817\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzqpt"
Jul 22 01:47:29.698218 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.697361    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 22 01:47:29.698218 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.697441    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-etc-openvswitch\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.698218 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.697471    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c4290331-2538-4205-b3d0-58ab1a1126b6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pbf4b\" (UID: \"c4290331-2538-4205-b3d0-58ab1a1126b6\") " pod="openshift-multus/multus-additional-cni-plugins-pbf4b"
Jul 22 01:47:29.698218 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.697502    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/04c89d69-a674-49e4-9630-9302cec0fad4-konnectivity-ca\") pod \"konnectivity-agent-rd82t\" (UID: \"04c89d69-a674-49e4-9630-9302cec0fad4\") " pod="kube-system/konnectivity-agent-rd82t"
Jul 22 01:47:29.698218 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.697550    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2553bf24-562e-4e3b-b5d3-d89687dd807f-multus-socket-dir-parent\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.698218 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.697576    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2553bf24-562e-4e3b-b5d3-d89687dd807f-host-run-multus-certs\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.698218 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.697603    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-host-cni-netd\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.698218 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.697629    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk66j\" (UniqueName: \"kubernetes.io/projected/22905b02-8ed1-4c26-92ae-7282baf885cd-kube-api-access-wk66j\") pod \"network-metrics-daemon-t7sb9\" (UID: \"22905b02-8ed1-4c26-92ae-7282baf885cd\") " pod="openshift-multus/network-metrics-daemon-t7sb9"
Jul 22 01:47:29.698218 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.697657    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9n9t\" (UniqueName: \"kubernetes.io/projected/2393693b-9aef-40c6-ab3f-df34a7500ed8-kube-api-access-j9n9t\") pod \"node-ca-drsr9\" (UID: \"2393693b-9aef-40c6-ab3f-df34a7500ed8\") " pod="openshift-image-registry/node-ca-drsr9"
Jul 22 01:47:29.698218 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.697748    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-etc-openvswitch\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.698218 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.697811    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/00689b5e-a714-4b23-8b13-b16b3c05c817-device-dir\") pod \"aws-ebs-csi-driver-node-zzqpt\" (UID: \"00689b5e-a714-4b23-8b13-b16b3c05c817\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzqpt"
Jul 22 01:47:29.698830 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.697840    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/00689b5e-a714-4b23-8b13-b16b3c05c817-etc-selinux\") pod \"aws-ebs-csi-driver-node-zzqpt\" (UID: \"00689b5e-a714-4b23-8b13-b16b3c05c817\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzqpt"
Jul 22 01:47:29.698830 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.697857    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-host-run-ovn-kubernetes\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.698830 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.698357    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c4290331-2538-4205-b3d0-58ab1a1126b6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pbf4b\" (UID: \"c4290331-2538-4205-b3d0-58ab1a1126b6\") " pod="openshift-multus/multus-additional-cni-plugins-pbf4b"
Jul 22 01:47:29.698830 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.698424    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2553bf24-562e-4e3b-b5d3-d89687dd807f-multus-socket-dir-parent\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.698830 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.698433    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2553bf24-562e-4e3b-b5d3-d89687dd807f-multus-daemon-config\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.698830 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.698491    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2553bf24-562e-4e3b-b5d3-d89687dd807f-host-run-multus-certs\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.698830 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.698500    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-host-kubelet\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.698830 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.698530    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-host-cni-netd\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.698830 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.698539    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.698830 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.698572    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2553bf24-562e-4e3b-b5d3-d89687dd807f-host-var-lib-cni-bin\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.698830 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.698575    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ce13bc5e-08ec-4b2b-a9fb-63ac3c1a34f1-iptables-alerter-script\") pod \"iptables-alerter-pmrf8\" (UID: \"ce13bc5e-08ec-4b2b-a9fb-63ac3c1a34f1\") " pod="openshift-network-operator/iptables-alerter-pmrf8"
Jul 22 01:47:29.698830 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.698613    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2553bf24-562e-4e3b-b5d3-d89687dd807f-os-release\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.698830 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.698647    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/00689b5e-a714-4b23-8b13-b16b3c05c817-sys-fs\") pod \"aws-ebs-csi-driver-node-zzqpt\" (UID: \"00689b5e-a714-4b23-8b13-b16b3c05c817\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzqpt"
Jul 22 01:47:29.698830 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.698715    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2553bf24-562e-4e3b-b5d3-d89687dd807f-os-release\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.698830 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.698758    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-host-run-netns\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.698830 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.698780    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/00689b5e-a714-4b23-8b13-b16b3c05c817-sys-fs\") pod \"aws-ebs-csi-driver-node-zzqpt\" (UID: \"00689b5e-a714-4b23-8b13-b16b3c05c817\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzqpt"
Jul 22 01:47:29.698830 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.698812    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-run-openvswitch\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.699377 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.698839    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-env-overrides\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.699377 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.698884    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c4290331-2538-4205-b3d0-58ab1a1126b6-cnibin\") pod \"multus-additional-cni-plugins-pbf4b\" (UID: \"c4290331-2538-4205-b3d0-58ab1a1126b6\") " pod="openshift-multus/multus-additional-cni-plugins-pbf4b"
Jul 22 01:47:29.699377 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.698910    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-run-openvswitch\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.699377 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.698911    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c4290331-2538-4205-b3d0-58ab1a1126b6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pbf4b\" (UID: \"c4290331-2538-4205-b3d0-58ab1a1126b6\") " pod="openshift-multus/multus-additional-cni-plugins-pbf4b"
Jul 22 01:47:29.699377 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.698963    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c4290331-2538-4205-b3d0-58ab1a1126b6-cnibin\") pod \"multus-additional-cni-plugins-pbf4b\" (UID: \"c4290331-2538-4205-b3d0-58ab1a1126b6\") " pod="openshift-multus/multus-additional-cni-plugins-pbf4b"
Jul 22 01:47:29.699377 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.699003    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-host-run-netns\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.699377 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.699032    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/00689b5e-a714-4b23-8b13-b16b3c05c817-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zzqpt\" (UID: \"00689b5e-a714-4b23-8b13-b16b3c05c817\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzqpt"
Jul 22 01:47:29.699377 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.699060    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c4290331-2538-4205-b3d0-58ab1a1126b6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pbf4b\" (UID: \"c4290331-2538-4205-b3d0-58ab1a1126b6\") " pod="openshift-multus/multus-additional-cni-plugins-pbf4b"
Jul 22 01:47:29.699377 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.699071    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g6gd\" (UniqueName: \"kubernetes.io/projected/c4290331-2538-4205-b3d0-58ab1a1126b6-kube-api-access-8g6gd\") pod \"multus-additional-cni-plugins-pbf4b\" (UID: \"c4290331-2538-4205-b3d0-58ab1a1126b6\") " pod="openshift-multus/multus-additional-cni-plugins-pbf4b"
Jul 22 01:47:29.699377 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.699342    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/00689b5e-a714-4b23-8b13-b16b3c05c817-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zzqpt\" (UID: \"00689b5e-a714-4b23-8b13-b16b3c05c817\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzqpt"
Jul 22 01:47:29.699723 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.699407    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6whb\" (UniqueName: \"kubernetes.io/projected/08637867-465c-4df3-b707-ef1e33b64ed2-kube-api-access-z6whb\") pod \"node-resolver-4xkbt\" (UID: \"08637867-465c-4df3-b707-ef1e33b64ed2\") " pod="openshift-dns/node-resolver-4xkbt"
Jul 22 01:47:29.699723 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.699461    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2553bf24-562e-4e3b-b5d3-d89687dd807f-system-cni-dir\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.699723 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.699488    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-host-slash\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.699723 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.699531    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-ovnkube-config\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.699723 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.699560    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-env-overrides\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.699723 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.699560    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c4290331-2538-4205-b3d0-58ab1a1126b6-os-release\") pod \"multus-additional-cni-plugins-pbf4b\" (UID: \"c4290331-2538-4205-b3d0-58ab1a1126b6\") " pod="openshift-multus/multus-additional-cni-plugins-pbf4b"
Jul 22 01:47:29.699723 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.699602    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/08637867-465c-4df3-b707-ef1e33b64ed2-hosts-file\") pod \"node-resolver-4xkbt\" (UID: \"08637867-465c-4df3-b707-ef1e33b64ed2\") " pod="openshift-dns/node-resolver-4xkbt"
Jul 22 01:47:29.699723 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.699623    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c4290331-2538-4205-b3d0-58ab1a1126b6-os-release\") pod \"multus-additional-cni-plugins-pbf4b\" (UID: \"c4290331-2538-4205-b3d0-58ab1a1126b6\") " pod="openshift-multus/multus-additional-cni-plugins-pbf4b"
Jul 22 01:47:29.699723 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.699638    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22905b02-8ed1-4c26-92ae-7282baf885cd-metrics-certs\") pod \"network-metrics-daemon-t7sb9\" (UID: \"22905b02-8ed1-4c26-92ae-7282baf885cd\") " pod="openshift-multus/network-metrics-daemon-t7sb9"
Jul 22 01:47:29.699723 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.699666    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-host-slash\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.699723 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.699678    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2553bf24-562e-4e3b-b5d3-d89687dd807f-host-run-netns\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.699723 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.699711    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2553bf24-562e-4e3b-b5d3-d89687dd807f-system-cni-dir\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.700177 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.699749    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/91595658-5e3d-40eb-9763-67b1f193589d-run\") pod \"tuned-sq599\" (UID: \"91595658-5e3d-40eb-9763-67b1f193589d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq599"
Jul 22 01:47:29.700177 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.699775    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91595658-5e3d-40eb-9763-67b1f193589d-host\") pod \"tuned-sq599\" (UID: \"91595658-5e3d-40eb-9763-67b1f193589d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq599"
Jul 22 01:47:29.700177 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.699803    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2553bf24-562e-4e3b-b5d3-d89687dd807f-cni-binary-copy\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.700177 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.699841    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2553bf24-562e-4e3b-b5d3-d89687dd807f-multus-conf-dir\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.700177 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.699864    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2553bf24-562e-4e3b-b5d3-d89687dd807f-host-run-netns\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.700177 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.699867    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-systemd-units\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.700177 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.699904    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-systemd-units\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.700177 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.699942    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c4290331-2538-4205-b3d0-58ab1a1126b6-cni-binary-copy\") pod \"multus-additional-cni-plugins-pbf4b\" (UID: \"c4290331-2538-4205-b3d0-58ab1a1126b6\") " pod="openshift-multus/multus-additional-cni-plugins-pbf4b"
Jul 22 01:47:29.700177 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.699983    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2f67b074-15ce-4e39-afdc-5c30c8addc97-dbus\") pod \"global-pull-secret-syncer-mjkvg\" (UID: \"2f67b074-15ce-4e39-afdc-5c30c8addc97\") " pod="kube-system/global-pull-secret-syncer-mjkvg"
Jul 22 01:47:29.700177 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.700009    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91595658-5e3d-40eb-9763-67b1f193589d-host\") pod \"tuned-sq599\" (UID: \"91595658-5e3d-40eb-9763-67b1f193589d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq599"
Jul 22 01:47:29.700177 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.700013    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/91595658-5e3d-40eb-9763-67b1f193589d-etc-tuned\") pod \"tuned-sq599\" (UID: \"91595658-5e3d-40eb-9763-67b1f193589d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq599"
Jul 22 01:47:29.700177 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.700053    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-run-systemd\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.700766 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.700344    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-run-systemd\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.700766 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.700390    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-ovnkube-script-lib\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.700766 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.700421    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2f67b074-15ce-4e39-afdc-5c30c8addc97-kubelet-config\") pod \"global-pull-secret-syncer-mjkvg\" (UID: \"2f67b074-15ce-4e39-afdc-5c30c8addc97\") " pod="kube-system/global-pull-secret-syncer-mjkvg"
Jul 22 01:47:29.700766 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.700473    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m77n\" (UniqueName: \"kubernetes.io/projected/ce13bc5e-08ec-4b2b-a9fb-63ac3c1a34f1-kube-api-access-6m77n\") pod \"iptables-alerter-pmrf8\" (UID: \"ce13bc5e-08ec-4b2b-a9fb-63ac3c1a34f1\") " pod="openshift-network-operator/iptables-alerter-pmrf8"
Jul 22 01:47:29.700766 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.700503    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-log-socket\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.700766 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.700532    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-ovnkube-config\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.700766 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.700579    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/04c89d69-a674-49e4-9630-9302cec0fad4-agent-certs\") pod \"konnectivity-agent-rd82t\" (UID: \"04c89d69-a674-49e4-9630-9302cec0fad4\") " pod="kube-system/konnectivity-agent-rd82t"
Jul 22 01:47:29.700766 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.700611    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/91595658-5e3d-40eb-9763-67b1f193589d-etc-sysctl-conf\") pod \"tuned-sq599\" (UID: \"91595658-5e3d-40eb-9763-67b1f193589d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq599"
Jul 22 01:47:29.700766 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.700640    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-ovn-node-metrics-cert\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.700766 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.700666    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2f67b074-15ce-4e39-afdc-5c30c8addc97-original-pull-secret\") pod \"global-pull-secret-syncer-mjkvg\" (UID: \"2f67b074-15ce-4e39-afdc-5c30c8addc97\") " pod="kube-system/global-pull-secret-syncer-mjkvg"
Jul 22 01:47:29.700766 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.700693    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2393693b-9aef-40c6-ab3f-df34a7500ed8-host\") pod \"node-ca-drsr9\" (UID: \"2393693b-9aef-40c6-ab3f-df34a7500ed8\") " pod="openshift-image-registry/node-ca-drsr9"
Jul 22 01:47:29.700766 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.700769    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2553bf24-562e-4e3b-b5d3-d89687dd807f-host-var-lib-cni-multus\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.701395 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.700778    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c4290331-2538-4205-b3d0-58ab1a1126b6-cni-binary-copy\") pod \"multus-additional-cni-plugins-pbf4b\" (UID: \"c4290331-2538-4205-b3d0-58ab1a1126b6\") " pod="openshift-multus/multus-additional-cni-plugins-pbf4b"
Jul 22 01:47:29.701395 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.700838    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2553bf24-562e-4e3b-b5d3-d89687dd807f-multus-conf-dir\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.701395 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.700868    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2553bf24-562e-4e3b-b5d3-d89687dd807f-host-var-lib-cni-multus\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.701395 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.700910    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-log-socket\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.701395 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.700981    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/91595658-5e3d-40eb-9763-67b1f193589d-etc-sysctl-conf\") pod \"tuned-sq599\" (UID: \"91595658-5e3d-40eb-9763-67b1f193589d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq599"
Jul 22 01:47:29.701395 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.701015    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2553bf24-562e-4e3b-b5d3-d89687dd807f-cni-binary-copy\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.701395 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.701066    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/91595658-5e3d-40eb-9763-67b1f193589d-run\") pod \"tuned-sq599\" (UID: \"91595658-5e3d-40eb-9763-67b1f193589d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq599"
Jul 22 01:47:29.701834 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.701434    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-ovnkube-script-lib\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.703293 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.703243    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91595658-5e3d-40eb-9763-67b1f193589d-tmp\") pod \"tuned-sq599\" (UID: \"91595658-5e3d-40eb-9763-67b1f193589d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq599"
Jul 22 01:47:29.703438 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.703415    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/91595658-5e3d-40eb-9763-67b1f193589d-etc-tuned\") pod \"tuned-sq599\" (UID: \"91595658-5e3d-40eb-9763-67b1f193589d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq599"
Jul 22 01:47:29.703939 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.703917    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-ovn-node-metrics-cert\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.705625 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.705599    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8ctp\" (UniqueName: \"kubernetes.io/projected/91595658-5e3d-40eb-9763-67b1f193589d-kube-api-access-f8ctp\") pod \"tuned-sq599\" (UID: \"91595658-5e3d-40eb-9763-67b1f193589d\") " pod="openshift-cluster-node-tuning-operator/tuned-sq599"
Jul 22 01:47:29.707219 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.707173    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62fx2\" (UniqueName: \"kubernetes.io/projected/00689b5e-a714-4b23-8b13-b16b3c05c817-kube-api-access-62fx2\") pod \"aws-ebs-csi-driver-node-zzqpt\" (UID: \"00689b5e-a714-4b23-8b13-b16b3c05c817\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzqpt"
Jul 22 01:47:29.707343 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.707320    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4smjw\" (UniqueName: \"kubernetes.io/projected/2553bf24-562e-4e3b-b5d3-d89687dd807f-kube-api-access-4smjw\") pod \"multus-78zsq\" (UID: \"2553bf24-562e-4e3b-b5d3-d89687dd807f\") " pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.708033 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.708009    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt77g\" (UniqueName: \"kubernetes.io/projected/7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17-kube-api-access-tt77g\") pod \"ovnkube-node-bpxz8\" (UID: \"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.708791 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.708760    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g6gd\" (UniqueName: \"kubernetes.io/projected/c4290331-2538-4205-b3d0-58ab1a1126b6-kube-api-access-8g6gd\") pod \"multus-additional-cni-plugins-pbf4b\" (UID: \"c4290331-2538-4205-b3d0-58ab1a1126b6\") " pod="openshift-multus/multus-additional-cni-plugins-pbf4b"
Jul 22 01:47:29.801456 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.801422    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/08637867-465c-4df3-b707-ef1e33b64ed2-hosts-file\") pod \"node-resolver-4xkbt\" (UID: \"08637867-465c-4df3-b707-ef1e33b64ed2\") " pod="openshift-dns/node-resolver-4xkbt"
Jul 22 01:47:29.801456 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.801463    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22905b02-8ed1-4c26-92ae-7282baf885cd-metrics-certs\") pod \"network-metrics-daemon-t7sb9\" (UID: \"22905b02-8ed1-4c26-92ae-7282baf885cd\") " pod="openshift-multus/network-metrics-daemon-t7sb9"
Jul 22 01:47:29.801719 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.801502    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2f67b074-15ce-4e39-afdc-5c30c8addc97-dbus\") pod \"global-pull-secret-syncer-mjkvg\" (UID: \"2f67b074-15ce-4e39-afdc-5c30c8addc97\") " pod="kube-system/global-pull-secret-syncer-mjkvg"
Jul 22 01:47:29.801719 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.801557    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2f67b074-15ce-4e39-afdc-5c30c8addc97-kubelet-config\") pod \"global-pull-secret-syncer-mjkvg\" (UID: \"2f67b074-15ce-4e39-afdc-5c30c8addc97\") " pod="kube-system/global-pull-secret-syncer-mjkvg"
Jul 22 01:47:29.801719 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.801579    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/08637867-465c-4df3-b707-ef1e33b64ed2-hosts-file\") pod \"node-resolver-4xkbt\" (UID: \"08637867-465c-4df3-b707-ef1e33b64ed2\") " pod="openshift-dns/node-resolver-4xkbt"
Jul 22 01:47:29.801719 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.801603    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m77n\" (UniqueName: \"kubernetes.io/projected/ce13bc5e-08ec-4b2b-a9fb-63ac3c1a34f1-kube-api-access-6m77n\") pod \"iptables-alerter-pmrf8\" (UID: \"ce13bc5e-08ec-4b2b-a9fb-63ac3c1a34f1\") " pod="openshift-network-operator/iptables-alerter-pmrf8"
Jul 22 01:47:29.801719 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.801640    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/04c89d69-a674-49e4-9630-9302cec0fad4-agent-certs\") pod \"konnectivity-agent-rd82t\" (UID: \"04c89d69-a674-49e4-9630-9302cec0fad4\") " pod="kube-system/konnectivity-agent-rd82t"
Jul 22 01:47:29.801719 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.801670    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2f67b074-15ce-4e39-afdc-5c30c8addc97-original-pull-secret\") pod \"global-pull-secret-syncer-mjkvg\" (UID: \"2f67b074-15ce-4e39-afdc-5c30c8addc97\") " pod="kube-system/global-pull-secret-syncer-mjkvg"
Jul 22 01:47:29.801719 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:29.801678    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 01:47:29.801719 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.801680    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2f67b074-15ce-4e39-afdc-5c30c8addc97-kubelet-config\") pod \"global-pull-secret-syncer-mjkvg\" (UID: \"2f67b074-15ce-4e39-afdc-5c30c8addc97\") " pod="kube-system/global-pull-secret-syncer-mjkvg"
Jul 22 01:47:29.801719 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.801690    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2393693b-9aef-40c6-ab3f-df34a7500ed8-host\") pod \"node-ca-drsr9\" (UID: \"2393693b-9aef-40c6-ab3f-df34a7500ed8\") " pod="openshift-image-registry/node-ca-drsr9"
Jul 22 01:47:29.802176 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.801730    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2393693b-9aef-40c6-ab3f-df34a7500ed8-host\") pod \"node-ca-drsr9\" (UID: \"2393693b-9aef-40c6-ab3f-df34a7500ed8\") " pod="openshift-image-registry/node-ca-drsr9"
Jul 22 01:47:29.802176 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.801586    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2f67b074-15ce-4e39-afdc-5c30c8addc97-dbus\") pod \"global-pull-secret-syncer-mjkvg\" (UID: \"2f67b074-15ce-4e39-afdc-5c30c8addc97\") " pod="kube-system/global-pull-secret-syncer-mjkvg"
Jul 22 01:47:29.802176 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:29.801752    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22905b02-8ed1-4c26-92ae-7282baf885cd-metrics-certs podName:22905b02-8ed1-4c26-92ae-7282baf885cd nodeName:}" failed. No retries permitted until 2026-07-22 01:47:30.301727887 +0000 UTC m=+3.145557019 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/22905b02-8ed1-4c26-92ae-7282baf885cd-metrics-certs") pod "network-metrics-daemon-t7sb9" (UID: "22905b02-8ed1-4c26-92ae-7282baf885cd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 01:47:29.802176 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.801776    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce13bc5e-08ec-4b2b-a9fb-63ac3c1a34f1-host-slash\") pod \"iptables-alerter-pmrf8\" (UID: \"ce13bc5e-08ec-4b2b-a9fb-63ac3c1a34f1\") " pod="openshift-network-operator/iptables-alerter-pmrf8"
Jul 22 01:47:29.802176 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:29.801812    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 22 01:47:29.802176 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.801819    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2393693b-9aef-40c6-ab3f-df34a7500ed8-serviceca\") pod \"node-ca-drsr9\" (UID: \"2393693b-9aef-40c6-ab3f-df34a7500ed8\") " pod="openshift-image-registry/node-ca-drsr9"
Jul 22 01:47:29.802176 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.801854    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08637867-465c-4df3-b707-ef1e33b64ed2-tmp-dir\") pod \"node-resolver-4xkbt\" (UID: \"08637867-465c-4df3-b707-ef1e33b64ed2\") " pod="openshift-dns/node-resolver-4xkbt"
Jul 22 01:47:29.802176 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:29.801883    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f67b074-15ce-4e39-afdc-5c30c8addc97-original-pull-secret podName:2f67b074-15ce-4e39-afdc-5c30c8addc97 nodeName:}" failed. No retries permitted until 2026-07-22 01:47:30.301862358 +0000 UTC m=+3.145691493 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2f67b074-15ce-4e39-afdc-5c30c8addc97-original-pull-secret") pod "global-pull-secret-syncer-mjkvg" (UID: "2f67b074-15ce-4e39-afdc-5c30c8addc97") : object "kube-system"/"original-pull-secret" not registered
Jul 22 01:47:29.802176 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.801880    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ce13bc5e-08ec-4b2b-a9fb-63ac3c1a34f1-host-slash\") pod \"iptables-alerter-pmrf8\" (UID: \"ce13bc5e-08ec-4b2b-a9fb-63ac3c1a34f1\") " pod="openshift-network-operator/iptables-alerter-pmrf8"
Jul 22 01:47:29.802176 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.801926    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7gqd\" (UniqueName: \"kubernetes.io/projected/e6cd3d7c-b979-4d69-aad7-68ab39aed83d-kube-api-access-z7gqd\") pod \"network-check-target-fltbj\" (UID: \"e6cd3d7c-b979-4d69-aad7-68ab39aed83d\") " pod="openshift-network-diagnostics/network-check-target-fltbj"
Jul 22 01:47:29.802176 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.801979    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/04c89d69-a674-49e4-9630-9302cec0fad4-konnectivity-ca\") pod \"konnectivity-agent-rd82t\" (UID: \"04c89d69-a674-49e4-9630-9302cec0fad4\") " pod="kube-system/konnectivity-agent-rd82t"
Jul 22 01:47:29.802176 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.802168    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk66j\" (UniqueName: \"kubernetes.io/projected/22905b02-8ed1-4c26-92ae-7282baf885cd-kube-api-access-wk66j\") pod \"network-metrics-daemon-t7sb9\" (UID: \"22905b02-8ed1-4c26-92ae-7282baf885cd\") " pod="openshift-multus/network-metrics-daemon-t7sb9"
Jul 22 01:47:29.802758 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.802201    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9n9t\" (UniqueName: \"kubernetes.io/projected/2393693b-9aef-40c6-ab3f-df34a7500ed8-kube-api-access-j9n9t\") pod \"node-ca-drsr9\" (UID: \"2393693b-9aef-40c6-ab3f-df34a7500ed8\") " pod="openshift-image-registry/node-ca-drsr9"
Jul 22 01:47:29.802758 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.802226    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08637867-465c-4df3-b707-ef1e33b64ed2-tmp-dir\") pod \"node-resolver-4xkbt\" (UID: \"08637867-465c-4df3-b707-ef1e33b64ed2\") " pod="openshift-dns/node-resolver-4xkbt"
Jul 22 01:47:29.802758 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.802237    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ce13bc5e-08ec-4b2b-a9fb-63ac3c1a34f1-iptables-alerter-script\") pod \"iptables-alerter-pmrf8\" (UID: \"ce13bc5e-08ec-4b2b-a9fb-63ac3c1a34f1\") " pod="openshift-network-operator/iptables-alerter-pmrf8"
Jul 22 01:47:29.802758 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.802287    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6whb\" (UniqueName: \"kubernetes.io/projected/08637867-465c-4df3-b707-ef1e33b64ed2-kube-api-access-z6whb\") pod \"node-resolver-4xkbt\" (UID: \"08637867-465c-4df3-b707-ef1e33b64ed2\") " pod="openshift-dns/node-resolver-4xkbt"
Jul 22 01:47:29.802758 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.802404    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2393693b-9aef-40c6-ab3f-df34a7500ed8-serviceca\") pod \"node-ca-drsr9\" (UID: \"2393693b-9aef-40c6-ab3f-df34a7500ed8\") " pod="openshift-image-registry/node-ca-drsr9"
Jul 22 01:47:29.802758 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.802576    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/04c89d69-a674-49e4-9630-9302cec0fad4-konnectivity-ca\") pod \"konnectivity-agent-rd82t\" (UID: \"04c89d69-a674-49e4-9630-9302cec0fad4\") " pod="kube-system/konnectivity-agent-rd82t"
Jul 22 01:47:29.802956 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.802820    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ce13bc5e-08ec-4b2b-a9fb-63ac3c1a34f1-iptables-alerter-script\") pod \"iptables-alerter-pmrf8\" (UID: \"ce13bc5e-08ec-4b2b-a9fb-63ac3c1a34f1\") " pod="openshift-network-operator/iptables-alerter-pmrf8"
Jul 22 01:47:29.805038 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.805014    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/04c89d69-a674-49e4-9630-9302cec0fad4-agent-certs\") pod \"konnectivity-agent-rd82t\" (UID: \"04c89d69-a674-49e4-9630-9302cec0fad4\") " pod="kube-system/konnectivity-agent-rd82t"
Jul 22 01:47:29.811613 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:29.811533    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 22 01:47:29.811613 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:29.811561    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 22 01:47:29.811613 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:29.811577    2563 projected.go:196] Error preparing data for projected volume kube-api-access-z7gqd for pod openshift-network-diagnostics/network-check-target-fltbj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 01:47:29.811867 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:29.811647    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6cd3d7c-b979-4d69-aad7-68ab39aed83d-kube-api-access-z7gqd podName:e6cd3d7c-b979-4d69-aad7-68ab39aed83d nodeName:}" failed. No retries permitted until 2026-07-22 01:47:30.311627741 +0000 UTC m=+3.155456875 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-z7gqd" (UniqueName: "kubernetes.io/projected/e6cd3d7c-b979-4d69-aad7-68ab39aed83d-kube-api-access-z7gqd") pod "network-check-target-fltbj" (UID: "e6cd3d7c-b979-4d69-aad7-68ab39aed83d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 01:47:29.814052 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.814021    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9n9t\" (UniqueName: \"kubernetes.io/projected/2393693b-9aef-40c6-ab3f-df34a7500ed8-kube-api-access-j9n9t\") pod \"node-ca-drsr9\" (UID: \"2393693b-9aef-40c6-ab3f-df34a7500ed8\") " pod="openshift-image-registry/node-ca-drsr9"
Jul 22 01:47:29.814182 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.814150    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6whb\" (UniqueName: \"kubernetes.io/projected/08637867-465c-4df3-b707-ef1e33b64ed2-kube-api-access-z6whb\") pod \"node-resolver-4xkbt\" (UID: \"08637867-465c-4df3-b707-ef1e33b64ed2\") " pod="openshift-dns/node-resolver-4xkbt"
Jul 22 01:47:29.814646 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.814613    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m77n\" (UniqueName: \"kubernetes.io/projected/ce13bc5e-08ec-4b2b-a9fb-63ac3c1a34f1-kube-api-access-6m77n\") pod \"iptables-alerter-pmrf8\" (UID: \"ce13bc5e-08ec-4b2b-a9fb-63ac3c1a34f1\") " pod="openshift-network-operator/iptables-alerter-pmrf8"
Jul 22 01:47:29.814744 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.814654    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk66j\" (UniqueName: \"kubernetes.io/projected/22905b02-8ed1-4c26-92ae-7282baf885cd-kube-api-access-wk66j\") pod \"network-metrics-daemon-t7sb9\" (UID: \"22905b02-8ed1-4c26-92ae-7282baf885cd\") " pod="openshift-multus/network-metrics-daemon-t7sb9"
Jul 22 01:47:29.882252 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.882216    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 22 01:47:29.889352 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.889325    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sq599"
Jul 22 01:47:29.900185 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.900160    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-78zsq"
Jul 22 01:47:29.911673 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.911648    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzqpt"
Jul 22 01:47:29.918676 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.918654    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:29.928812 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.928785    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pbf4b"
Jul 22 01:47:29.935943 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.935912    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rd82t"
Jul 22 01:47:29.945993 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.945967    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pmrf8"
Jul 22 01:47:29.954230 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.954201    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4xkbt"
Jul 22 01:47:29.961996 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:29.961970    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-drsr9"
Jul 22 01:47:30.306587 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:30.306489    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22905b02-8ed1-4c26-92ae-7282baf885cd-metrics-certs\") pod \"network-metrics-daemon-t7sb9\" (UID: \"22905b02-8ed1-4c26-92ae-7282baf885cd\") " pod="openshift-multus/network-metrics-daemon-t7sb9"
Jul 22 01:47:30.306587 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:30.306549    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2f67b074-15ce-4e39-afdc-5c30c8addc97-original-pull-secret\") pod \"global-pull-secret-syncer-mjkvg\" (UID: \"2f67b074-15ce-4e39-afdc-5c30c8addc97\") " pod="kube-system/global-pull-secret-syncer-mjkvg"
Jul 22 01:47:30.306830 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:30.306669    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 22 01:47:30.306830 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:30.306680    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 01:47:30.306830 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:30.306736    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f67b074-15ce-4e39-afdc-5c30c8addc97-original-pull-secret podName:2f67b074-15ce-4e39-afdc-5c30c8addc97 nodeName:}" failed. No retries permitted until 2026-07-22 01:47:31.306716322 +0000 UTC m=+4.150545458 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2f67b074-15ce-4e39-afdc-5c30c8addc97-original-pull-secret") pod "global-pull-secret-syncer-mjkvg" (UID: "2f67b074-15ce-4e39-afdc-5c30c8addc97") : object "kube-system"/"original-pull-secret" not registered
Jul 22 01:47:30.306830 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:30.306758    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22905b02-8ed1-4c26-92ae-7282baf885cd-metrics-certs podName:22905b02-8ed1-4c26-92ae-7282baf885cd nodeName:}" failed. No retries permitted until 2026-07-22 01:47:31.306746148 +0000 UTC m=+4.150575261 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/22905b02-8ed1-4c26-92ae-7282baf885cd-metrics-certs") pod "network-metrics-daemon-t7sb9" (UID: "22905b02-8ed1-4c26-92ae-7282baf885cd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 01:47:30.407457 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:30.407378    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7gqd\" (UniqueName: \"kubernetes.io/projected/e6cd3d7c-b979-4d69-aad7-68ab39aed83d-kube-api-access-z7gqd\") pod \"network-check-target-fltbj\" (UID: \"e6cd3d7c-b979-4d69-aad7-68ab39aed83d\") " pod="openshift-network-diagnostics/network-check-target-fltbj"
Jul 22 01:47:30.407646 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:30.407523    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 22 01:47:30.407646 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:30.407547    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 22 01:47:30.407646 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:30.407558    2563 projected.go:196] Error preparing data for projected volume kube-api-access-z7gqd for pod openshift-network-diagnostics/network-check-target-fltbj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 01:47:30.407646 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:30.407620    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6cd3d7c-b979-4d69-aad7-68ab39aed83d-kube-api-access-z7gqd podName:e6cd3d7c-b979-4d69-aad7-68ab39aed83d nodeName:}" failed. No retries permitted until 2026-07-22 01:47:31.407605309 +0000 UTC m=+4.251434424 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-z7gqd" (UniqueName: "kubernetes.io/projected/e6cd3d7c-b979-4d69-aad7-68ab39aed83d-kube-api-access-z7gqd") pod "network-check-target-fltbj" (UID: "e6cd3d7c-b979-4d69-aad7-68ab39aed83d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 01:47:30.631741 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:30.631600    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 01:42:30 +0000 UTC" deadline="2028-02-26 21:04:36.049701617 +0000 UTC"
Jul 22 01:47:30.631741 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:30.631684    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14035h17m5.418021902s"
Jul 22 01:47:30.658203 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:30.658157    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fltbj"
Jul 22 01:47:30.658406 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:30.658290    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fltbj" podUID="e6cd3d7c-b979-4d69-aad7-68ab39aed83d"
Jul 22 01:47:30.816179 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:30.816129    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2553bf24_562e_4e3b_b5d3_d89687dd807f.slice/crio-58ebf72e5ad22bdc236f7438056e6716f689dbca8817ca2aa2c02265375d83f6 WatchSource:0}: Error finding container 58ebf72e5ad22bdc236f7438056e6716f689dbca8817ca2aa2c02265375d83f6: Status 404 returned error can't find the container with id 58ebf72e5ad22bdc236f7438056e6716f689dbca8817ca2aa2c02265375d83f6
Jul 22 01:47:30.820779 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:30.820753    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08637867_465c_4df3_b707_ef1e33b64ed2.slice/crio-6d4c9ab2c9b82abebaea2b9e9492e114bf49e7dbe25fd1b381df52a130528bb7 WatchSource:0}: Error finding container 6d4c9ab2c9b82abebaea2b9e9492e114bf49e7dbe25fd1b381df52a130528bb7: Status 404 returned error can't find the container with id 6d4c9ab2c9b82abebaea2b9e9492e114bf49e7dbe25fd1b381df52a130528bb7
Jul 22 01:47:30.822160 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:30.822134    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04c89d69_a674_49e4_9630_9302cec0fad4.slice/crio-b402de84d2bb12adf0a00b13c2c3617b709421c1801110342b561b2cc24b053e WatchSource:0}: Error finding container b402de84d2bb12adf0a00b13c2c3617b709421c1801110342b561b2cc24b053e: Status 404 returned error can't find the container with id b402de84d2bb12adf0a00b13c2c3617b709421c1801110342b561b2cc24b053e
Jul 22 01:47:30.845735 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:30.845701    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce13bc5e_08ec_4b2b_a9fb_63ac3c1a34f1.slice/crio-bbc264d3d4184e610ccfed28ab66418837f53ecd58b68ed606b2187d366c61eb WatchSource:0}: Error finding container bbc264d3d4184e610ccfed28ab66418837f53ecd58b68ed606b2187d366c61eb: Status 404 returned error can't find the container with id bbc264d3d4184e610ccfed28ab66418837f53ecd58b68ed606b2187d366c61eb
Jul 22 01:47:30.846413 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:30.846384    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00689b5e_a714_4b23_8b13_b16b3c05c817.slice/crio-c33e3fe9fe146a71644fff8d3db39777c2e1f96b13b4bf9f44e00a4fe0345e05 WatchSource:0}: Error finding container c33e3fe9fe146a71644fff8d3db39777c2e1f96b13b4bf9f44e00a4fe0345e05: Status 404 returned error can't find the container with id c33e3fe9fe146a71644fff8d3db39777c2e1f96b13b4bf9f44e00a4fe0345e05
Jul 22 01:47:30.847158 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:30.847134    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4290331_2538_4205_b3d0_58ab1a1126b6.slice/crio-c47f77734581222fdc050813a6818de36ee4d60098a0bcc11e728a0a239a1ea0 WatchSource:0}: Error finding container c47f77734581222fdc050813a6818de36ee4d60098a0bcc11e728a0a239a1ea0: Status 404 returned error can't find the container with id c47f77734581222fdc050813a6818de36ee4d60098a0bcc11e728a0a239a1ea0
Jul 22 01:47:30.848241 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:30.848190    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ec6fec7_2d6d_4522_ae0b_6a53e98aaa17.slice/crio-4cff8cfc9c601d1e77b43c7c1b318f2be8698381d8878943113cf2280a6aa2bf WatchSource:0}: Error finding container 4cff8cfc9c601d1e77b43c7c1b318f2be8698381d8878943113cf2280a6aa2bf: Status 404 returned error can't find the container with id 4cff8cfc9c601d1e77b43c7c1b318f2be8698381d8878943113cf2280a6aa2bf
Jul 22 01:47:30.850277 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:30.849948    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91595658_5e3d_40eb_9763_67b1f193589d.slice/crio-965540e81b580ea9b9154d52840ae0ae8bbb94f3aca9e9575a46373098a43e1e WatchSource:0}: Error finding container 965540e81b580ea9b9154d52840ae0ae8bbb94f3aca9e9575a46373098a43e1e: Status 404 returned error can't find the container with id 965540e81b580ea9b9154d52840ae0ae8bbb94f3aca9e9575a46373098a43e1e
Jul 22 01:47:30.851066 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:47:30.851046    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2393693b_9aef_40c6_ab3f_df34a7500ed8.slice/crio-97cd3e53a3ecab8d065b2f573f25f9bcdf26980c87a560379fbe6efa0aafd3b2 WatchSource:0}: Error finding container 97cd3e53a3ecab8d065b2f573f25f9bcdf26980c87a560379fbe6efa0aafd3b2: Status 404 returned error can't find the container with id 97cd3e53a3ecab8d065b2f573f25f9bcdf26980c87a560379fbe6efa0aafd3b2
Jul 22 01:47:31.316089 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:31.315747    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22905b02-8ed1-4c26-92ae-7282baf885cd-metrics-certs\") pod \"network-metrics-daemon-t7sb9\" (UID: \"22905b02-8ed1-4c26-92ae-7282baf885cd\") " pod="openshift-multus/network-metrics-daemon-t7sb9"
Jul 22 01:47:31.316089 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:31.316056    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2f67b074-15ce-4e39-afdc-5c30c8addc97-original-pull-secret\") pod \"global-pull-secret-syncer-mjkvg\" (UID: \"2f67b074-15ce-4e39-afdc-5c30c8addc97\") " pod="kube-system/global-pull-secret-syncer-mjkvg"
Jul 22 01:47:31.316322 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:31.316270    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 22 01:47:31.316385 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:31.316339    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f67b074-15ce-4e39-afdc-5c30c8addc97-original-pull-secret podName:2f67b074-15ce-4e39-afdc-5c30c8addc97 nodeName:}" failed. No retries permitted until 2026-07-22 01:47:33.316319484 +0000 UTC m=+6.160148603 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2f67b074-15ce-4e39-afdc-5c30c8addc97-original-pull-secret") pod "global-pull-secret-syncer-mjkvg" (UID: "2f67b074-15ce-4e39-afdc-5c30c8addc97") : object "kube-system"/"original-pull-secret" not registered
Jul 22 01:47:31.316922 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:31.316790    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 01:47:31.316922 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:31.316862    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22905b02-8ed1-4c26-92ae-7282baf885cd-metrics-certs podName:22905b02-8ed1-4c26-92ae-7282baf885cd nodeName:}" failed. No retries permitted until 2026-07-22 01:47:33.316840695 +0000 UTC m=+6.160669825 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/22905b02-8ed1-4c26-92ae-7282baf885cd-metrics-certs") pod "network-metrics-daemon-t7sb9" (UID: "22905b02-8ed1-4c26-92ae-7282baf885cd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 01:47:31.417289 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:31.417222    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7gqd\" (UniqueName: \"kubernetes.io/projected/e6cd3d7c-b979-4d69-aad7-68ab39aed83d-kube-api-access-z7gqd\") pod \"network-check-target-fltbj\" (UID: \"e6cd3d7c-b979-4d69-aad7-68ab39aed83d\") " pod="openshift-network-diagnostics/network-check-target-fltbj"
Jul 22 01:47:31.417526 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:31.417481    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 22 01:47:31.417526 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:31.417506    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 22 01:47:31.417526 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:31.417522    2563 projected.go:196] Error preparing data for projected volume kube-api-access-z7gqd for pod openshift-network-diagnostics/network-check-target-fltbj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 01:47:31.417804 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:31.417587    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6cd3d7c-b979-4d69-aad7-68ab39aed83d-kube-api-access-z7gqd podName:e6cd3d7c-b979-4d69-aad7-68ab39aed83d nodeName:}" failed. No retries permitted until 2026-07-22 01:47:33.417565275 +0000 UTC m=+6.261394389 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-z7gqd" (UniqueName: "kubernetes.io/projected/e6cd3d7c-b979-4d69-aad7-68ab39aed83d-kube-api-access-z7gqd") pod "network-check-target-fltbj" (UID: "e6cd3d7c-b979-4d69-aad7-68ab39aed83d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 01:47:31.632953 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:31.632858    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 01:42:30 +0000 UTC" deadline="2028-02-10 12:51:33.576344445 +0000 UTC"
Jul 22 01:47:31.632953 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:31.632901    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13643h4m1.943448808s"
Jul 22 01:47:31.661148 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:31.660377    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mjkvg"
Jul 22 01:47:31.661148 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:31.660535    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mjkvg" podUID="2f67b074-15ce-4e39-afdc-5c30c8addc97"
Jul 22 01:47:31.661148 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:31.660947    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t7sb9"
Jul 22 01:47:31.661148 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:31.661056    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t7sb9" podUID="22905b02-8ed1-4c26-92ae-7282baf885cd"
Jul 22 01:47:31.672314 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:31.672273    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pmrf8" event={"ID":"ce13bc5e-08ec-4b2b-a9fb-63ac3c1a34f1","Type":"ContainerStarted","Data":"bbc264d3d4184e610ccfed28ab66418837f53ecd58b68ed606b2187d366c61eb"}
Jul 22 01:47:31.675209 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:31.675130    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rd82t" event={"ID":"04c89d69-a674-49e4-9630-9302cec0fad4","Type":"ContainerStarted","Data":"b402de84d2bb12adf0a00b13c2c3617b709421c1801110342b561b2cc24b053e"}
Jul 22 01:47:31.679230 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:31.679197    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-78zsq" event={"ID":"2553bf24-562e-4e3b-b5d3-d89687dd807f","Type":"ContainerStarted","Data":"58ebf72e5ad22bdc236f7438056e6716f689dbca8817ca2aa2c02265375d83f6"}
Jul 22 01:47:31.686013 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:31.685875    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8" event={"ID":"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17","Type":"ContainerStarted","Data":"4cff8cfc9c601d1e77b43c7c1b318f2be8698381d8878943113cf2280a6aa2bf"}
Jul 22 01:47:31.692001 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:31.691918    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzqpt" event={"ID":"00689b5e-a714-4b23-8b13-b16b3c05c817","Type":"ContainerStarted","Data":"c33e3fe9fe146a71644fff8d3db39777c2e1f96b13b4bf9f44e00a4fe0345e05"}
Jul 22 01:47:31.698702 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:31.698449    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4xkbt" event={"ID":"08637867-465c-4df3-b707-ef1e33b64ed2","Type":"ContainerStarted","Data":"6d4c9ab2c9b82abebaea2b9e9492e114bf49e7dbe25fd1b381df52a130528bb7"}
Jul 22 01:47:31.721949 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:31.721879    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-230.ec2.internal" event={"ID":"a67e292d73c90fc17a80248cb369ca7d","Type":"ContainerStarted","Data":"0c1cd8aa2505960f149fc2385ffccf622d641938ca15b5f1939998aa9f07fe61"}
Jul 22 01:47:31.732067 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:31.732029    2563 generic.go:356] "Generic (PLEG): container finished" podID="f96cb24b354bdc0f39f6acdd48c651dc" containerID="9edf2fcf75de7fd25c73571177db44d9f72c25c854798e8608f6aa5fa73c2d91" exitCode=0
Jul 22 01:47:31.732302 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:31.732129    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-230.ec2.internal" event={"ID":"f96cb24b354bdc0f39f6acdd48c651dc","Type":"ContainerDied","Data":"9edf2fcf75de7fd25c73571177db44d9f72c25c854798e8608f6aa5fa73c2d91"}
Jul 22 01:47:31.756267 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:31.756226    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-drsr9" event={"ID":"2393693b-9aef-40c6-ab3f-df34a7500ed8","Type":"ContainerStarted","Data":"97cd3e53a3ecab8d065b2f573f25f9bcdf26980c87a560379fbe6efa0aafd3b2"}
Jul 22 01:47:31.761115 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:31.761055    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-230.ec2.internal" podStartSLOduration=2.761033009 podStartE2EDuration="2.761033009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:47:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:47:31.740490751 +0000 UTC m=+4.584319890" watchObservedRunningTime="2026-07-22 01:47:31.761033009 +0000 UTC m=+4.604862143"
Jul 22 01:47:31.763412 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:31.763383    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sq599" event={"ID":"91595658-5e3d-40eb-9763-67b1f193589d","Type":"ContainerStarted","Data":"965540e81b580ea9b9154d52840ae0ae8bbb94f3aca9e9575a46373098a43e1e"}
Jul 22 01:47:31.770325 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:31.768399    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pbf4b" event={"ID":"c4290331-2538-4205-b3d0-58ab1a1126b6","Type":"ContainerStarted","Data":"c47f77734581222fdc050813a6818de36ee4d60098a0bcc11e728a0a239a1ea0"}
Jul 22 01:47:32.658087 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:32.658055    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fltbj"
Jul 22 01:47:32.658704 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:32.658190    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fltbj" podUID="e6cd3d7c-b979-4d69-aad7-68ab39aed83d"
Jul 22 01:47:32.794250 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:32.794145    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-230.ec2.internal" event={"ID":"f96cb24b354bdc0f39f6acdd48c651dc","Type":"ContainerStarted","Data":"597e2296062590fb5e597979d82dcc6feb0ea52b6b364805f4131316b2fdf86b"}
Jul 22 01:47:33.332574 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:33.332538    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22905b02-8ed1-4c26-92ae-7282baf885cd-metrics-certs\") pod \"network-metrics-daemon-t7sb9\" (UID: \"22905b02-8ed1-4c26-92ae-7282baf885cd\") " pod="openshift-multus/network-metrics-daemon-t7sb9"
Jul 22 01:47:33.332752 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:33.332596    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2f67b074-15ce-4e39-afdc-5c30c8addc97-original-pull-secret\") pod \"global-pull-secret-syncer-mjkvg\" (UID: \"2f67b074-15ce-4e39-afdc-5c30c8addc97\") " pod="kube-system/global-pull-secret-syncer-mjkvg"
Jul 22 01:47:33.332752 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:33.332729    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 22 01:47:33.332874 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:33.332791    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f67b074-15ce-4e39-afdc-5c30c8addc97-original-pull-secret podName:2f67b074-15ce-4e39-afdc-5c30c8addc97 nodeName:}" failed. No retries permitted until 2026-07-22 01:47:37.332772608 +0000 UTC m=+10.176601722 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2f67b074-15ce-4e39-afdc-5c30c8addc97-original-pull-secret") pod "global-pull-secret-syncer-mjkvg" (UID: "2f67b074-15ce-4e39-afdc-5c30c8addc97") : object "kube-system"/"original-pull-secret" not registered
Jul 22 01:47:33.333235 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:33.333210    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 01:47:33.333357 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:33.333274    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22905b02-8ed1-4c26-92ae-7282baf885cd-metrics-certs podName:22905b02-8ed1-4c26-92ae-7282baf885cd nodeName:}" failed. No retries permitted until 2026-07-22 01:47:37.333258192 +0000 UTC m=+10.177087307 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/22905b02-8ed1-4c26-92ae-7282baf885cd-metrics-certs") pod "network-metrics-daemon-t7sb9" (UID: "22905b02-8ed1-4c26-92ae-7282baf885cd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 01:47:33.434198 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:33.433964    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7gqd\" (UniqueName: \"kubernetes.io/projected/e6cd3d7c-b979-4d69-aad7-68ab39aed83d-kube-api-access-z7gqd\") pod \"network-check-target-fltbj\" (UID: \"e6cd3d7c-b979-4d69-aad7-68ab39aed83d\") " pod="openshift-network-diagnostics/network-check-target-fltbj"
Jul 22 01:47:33.434198 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:33.434117    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 22 01:47:33.434198 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:33.434139    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 22 01:47:33.434198 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:33.434152    2563 projected.go:196] Error preparing data for projected volume kube-api-access-z7gqd for pod openshift-network-diagnostics/network-check-target-fltbj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 01:47:33.434198 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:33.434206    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6cd3d7c-b979-4d69-aad7-68ab39aed83d-kube-api-access-z7gqd podName:e6cd3d7c-b979-4d69-aad7-68ab39aed83d nodeName:}" failed. No retries permitted until 2026-07-22 01:47:37.434188168 +0000 UTC m=+10.278017283 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-z7gqd" (UniqueName: "kubernetes.io/projected/e6cd3d7c-b979-4d69-aad7-68ab39aed83d-kube-api-access-z7gqd") pod "network-check-target-fltbj" (UID: "e6cd3d7c-b979-4d69-aad7-68ab39aed83d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 01:47:33.659158 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:33.658546    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t7sb9"
Jul 22 01:47:33.659158 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:33.658709    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t7sb9" podUID="22905b02-8ed1-4c26-92ae-7282baf885cd"
Jul 22 01:47:33.659158 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:33.658765    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mjkvg"
Jul 22 01:47:33.659158 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:33.658896    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mjkvg" podUID="2f67b074-15ce-4e39-afdc-5c30c8addc97"
Jul 22 01:47:34.658341 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:34.657831    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fltbj"
Jul 22 01:47:34.658341 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:34.657973    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fltbj" podUID="e6cd3d7c-b979-4d69-aad7-68ab39aed83d"
Jul 22 01:47:35.659960 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:35.657636    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t7sb9"
Jul 22 01:47:35.659960 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:35.657777    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t7sb9" podUID="22905b02-8ed1-4c26-92ae-7282baf885cd"
Jul 22 01:47:35.659960 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:35.658162    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mjkvg"
Jul 22 01:47:35.659960 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:35.658255    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mjkvg" podUID="2f67b074-15ce-4e39-afdc-5c30c8addc97"
Jul 22 01:47:36.657590 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:36.657079    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fltbj"
Jul 22 01:47:36.657590 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:36.657204    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fltbj" podUID="e6cd3d7c-b979-4d69-aad7-68ab39aed83d"
Jul 22 01:47:37.369638 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:37.369598    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22905b02-8ed1-4c26-92ae-7282baf885cd-metrics-certs\") pod \"network-metrics-daemon-t7sb9\" (UID: \"22905b02-8ed1-4c26-92ae-7282baf885cd\") " pod="openshift-multus/network-metrics-daemon-t7sb9"
Jul 22 01:47:37.370143 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:37.369661    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2f67b074-15ce-4e39-afdc-5c30c8addc97-original-pull-secret\") pod \"global-pull-secret-syncer-mjkvg\" (UID: \"2f67b074-15ce-4e39-afdc-5c30c8addc97\") " pod="kube-system/global-pull-secret-syncer-mjkvg"
Jul 22 01:47:37.370143 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:37.369796    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 01:47:37.370143 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:37.369874    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22905b02-8ed1-4c26-92ae-7282baf885cd-metrics-certs podName:22905b02-8ed1-4c26-92ae-7282baf885cd nodeName:}" failed. No retries permitted until 2026-07-22 01:47:45.369851007 +0000 UTC m=+18.213680129 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/22905b02-8ed1-4c26-92ae-7282baf885cd-metrics-certs") pod "network-metrics-daemon-t7sb9" (UID: "22905b02-8ed1-4c26-92ae-7282baf885cd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 01:47:37.370143 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:37.369796    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 22 01:47:37.370143 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:37.369923    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f67b074-15ce-4e39-afdc-5c30c8addc97-original-pull-secret podName:2f67b074-15ce-4e39-afdc-5c30c8addc97 nodeName:}" failed. No retries permitted until 2026-07-22 01:47:45.369910348 +0000 UTC m=+18.213739478 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2f67b074-15ce-4e39-afdc-5c30c8addc97-original-pull-secret") pod "global-pull-secret-syncer-mjkvg" (UID: "2f67b074-15ce-4e39-afdc-5c30c8addc97") : object "kube-system"/"original-pull-secret" not registered
Jul 22 01:47:37.471055 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:37.470828    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7gqd\" (UniqueName: \"kubernetes.io/projected/e6cd3d7c-b979-4d69-aad7-68ab39aed83d-kube-api-access-z7gqd\") pod \"network-check-target-fltbj\" (UID: \"e6cd3d7c-b979-4d69-aad7-68ab39aed83d\") " pod="openshift-network-diagnostics/network-check-target-fltbj"
Jul 22 01:47:37.471055 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:37.471036    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 22 01:47:37.471055 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:37.471059    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 22 01:47:37.471353 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:37.471073    2563 projected.go:196] Error preparing data for projected volume kube-api-access-z7gqd for pod openshift-network-diagnostics/network-check-target-fltbj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 01:47:37.471353 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:37.471134    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6cd3d7c-b979-4d69-aad7-68ab39aed83d-kube-api-access-z7gqd podName:e6cd3d7c-b979-4d69-aad7-68ab39aed83d nodeName:}" failed. No retries permitted until 2026-07-22 01:47:45.471115155 +0000 UTC m=+18.314944269 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-z7gqd" (UniqueName: "kubernetes.io/projected/e6cd3d7c-b979-4d69-aad7-68ab39aed83d-kube-api-access-z7gqd") pod "network-check-target-fltbj" (UID: "e6cd3d7c-b979-4d69-aad7-68ab39aed83d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 01:47:37.657335 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:37.657228    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t7sb9"
Jul 22 01:47:37.657335 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:37.657222    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mjkvg"
Jul 22 01:47:37.658360 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:37.658326    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t7sb9" podUID="22905b02-8ed1-4c26-92ae-7282baf885cd"
Jul 22 01:47:37.658523 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:37.658427    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mjkvg" podUID="2f67b074-15ce-4e39-afdc-5c30c8addc97"
Jul 22 01:47:38.657738 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:38.657286    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fltbj"
Jul 22 01:47:38.657738 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:38.657433    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fltbj" podUID="e6cd3d7c-b979-4d69-aad7-68ab39aed83d"
Jul 22 01:47:39.657299 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:39.657263    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t7sb9"
Jul 22 01:47:39.657477 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:39.657422    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t7sb9" podUID="22905b02-8ed1-4c26-92ae-7282baf885cd"
Jul 22 01:47:39.657908 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:39.657763    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mjkvg"
Jul 22 01:47:39.657908 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:39.657866    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mjkvg" podUID="2f67b074-15ce-4e39-afdc-5c30c8addc97"
Jul 22 01:47:40.657767 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:40.657735    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fltbj"
Jul 22 01:47:40.657936 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:40.657861    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fltbj" podUID="e6cd3d7c-b979-4d69-aad7-68ab39aed83d"
Jul 22 01:47:41.657909 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:41.657879    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mjkvg"
Jul 22 01:47:41.658074 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:41.658016    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mjkvg" podUID="2f67b074-15ce-4e39-afdc-5c30c8addc97"
Jul 22 01:47:41.658483 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:41.658092    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t7sb9"
Jul 22 01:47:41.658483 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:41.658192    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t7sb9" podUID="22905b02-8ed1-4c26-92ae-7282baf885cd"
Jul 22 01:47:42.657298 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:42.657262    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fltbj"
Jul 22 01:47:42.657502 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:42.657389    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fltbj" podUID="e6cd3d7c-b979-4d69-aad7-68ab39aed83d"
Jul 22 01:47:43.657856 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:43.657823    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mjkvg"
Jul 22 01:47:43.658285 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:43.657823    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t7sb9"
Jul 22 01:47:43.658285 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:43.657965    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mjkvg" podUID="2f67b074-15ce-4e39-afdc-5c30c8addc97"
Jul 22 01:47:43.658285 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:43.658032    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t7sb9" podUID="22905b02-8ed1-4c26-92ae-7282baf885cd"
Jul 22 01:47:44.657842 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:44.657808    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fltbj"
Jul 22 01:47:44.658042 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:44.657917    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fltbj" podUID="e6cd3d7c-b979-4d69-aad7-68ab39aed83d"
Jul 22 01:47:45.433439 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:45.433405    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22905b02-8ed1-4c26-92ae-7282baf885cd-metrics-certs\") pod \"network-metrics-daemon-t7sb9\" (UID: \"22905b02-8ed1-4c26-92ae-7282baf885cd\") " pod="openshift-multus/network-metrics-daemon-t7sb9"
Jul 22 01:47:45.433439 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:45.433448    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2f67b074-15ce-4e39-afdc-5c30c8addc97-original-pull-secret\") pod \"global-pull-secret-syncer-mjkvg\" (UID: \"2f67b074-15ce-4e39-afdc-5c30c8addc97\") " pod="kube-system/global-pull-secret-syncer-mjkvg"
Jul 22 01:47:45.433681 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:45.433553    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 01:47:45.433681 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:45.433621    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22905b02-8ed1-4c26-92ae-7282baf885cd-metrics-certs podName:22905b02-8ed1-4c26-92ae-7282baf885cd nodeName:}" failed. No retries permitted until 2026-07-22 01:48:01.43360106 +0000 UTC m=+34.277430186 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/22905b02-8ed1-4c26-92ae-7282baf885cd-metrics-certs") pod "network-metrics-daemon-t7sb9" (UID: "22905b02-8ed1-4c26-92ae-7282baf885cd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 01:47:45.433681 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:45.433557    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 22 01:47:45.433831 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:45.433694    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f67b074-15ce-4e39-afdc-5c30c8addc97-original-pull-secret podName:2f67b074-15ce-4e39-afdc-5c30c8addc97 nodeName:}" failed. No retries permitted until 2026-07-22 01:48:01.433677393 +0000 UTC m=+34.277506503 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2f67b074-15ce-4e39-afdc-5c30c8addc97-original-pull-secret") pod "global-pull-secret-syncer-mjkvg" (UID: "2f67b074-15ce-4e39-afdc-5c30c8addc97") : object "kube-system"/"original-pull-secret" not registered
Jul 22 01:47:45.533929 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:45.533901    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7gqd\" (UniqueName: \"kubernetes.io/projected/e6cd3d7c-b979-4d69-aad7-68ab39aed83d-kube-api-access-z7gqd\") pod \"network-check-target-fltbj\" (UID: \"e6cd3d7c-b979-4d69-aad7-68ab39aed83d\") " pod="openshift-network-diagnostics/network-check-target-fltbj"
Jul 22 01:47:45.534090 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:45.534032    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 22 01:47:45.534090 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:45.534064    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 22 01:47:45.534090 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:45.534078    2563 projected.go:196] Error preparing data for projected volume kube-api-access-z7gqd for pod openshift-network-diagnostics/network-check-target-fltbj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 01:47:45.534216 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:45.534130    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6cd3d7c-b979-4d69-aad7-68ab39aed83d-kube-api-access-z7gqd podName:e6cd3d7c-b979-4d69-aad7-68ab39aed83d nodeName:}" failed. No retries permitted until 2026-07-22 01:48:01.534115887 +0000 UTC m=+34.377945010 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-z7gqd" (UniqueName: "kubernetes.io/projected/e6cd3d7c-b979-4d69-aad7-68ab39aed83d-kube-api-access-z7gqd") pod "network-check-target-fltbj" (UID: "e6cd3d7c-b979-4d69-aad7-68ab39aed83d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 01:47:45.657208 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:45.657178    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mjkvg"
Jul 22 01:47:45.657357 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:45.657177    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t7sb9"
Jul 22 01:47:45.657357 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:45.657287    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mjkvg" podUID="2f67b074-15ce-4e39-afdc-5c30c8addc97"
Jul 22 01:47:45.657457 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:45.657410    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t7sb9" podUID="22905b02-8ed1-4c26-92ae-7282baf885cd"
Jul 22 01:47:46.658051 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:46.658013    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fltbj"
Jul 22 01:47:46.658533 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:46.658156    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fltbj" podUID="e6cd3d7c-b979-4d69-aad7-68ab39aed83d"
Jul 22 01:47:47.658648 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:47.658616    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t7sb9"
Jul 22 01:47:47.659068 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:47.658724    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t7sb9" podUID="22905b02-8ed1-4c26-92ae-7282baf885cd"
Jul 22 01:47:47.659068 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:47.658769    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mjkvg"
Jul 22 01:47:47.659068 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:47.658883    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mjkvg" podUID="2f67b074-15ce-4e39-afdc-5c30c8addc97"
Jul 22 01:47:48.657613 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:48.657581    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fltbj"
Jul 22 01:47:48.657770 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:48.657706    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fltbj" podUID="e6cd3d7c-b979-4d69-aad7-68ab39aed83d"
Jul 22 01:47:49.657620 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:49.657584    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t7sb9"
Jul 22 01:47:49.658089 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:49.657720    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t7sb9" podUID="22905b02-8ed1-4c26-92ae-7282baf885cd"
Jul 22 01:47:49.658089 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:49.657751    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mjkvg"
Jul 22 01:47:49.658089 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:49.657832    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mjkvg" podUID="2f67b074-15ce-4e39-afdc-5c30c8addc97"
Jul 22 01:47:50.657298 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:50.657143    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fltbj"
Jul 22 01:47:50.657422 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:50.657372    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fltbj" podUID="e6cd3d7c-b979-4d69-aad7-68ab39aed83d"
Jul 22 01:47:50.832180 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:50.832139    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-drsr9" event={"ID":"2393693b-9aef-40c6-ab3f-df34a7500ed8","Type":"ContainerStarted","Data":"efc0a026fc96ef18a879869c4bebffbd806f29a8130ac1b5af2fa477f310e2a9"}
Jul 22 01:47:50.834148 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:50.834121    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sq599" event={"ID":"91595658-5e3d-40eb-9763-67b1f193589d","Type":"ContainerStarted","Data":"a712c15ad8b27edde7aeb7e82441e5b8e7d340bbe10ebdd68b410c48c76921b6"}
Jul 22 01:47:50.836038 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:50.836001    2563 generic.go:356] "Generic (PLEG): container finished" podID="c4290331-2538-4205-b3d0-58ab1a1126b6" containerID="248511652a6f0db530afc85444ba52b1474897f1ec22501c55167762bc07537a" exitCode=0
Jul 22 01:47:50.836153 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:50.836089    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pbf4b" event={"ID":"c4290331-2538-4205-b3d0-58ab1a1126b6","Type":"ContainerDied","Data":"248511652a6f0db530afc85444ba52b1474897f1ec22501c55167762bc07537a"}
Jul 22 01:47:50.838041 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:50.838014    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rd82t" event={"ID":"04c89d69-a674-49e4-9630-9302cec0fad4","Type":"ContainerStarted","Data":"eccacee2405fc42740e8e270fc6703ffe946aaac5ab234b462e766a17a7d22c8"}
Jul 22 01:47:50.839748 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:50.839705    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-78zsq" event={"ID":"2553bf24-562e-4e3b-b5d3-d89687dd807f","Type":"ContainerStarted","Data":"4f4773046b29e3b2f78e5526ed6590c94bfa8c4c656a900b3f76405734371cc5"}
Jul 22 01:47:50.843431 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:50.843404    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8" event={"ID":"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17","Type":"ContainerStarted","Data":"b89353057f2b11f6ab83e328acd71a0fd62927cdb38fc5a3cedb57ca606b16a5"}
Jul 22 01:47:50.843543 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:50.843489    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8" event={"ID":"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17","Type":"ContainerStarted","Data":"366637807ae4baf6ba8b37dffbf2e2fbde60a647876ae28f543063f313b15434"}
Jul 22 01:47:50.843600 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:50.843543    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8" event={"ID":"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17","Type":"ContainerStarted","Data":"5c0f5f0e044d767c57096cc37398e84dd29d3e5a90a4a1c48db4b44d79439e83"}
Jul 22 01:47:50.843600 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:50.843589    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8" event={"ID":"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17","Type":"ContainerStarted","Data":"8dd4302d5424d231250d02f92bd71a20563ea855dfee12c10baef818c22b885a"}
Jul 22 01:47:50.843677 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:50.843635    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8" event={"ID":"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17","Type":"ContainerStarted","Data":"89fa1eeb3744724cf8aa0fb8081d75db08ae7bc891b6c1129fbb1df2981b59a7"}
Jul 22 01:47:50.843713 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:50.843685    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8" event={"ID":"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17","Type":"ContainerStarted","Data":"b58594fef9d7680b5f13f97f6dc3a0da36b7805886100467a4842a015aa1abf3"}
Jul 22 01:47:50.845713 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:50.845675    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzqpt" event={"ID":"00689b5e-a714-4b23-8b13-b16b3c05c817","Type":"ContainerStarted","Data":"8d00b1aefba072b0e4a472cf86618b56da75c64675b6ff9a76608e3c0aa9018a"}
Jul 22 01:47:50.847113 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:50.847074    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-drsr9" podStartSLOduration=3.599885795 podStartE2EDuration="22.847059189s" totalImagesPullingTime="19.247173394s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:47:50.846646876 +0000 UTC m=+23.690476007" watchObservedRunningTime="2026-07-22 01:47:50.847059189 +0000 UTC m=+23.690888324"
Jul 22 01:47:50.847743 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:50.847330    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4xkbt" event={"ID":"08637867-465c-4df3-b707-ef1e33b64ed2","Type":"ContainerStarted","Data":"e40e9657f0eea937380c1e2f5e960d3baa5f627014a9763cd198c6cbb446cbd5"}
Jul 22 01:47:50.847743 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:50.847601    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-230.ec2.internal" podStartSLOduration=21.847591219999998 podStartE2EDuration="21.84759122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:47:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:47:32.810153293 +0000 UTC m=+5.653982430" watchObservedRunningTime="2026-07-22 01:47:50.84759122 +0000 UTC m=+23.691420353"
Jul 22 01:47:50.861335 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:50.861287    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-sq599" podStartSLOduration=3.611937807 podStartE2EDuration="22.861273746s" totalImagesPullingTime="19.249335939s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:47:50.860916819 +0000 UTC m=+23.704745952" watchObservedRunningTime="2026-07-22 01:47:50.861273746 +0000 UTC m=+23.705102882"
Jul 22 01:47:50.884638 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:50.884561    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-78zsq" podStartSLOduration=3.55247291 podStartE2EDuration="22.884548458s" totalImagesPullingTime="19.332075548s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:47:50.884355432 +0000 UTC m=+23.728184567" watchObservedRunningTime="2026-07-22 01:47:50.884548458 +0000 UTC m=+23.728377589"
Jul 22 01:47:50.918328 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:50.918288    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rd82t" podStartSLOduration=11.720023657 podStartE2EDuration="22.918275073s" totalImagesPullingTime="11.198251416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:47:50.918142822 +0000 UTC m=+23.761971947" watchObservedRunningTime="2026-07-22 01:47:50.918275073 +0000 UTC m=+23.762104205"
Jul 22 01:47:50.945130 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:50.945081    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4xkbt" podStartSLOduration=3.687882197 podStartE2EDuration="22.945067608s" totalImagesPullingTime="19.257185411s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:47:50.944672042 +0000 UTC m=+23.788501174" watchObservedRunningTime="2026-07-22 01:47:50.945067608 +0000 UTC m=+23.788896740"
Jul 22 01:47:51.637290 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:51.637265    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 22 01:47:51.657612 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:51.657588    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mjkvg"
Jul 22 01:47:51.657729 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:51.657597    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t7sb9"
Jul 22 01:47:51.657729 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:51.657714    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mjkvg" podUID="2f67b074-15ce-4e39-afdc-5c30c8addc97"
Jul 22 01:47:51.657855 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:51.657815    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t7sb9" podUID="22905b02-8ed1-4c26-92ae-7282baf885cd"
Jul 22 01:47:51.660607 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:51.660510    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-22T01:47:51.637286058Z","UUID":"6dd0f329-db55-4b68-9dc5-d9367939a425","Handler":null,"Name":"","Endpoint":""}
Jul 22 01:47:51.662002 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:51.661984    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 22 01:47:51.662101 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:51.662022    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 22 01:47:51.681763 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:51.681738    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rd82t"
Jul 22 01:47:51.682309 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:51.682290    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rd82t"
Jul 22 01:47:51.852066 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:51.852028    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzqpt" event={"ID":"00689b5e-a714-4b23-8b13-b16b3c05c817","Type":"ContainerStarted","Data":"c71a0e53d8b07fe423b521046364c0553e01557fd4b82ce01786b5cd4139a2d2"}
Jul 22 01:47:51.853997 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:51.853954    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pmrf8" event={"ID":"ce13bc5e-08ec-4b2b-a9fb-63ac3c1a34f1","Type":"ContainerStarted","Data":"7e0bc72ac94065f7e9e476413e88e0c72ca134e32e9882c75dcd73307e797ea5"}
Jul 22 01:47:51.854773 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:51.854726    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rd82t"
Jul 22 01:47:51.855304 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:51.855279    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rd82t"
Jul 22 01:47:51.879222 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:51.879181    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pmrf8" podStartSLOduration=4.625801883 podStartE2EDuration="23.879168324s" totalImagesPullingTime="19.253366441s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:47:51.878885162 +0000 UTC m=+24.722714295" watchObservedRunningTime="2026-07-22 01:47:51.879168324 +0000 UTC m=+24.722997457"
Jul 22 01:47:52.657518 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:52.657483    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fltbj"
Jul 22 01:47:52.657911 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:52.657615    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fltbj" podUID="e6cd3d7c-b979-4d69-aad7-68ab39aed83d"
Jul 22 01:47:53.657734 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:53.657551    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t7sb9"
Jul 22 01:47:53.658251 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:53.657621    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mjkvg"
Jul 22 01:47:53.658251 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:53.657838    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t7sb9" podUID="22905b02-8ed1-4c26-92ae-7282baf885cd"
Jul 22 01:47:53.658251 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:53.657912    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mjkvg" podUID="2f67b074-15ce-4e39-afdc-5c30c8addc97"
Jul 22 01:47:53.863003 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:53.862966    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8" event={"ID":"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17","Type":"ContainerStarted","Data":"3bd4a27d9d007f85ae27a7496e82b75111c54f0b14c4fdaa4ebbb0cd1404cf55"}
Jul 22 01:47:53.865290 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:53.865258    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzqpt" event={"ID":"00689b5e-a714-4b23-8b13-b16b3c05c817","Type":"ContainerStarted","Data":"77108e318589e2d75f93fbbcdcf42598845ed3c08637cba4d1fa81ea32ceaf8b"}
Jul 22 01:47:53.892395 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:53.892334    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zzqpt" podStartSLOduration=4.306601461 podStartE2EDuration="25.892315565s" totalImagesPullingTime="21.585714104s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:47:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:47:53.892164116 +0000 UTC m=+26.735993252" watchObservedRunningTime="2026-07-22 01:47:53.892315565 +0000 UTC m=+26.736144696"
Jul 22 01:47:54.657489 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:54.657455    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fltbj"
Jul 22 01:47:54.657668 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:54.657644    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fltbj" podUID="e6cd3d7c-b979-4d69-aad7-68ab39aed83d"
Jul 22 01:47:55.657953 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:55.657695    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mjkvg"
Jul 22 01:47:55.657953 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:55.657695    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t7sb9"
Jul 22 01:47:55.658750 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:55.657991    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mjkvg" podUID="2f67b074-15ce-4e39-afdc-5c30c8addc97"
Jul 22 01:47:55.658750 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:55.658110    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t7sb9" podUID="22905b02-8ed1-4c26-92ae-7282baf885cd"
Jul 22 01:47:55.871871 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:55.871840    2563 generic.go:356] "Generic (PLEG): container finished" podID="c4290331-2538-4205-b3d0-58ab1a1126b6" containerID="3665807f8077d824f31728c933d062b2c5b23103949497612cd48f88d645289d" exitCode=0
Jul 22 01:47:55.872058 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:55.871930    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pbf4b" event={"ID":"c4290331-2538-4205-b3d0-58ab1a1126b6","Type":"ContainerDied","Data":"3665807f8077d824f31728c933d062b2c5b23103949497612cd48f88d645289d"}
Jul 22 01:47:55.875944 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:55.875912    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8" event={"ID":"7ec6fec7-2d6d-4522-ae0b-6a53e98aaa17","Type":"ContainerStarted","Data":"192a30e3e9c5e864d3136548c5bf802692ae599be2d3a648dbd820ce55a8d336"}
Jul 22 01:47:55.876192 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:55.876166    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:55.876336 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:55.876249    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:55.876336 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:55.876297    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:55.892058 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:55.892035    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:55.896888 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:55.896868    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:47:55.927245 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:55.926952    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8" podStartSLOduration=8.607205972 podStartE2EDuration="27.92692525s" totalImagesPullingTime="19.319719278s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:47:55.926833785 +0000 UTC m=+28.770662915" watchObservedRunningTime="2026-07-22 01:47:55.92692525 +0000 UTC m=+28.770754373"
Jul 22 01:47:56.657648 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:56.657616    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fltbj"
Jul 22 01:47:56.657944 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:56.657728    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fltbj" podUID="e6cd3d7c-b979-4d69-aad7-68ab39aed83d"
Jul 22 01:47:57.323391 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:57.323282    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mjkvg"]
Jul 22 01:47:57.323818 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:57.323439    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mjkvg"
Jul 22 01:47:57.323818 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:57.323548    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mjkvg" podUID="2f67b074-15ce-4e39-afdc-5c30c8addc97"
Jul 22 01:47:57.326854 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:57.326830    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fltbj"]
Jul 22 01:47:57.327005 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:57.326924    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fltbj"
Jul 22 01:47:57.327047 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:57.327021    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fltbj" podUID="e6cd3d7c-b979-4d69-aad7-68ab39aed83d"
Jul 22 01:47:57.327403 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:57.327363    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-t7sb9"]
Jul 22 01:47:57.327514 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:57.327486    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t7sb9"
Jul 22 01:47:57.327597 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:57.327579    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t7sb9" podUID="22905b02-8ed1-4c26-92ae-7282baf885cd"
Jul 22 01:47:57.881720 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:57.881683    2563 generic.go:356] "Generic (PLEG): container finished" podID="c4290331-2538-4205-b3d0-58ab1a1126b6" containerID="144925f4f21e6429c9e4f06983757bc5dcde86197f7baeea3206d07ed266a036" exitCode=0
Jul 22 01:47:57.881890 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:57.881761    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pbf4b" event={"ID":"c4290331-2538-4205-b3d0-58ab1a1126b6","Type":"ContainerDied","Data":"144925f4f21e6429c9e4f06983757bc5dcde86197f7baeea3206d07ed266a036"}
Jul 22 01:47:58.657884 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:58.657713    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t7sb9"
Jul 22 01:47:58.658326 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:58.657721    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mjkvg"
Jul 22 01:47:58.658326 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:58.657989    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t7sb9" podUID="22905b02-8ed1-4c26-92ae-7282baf885cd"
Jul 22 01:47:58.658326 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:58.657763    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fltbj"
Jul 22 01:47:58.658326 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:58.658037    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mjkvg" podUID="2f67b074-15ce-4e39-afdc-5c30c8addc97"
Jul 22 01:47:58.658326 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:47:58.658097    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fltbj" podUID="e6cd3d7c-b979-4d69-aad7-68ab39aed83d"
Jul 22 01:47:59.888556 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:59.888518    2563 generic.go:356] "Generic (PLEG): container finished" podID="c4290331-2538-4205-b3d0-58ab1a1126b6" containerID="13a699bad0c933c7e540fdfa1d6929e307e38c9a11640081485bc6a0698d586d" exitCode=0
Jul 22 01:47:59.888987 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:47:59.888576    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pbf4b" event={"ID":"c4290331-2538-4205-b3d0-58ab1a1126b6","Type":"ContainerDied","Data":"13a699bad0c933c7e540fdfa1d6929e307e38c9a11640081485bc6a0698d586d"}
Jul 22 01:48:00.658097 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:00.658059    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t7sb9"
Jul 22 01:48:00.658279 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:00.658059    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mjkvg"
Jul 22 01:48:00.658279 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:00.658185    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t7sb9" podUID="22905b02-8ed1-4c26-92ae-7282baf885cd"
Jul 22 01:48:00.658392 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:00.658287    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mjkvg" podUID="2f67b074-15ce-4e39-afdc-5c30c8addc97"
Jul 22 01:48:00.658392 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:00.658063    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fltbj"
Jul 22 01:48:00.658472 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:00.658403    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fltbj" podUID="e6cd3d7c-b979-4d69-aad7-68ab39aed83d"
Jul 22 01:48:01.451531 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:01.451491    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22905b02-8ed1-4c26-92ae-7282baf885cd-metrics-certs\") pod \"network-metrics-daemon-t7sb9\" (UID: \"22905b02-8ed1-4c26-92ae-7282baf885cd\") " pod="openshift-multus/network-metrics-daemon-t7sb9"
Jul 22 01:48:01.451531 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:01.451541    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2f67b074-15ce-4e39-afdc-5c30c8addc97-original-pull-secret\") pod \"global-pull-secret-syncer-mjkvg\" (UID: \"2f67b074-15ce-4e39-afdc-5c30c8addc97\") " pod="kube-system/global-pull-secret-syncer-mjkvg"
Jul 22 01:48:01.452009 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:01.451653    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 22 01:48:01.452009 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:01.451661    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 01:48:01.452009 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:01.451716    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22905b02-8ed1-4c26-92ae-7282baf885cd-metrics-certs podName:22905b02-8ed1-4c26-92ae-7282baf885cd nodeName:}" failed. No retries permitted until 2026-07-22 01:48:33.451701161 +0000 UTC m=+66.295530277 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/22905b02-8ed1-4c26-92ae-7282baf885cd-metrics-certs") pod "network-metrics-daemon-t7sb9" (UID: "22905b02-8ed1-4c26-92ae-7282baf885cd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 01:48:01.452009 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:01.451731    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f67b074-15ce-4e39-afdc-5c30c8addc97-original-pull-secret podName:2f67b074-15ce-4e39-afdc-5c30c8addc97 nodeName:}" failed. No retries permitted until 2026-07-22 01:48:33.451724138 +0000 UTC m=+66.295553247 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2f67b074-15ce-4e39-afdc-5c30c8addc97-original-pull-secret") pod "global-pull-secret-syncer-mjkvg" (UID: "2f67b074-15ce-4e39-afdc-5c30c8addc97") : object "kube-system"/"original-pull-secret" not registered
Jul 22 01:48:01.552736 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:01.552698    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7gqd\" (UniqueName: \"kubernetes.io/projected/e6cd3d7c-b979-4d69-aad7-68ab39aed83d-kube-api-access-z7gqd\") pod \"network-check-target-fltbj\" (UID: \"e6cd3d7c-b979-4d69-aad7-68ab39aed83d\") " pod="openshift-network-diagnostics/network-check-target-fltbj"
Jul 22 01:48:01.552920 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:01.552874    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 22 01:48:01.552920 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:01.552901    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 22 01:48:01.552920 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:01.552914    2563 projected.go:196] Error preparing data for projected volume kube-api-access-z7gqd for pod openshift-network-diagnostics/network-check-target-fltbj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 01:48:01.553098 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:01.552981    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6cd3d7c-b979-4d69-aad7-68ab39aed83d-kube-api-access-z7gqd podName:e6cd3d7c-b979-4d69-aad7-68ab39aed83d nodeName:}" failed. No retries permitted until 2026-07-22 01:48:33.552966206 +0000 UTC m=+66.396795328 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-z7gqd" (UniqueName: "kubernetes.io/projected/e6cd3d7c-b979-4d69-aad7-68ab39aed83d-kube-api-access-z7gqd") pod "network-check-target-fltbj" (UID: "e6cd3d7c-b979-4d69-aad7-68ab39aed83d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 01:48:02.657812 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:02.657778    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t7sb9"
Jul 22 01:48:02.657812 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:02.657806    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fltbj"
Jul 22 01:48:02.658529 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:02.657812    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mjkvg"
Jul 22 01:48:02.658529 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:02.657906    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t7sb9" podUID="22905b02-8ed1-4c26-92ae-7282baf885cd"
Jul 22 01:48:02.658529 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:02.657991    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fltbj" podUID="e6cd3d7c-b979-4d69-aad7-68ab39aed83d"
Jul 22 01:48:02.658529 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:02.658082    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mjkvg" podUID="2f67b074-15ce-4e39-afdc-5c30c8addc97"
Jul 22 01:48:02.961655 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:02.961615    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-230.ec2.internal" event="NodeReady"
Jul 22 01:48:02.961849 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:02.961776    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 22 01:48:02.993802 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:02.993768    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6f7bd956-kxk6z"]
Jul 22 01:48:02.999858 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:02.999831    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f7bd956-kxk6z"
Jul 22 01:48:03.002109 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.002081    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 22 01:48:03.002355 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.002328    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 22 01:48:03.002510 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.002493    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 22 01:48:03.002602 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.002583    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-c2smp\""
Jul 22 01:48:03.007613 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.007562    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wkdlw"]
Jul 22 01:48:03.008018 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.007996    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 22 01:48:03.011460 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.011437    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mhjk2"]
Jul 22 01:48:03.011608 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.011586    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wkdlw"
Jul 22 01:48:03.015352 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.014023    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hpqgw\""
Jul 22 01:48:03.015352 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.014275    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 22 01:48:03.015352 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.014530    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 22 01:48:03.016854 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.016678    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f7bd956-kxk6z"]
Jul 22 01:48:03.016854 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.016803    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mhjk2"
Jul 22 01:48:03.019736 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.019715    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mhjk2"]
Jul 22 01:48:03.021903 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.021628    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 22 01:48:03.022104 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.022080    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 22 01:48:03.022201 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.022102    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 22 01:48:03.022908 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.022889    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-v5h9n\""
Jul 22 01:48:03.024139 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.024114    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wkdlw"]
Jul 22 01:48:03.165910 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.165871    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4f988b4-1014-4229-ba21-062a1dc87923-bound-sa-token\") pod \"image-registry-6f7bd956-kxk6z\" (UID: \"c4f988b4-1014-4229-ba21-062a1dc87923\") " pod="openshift-image-registry/image-registry-6f7bd956-kxk6z"
Jul 22 01:48:03.165910 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.165917    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41d01142-330d-46a2-8625-9e5b9ba1e2bf-metrics-tls\") pod \"dns-default-wkdlw\" (UID: \"41d01142-330d-46a2-8625-9e5b9ba1e2bf\") " pod="openshift-dns/dns-default-wkdlw"
Jul 22 01:48:03.166169 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.165953    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpbws\" (UniqueName: \"kubernetes.io/projected/6cc54a54-1e35-414c-ac99-ce76586fe402-kube-api-access-fpbws\") pod \"ingress-canary-mhjk2\" (UID: \"6cc54a54-1e35-414c-ac99-ce76586fe402\") " pod="openshift-ingress-canary/ingress-canary-mhjk2"
Jul 22 01:48:03.166169 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.166043    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/41d01142-330d-46a2-8625-9e5b9ba1e2bf-config-volume\") pod \"dns-default-wkdlw\" (UID: \"41d01142-330d-46a2-8625-9e5b9ba1e2bf\") " pod="openshift-dns/dns-default-wkdlw"
Jul 22 01:48:03.166169 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.166082    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b48r\" (UniqueName: \"kubernetes.io/projected/41d01142-330d-46a2-8625-9e5b9ba1e2bf-kube-api-access-5b48r\") pod \"dns-default-wkdlw\" (UID: \"41d01142-330d-46a2-8625-9e5b9ba1e2bf\") " pod="openshift-dns/dns-default-wkdlw"
Jul 22 01:48:03.166169 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.166135    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41d01142-330d-46a2-8625-9e5b9ba1e2bf-tmp-dir\") pod \"dns-default-wkdlw\" (UID: \"41d01142-330d-46a2-8625-9e5b9ba1e2bf\") " pod="openshift-dns/dns-default-wkdlw"
Jul 22 01:48:03.166367 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.166194    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c4f988b4-1014-4229-ba21-062a1dc87923-trusted-ca\") pod \"image-registry-6f7bd956-kxk6z\" (UID: \"c4f988b4-1014-4229-ba21-062a1dc87923\") " pod="openshift-image-registry/image-registry-6f7bd956-kxk6z"
Jul 22 01:48:03.166367 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.166288    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c4f988b4-1014-4229-ba21-062a1dc87923-image-registry-private-configuration\") pod \"image-registry-6f7bd956-kxk6z\" (UID: \"c4f988b4-1014-4229-ba21-062a1dc87923\") " pod="openshift-image-registry/image-registry-6f7bd956-kxk6z"
Jul 22 01:48:03.166367 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.166324    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c4f988b4-1014-4229-ba21-062a1dc87923-ca-trust-extracted\") pod \"image-registry-6f7bd956-kxk6z\" (UID: \"c4f988b4-1014-4229-ba21-062a1dc87923\") " pod="openshift-image-registry/image-registry-6f7bd956-kxk6z"
Jul 22 01:48:03.166367 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.166349    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdsvh\" (UniqueName: \"kubernetes.io/projected/c4f988b4-1014-4229-ba21-062a1dc87923-kube-api-access-gdsvh\") pod \"image-registry-6f7bd956-kxk6z\" (UID: \"c4f988b4-1014-4229-ba21-062a1dc87923\") " pod="openshift-image-registry/image-registry-6f7bd956-kxk6z"
Jul 22 01:48:03.166564 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.166404    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6cc54a54-1e35-414c-ac99-ce76586fe402-cert\") pod \"ingress-canary-mhjk2\" (UID: \"6cc54a54-1e35-414c-ac99-ce76586fe402\") " pod="openshift-ingress-canary/ingress-canary-mhjk2"
Jul 22 01:48:03.166564 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.166442    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4f988b4-1014-4229-ba21-062a1dc87923-registry-tls\") pod \"image-registry-6f7bd956-kxk6z\" (UID: \"c4f988b4-1014-4229-ba21-062a1dc87923\") " pod="openshift-image-registry/image-registry-6f7bd956-kxk6z"
Jul 22 01:48:03.166564 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.166458    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c4f988b4-1014-4229-ba21-062a1dc87923-registry-certificates\") pod \"image-registry-6f7bd956-kxk6z\" (UID: \"c4f988b4-1014-4229-ba21-062a1dc87923\") " pod="openshift-image-registry/image-registry-6f7bd956-kxk6z"
Jul 22 01:48:03.166564 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.166478    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c4f988b4-1014-4229-ba21-062a1dc87923-installation-pull-secrets\") pod \"image-registry-6f7bd956-kxk6z\" (UID: \"c4f988b4-1014-4229-ba21-062a1dc87923\") " pod="openshift-image-registry/image-registry-6f7bd956-kxk6z"
Jul 22 01:48:03.267203 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.267111    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c4f988b4-1014-4229-ba21-062a1dc87923-trusted-ca\") pod \"image-registry-6f7bd956-kxk6z\" (UID: \"c4f988b4-1014-4229-ba21-062a1dc87923\") " pod="openshift-image-registry/image-registry-6f7bd956-kxk6z"
Jul 22 01:48:03.267203 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.267164    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c4f988b4-1014-4229-ba21-062a1dc87923-image-registry-private-configuration\") pod \"image-registry-6f7bd956-kxk6z\" (UID: \"c4f988b4-1014-4229-ba21-062a1dc87923\") " pod="openshift-image-registry/image-registry-6f7bd956-kxk6z"
Jul 22 01:48:03.267203 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.267197    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c4f988b4-1014-4229-ba21-062a1dc87923-ca-trust-extracted\") pod \"image-registry-6f7bd956-kxk6z\" (UID: \"c4f988b4-1014-4229-ba21-062a1dc87923\") " pod="openshift-image-registry/image-registry-6f7bd956-kxk6z"
Jul 22 01:48:03.267523 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.267224    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdsvh\" (UniqueName: \"kubernetes.io/projected/c4f988b4-1014-4229-ba21-062a1dc87923-kube-api-access-gdsvh\") pod \"image-registry-6f7bd956-kxk6z\" (UID: \"c4f988b4-1014-4229-ba21-062a1dc87923\") " pod="openshift-image-registry/image-registry-6f7bd956-kxk6z"
Jul 22 01:48:03.267523 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.267256    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6cc54a54-1e35-414c-ac99-ce76586fe402-cert\") pod \"ingress-canary-mhjk2\" (UID: \"6cc54a54-1e35-414c-ac99-ce76586fe402\") " pod="openshift-ingress-canary/ingress-canary-mhjk2"
Jul 22 01:48:03.267523 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.267502    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4f988b4-1014-4229-ba21-062a1dc87923-registry-tls\") pod \"image-registry-6f7bd956-kxk6z\" (UID: \"c4f988b4-1014-4229-ba21-062a1dc87923\") " pod="openshift-image-registry/image-registry-6f7bd956-kxk6z"
Jul 22 01:48:03.267652 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.267538    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c4f988b4-1014-4229-ba21-062a1dc87923-registry-certificates\") pod \"image-registry-6f7bd956-kxk6z\" (UID: \"c4f988b4-1014-4229-ba21-062a1dc87923\") " pod="openshift-image-registry/image-registry-6f7bd956-kxk6z"
Jul 22 01:48:03.267652 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.267563    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c4f988b4-1014-4229-ba21-062a1dc87923-installation-pull-secrets\") pod \"image-registry-6f7bd956-kxk6z\" (UID: \"c4f988b4-1014-4229-ba21-062a1dc87923\") " pod="openshift-image-registry/image-registry-6f7bd956-kxk6z"
Jul 22 01:48:03.267652 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:03.267641    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 22 01:48:03.267834 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:03.267661    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f7bd956-kxk6z: secret "image-registry-tls" not found
Jul 22 01:48:03.267834 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.267668    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c4f988b4-1014-4229-ba21-062a1dc87923-ca-trust-extracted\") pod \"image-registry-6f7bd956-kxk6z\" (UID: \"c4f988b4-1014-4229-ba21-062a1dc87923\") " pod="openshift-image-registry/image-registry-6f7bd956-kxk6z"
Jul 22 01:48:03.267834 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:03.267722    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4f988b4-1014-4229-ba21-062a1dc87923-registry-tls podName:c4f988b4-1014-4229-ba21-062a1dc87923 nodeName:}" failed. No retries permitted until 2026-07-22 01:48:03.767706233 +0000 UTC m=+36.611535349 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c4f988b4-1014-4229-ba21-062a1dc87923-registry-tls") pod "image-registry-6f7bd956-kxk6z" (UID: "c4f988b4-1014-4229-ba21-062a1dc87923") : secret "image-registry-tls" not found
Jul 22 01:48:03.267834 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.267755    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4f988b4-1014-4229-ba21-062a1dc87923-bound-sa-token\") pod \"image-registry-6f7bd956-kxk6z\" (UID: \"c4f988b4-1014-4229-ba21-062a1dc87923\") " pod="openshift-image-registry/image-registry-6f7bd956-kxk6z"
Jul 22 01:48:03.267834 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.267787    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41d01142-330d-46a2-8625-9e5b9ba1e2bf-metrics-tls\") pod \"dns-default-wkdlw\" (UID: \"41d01142-330d-46a2-8625-9e5b9ba1e2bf\") " pod="openshift-dns/dns-default-wkdlw"
Jul 22 01:48:03.267834 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.267824    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpbws\" (UniqueName: \"kubernetes.io/projected/6cc54a54-1e35-414c-ac99-ce76586fe402-kube-api-access-fpbws\") pod \"ingress-canary-mhjk2\" (UID: \"6cc54a54-1e35-414c-ac99-ce76586fe402\") " pod="openshift-ingress-canary/ingress-canary-mhjk2"
Jul 22 01:48:03.268076 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.267854    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/41d01142-330d-46a2-8625-9e5b9ba1e2bf-config-volume\") pod \"dns-default-wkdlw\" (UID: \"41d01142-330d-46a2-8625-9e5b9ba1e2bf\") " pod="openshift-dns/dns-default-wkdlw"
Jul 22 01:48:03.268076 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.267880    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b48r\" (UniqueName: \"kubernetes.io/projected/41d01142-330d-46a2-8625-9e5b9ba1e2bf-kube-api-access-5b48r\") pod \"dns-default-wkdlw\" (UID: \"41d01142-330d-46a2-8625-9e5b9ba1e2bf\") " pod="openshift-dns/dns-default-wkdlw"
Jul 22 01:48:03.268076 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.267915    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41d01142-330d-46a2-8625-9e5b9ba1e2bf-tmp-dir\") pod \"dns-default-wkdlw\" (UID: \"41d01142-330d-46a2-8625-9e5b9ba1e2bf\") " pod="openshift-dns/dns-default-wkdlw"
Jul 22 01:48:03.268169 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.268143    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c4f988b4-1014-4229-ba21-062a1dc87923-registry-certificates\") pod \"image-registry-6f7bd956-kxk6z\" (UID: \"c4f988b4-1014-4229-ba21-062a1dc87923\") " pod="openshift-image-registry/image-registry-6f7bd956-kxk6z"
Jul 22 01:48:03.268235 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.268212    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c4f988b4-1014-4229-ba21-062a1dc87923-trusted-ca\") pod \"image-registry-6f7bd956-kxk6z\" (UID: \"c4f988b4-1014-4229-ba21-062a1dc87923\") " pod="openshift-image-registry/image-registry-6f7bd956-kxk6z"
Jul 22 01:48:03.268289 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:03.268253    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 22 01:48:03.268289 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.268258    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41d01142-330d-46a2-8625-9e5b9ba1e2bf-tmp-dir\") pod \"dns-default-wkdlw\" (UID: \"41d01142-330d-46a2-8625-9e5b9ba1e2bf\") " pod="openshift-dns/dns-default-wkdlw"
Jul 22 01:48:03.268393 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:03.268326    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41d01142-330d-46a2-8625-9e5b9ba1e2bf-metrics-tls podName:41d01142-330d-46a2-8625-9e5b9ba1e2bf nodeName:}" failed. No retries permitted until 2026-07-22 01:48:03.768309334 +0000 UTC m=+36.612138460 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/41d01142-330d-46a2-8625-9e5b9ba1e2bf-metrics-tls") pod "dns-default-wkdlw" (UID: "41d01142-330d-46a2-8625-9e5b9ba1e2bf") : secret "dns-default-metrics-tls" not found
Jul 22 01:48:03.268689 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.268663    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/41d01142-330d-46a2-8625-9e5b9ba1e2bf-config-volume\") pod \"dns-default-wkdlw\" (UID: \"41d01142-330d-46a2-8625-9e5b9ba1e2bf\") " pod="openshift-dns/dns-default-wkdlw"
Jul 22 01:48:03.272458 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.272435    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c4f988b4-1014-4229-ba21-062a1dc87923-image-registry-private-configuration\") pod \"image-registry-6f7bd956-kxk6z\" (UID: \"c4f988b4-1014-4229-ba21-062a1dc87923\") " pod="openshift-image-registry/image-registry-6f7bd956-kxk6z"
Jul 22 01:48:03.272577 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.272438    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6cc54a54-1e35-414c-ac99-ce76586fe402-cert\") pod \"ingress-canary-mhjk2\" (UID: \"6cc54a54-1e35-414c-ac99-ce76586fe402\") " pod="openshift-ingress-canary/ingress-canary-mhjk2"
Jul 22 01:48:03.272577 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.272475    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c4f988b4-1014-4229-ba21-062a1dc87923-installation-pull-secrets\") pod \"image-registry-6f7bd956-kxk6z\" (UID: \"c4f988b4-1014-4229-ba21-062a1dc87923\") " pod="openshift-image-registry/image-registry-6f7bd956-kxk6z"
Jul 22 01:48:03.276247 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.276221    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b48r\" (UniqueName: \"kubernetes.io/projected/41d01142-330d-46a2-8625-9e5b9ba1e2bf-kube-api-access-5b48r\") pod \"dns-default-wkdlw\" (UID: \"41d01142-330d-46a2-8625-9e5b9ba1e2bf\") " pod="openshift-dns/dns-default-wkdlw"
Jul 22 01:48:03.276247 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.276242    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpbws\" (UniqueName: \"kubernetes.io/projected/6cc54a54-1e35-414c-ac99-ce76586fe402-kube-api-access-fpbws\") pod \"ingress-canary-mhjk2\" (UID: \"6cc54a54-1e35-414c-ac99-ce76586fe402\") " pod="openshift-ingress-canary/ingress-canary-mhjk2"
Jul 22 01:48:03.276444 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.276426    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4f988b4-1014-4229-ba21-062a1dc87923-bound-sa-token\") pod \"image-registry-6f7bd956-kxk6z\" (UID: \"c4f988b4-1014-4229-ba21-062a1dc87923\") " pod="openshift-image-registry/image-registry-6f7bd956-kxk6z"
Jul 22 01:48:03.276914 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.276898    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdsvh\" (UniqueName: \"kubernetes.io/projected/c4f988b4-1014-4229-ba21-062a1dc87923-kube-api-access-gdsvh\") pod \"image-registry-6f7bd956-kxk6z\" (UID: \"c4f988b4-1014-4229-ba21-062a1dc87923\") " pod="openshift-image-registry/image-registry-6f7bd956-kxk6z"
Jul 22 01:48:03.354427 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.354387    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mhjk2"
Jul 22 01:48:03.772487 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.772446    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4f988b4-1014-4229-ba21-062a1dc87923-registry-tls\") pod \"image-registry-6f7bd956-kxk6z\" (UID: \"c4f988b4-1014-4229-ba21-062a1dc87923\") " pod="openshift-image-registry/image-registry-6f7bd956-kxk6z"
Jul 22 01:48:03.773023 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:03.772511    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41d01142-330d-46a2-8625-9e5b9ba1e2bf-metrics-tls\") pod \"dns-default-wkdlw\" (UID: \"41d01142-330d-46a2-8625-9e5b9ba1e2bf\") " pod="openshift-dns/dns-default-wkdlw"
Jul 22 01:48:03.773023 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:03.772608    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 22 01:48:03.773023 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:03.772630    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f7bd956-kxk6z: secret "image-registry-tls" not found
Jul 22 01:48:03.773023 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:03.772664    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 22 01:48:03.773023 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:03.772691    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4f988b4-1014-4229-ba21-062a1dc87923-registry-tls podName:c4f988b4-1014-4229-ba21-062a1dc87923 nodeName:}" failed. No retries permitted until 2026-07-22 01:48:04.772670182 +0000 UTC m=+37.616499291 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c4f988b4-1014-4229-ba21-062a1dc87923-registry-tls") pod "image-registry-6f7bd956-kxk6z" (UID: "c4f988b4-1014-4229-ba21-062a1dc87923") : secret "image-registry-tls" not found
Jul 22 01:48:03.773023 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:03.772723    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41d01142-330d-46a2-8625-9e5b9ba1e2bf-metrics-tls podName:41d01142-330d-46a2-8625-9e5b9ba1e2bf nodeName:}" failed. No retries permitted until 2026-07-22 01:48:04.77270724 +0000 UTC m=+37.616536350 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/41d01142-330d-46a2-8625-9e5b9ba1e2bf-metrics-tls") pod "dns-default-wkdlw" (UID: "41d01142-330d-46a2-8625-9e5b9ba1e2bf") : secret "dns-default-metrics-tls" not found
Jul 22 01:48:04.125782 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:04.125625    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mhjk2"]
Jul 22 01:48:04.128826 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:48:04.128800    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cc54a54_1e35_414c_ac99_ce76586fe402.slice/crio-72ea44afb3c904ec3676ad8a54ee49af5d48fa46a451bb60c72c2e4f44423de4 WatchSource:0}: Error finding container 72ea44afb3c904ec3676ad8a54ee49af5d48fa46a451bb60c72c2e4f44423de4: Status 404 returned error can't find the container with id 72ea44afb3c904ec3676ad8a54ee49af5d48fa46a451bb60c72c2e4f44423de4
Jul 22 01:48:04.658028 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:04.657992    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mjkvg"
Jul 22 01:48:04.658254 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:04.658204    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fltbj"
Jul 22 01:48:04.658513 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:04.658449    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t7sb9"
Jul 22 01:48:04.660836 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:04.660811    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 22 01:48:04.661545 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:04.661518    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 22 01:48:04.661644 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:04.661547    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 22 01:48:04.661644 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:04.661590    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rnbhf\""
Jul 22 01:48:04.661752 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:04.661651    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-b6r65\""
Jul 22 01:48:04.661860 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:04.661844    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 22 01:48:04.779316 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:04.779283    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41d01142-330d-46a2-8625-9e5b9ba1e2bf-metrics-tls\") pod \"dns-default-wkdlw\" (UID: \"41d01142-330d-46a2-8625-9e5b9ba1e2bf\") " pod="openshift-dns/dns-default-wkdlw"
Jul 22 01:48:04.779808 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:04.779415    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4f988b4-1014-4229-ba21-062a1dc87923-registry-tls\") pod \"image-registry-6f7bd956-kxk6z\" (UID: \"c4f988b4-1014-4229-ba21-062a1dc87923\") " pod="openshift-image-registry/image-registry-6f7bd956-kxk6z"
Jul 22 01:48:04.779808 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:04.779539    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 22 01:48:04.779808 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:04.779555    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f7bd956-kxk6z: secret "image-registry-tls" not found
Jul 22 01:48:04.779808 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:04.779614    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4f988b4-1014-4229-ba21-062a1dc87923-registry-tls podName:c4f988b4-1014-4229-ba21-062a1dc87923 nodeName:}" failed. No retries permitted until 2026-07-22 01:48:06.77959443 +0000 UTC m=+39.623423560 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c4f988b4-1014-4229-ba21-062a1dc87923-registry-tls") pod "image-registry-6f7bd956-kxk6z" (UID: "c4f988b4-1014-4229-ba21-062a1dc87923") : secret "image-registry-tls" not found
Jul 22 01:48:04.780069 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:04.779984    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 22 01:48:04.780069 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:04.780038    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41d01142-330d-46a2-8625-9e5b9ba1e2bf-metrics-tls podName:41d01142-330d-46a2-8625-9e5b9ba1e2bf nodeName:}" failed. No retries permitted until 2026-07-22 01:48:06.780024849 +0000 UTC m=+39.623853972 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/41d01142-330d-46a2-8625-9e5b9ba1e2bf-metrics-tls") pod "dns-default-wkdlw" (UID: "41d01142-330d-46a2-8625-9e5b9ba1e2bf") : secret "dns-default-metrics-tls" not found
Jul 22 01:48:04.907134 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:04.907100    2563 generic.go:356] "Generic (PLEG): container finished" podID="c4290331-2538-4205-b3d0-58ab1a1126b6" containerID="ef6a5473dc2ccd2874e8c2f30f7b6b25e8bcaad580872c6629e752aadd3509cb" exitCode=0
Jul 22 01:48:04.907310 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:04.907203    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pbf4b" event={"ID":"c4290331-2538-4205-b3d0-58ab1a1126b6","Type":"ContainerDied","Data":"ef6a5473dc2ccd2874e8c2f30f7b6b25e8bcaad580872c6629e752aadd3509cb"}
Jul 22 01:48:04.908423 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:04.908344    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mhjk2" event={"ID":"6cc54a54-1e35-414c-ac99-ce76586fe402","Type":"ContainerStarted","Data":"72ea44afb3c904ec3676ad8a54ee49af5d48fa46a451bb60c72c2e4f44423de4"}
Jul 22 01:48:05.915128 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:05.915086    2563 generic.go:356] "Generic (PLEG): container finished" podID="c4290331-2538-4205-b3d0-58ab1a1126b6" containerID="65528a11eb0d11557a851d0e2a8468ad56246235786b9d90717150525a2a1ffd" exitCode=0
Jul 22 01:48:05.915588 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:05.915174    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pbf4b" event={"ID":"c4290331-2538-4205-b3d0-58ab1a1126b6","Type":"ContainerDied","Data":"65528a11eb0d11557a851d0e2a8468ad56246235786b9d90717150525a2a1ffd"}
Jul 22 01:48:06.799953 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:06.799772    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4f988b4-1014-4229-ba21-062a1dc87923-registry-tls\") pod \"image-registry-6f7bd956-kxk6z\" (UID: \"c4f988b4-1014-4229-ba21-062a1dc87923\") " pod="openshift-image-registry/image-registry-6f7bd956-kxk6z"
Jul 22 01:48:06.800091 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:06.799975    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41d01142-330d-46a2-8625-9e5b9ba1e2bf-metrics-tls\") pod \"dns-default-wkdlw\" (UID: \"41d01142-330d-46a2-8625-9e5b9ba1e2bf\") " pod="openshift-dns/dns-default-wkdlw"
Jul 22 01:48:06.800091 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:06.799930    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 22 01:48:06.800091 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:06.800038    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f7bd956-kxk6z: secret "image-registry-tls" not found
Jul 22 01:48:06.800091 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:06.800064    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 22 01:48:06.800224 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:06.800095    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4f988b4-1014-4229-ba21-062a1dc87923-registry-tls podName:c4f988b4-1014-4229-ba21-062a1dc87923 nodeName:}" failed. No retries permitted until 2026-07-22 01:48:10.800079744 +0000 UTC m=+43.643908854 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c4f988b4-1014-4229-ba21-062a1dc87923-registry-tls") pod "image-registry-6f7bd956-kxk6z" (UID: "c4f988b4-1014-4229-ba21-062a1dc87923") : secret "image-registry-tls" not found
Jul 22 01:48:06.800224 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:06.800107    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41d01142-330d-46a2-8625-9e5b9ba1e2bf-metrics-tls podName:41d01142-330d-46a2-8625-9e5b9ba1e2bf nodeName:}" failed. No retries permitted until 2026-07-22 01:48:10.800101024 +0000 UTC m=+43.643930133 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/41d01142-330d-46a2-8625-9e5b9ba1e2bf-metrics-tls") pod "dns-default-wkdlw" (UID: "41d01142-330d-46a2-8625-9e5b9ba1e2bf") : secret "dns-default-metrics-tls" not found
Jul 22 01:48:06.919039 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:06.918973    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mhjk2" event={"ID":"6cc54a54-1e35-414c-ac99-ce76586fe402","Type":"ContainerStarted","Data":"3c383ce1bbb85af659c1dc4de94f2ef83c930f2131fd0cbc377513a85c3c9e45"}
Jul 22 01:48:06.922021 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:06.921974    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pbf4b" event={"ID":"c4290331-2538-4205-b3d0-58ab1a1126b6","Type":"ContainerStarted","Data":"01f85a29e393b6ce2957e4417b427bd06552abe374e28881b9e5e1f58c368c62"}
Jul 22 01:48:06.934223 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:06.934169    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mhjk2" podStartSLOduration=2.682911003 podStartE2EDuration="4.934126212s" totalImagesPullingTime="2.251215209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:48:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:48:06.933441163 +0000 UTC m=+39.777270297" watchObservedRunningTime="2026-07-22 01:48:06.934126212 +0000 UTC m=+39.777955335"
Jul 22 01:48:06.953348 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:06.953301    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-pbf4b" podStartSLOduration=9.084604292 podStartE2EDuration="38.95328396s" totalImagesPullingTime="29.868679668s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:47:28 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:48:06.95209738 +0000 UTC m=+39.795926511" watchObservedRunningTime="2026-07-22 01:48:06.95328396 +0000 UTC m=+39.797113115"
Jul 22 01:48:09.693359 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:09.693317    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jrf2p"]
Jul 22 01:48:09.725807 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:09.725772    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jrf2p"]
Jul 22 01:48:09.725939 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:09.725886    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jrf2p"
Jul 22 01:48:09.728042 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:09.728010    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 22 01:48:09.728161 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:09.728069    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 22 01:48:09.728161 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:09.728118    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l2nw2\""
Jul 22 01:48:09.728984 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:09.728964    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 22 01:48:09.728984 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:09.728976    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 22 01:48:09.822907 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:09.822866    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3089a472-02e2-4cf4-994d-146d917dd72d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jrf2p\" (UID: \"3089a472-02e2-4cf4-994d-146d917dd72d\") " pod="openshift-insights/insights-runtime-extractor-jrf2p"
Jul 22 01:48:09.823096 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:09.822929    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3089a472-02e2-4cf4-994d-146d917dd72d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jrf2p\" (UID: \"3089a472-02e2-4cf4-994d-146d917dd72d\") " pod="openshift-insights/insights-runtime-extractor-jrf2p"
Jul 22 01:48:09.823096 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:09.822954    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3089a472-02e2-4cf4-994d-146d917dd72d-crio-socket\") pod \"insights-runtime-extractor-jrf2p\" (UID: \"3089a472-02e2-4cf4-994d-146d917dd72d\") " pod="openshift-insights/insights-runtime-extractor-jrf2p"
Jul 22 01:48:09.823096 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:09.823016    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3089a472-02e2-4cf4-994d-146d917dd72d-data-volume\") pod \"insights-runtime-extractor-jrf2p\" (UID: \"3089a472-02e2-4cf4-994d-146d917dd72d\") " pod="openshift-insights/insights-runtime-extractor-jrf2p"
Jul 22 01:48:09.823227 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:09.823094    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl8ck\" (UniqueName: \"kubernetes.io/projected/3089a472-02e2-4cf4-994d-146d917dd72d-kube-api-access-sl8ck\") pod \"insights-runtime-extractor-jrf2p\" (UID: \"3089a472-02e2-4cf4-994d-146d917dd72d\") " pod="openshift-insights/insights-runtime-extractor-jrf2p"
Jul 22 01:48:09.924333 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:09.924291    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3089a472-02e2-4cf4-994d-146d917dd72d-data-volume\") pod \"insights-runtime-extractor-jrf2p\" (UID: \"3089a472-02e2-4cf4-994d-146d917dd72d\") " pod="openshift-insights/insights-runtime-extractor-jrf2p"
Jul 22 01:48:09.924333 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:09.924332    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl8ck\" (UniqueName: \"kubernetes.io/projected/3089a472-02e2-4cf4-994d-146d917dd72d-kube-api-access-sl8ck\") pod \"insights-runtime-extractor-jrf2p\" (UID: \"3089a472-02e2-4cf4-994d-146d917dd72d\") " pod="openshift-insights/insights-runtime-extractor-jrf2p"
Jul 22 01:48:09.924547 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:09.924394    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3089a472-02e2-4cf4-994d-146d917dd72d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jrf2p\" (UID: \"3089a472-02e2-4cf4-994d-146d917dd72d\") " pod="openshift-insights/insights-runtime-extractor-jrf2p"
Jul 22 01:48:09.924547 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:09.924425    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3089a472-02e2-4cf4-994d-146d917dd72d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jrf2p\" (UID: \"3089a472-02e2-4cf4-994d-146d917dd72d\") " pod="openshift-insights/insights-runtime-extractor-jrf2p"
Jul 22 01:48:09.924547 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:09.924453    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3089a472-02e2-4cf4-994d-146d917dd72d-crio-socket\") pod \"insights-runtime-extractor-jrf2p\" (UID: \"3089a472-02e2-4cf4-994d-146d917dd72d\") " pod="openshift-insights/insights-runtime-extractor-jrf2p"
Jul 22 01:48:09.924547 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:09.924523    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 22 01:48:09.924677 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:09.924603    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3089a472-02e2-4cf4-994d-146d917dd72d-kube-rbac-proxy-cm podName:3089a472-02e2-4cf4-994d-146d917dd72d nodeName:}" failed. No retries permitted until 2026-07-22 01:48:10.424588075 +0000 UTC m=+43.268417197 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3089a472-02e2-4cf4-994d-146d917dd72d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jrf2p" (UID: "3089a472-02e2-4cf4-994d-146d917dd72d") : configmap "kube-rbac-proxy" not found
Jul 22 01:48:09.924677 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:09.924529    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 22 01:48:09.924677 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:09.924660    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3089a472-02e2-4cf4-994d-146d917dd72d-crio-socket\") pod \"insights-runtime-extractor-jrf2p\" (UID: \"3089a472-02e2-4cf4-994d-146d917dd72d\") " pod="openshift-insights/insights-runtime-extractor-jrf2p"
Jul 22 01:48:09.924786 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:09.924687    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3089a472-02e2-4cf4-994d-146d917dd72d-insights-runtime-extractor-tls podName:3089a472-02e2-4cf4-994d-146d917dd72d nodeName:}" failed. No retries permitted until 2026-07-22 01:48:10.424670477 +0000 UTC m=+43.268499596 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3089a472-02e2-4cf4-994d-146d917dd72d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jrf2p" (UID: "3089a472-02e2-4cf4-994d-146d917dd72d") : secret "insights-runtime-extractor-tls" not found
Jul 22 01:48:09.924786 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:09.924696    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3089a472-02e2-4cf4-994d-146d917dd72d-data-volume\") pod \"insights-runtime-extractor-jrf2p\" (UID: \"3089a472-02e2-4cf4-994d-146d917dd72d\") " pod="openshift-insights/insights-runtime-extractor-jrf2p"
Jul 22 01:48:09.934396 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:09.934346    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl8ck\" (UniqueName: \"kubernetes.io/projected/3089a472-02e2-4cf4-994d-146d917dd72d-kube-api-access-sl8ck\") pod \"insights-runtime-extractor-jrf2p\" (UID: \"3089a472-02e2-4cf4-994d-146d917dd72d\") " pod="openshift-insights/insights-runtime-extractor-jrf2p"
Jul 22 01:48:10.213206 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:10.213175    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-kqzt5"]
Jul 22 01:48:10.234915 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:10.234885    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-kqzt5"]
Jul 22 01:48:10.235060 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:10.234995    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-kqzt5"
Jul 22 01:48:10.237434 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:10.237405    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 22 01:48:10.237589 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:10.237448    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 22 01:48:10.237589 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:10.237448    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-tgwr9\""
Jul 22 01:48:10.428766 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:10.428727    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw9gc\" (UniqueName: \"kubernetes.io/projected/df76e48e-f4a0-487f-ab1b-0b8d64461913-kube-api-access-jw9gc\") pod \"migrator-7cc9c7ffd7-kqzt5\" (UID: \"df76e48e-f4a0-487f-ab1b-0b8d64461913\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-kqzt5"
Jul 22 01:48:10.428980 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:10.428801    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3089a472-02e2-4cf4-994d-146d917dd72d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jrf2p\" (UID: \"3089a472-02e2-4cf4-994d-146d917dd72d\") " pod="openshift-insights/insights-runtime-extractor-jrf2p"
Jul 22 01:48:10.428980 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:10.428838    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3089a472-02e2-4cf4-994d-146d917dd72d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jrf2p\" (UID: \"3089a472-02e2-4cf4-994d-146d917dd72d\") " pod="openshift-insights/insights-runtime-extractor-jrf2p"
Jul 22 01:48:10.428980 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:10.428914    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 22 01:48:10.428980 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:10.428949    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 22 01:48:10.428980 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:10.428974    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3089a472-02e2-4cf4-994d-146d917dd72d-kube-rbac-proxy-cm podName:3089a472-02e2-4cf4-994d-146d917dd72d nodeName:}" failed. No retries permitted until 2026-07-22 01:48:11.428960457 +0000 UTC m=+44.272789580 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3089a472-02e2-4cf4-994d-146d917dd72d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jrf2p" (UID: "3089a472-02e2-4cf4-994d-146d917dd72d") : configmap "kube-rbac-proxy" not found
Jul 22 01:48:10.429179 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:10.429017    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3089a472-02e2-4cf4-994d-146d917dd72d-insights-runtime-extractor-tls podName:3089a472-02e2-4cf4-994d-146d917dd72d nodeName:}" failed. No retries permitted until 2026-07-22 01:48:11.428998751 +0000 UTC m=+44.272827871 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3089a472-02e2-4cf4-994d-146d917dd72d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jrf2p" (UID: "3089a472-02e2-4cf4-994d-146d917dd72d") : secret "insights-runtime-extractor-tls" not found
Jul 22 01:48:10.529721 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:10.529629    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw9gc\" (UniqueName: \"kubernetes.io/projected/df76e48e-f4a0-487f-ab1b-0b8d64461913-kube-api-access-jw9gc\") pod \"migrator-7cc9c7ffd7-kqzt5\" (UID: \"df76e48e-f4a0-487f-ab1b-0b8d64461913\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-kqzt5"
Jul 22 01:48:10.540604 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:10.540580    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw9gc\" (UniqueName: \"kubernetes.io/projected/df76e48e-f4a0-487f-ab1b-0b8d64461913-kube-api-access-jw9gc\") pod \"migrator-7cc9c7ffd7-kqzt5\" (UID: \"df76e48e-f4a0-487f-ab1b-0b8d64461913\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-kqzt5"
Jul 22 01:48:10.543460 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:10.543439    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-kqzt5"
Jul 22 01:48:10.686448 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:10.686415    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-kqzt5"]
Jul 22 01:48:10.690112 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:48:10.690077    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf76e48e_f4a0_487f_ab1b_0b8d64461913.slice/crio-0df09035f0f7ec79b4d6b555dada301f504b413dcc6ff3a889ab42eba97e0561 WatchSource:0}: Error finding container 0df09035f0f7ec79b4d6b555dada301f504b413dcc6ff3a889ab42eba97e0561: Status 404 returned error can't find the container with id 0df09035f0f7ec79b4d6b555dada301f504b413dcc6ff3a889ab42eba97e0561
Jul 22 01:48:10.832424 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:10.832326    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4f988b4-1014-4229-ba21-062a1dc87923-registry-tls\") pod \"image-registry-6f7bd956-kxk6z\" (UID: \"c4f988b4-1014-4229-ba21-062a1dc87923\") " pod="openshift-image-registry/image-registry-6f7bd956-kxk6z"
Jul 22 01:48:10.832424 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:10.832402    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41d01142-330d-46a2-8625-9e5b9ba1e2bf-metrics-tls\") pod \"dns-default-wkdlw\" (UID: \"41d01142-330d-46a2-8625-9e5b9ba1e2bf\") " pod="openshift-dns/dns-default-wkdlw"
Jul 22 01:48:10.832813 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:10.832490    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 22 01:48:10.832813 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:10.832512    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f7bd956-kxk6z: secret "image-registry-tls" not found
Jul 22 01:48:10.832813 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:10.832538    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 22 01:48:10.832813 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:10.832570    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4f988b4-1014-4229-ba21-062a1dc87923-registry-tls podName:c4f988b4-1014-4229-ba21-062a1dc87923 nodeName:}" failed. No retries permitted until 2026-07-22 01:48:18.832553473 +0000 UTC m=+51.676382584 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c4f988b4-1014-4229-ba21-062a1dc87923-registry-tls") pod "image-registry-6f7bd956-kxk6z" (UID: "c4f988b4-1014-4229-ba21-062a1dc87923") : secret "image-registry-tls" not found
Jul 22 01:48:10.832813 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:10.832587    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41d01142-330d-46a2-8625-9e5b9ba1e2bf-metrics-tls podName:41d01142-330d-46a2-8625-9e5b9ba1e2bf nodeName:}" failed. No retries permitted until 2026-07-22 01:48:18.832575297 +0000 UTC m=+51.676404408 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/41d01142-330d-46a2-8625-9e5b9ba1e2bf-metrics-tls") pod "dns-default-wkdlw" (UID: "41d01142-330d-46a2-8625-9e5b9ba1e2bf") : secret "dns-default-metrics-tls" not found
Jul 22 01:48:10.931770 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:10.931737    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-kqzt5" event={"ID":"df76e48e-f4a0-487f-ab1b-0b8d64461913","Type":"ContainerStarted","Data":"0df09035f0f7ec79b4d6b555dada301f504b413dcc6ff3a889ab42eba97e0561"}
Jul 22 01:48:11.091805 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:11.091719    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4xkbt_08637867-465c-4df3-b707-ef1e33b64ed2/dns-node-resolver/0.log"
Jul 22 01:48:11.436844 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:11.436806    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3089a472-02e2-4cf4-994d-146d917dd72d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jrf2p\" (UID: \"3089a472-02e2-4cf4-994d-146d917dd72d\") " pod="openshift-insights/insights-runtime-extractor-jrf2p"
Jul 22 01:48:11.437027 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:11.436912    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3089a472-02e2-4cf4-994d-146d917dd72d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jrf2p\" (UID: \"3089a472-02e2-4cf4-994d-146d917dd72d\") " pod="openshift-insights/insights-runtime-extractor-jrf2p"
Jul 22 01:48:11.437027 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:11.436943    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 22 01:48:11.437027 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:11.437015    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3089a472-02e2-4cf4-994d-146d917dd72d-kube-rbac-proxy-cm podName:3089a472-02e2-4cf4-994d-146d917dd72d nodeName:}" failed. No retries permitted until 2026-07-22 01:48:13.437000383 +0000 UTC m=+46.280829493 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3089a472-02e2-4cf4-994d-146d917dd72d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jrf2p" (UID: "3089a472-02e2-4cf4-994d-146d917dd72d") : configmap "kube-rbac-proxy" not found
Jul 22 01:48:11.437146 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:11.437042    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 22 01:48:11.437146 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:11.437105    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3089a472-02e2-4cf4-994d-146d917dd72d-insights-runtime-extractor-tls podName:3089a472-02e2-4cf4-994d-146d917dd72d nodeName:}" failed. No retries permitted until 2026-07-22 01:48:13.437091649 +0000 UTC m=+46.280920778 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3089a472-02e2-4cf4-994d-146d917dd72d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jrf2p" (UID: "3089a472-02e2-4cf4-994d-146d917dd72d") : secret "insights-runtime-extractor-tls" not found
Jul 22 01:48:12.474270 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:12.474237    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-42lj8"]
Jul 22 01:48:12.487911 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:12.487881    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-42lj8"]
Jul 22 01:48:12.488014 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:12.487899    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-42lj8"
Jul 22 01:48:12.490256 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:12.490160    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-nswsw\""
Jul 22 01:48:12.491316 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:12.491298    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 22 01:48:12.491817 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:12.491654    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 22 01:48:12.491817 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:12.491659    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 22 01:48:12.491817 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:12.491724    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 22 01:48:12.493941 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:12.493924    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-drsr9_2393693b-9aef-40c6-ab3f-df34a7500ed8/node-ca/0.log"
Jul 22 01:48:12.647929 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:12.647843    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b86fm\" (UniqueName: \"kubernetes.io/projected/21ee66ba-898e-479e-9c1d-c36941aeb13a-kube-api-access-b86fm\") pod \"service-ca-58598695f7-42lj8\" (UID: \"21ee66ba-898e-479e-9c1d-c36941aeb13a\") " pod="openshift-service-ca/service-ca-58598695f7-42lj8"
Jul 22 01:48:12.647929 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:12.647910    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/21ee66ba-898e-479e-9c1d-c36941aeb13a-signing-cabundle\") pod \"service-ca-58598695f7-42lj8\" (UID: \"21ee66ba-898e-479e-9c1d-c36941aeb13a\") " pod="openshift-service-ca/service-ca-58598695f7-42lj8"
Jul 22 01:48:12.648117 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:12.648019    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/21ee66ba-898e-479e-9c1d-c36941aeb13a-signing-key\") pod \"service-ca-58598695f7-42lj8\" (UID: \"21ee66ba-898e-479e-9c1d-c36941aeb13a\") " pod="openshift-service-ca/service-ca-58598695f7-42lj8"
Jul 22 01:48:12.748837 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:12.748801    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/21ee66ba-898e-479e-9c1d-c36941aeb13a-signing-cabundle\") pod \"service-ca-58598695f7-42lj8\" (UID: \"21ee66ba-898e-479e-9c1d-c36941aeb13a\") " pod="openshift-service-ca/service-ca-58598695f7-42lj8"
Jul 22 01:48:12.749042 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:12.748865    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/21ee66ba-898e-479e-9c1d-c36941aeb13a-signing-key\") pod \"service-ca-58598695f7-42lj8\" (UID: \"21ee66ba-898e-479e-9c1d-c36941aeb13a\") " pod="openshift-service-ca/service-ca-58598695f7-42lj8"
Jul 22 01:48:12.749111 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:12.749051    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b86fm\" (UniqueName: \"kubernetes.io/projected/21ee66ba-898e-479e-9c1d-c36941aeb13a-kube-api-access-b86fm\") pod \"service-ca-58598695f7-42lj8\" (UID: \"21ee66ba-898e-479e-9c1d-c36941aeb13a\") " pod="openshift-service-ca/service-ca-58598695f7-42lj8"
Jul 22 01:48:12.749517 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:12.749491    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/21ee66ba-898e-479e-9c1d-c36941aeb13a-signing-cabundle\") pod \"service-ca-58598695f7-42lj8\" (UID: \"21ee66ba-898e-479e-9c1d-c36941aeb13a\") " pod="openshift-service-ca/service-ca-58598695f7-42lj8"
Jul 22 01:48:12.751354 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:12.751322    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/21ee66ba-898e-479e-9c1d-c36941aeb13a-signing-key\") pod \"service-ca-58598695f7-42lj8\" (UID: \"21ee66ba-898e-479e-9c1d-c36941aeb13a\") " pod="openshift-service-ca/service-ca-58598695f7-42lj8"
Jul 22 01:48:12.758353 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:12.758331    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b86fm\" (UniqueName: \"kubernetes.io/projected/21ee66ba-898e-479e-9c1d-c36941aeb13a-kube-api-access-b86fm\") pod \"service-ca-58598695f7-42lj8\" (UID: \"21ee66ba-898e-479e-9c1d-c36941aeb13a\") " pod="openshift-service-ca/service-ca-58598695f7-42lj8"
Jul 22 01:48:12.797482 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:12.797449    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-42lj8"
Jul 22 01:48:12.932768 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:12.932735    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-42lj8"]
Jul 22 01:48:12.936345 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:48:12.936319    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ee66ba_898e_479e_9c1d_c36941aeb13a.slice/crio-bf835125bf5af681253e4ce751db8c55f94b6cc2e8af2b4ded2be1a7fb362f98 WatchSource:0}: Error finding container bf835125bf5af681253e4ce751db8c55f94b6cc2e8af2b4ded2be1a7fb362f98: Status 404 returned error can't find the container with id bf835125bf5af681253e4ce751db8c55f94b6cc2e8af2b4ded2be1a7fb362f98
Jul 22 01:48:12.938258 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:12.938230    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-kqzt5" event={"ID":"df76e48e-f4a0-487f-ab1b-0b8d64461913","Type":"ContainerStarted","Data":"1a41d73151df61a5a53b1b609599bc477cf412cf52cb0d4e7f180796b25c8329"}
Jul 22 01:48:12.938337 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:12.938282    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-kqzt5" event={"ID":"df76e48e-f4a0-487f-ab1b-0b8d64461913","Type":"ContainerStarted","Data":"c87d9873dc0d9d813190418947c1820f14e2f8c161efe2cfb6b5cd8c36df5297"}
Jul 22 01:48:12.958421 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:12.958354    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-kqzt5" podStartSLOduration=1.251687943 podStartE2EDuration="2.958338802s" totalImagesPullingTime="1.706650859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:48:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:48:12.957037084 +0000 UTC m=+45.800866216" watchObservedRunningTime="2026-07-22 01:48:12.958338802 +0000 UTC m=+45.802167934"
Jul 22 01:48:13.093788 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:13.093731    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mhjk2_6cc54a54-1e35-414c-ac99-ce76586fe402/serve-healthcheck-canary/0.log"
Jul 22 01:48:13.454859 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:13.454826    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3089a472-02e2-4cf4-994d-146d917dd72d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jrf2p\" (UID: \"3089a472-02e2-4cf4-994d-146d917dd72d\") " pod="openshift-insights/insights-runtime-extractor-jrf2p"
Jul 22 01:48:13.455060 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:13.454877    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3089a472-02e2-4cf4-994d-146d917dd72d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jrf2p\" (UID: \"3089a472-02e2-4cf4-994d-146d917dd72d\") " pod="openshift-insights/insights-runtime-extractor-jrf2p"
Jul 22 01:48:13.455060 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:13.455027    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 22 01:48:13.455128 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:13.455089    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3089a472-02e2-4cf4-994d-146d917dd72d-insights-runtime-extractor-tls podName:3089a472-02e2-4cf4-994d-146d917dd72d nodeName:}" failed. No retries permitted until 2026-07-22 01:48:17.455075545 +0000 UTC m=+50.298904671 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3089a472-02e2-4cf4-994d-146d917dd72d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jrf2p" (UID: "3089a472-02e2-4cf4-994d-146d917dd72d") : secret "insights-runtime-extractor-tls" not found
Jul 22 01:48:13.455469 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:13.455452    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3089a472-02e2-4cf4-994d-146d917dd72d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jrf2p\" (UID: \"3089a472-02e2-4cf4-994d-146d917dd72d\") " pod="openshift-insights/insights-runtime-extractor-jrf2p"
Jul 22 01:48:13.943034 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:13.942986    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-42lj8" event={"ID":"21ee66ba-898e-479e-9c1d-c36941aeb13a","Type":"ContainerStarted","Data":"bf835125bf5af681253e4ce751db8c55f94b6cc2e8af2b4ded2be1a7fb362f98"}
Jul 22 01:48:15.949466 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:15.949424    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-42lj8" event={"ID":"21ee66ba-898e-479e-9c1d-c36941aeb13a","Type":"ContainerStarted","Data":"e6173c9415f5b4f37a66e392496128723cf86cebd02ab939c5d4eb1a09885a6e"}
Jul 22 01:48:15.967921 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:15.967863    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-42lj8" podStartSLOduration=1.5217357969999998 podStartE2EDuration="3.967845949s" totalImagesPullingTime="2.446110152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:48:15.967548421 +0000 UTC m=+48.811377553" watchObservedRunningTime="2026-07-22 01:48:15.967845949 +0000 UTC m=+48.811675084"
Jul 22 01:48:17.479591 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:17.479550    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3089a472-02e2-4cf4-994d-146d917dd72d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jrf2p\" (UID: \"3089a472-02e2-4cf4-994d-146d917dd72d\") " pod="openshift-insights/insights-runtime-extractor-jrf2p"
Jul 22 01:48:17.479981 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:17.479694    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 22 01:48:17.479981 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:17.479762    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3089a472-02e2-4cf4-994d-146d917dd72d-insights-runtime-extractor-tls podName:3089a472-02e2-4cf4-994d-146d917dd72d nodeName:}" failed. No retries permitted until 2026-07-22 01:48:25.479745116 +0000 UTC m=+58.323574229 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3089a472-02e2-4cf4-994d-146d917dd72d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jrf2p" (UID: "3089a472-02e2-4cf4-994d-146d917dd72d") : secret "insights-runtime-extractor-tls" not found
Jul 22 01:48:18.889690 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:18.889654    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41d01142-330d-46a2-8625-9e5b9ba1e2bf-metrics-tls\") pod \"dns-default-wkdlw\" (UID: \"41d01142-330d-46a2-8625-9e5b9ba1e2bf\") " pod="openshift-dns/dns-default-wkdlw"
Jul 22 01:48:18.890075 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:18.889763    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4f988b4-1014-4229-ba21-062a1dc87923-registry-tls\") pod \"image-registry-6f7bd956-kxk6z\" (UID: \"c4f988b4-1014-4229-ba21-062a1dc87923\") " pod="openshift-image-registry/image-registry-6f7bd956-kxk6z"
Jul 22 01:48:18.890075 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:18.889823    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 22 01:48:18.890075 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:18.889856    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 22 01:48:18.890075 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:18.889868    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f7bd956-kxk6z: secret "image-registry-tls" not found
Jul 22 01:48:18.890075 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:18.889894    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41d01142-330d-46a2-8625-9e5b9ba1e2bf-metrics-tls podName:41d01142-330d-46a2-8625-9e5b9ba1e2bf nodeName:}" failed. No retries permitted until 2026-07-22 01:48:34.889874346 +0000 UTC m=+67.733703479 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/41d01142-330d-46a2-8625-9e5b9ba1e2bf-metrics-tls") pod "dns-default-wkdlw" (UID: "41d01142-330d-46a2-8625-9e5b9ba1e2bf") : secret "dns-default-metrics-tls" not found
Jul 22 01:48:18.890075 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:18.889913    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4f988b4-1014-4229-ba21-062a1dc87923-registry-tls podName:c4f988b4-1014-4229-ba21-062a1dc87923 nodeName:}" failed. No retries permitted until 2026-07-22 01:48:34.88990456 +0000 UTC m=+67.733733669 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c4f988b4-1014-4229-ba21-062a1dc87923-registry-tls") pod "image-registry-6f7bd956-kxk6z" (UID: "c4f988b4-1014-4229-ba21-062a1dc87923") : secret "image-registry-tls" not found
Jul 22 01:48:25.534505 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:25.534465    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3089a472-02e2-4cf4-994d-146d917dd72d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jrf2p\" (UID: \"3089a472-02e2-4cf4-994d-146d917dd72d\") " pod="openshift-insights/insights-runtime-extractor-jrf2p"
Jul 22 01:48:25.536719 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:25.536694    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3089a472-02e2-4cf4-994d-146d917dd72d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jrf2p\" (UID: \"3089a472-02e2-4cf4-994d-146d917dd72d\") " pod="openshift-insights/insights-runtime-extractor-jrf2p"
Jul 22 01:48:25.634958 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:25.634936    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jrf2p"
Jul 22 01:48:25.760316 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:25.760288    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jrf2p"]
Jul 22 01:48:25.763999 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:48:25.763966    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3089a472_02e2_4cf4_994d_146d917dd72d.slice/crio-09f350f2b72a69286b399ad073e2841dcef5565fc3d3cd7007a05a29bbd87946 WatchSource:0}: Error finding container 09f350f2b72a69286b399ad073e2841dcef5565fc3d3cd7007a05a29bbd87946: Status 404 returned error can't find the container with id 09f350f2b72a69286b399ad073e2841dcef5565fc3d3cd7007a05a29bbd87946
Jul 22 01:48:25.978185 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:25.978156    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jrf2p" event={"ID":"3089a472-02e2-4cf4-994d-146d917dd72d","Type":"ContainerStarted","Data":"09f350f2b72a69286b399ad073e2841dcef5565fc3d3cd7007a05a29bbd87946"}
Jul 22 01:48:27.905072 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:27.905047    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bpxz8"
Jul 22 01:48:30.992192 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:30.992125    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jrf2p" event={"ID":"3089a472-02e2-4cf4-994d-146d917dd72d","Type":"ContainerStarted","Data":"65320a20b83792ab48f1841f8f2c01cdcc7365df9c5329791f3055fe74f6dd12"}
Jul 22 01:48:32.028224 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.028188    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f7bd956-kxk6z"]
Jul 22 01:48:32.028645 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:32.028390    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-6f7bd956-kxk6z" podUID="c4f988b4-1014-4229-ba21-062a1dc87923"
Jul 22 01:48:32.058663 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.058628    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-94ff6d9d8-4zdlm"]
Jul 22 01:48:32.061496 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.061475    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdd8f9466-6d58j"]
Jul 22 01:48:32.061620 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.061600    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-94ff6d9d8-4zdlm"
Jul 22 01:48:32.064127 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.064108    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 22 01:48:32.064387 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.064352    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 22 01:48:32.064495 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.064448    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 22 01:48:32.064562 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.064518    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-vxpll\""
Jul 22 01:48:32.064846 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.064830    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdd8f9466-6d58j"
Jul 22 01:48:32.066242 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.066225    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 22 01:48:32.069168 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.069136    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 22 01:48:32.069168 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.069158    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 22 01:48:32.069334 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.069165    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 22 01:48:32.069334 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.069136    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 22 01:48:32.074284 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.074263    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-94ff6d9d8-4zdlm"]
Jul 22 01:48:32.078654 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.078609    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdd8f9466-6d58j"]
Jul 22 01:48:32.082362 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.082341    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/37dea6a4-f522-4729-ac7f-3892ba6b3409-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-bdd8f9466-6d58j\" (UID: \"37dea6a4-f522-4729-ac7f-3892ba6b3409\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdd8f9466-6d58j"
Jul 22 01:48:32.082484 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.082468    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/37dea6a4-f522-4729-ac7f-3892ba6b3409-additional-service-ca\") pod \"cluster-proxy-proxy-agent-bdd8f9466-6d58j\" (UID: \"37dea6a4-f522-4729-ac7f-3892ba6b3409\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdd8f9466-6d58j"
Jul 22 01:48:32.082545 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.082508    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/37dea6a4-f522-4729-ac7f-3892ba6b3409-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-bdd8f9466-6d58j\" (UID: \"37dea6a4-f522-4729-ac7f-3892ba6b3409\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdd8f9466-6d58j"
Jul 22 01:48:32.082596 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.082550    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsj4w\" (UniqueName: \"kubernetes.io/projected/37dea6a4-f522-4729-ac7f-3892ba6b3409-kube-api-access-tsj4w\") pod \"cluster-proxy-proxy-agent-bdd8f9466-6d58j\" (UID: \"37dea6a4-f522-4729-ac7f-3892ba6b3409\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdd8f9466-6d58j"
Jul 22 01:48:32.082596 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.082581    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/37dea6a4-f522-4729-ac7f-3892ba6b3409-hub\") pod \"cluster-proxy-proxy-agent-bdd8f9466-6d58j\" (UID: \"37dea6a4-f522-4729-ac7f-3892ba6b3409\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdd8f9466-6d58j"
Jul 22 01:48:32.082666 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.082628    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2t6c\" (UniqueName: \"kubernetes.io/projected/9a57a52a-57b5-4435-bd9d-62334152a442-kube-api-access-j2t6c\") pod \"managed-serviceaccount-addon-agent-94ff6d9d8-4zdlm\" (UID: \"9a57a52a-57b5-4435-bd9d-62334152a442\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-94ff6d9d8-4zdlm"
Jul 22 01:48:32.082714 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.082694    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9a57a52a-57b5-4435-bd9d-62334152a442-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-94ff6d9d8-4zdlm\" (UID: \"9a57a52a-57b5-4435-bd9d-62334152a442\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-94ff6d9d8-4zdlm"
Jul 22 01:48:32.082816 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.082794    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/37dea6a4-f522-4729-ac7f-3892ba6b3409-ca\") pod \"cluster-proxy-proxy-agent-bdd8f9466-6d58j\" (UID: \"37dea6a4-f522-4729-ac7f-3892ba6b3409\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdd8f9466-6d58j"
Jul 22 01:48:32.158800 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.158772    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-9d8c7d849-kjrp7"]
Jul 22 01:48:32.163225 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.163202    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9d8c7d849-kjrp7"
Jul 22 01:48:32.170132 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.170106    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-9d8c7d849-kjrp7"]
Jul 22 01:48:32.184091 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.184071    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/56de72cc-8504-438a-adce-ea7a32dc8bfa-image-registry-private-configuration\") pod \"image-registry-9d8c7d849-kjrp7\" (UID: \"56de72cc-8504-438a-adce-ea7a32dc8bfa\") " pod="openshift-image-registry/image-registry-9d8c7d849-kjrp7"
Jul 22 01:48:32.184171 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.184098    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/56de72cc-8504-438a-adce-ea7a32dc8bfa-installation-pull-secrets\") pod \"image-registry-9d8c7d849-kjrp7\" (UID: \"56de72cc-8504-438a-adce-ea7a32dc8bfa\") " pod="openshift-image-registry/image-registry-9d8c7d849-kjrp7"
Jul 22 01:48:32.184171 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.184117    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/56de72cc-8504-438a-adce-ea7a32dc8bfa-bound-sa-token\") pod \"image-registry-9d8c7d849-kjrp7\" (UID: \"56de72cc-8504-438a-adce-ea7a32dc8bfa\") " pod="openshift-image-registry/image-registry-9d8c7d849-kjrp7"
Jul 22 01:48:32.184171 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.184142    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/37dea6a4-f522-4729-ac7f-3892ba6b3409-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-bdd8f9466-6d58j\" (UID: \"37dea6a4-f522-4729-ac7f-3892ba6b3409\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdd8f9466-6d58j"
Jul 22 01:48:32.184273 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.184195    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56de72cc-8504-438a-adce-ea7a32dc8bfa-registry-tls\") pod \"image-registry-9d8c7d849-kjrp7\" (UID: \"56de72cc-8504-438a-adce-ea7a32dc8bfa\") " pod="openshift-image-registry/image-registry-9d8c7d849-kjrp7"
Jul 22 01:48:32.184273 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.184240    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/37dea6a4-f522-4729-ac7f-3892ba6b3409-additional-service-ca\") pod \"cluster-proxy-proxy-agent-bdd8f9466-6d58j\" (UID: \"37dea6a4-f522-4729-ac7f-3892ba6b3409\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdd8f9466-6d58j"
Jul 22 01:48:32.184273 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.184264    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/37dea6a4-f522-4729-ac7f-3892ba6b3409-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-bdd8f9466-6d58j\" (UID: \"37dea6a4-f522-4729-ac7f-3892ba6b3409\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdd8f9466-6d58j"
Jul 22 01:48:32.184369 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.184285    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsj4w\" (UniqueName: \"kubernetes.io/projected/37dea6a4-f522-4729-ac7f-3892ba6b3409-kube-api-access-tsj4w\") pod \"cluster-proxy-proxy-agent-bdd8f9466-6d58j\" (UID: \"37dea6a4-f522-4729-ac7f-3892ba6b3409\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdd8f9466-6d58j"
Jul 22 01:48:32.184430 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.184415    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/37dea6a4-f522-4729-ac7f-3892ba6b3409-hub\") pod \"cluster-proxy-proxy-agent-bdd8f9466-6d58j\" (UID: \"37dea6a4-f522-4729-ac7f-3892ba6b3409\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdd8f9466-6d58j"
Jul 22 01:48:32.184476 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.184452    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dlfj\" (UniqueName: \"kubernetes.io/projected/56de72cc-8504-438a-adce-ea7a32dc8bfa-kube-api-access-6dlfj\") pod \"image-registry-9d8c7d849-kjrp7\" (UID: \"56de72cc-8504-438a-adce-ea7a32dc8bfa\") " pod="openshift-image-registry/image-registry-9d8c7d849-kjrp7"
Jul 22 01:48:32.184526 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.184504    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2t6c\" (UniqueName: \"kubernetes.io/projected/9a57a52a-57b5-4435-bd9d-62334152a442-kube-api-access-j2t6c\") pod \"managed-serviceaccount-addon-agent-94ff6d9d8-4zdlm\" (UID: \"9a57a52a-57b5-4435-bd9d-62334152a442\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-94ff6d9d8-4zdlm"
Jul 22 01:48:32.184577 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.184534    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9a57a52a-57b5-4435-bd9d-62334152a442-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-94ff6d9d8-4zdlm\" (UID: \"9a57a52a-57b5-4435-bd9d-62334152a442\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-94ff6d9d8-4zdlm"
Jul 22 01:48:32.184577 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.184562    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/56de72cc-8504-438a-adce-ea7a32dc8bfa-registry-certificates\") pod \"image-registry-9d8c7d849-kjrp7\" (UID: \"56de72cc-8504-438a-adce-ea7a32dc8bfa\") " pod="openshift-image-registry/image-registry-9d8c7d849-kjrp7"
Jul 22 01:48:32.184729 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.184605    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/56de72cc-8504-438a-adce-ea7a32dc8bfa-ca-trust-extracted\") pod \"image-registry-9d8c7d849-kjrp7\" (UID: \"56de72cc-8504-438a-adce-ea7a32dc8bfa\") " pod="openshift-image-registry/image-registry-9d8c7d849-kjrp7"
Jul 22 01:48:32.184729 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.184631    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/56de72cc-8504-438a-adce-ea7a32dc8bfa-trusted-ca\") pod \"image-registry-9d8c7d849-kjrp7\" (UID: \"56de72cc-8504-438a-adce-ea7a32dc8bfa\") " pod="openshift-image-registry/image-registry-9d8c7d849-kjrp7"
Jul 22 01:48:32.184729 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.184656    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/37dea6a4-f522-4729-ac7f-3892ba6b3409-ca\") pod \"cluster-proxy-proxy-agent-bdd8f9466-6d58j\" (UID: \"37dea6a4-f522-4729-ac7f-3892ba6b3409\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdd8f9466-6d58j"
Jul 22 01:48:32.184950 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.184930    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/37dea6a4-f522-4729-ac7f-3892ba6b3409-additional-service-ca\") pod \"cluster-proxy-proxy-agent-bdd8f9466-6d58j\" (UID: \"37dea6a4-f522-4729-ac7f-3892ba6b3409\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdd8f9466-6d58j"
Jul 22 01:48:32.186765 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.186740    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/37dea6a4-f522-4729-ac7f-3892ba6b3409-hub\") pod \"cluster-proxy-proxy-agent-bdd8f9466-6d58j\" (UID: \"37dea6a4-f522-4729-ac7f-3892ba6b3409\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdd8f9466-6d58j"
Jul 22 01:48:32.186960 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.186936    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/37dea6a4-f522-4729-ac7f-3892ba6b3409-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-bdd8f9466-6d58j\" (UID: \"37dea6a4-f522-4729-ac7f-3892ba6b3409\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdd8f9466-6d58j"
Jul 22 01:48:32.187157 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.187137    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/37dea6a4-f522-4729-ac7f-3892ba6b3409-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-bdd8f9466-6d58j\" (UID: \"37dea6a4-f522-4729-ac7f-3892ba6b3409\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdd8f9466-6d58j"
Jul 22 01:48:32.187226 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.187165    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9a57a52a-57b5-4435-bd9d-62334152a442-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-94ff6d9d8-4zdlm\" (UID: \"9a57a52a-57b5-4435-bd9d-62334152a442\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-94ff6d9d8-4zdlm"
Jul 22 01:48:32.187304 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.187288    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/37dea6a4-f522-4729-ac7f-3892ba6b3409-ca\") pod \"cluster-proxy-proxy-agent-bdd8f9466-6d58j\" (UID: \"37dea6a4-f522-4729-ac7f-3892ba6b3409\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdd8f9466-6d58j"
Jul 22 01:48:32.202732 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.202712    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsj4w\" (UniqueName: \"kubernetes.io/projected/37dea6a4-f522-4729-ac7f-3892ba6b3409-kube-api-access-tsj4w\") pod \"cluster-proxy-proxy-agent-bdd8f9466-6d58j\" (UID: \"37dea6a4-f522-4729-ac7f-3892ba6b3409\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdd8f9466-6d58j"
Jul 22 01:48:32.205398 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.205357    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2t6c\" (UniqueName: \"kubernetes.io/projected/9a57a52a-57b5-4435-bd9d-62334152a442-kube-api-access-j2t6c\") pod \"managed-serviceaccount-addon-agent-94ff6d9d8-4zdlm\" (UID: \"9a57a52a-57b5-4435-bd9d-62334152a442\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-94ff6d9d8-4zdlm"
Jul 22 01:48:32.285994 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.285922    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dlfj\" (UniqueName: \"kubernetes.io/projected/56de72cc-8504-438a-adce-ea7a32dc8bfa-kube-api-access-6dlfj\") pod \"image-registry-9d8c7d849-kjrp7\" (UID: \"56de72cc-8504-438a-adce-ea7a32dc8bfa\") " pod="openshift-image-registry/image-registry-9d8c7d849-kjrp7"
Jul 22 01:48:32.285994 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.285986    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/56de72cc-8504-438a-adce-ea7a32dc8bfa-registry-certificates\") pod \"image-registry-9d8c7d849-kjrp7\" (UID: \"56de72cc-8504-438a-adce-ea7a32dc8bfa\") " pod="openshift-image-registry/image-registry-9d8c7d849-kjrp7"
Jul 22 01:48:32.286183 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.286025    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/56de72cc-8504-438a-adce-ea7a32dc8bfa-ca-trust-extracted\") pod \"image-registry-9d8c7d849-kjrp7\" (UID: \"56de72cc-8504-438a-adce-ea7a32dc8bfa\") " pod="openshift-image-registry/image-registry-9d8c7d849-kjrp7"
Jul 22 01:48:32.286183 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.286053    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/56de72cc-8504-438a-adce-ea7a32dc8bfa-trusted-ca\") pod \"image-registry-9d8c7d849-kjrp7\" (UID: \"56de72cc-8504-438a-adce-ea7a32dc8bfa\") " pod="openshift-image-registry/image-registry-9d8c7d849-kjrp7"
Jul 22 01:48:32.286183 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.286114    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/56de72cc-8504-438a-adce-ea7a32dc8bfa-image-registry-private-configuration\") pod \"image-registry-9d8c7d849-kjrp7\" (UID: \"56de72cc-8504-438a-adce-ea7a32dc8bfa\") " pod="openshift-image-registry/image-registry-9d8c7d849-kjrp7"
Jul 22 01:48:32.286183 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.286141    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/56de72cc-8504-438a-adce-ea7a32dc8bfa-installation-pull-secrets\") pod \"image-registry-9d8c7d849-kjrp7\" (UID: \"56de72cc-8504-438a-adce-ea7a32dc8bfa\") " pod="openshift-image-registry/image-registry-9d8c7d849-kjrp7"
Jul 22 01:48:32.286466 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.286430    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/56de72cc-8504-438a-adce-ea7a32dc8bfa-ca-trust-extracted\") pod \"image-registry-9d8c7d849-kjrp7\" (UID: \"56de72cc-8504-438a-adce-ea7a32dc8bfa\") " pod="openshift-image-registry/image-registry-9d8c7d849-kjrp7"
Jul 22 01:48:32.286466 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.286425    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/56de72cc-8504-438a-adce-ea7a32dc8bfa-bound-sa-token\") pod \"image-registry-9d8c7d849-kjrp7\" (UID: \"56de72cc-8504-438a-adce-ea7a32dc8bfa\") " pod="openshift-image-registry/image-registry-9d8c7d849-kjrp7"
Jul 22 01:48:32.286618 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.286511    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56de72cc-8504-438a-adce-ea7a32dc8bfa-registry-tls\") pod \"image-registry-9d8c7d849-kjrp7\" (UID: \"56de72cc-8504-438a-adce-ea7a32dc8bfa\") " pod="openshift-image-registry/image-registry-9d8c7d849-kjrp7"
Jul 22 01:48:32.286934 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.286908    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/56de72cc-8504-438a-adce-ea7a32dc8bfa-registry-certificates\") pod \"image-registry-9d8c7d849-kjrp7\" (UID: \"56de72cc-8504-438a-adce-ea7a32dc8bfa\") " pod="openshift-image-registry/image-registry-9d8c7d849-kjrp7"
Jul 22 01:48:32.287089 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.287041    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/56de72cc-8504-438a-adce-ea7a32dc8bfa-trusted-ca\") pod \"image-registry-9d8c7d849-kjrp7\" (UID: \"56de72cc-8504-438a-adce-ea7a32dc8bfa\") " pod="openshift-image-registry/image-registry-9d8c7d849-kjrp7"
Jul 22 01:48:32.288653 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.288635    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/56de72cc-8504-438a-adce-ea7a32dc8bfa-installation-pull-secrets\") pod \"image-registry-9d8c7d849-kjrp7\" (UID: \"56de72cc-8504-438a-adce-ea7a32dc8bfa\") " pod="openshift-image-registry/image-registry-9d8c7d849-kjrp7"
Jul 22 01:48:32.288892 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.288870    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56de72cc-8504-438a-adce-ea7a32dc8bfa-registry-tls\") pod \"image-registry-9d8c7d849-kjrp7\" (UID: \"56de72cc-8504-438a-adce-ea7a32dc8bfa\") " pod="openshift-image-registry/image-registry-9d8c7d849-kjrp7"
Jul 22 01:48:32.289253 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.289233    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/56de72cc-8504-438a-adce-ea7a32dc8bfa-image-registry-private-configuration\") pod \"image-registry-9d8c7d849-kjrp7\" (UID: \"56de72cc-8504-438a-adce-ea7a32dc8bfa\") " pod="openshift-image-registry/image-registry-9d8c7d849-kjrp7"
Jul 22 01:48:32.296765 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.296743    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/56de72cc-8504-438a-adce-ea7a32dc8bfa-bound-sa-token\") pod \"image-registry-9d8c7d849-kjrp7\" (UID: \"56de72cc-8504-438a-adce-ea7a32dc8bfa\") " pod="openshift-image-registry/image-registry-9d8c7d849-kjrp7"
Jul 22 01:48:32.297175 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.297153    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dlfj\" (UniqueName: \"kubernetes.io/projected/56de72cc-8504-438a-adce-ea7a32dc8bfa-kube-api-access-6dlfj\") pod \"image-registry-9d8c7d849-kjrp7\" (UID: \"56de72cc-8504-438a-adce-ea7a32dc8bfa\") " pod="openshift-image-registry/image-registry-9d8c7d849-kjrp7"
Jul 22 01:48:32.382185 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.382162    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-94ff6d9d8-4zdlm"
Jul 22 01:48:32.387754 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.387729    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdd8f9466-6d58j"
Jul 22 01:48:32.483685 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.483658    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-c2smp\""
Jul 22 01:48:32.491890 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.491869    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9d8c7d849-kjrp7"
Jul 22 01:48:32.998109 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:32.998081    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f7bd956-kxk6z"
Jul 22 01:48:33.013875 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:33.013854    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f7bd956-kxk6z"
Jul 22 01:48:33.093923 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:33.093897    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4f988b4-1014-4229-ba21-062a1dc87923-bound-sa-token\") pod \"c4f988b4-1014-4229-ba21-062a1dc87923\" (UID: \"c4f988b4-1014-4229-ba21-062a1dc87923\") "
Jul 22 01:48:33.094333 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:33.093943    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c4f988b4-1014-4229-ba21-062a1dc87923-registry-certificates\") pod \"c4f988b4-1014-4229-ba21-062a1dc87923\" (UID: \"c4f988b4-1014-4229-ba21-062a1dc87923\") "
Jul 22 01:48:33.094333 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:33.094005    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c4f988b4-1014-4229-ba21-062a1dc87923-image-registry-private-configuration\") pod \"c4f988b4-1014-4229-ba21-062a1dc87923\" (UID: \"c4f988b4-1014-4229-ba21-062a1dc87923\") "
Jul 22 01:48:33.094333 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:33.094037    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c4f988b4-1014-4229-ba21-062a1dc87923-installation-pull-secrets\") pod \"c4f988b4-1014-4229-ba21-062a1dc87923\" (UID: \"c4f988b4-1014-4229-ba21-062a1dc87923\") "
Jul 22 01:48:33.094333 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:33.094068    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c4f988b4-1014-4229-ba21-062a1dc87923-ca-trust-extracted\") pod \"c4f988b4-1014-4229-ba21-062a1dc87923\" (UID: \"c4f988b4-1014-4229-ba21-062a1dc87923\") "
Jul 22 01:48:33.094333 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:33.094095    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c4f988b4-1014-4229-ba21-062a1dc87923-trusted-ca\") pod \"c4f988b4-1014-4229-ba21-062a1dc87923\" (UID: \"c4f988b4-1014-4229-ba21-062a1dc87923\") "
Jul 22 01:48:33.094333 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:33.094154    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdsvh\" (UniqueName: \"kubernetes.io/projected/c4f988b4-1014-4229-ba21-062a1dc87923-kube-api-access-gdsvh\") pod \"c4f988b4-1014-4229-ba21-062a1dc87923\" (UID: \"c4f988b4-1014-4229-ba21-062a1dc87923\") "
Jul 22 01:48:33.095079 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:33.094396    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4f988b4-1014-4229-ba21-062a1dc87923-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c4f988b4-1014-4229-ba21-062a1dc87923" (UID: "c4f988b4-1014-4229-ba21-062a1dc87923"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 01:48:33.095079 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:33.094481    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4f988b4-1014-4229-ba21-062a1dc87923-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c4f988b4-1014-4229-ba21-062a1dc87923" (UID: "c4f988b4-1014-4229-ba21-062a1dc87923"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 01:48:33.095079 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:33.094774    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4f988b4-1014-4229-ba21-062a1dc87923-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c4f988b4-1014-4229-ba21-062a1dc87923" (UID: "c4f988b4-1014-4229-ba21-062a1dc87923"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 01:48:33.096838 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:33.096807    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4f988b4-1014-4229-ba21-062a1dc87923-kube-api-access-gdsvh" (OuterVolumeSpecName: "kube-api-access-gdsvh") pod "c4f988b4-1014-4229-ba21-062a1dc87923" (UID: "c4f988b4-1014-4229-ba21-062a1dc87923"). InnerVolumeSpecName "kube-api-access-gdsvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 01:48:33.096937 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:33.096811    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4f988b4-1014-4229-ba21-062a1dc87923-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c4f988b4-1014-4229-ba21-062a1dc87923" (UID: "c4f988b4-1014-4229-ba21-062a1dc87923"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 01:48:33.096937 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:33.096898    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4f988b4-1014-4229-ba21-062a1dc87923-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c4f988b4-1014-4229-ba21-062a1dc87923" (UID: "c4f988b4-1014-4229-ba21-062a1dc87923"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 01:48:33.096937 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:33.096910    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4f988b4-1014-4229-ba21-062a1dc87923-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c4f988b4-1014-4229-ba21-062a1dc87923" (UID: "c4f988b4-1014-4229-ba21-062a1dc87923"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 01:48:33.195310 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:33.195287    2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c4f988b4-1014-4229-ba21-062a1dc87923-image-registry-private-configuration\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:48:33.195442 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:33.195314    2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c4f988b4-1014-4229-ba21-062a1dc87923-installation-pull-secrets\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:48:33.195442 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:33.195327    2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c4f988b4-1014-4229-ba21-062a1dc87923-ca-trust-extracted\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:48:33.195442 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:33.195339    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c4f988b4-1014-4229-ba21-062a1dc87923-trusted-ca\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:48:33.195442 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:33.195353    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdsvh\" (UniqueName: \"kubernetes.io/projected/c4f988b4-1014-4229-ba21-062a1dc87923-kube-api-access-gdsvh\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:48:33.195442 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:33.195366    2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4f988b4-1014-4229-ba21-062a1dc87923-bound-sa-token\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:48:33.195442 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:33.195398    2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c4f988b4-1014-4229-ba21-062a1dc87923-registry-certificates\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:48:33.498476 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:33.498440    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22905b02-8ed1-4c26-92ae-7282baf885cd-metrics-certs\") pod \"network-metrics-daemon-t7sb9\" (UID: \"22905b02-8ed1-4c26-92ae-7282baf885cd\") " pod="openshift-multus/network-metrics-daemon-t7sb9"
Jul 22 01:48:33.498635 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:33.498492    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2f67b074-15ce-4e39-afdc-5c30c8addc97-original-pull-secret\") pod \"global-pull-secret-syncer-mjkvg\" (UID: \"2f67b074-15ce-4e39-afdc-5c30c8addc97\") " pod="kube-system/global-pull-secret-syncer-mjkvg"
Jul 22 01:48:33.500670 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:33.500650    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 22 01:48:33.500775 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:33.500651    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 22 01:48:33.510856 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:33.510833    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2f67b074-15ce-4e39-afdc-5c30c8addc97-original-pull-secret\") pod \"global-pull-secret-syncer-mjkvg\" (UID: \"2f67b074-15ce-4e39-afdc-5c30c8addc97\") " pod="kube-system/global-pull-secret-syncer-mjkvg"
Jul 22 01:48:33.510856 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:33.510852    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22905b02-8ed1-4c26-92ae-7282baf885cd-metrics-certs\") pod \"network-metrics-daemon-t7sb9\" (UID: \"22905b02-8ed1-4c26-92ae-7282baf885cd\") " pod="openshift-multus/network-metrics-daemon-t7sb9"
Jul 22 01:48:33.599357 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:33.599321    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7gqd\" (UniqueName: \"kubernetes.io/projected/e6cd3d7c-b979-4d69-aad7-68ab39aed83d-kube-api-access-z7gqd\") pod \"network-check-target-fltbj\" (UID: \"e6cd3d7c-b979-4d69-aad7-68ab39aed83d\") " pod="openshift-network-diagnostics/network-check-target-fltbj"
Jul 22 01:48:33.601845 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:33.601810    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 22 01:48:33.611742 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:33.611717    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 22 01:48:33.623324 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:33.623301    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7gqd\" (UniqueName: \"kubernetes.io/projected/e6cd3d7c-b979-4d69-aad7-68ab39aed83d-kube-api-access-z7gqd\") pod \"network-check-target-fltbj\" (UID: \"e6cd3d7c-b979-4d69-aad7-68ab39aed83d\") " pod="openshift-network-diagnostics/network-check-target-fltbj"
Jul 22 01:48:33.769423 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:33.769355    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mjkvg"
Jul 22 01:48:33.778538 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:33.778516    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-b6r65\""
Jul 22 01:48:33.784674 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:33.784650    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rnbhf\""
Jul 22 01:48:33.787309 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:33.787293    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fltbj"
Jul 22 01:48:33.793172 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:33.793151    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t7sb9"
Jul 22 01:48:34.000481 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:34.000455    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f7bd956-kxk6z"
Jul 22 01:48:34.037068 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:34.037006    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f7bd956-kxk6z"]
Jul 22 01:48:34.040785 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:34.040761    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6f7bd956-kxk6z"]
Jul 22 01:48:34.103347 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:34.103300    2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4f988b4-1014-4229-ba21-062a1dc87923-registry-tls\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:48:34.258420 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:34.257800    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-94ff6d9d8-4zdlm"]
Jul 22 01:48:34.285836 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:34.285471    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-9d8c7d849-kjrp7"]
Jul 22 01:48:34.303765 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:48:34.300469    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56de72cc_8504_438a_adce_ea7a32dc8bfa.slice/crio-bd13b395321842ab678d79944b27bb9765bf6f6d6c8e38765d22eb212f494514 WatchSource:0}: Error finding container bd13b395321842ab678d79944b27bb9765bf6f6d6c8e38765d22eb212f494514: Status 404 returned error can't find the container with id bd13b395321842ab678d79944b27bb9765bf6f6d6c8e38765d22eb212f494514
Jul 22 01:48:34.303765 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:34.303373    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fltbj"]
Jul 22 01:48:34.548061 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:34.547981    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-t7sb9"]
Jul 22 01:48:34.549807 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:34.549784    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mjkvg"]
Jul 22 01:48:34.552144 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:48:34.552121    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22905b02_8ed1_4c26_92ae_7282baf885cd.slice/crio-ada8752ed904913ce828d0a715ce4eed102997bcc500b5044384988269949019 WatchSource:0}: Error finding container ada8752ed904913ce828d0a715ce4eed102997bcc500b5044384988269949019: Status 404 returned error can't find the container with id ada8752ed904913ce828d0a715ce4eed102997bcc500b5044384988269949019
Jul 22 01:48:34.552854 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:48:34.552809    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f67b074_15ce_4e39_afdc_5c30c8addc97.slice/crio-92b05c9337f354917e15a50be95adf913a070d02878c2f7a0e3b9c71e0c05648 WatchSource:0}: Error finding container 92b05c9337f354917e15a50be95adf913a070d02878c2f7a0e3b9c71e0c05648: Status 404 returned error can't find the container with id 92b05c9337f354917e15a50be95adf913a070d02878c2f7a0e3b9c71e0c05648
Jul 22 01:48:34.567051 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:34.567002    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdd8f9466-6d58j"]
Jul 22 01:48:34.570634 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:48:34.570607    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37dea6a4_f522_4729_ac7f_3892ba6b3409.slice/crio-2e63ea7441d1f4a6e78d99d722558bd591c627e2b0b2e225fac84b639c73e55b WatchSource:0}: Error finding container 2e63ea7441d1f4a6e78d99d722558bd591c627e2b0b2e225fac84b639c73e55b: Status 404 returned error can't find the container with id 2e63ea7441d1f4a6e78d99d722558bd591c627e2b0b2e225fac84b639c73e55b
Jul 22 01:48:34.912628 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:34.912529    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41d01142-330d-46a2-8625-9e5b9ba1e2bf-metrics-tls\") pod \"dns-default-wkdlw\" (UID: \"41d01142-330d-46a2-8625-9e5b9ba1e2bf\") " pod="openshift-dns/dns-default-wkdlw"
Jul 22 01:48:34.915478 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:34.915453    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41d01142-330d-46a2-8625-9e5b9ba1e2bf-metrics-tls\") pod \"dns-default-wkdlw\" (UID: \"41d01142-330d-46a2-8625-9e5b9ba1e2bf\") " pod="openshift-dns/dns-default-wkdlw"
Jul 22 01:48:35.007583 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:35.007546    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jrf2p" event={"ID":"3089a472-02e2-4cf4-994d-146d917dd72d","Type":"ContainerStarted","Data":"ef42e9e73fe5d680cfe3dfe754eeb7dd0263e4eeac921553917e048f4bb58da8"}
Jul 22 01:48:35.009210 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:35.009151    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t7sb9" event={"ID":"22905b02-8ed1-4c26-92ae-7282baf885cd","Type":"ContainerStarted","Data":"ada8752ed904913ce828d0a715ce4eed102997bcc500b5044384988269949019"}
Jul 22 01:48:35.010587 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:35.010558    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fltbj" event={"ID":"e6cd3d7c-b979-4d69-aad7-68ab39aed83d","Type":"ContainerStarted","Data":"2ef4bdd3a4c3c822fa8848a9639dc4a2bf553460871e5e3157fb23ceee4f4df6"}
Jul 22 01:48:35.012030 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:35.011976    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdd8f9466-6d58j" event={"ID":"37dea6a4-f522-4729-ac7f-3892ba6b3409","Type":"ContainerStarted","Data":"2e63ea7441d1f4a6e78d99d722558bd591c627e2b0b2e225fac84b639c73e55b"}
Jul 22 01:48:35.013796 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:35.013740    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-94ff6d9d8-4zdlm" event={"ID":"9a57a52a-57b5-4435-bd9d-62334152a442","Type":"ContainerStarted","Data":"a4ab5f900eb57cc299d795fc9ce60d92225cde8455acf8d4d40357fcb93fc91f"}
Jul 22 01:48:35.016006 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:35.015974    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9d8c7d849-kjrp7" event={"ID":"56de72cc-8504-438a-adce-ea7a32dc8bfa","Type":"ContainerStarted","Data":"38bfbf45ac57943240ab485620aca37f04164375a964b2fcec99fe8a4d4d2fe9"}
Jul 22 01:48:35.016108 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:35.016024    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9d8c7d849-kjrp7" event={"ID":"56de72cc-8504-438a-adce-ea7a32dc8bfa","Type":"ContainerStarted","Data":"bd13b395321842ab678d79944b27bb9765bf6f6d6c8e38765d22eb212f494514"}
Jul 22 01:48:35.016392 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:35.016326    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-9d8c7d849-kjrp7"
Jul 22 01:48:35.017839 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:35.017807    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mjkvg" event={"ID":"2f67b074-15ce-4e39-afdc-5c30c8addc97","Type":"ContainerStarted","Data":"92b05c9337f354917e15a50be95adf913a070d02878c2f7a0e3b9c71e0c05648"}
Jul 22 01:48:35.034665 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:35.034599    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-9d8c7d849-kjrp7" podStartSLOduration=3.034580283 podStartE2EDuration="3.034580283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:48:35.033689329 +0000 UTC m=+67.877518463" watchObservedRunningTime="2026-07-22 01:48:35.034580283 +0000 UTC m=+67.878409417"
Jul 22 01:48:35.128306 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:35.128282    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hpqgw\""
Jul 22 01:48:35.137037 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:35.137016    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wkdlw"
Jul 22 01:48:35.325489 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:35.325415    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wkdlw"]
Jul 22 01:48:35.338986 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:48:35.338946    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41d01142_330d_46a2_8625_9e5b9ba1e2bf.slice/crio-2f5af21b73c35f3e472ff778f62fc6c76b514ec58becea1f67f81647ac27addd WatchSource:0}: Error finding container 2f5af21b73c35f3e472ff778f62fc6c76b514ec58becea1f67f81647ac27addd: Status 404 returned error can't find the container with id 2f5af21b73c35f3e472ff778f62fc6c76b514ec58becea1f67f81647ac27addd
Jul 22 01:48:35.663302 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:35.663257    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4f988b4-1014-4229-ba21-062a1dc87923" path="/var/lib/kubelet/pods/c4f988b4-1014-4229-ba21-062a1dc87923/volumes"
Jul 22 01:48:36.028665 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:36.028594    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wkdlw" event={"ID":"41d01142-330d-46a2-8625-9e5b9ba1e2bf","Type":"ContainerStarted","Data":"2f5af21b73c35f3e472ff778f62fc6c76b514ec58becea1f67f81647ac27addd"}
Jul 22 01:48:37.038862 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:37.038042    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t7sb9" event={"ID":"22905b02-8ed1-4c26-92ae-7282baf885cd","Type":"ContainerStarted","Data":"54b3d797046bf786dc8f6486dcd54158af9fc75560a738a5441ba8fe73d42740"}
Jul 22 01:48:37.038862 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:37.038145    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t7sb9" event={"ID":"22905b02-8ed1-4c26-92ae-7282baf885cd","Type":"ContainerStarted","Data":"cd81d9648f8b52a473cad3fc661c4fdac05d10496cbf36b74cbc75e63ef9dc29"}
Jul 22 01:48:37.063090 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:37.062591    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-t7sb9" podStartSLOduration=67.282305997 podStartE2EDuration="1m9.062540372s" totalImagesPullingTime="1.780234375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:48:37.061017082 +0000 UTC m=+69.904846212" watchObservedRunningTime="2026-07-22 01:48:37.062540372 +0000 UTC m=+69.906369509"
Jul 22 01:48:45.069789 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:45.069750    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jrf2p" event={"ID":"3089a472-02e2-4cf4-994d-146d917dd72d","Type":"ContainerStarted","Data":"c0c3cf54d691f1ff50d23fec266e540a8be9c26cae93d22d5ab722a91de4894b"}
Jul 22 01:48:45.071327 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:45.071304    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fltbj" event={"ID":"e6cd3d7c-b979-4d69-aad7-68ab39aed83d","Type":"ContainerStarted","Data":"a5363ea53168e7e495f8885c673564f6396a4583765e56316403bca53a52bda8"}
Jul 22 01:48:45.071452 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:45.071349    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fltbj"
Jul 22 01:48:45.073367 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:45.073345    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdd8f9466-6d58j" event={"ID":"37dea6a4-f522-4729-ac7f-3892ba6b3409","Type":"ContainerStarted","Data":"d93c00fcfae75af0f77e3316b35a874547c70926e9e0dc7d0d8b5550540036bd"}
Jul 22 01:48:45.073478 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:45.073395    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdd8f9466-6d58j" event={"ID":"37dea6a4-f522-4729-ac7f-3892ba6b3409","Type":"ContainerStarted","Data":"5f514577ec684a3125c165d6edffc5ac562748028347d5e8f6480e8664b66609"}
Jul 22 01:48:45.073478 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:45.073410    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdd8f9466-6d58j" event={"ID":"37dea6a4-f522-4729-ac7f-3892ba6b3409","Type":"ContainerStarted","Data":"ee515955b0f57118f2612a2ac02241fa327b2bd53b07e388757aa28171281a61"}
Jul 22 01:48:45.074867 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:45.074845    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-94ff6d9d8-4zdlm" event={"ID":"9a57a52a-57b5-4435-bd9d-62334152a442","Type":"ContainerStarted","Data":"c9a72afddf10b2fe111ebd9b92972beabf081077d0aae8578b1496b8a4cdc587"}
Jul 22 01:48:45.076514 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:45.076492    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wkdlw" event={"ID":"41d01142-330d-46a2-8625-9e5b9ba1e2bf","Type":"ContainerStarted","Data":"fd5f50632fe38ef3632ea9c606340dcc18a99c1dfbc4feca8f9b595f81794a3d"}
Jul 22 01:48:45.076629 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:45.076543    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wkdlw"
Jul 22 01:48:45.076629 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:45.076567    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wkdlw" event={"ID":"41d01142-330d-46a2-8625-9e5b9ba1e2bf","Type":"ContainerStarted","Data":"5c7d948e3c3feb41e5aba92afd2d46f889f3fb28ae5c9adbe412e1d08bfd76a1"}
Jul 22 01:48:45.077814 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:45.077795    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mjkvg" event={"ID":"2f67b074-15ce-4e39-afdc-5c30c8addc97","Type":"ContainerStarted","Data":"3b7bd9f8ae90d4cfa67e757fdcc7ab4de9a6976a3ab569bad3d8e13bcc7a448f"}
Jul 22 01:48:45.088306 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:45.088266    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jrf2p" podStartSLOduration=17.690834061 podStartE2EDuration="36.088252978s" totalImagesPullingTime="18.397418917s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:48:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:48:45.08665186 +0000 UTC m=+77.930480993" watchObservedRunningTime="2026-07-22 01:48:45.088252978 +0000 UTC m=+77.932082110"
Jul 22 01:48:45.101460 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:45.101420    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-94ff6d9d8-4zdlm" podStartSLOduration=3.077022313 podStartE2EDuration="13.101408932s" totalImagesPullingTime="10.024386619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:48:45.100499528 +0000 UTC m=+77.944328659" watchObservedRunningTime="2026-07-22 01:48:45.101408932 +0000 UTC m=+77.945238079"
Jul 22 01:48:45.136154 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:45.136106    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fltbj" podStartSLOduration=67.12497921 podStartE2EDuration="1m17.136092052s" totalImagesPullingTime="10.011112842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:48:45.135482438 +0000 UTC m=+77.979311570" watchObservedRunningTime="2026-07-22 01:48:45.136092052 +0000 UTC m=+77.979921184"
Jul 22 01:48:45.136298 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:45.136239    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bdd8f9466-6d58j" podStartSLOduration=3.369347485 podStartE2EDuration="13.136232213s" totalImagesPullingTime="9.766884728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:48:45.121338291 +0000 UTC m=+77.965167425" watchObservedRunningTime="2026-07-22 01:48:45.136232213 +0000 UTC m=+77.980061345"
Jul 22 01:48:45.150747 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:45.150698    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mjkvg" podStartSLOduration=67.369638575 podStartE2EDuration="1m17.150683165s" totalImagesPullingTime="9.78104459s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:48:45.150463265 +0000 UTC m=+77.994292398" watchObservedRunningTime="2026-07-22 01:48:45.150683165 +0000 UTC m=+77.994512296"
Jul 22 01:48:45.168020 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:45.167967    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-wkdlw" podStartSLOduration=34.191056671 podStartE2EDuration="43.167950887s" totalImagesPullingTime="8.976894216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:48:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:48:45.167614705 +0000 UTC m=+78.011443838" watchObservedRunningTime="2026-07-22 01:48:45.167950887 +0000 UTC m=+78.011780018"
Jul 22 01:48:52.497056 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:52.496934    2563 patch_prober.go:28] interesting pod/image-registry-9d8c7d849-kjrp7 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 22 01:48:52.497056 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:52.497006    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-9d8c7d849-kjrp7" podUID="56de72cc-8504-438a-adce-ea7a32dc8bfa" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 22 01:48:55.084241 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:55.084205    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wkdlw"
Jul 22 01:48:56.032875 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:56.032847    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-9d8c7d849-kjrp7"
Jul 22 01:48:57.159653 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:57.159616    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8lmr8"]
Jul 22 01:48:57.165557 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:57.165530    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8lmr8"
Jul 22 01:48:57.167941 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:57.167916    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 22 01:48:57.168195 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:57.168170    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 22 01:48:57.168302 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:57.168210    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 22 01:48:57.168302 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:57.168251    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xs2gv\""
Jul 22 01:48:57.168426 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:57.168308    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 22 01:48:57.168850 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:57.168823    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 22 01:48:57.168942 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:57.168882    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 22 01:48:57.287624 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:57.287593    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14610eac-7fab-4d74-ac5d-10ea5d09716b-metrics-client-ca\") pod \"node-exporter-8lmr8\" (UID: \"14610eac-7fab-4d74-ac5d-10ea5d09716b\") " pod="openshift-monitoring/node-exporter-8lmr8"
Jul 22 01:48:57.287624 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:57.287628    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/14610eac-7fab-4d74-ac5d-10ea5d09716b-root\") pod \"node-exporter-8lmr8\" (UID: \"14610eac-7fab-4d74-ac5d-10ea5d09716b\") " pod="openshift-monitoring/node-exporter-8lmr8"
Jul 22 01:48:57.287828 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:57.287646    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/14610eac-7fab-4d74-ac5d-10ea5d09716b-node-exporter-wtmp\") pod \"node-exporter-8lmr8\" (UID: \"14610eac-7fab-4d74-ac5d-10ea5d09716b\") " pod="openshift-monitoring/node-exporter-8lmr8"
Jul 22 01:48:57.287828 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:57.287674    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14610eac-7fab-4d74-ac5d-10ea5d09716b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8lmr8\" (UID: \"14610eac-7fab-4d74-ac5d-10ea5d09716b\") " pod="openshift-monitoring/node-exporter-8lmr8"
Jul 22 01:48:57.287828 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:57.287714    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/14610eac-7fab-4d74-ac5d-10ea5d09716b-node-exporter-textfile\") pod \"node-exporter-8lmr8\" (UID: \"14610eac-7fab-4d74-ac5d-10ea5d09716b\") " pod="openshift-monitoring/node-exporter-8lmr8"
Jul 22 01:48:57.287828 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:57.287730    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qf7l\" (UniqueName: \"kubernetes.io/projected/14610eac-7fab-4d74-ac5d-10ea5d09716b-kube-api-access-4qf7l\") pod \"node-exporter-8lmr8\" (UID: \"14610eac-7fab-4d74-ac5d-10ea5d09716b\") " pod="openshift-monitoring/node-exporter-8lmr8"
Jul 22 01:48:57.287828 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:57.287744    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14610eac-7fab-4d74-ac5d-10ea5d09716b-sys\") pod \"node-exporter-8lmr8\" (UID: \"14610eac-7fab-4d74-ac5d-10ea5d09716b\") " pod="openshift-monitoring/node-exporter-8lmr8"
Jul 22 01:48:57.287828 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:57.287803    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/14610eac-7fab-4d74-ac5d-10ea5d09716b-node-exporter-tls\") pod \"node-exporter-8lmr8\" (UID: \"14610eac-7fab-4d74-ac5d-10ea5d09716b\") " pod="openshift-monitoring/node-exporter-8lmr8"
Jul 22 01:48:57.287828 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:57.287824    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/14610eac-7fab-4d74-ac5d-10ea5d09716b-node-exporter-accelerators-collector-config\") pod \"node-exporter-8lmr8\" (UID: \"14610eac-7fab-4d74-ac5d-10ea5d09716b\") " pod="openshift-monitoring/node-exporter-8lmr8"
Jul 22 01:48:57.389007 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:57.388964    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14610eac-7fab-4d74-ac5d-10ea5d09716b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8lmr8\" (UID: \"14610eac-7fab-4d74-ac5d-10ea5d09716b\") " pod="openshift-monitoring/node-exporter-8lmr8"
Jul 22 01:48:57.389007 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:57.389010    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/14610eac-7fab-4d74-ac5d-10ea5d09716b-node-exporter-textfile\") pod \"node-exporter-8lmr8\" (UID: \"14610eac-7fab-4d74-ac5d-10ea5d09716b\") " pod="openshift-monitoring/node-exporter-8lmr8"
Jul 22 01:48:57.389255 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:57.389037    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qf7l\" (UniqueName: \"kubernetes.io/projected/14610eac-7fab-4d74-ac5d-10ea5d09716b-kube-api-access-4qf7l\") pod \"node-exporter-8lmr8\" (UID: \"14610eac-7fab-4d74-ac5d-10ea5d09716b\") " pod="openshift-monitoring/node-exporter-8lmr8"
Jul 22 01:48:57.389255 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:57.389151    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14610eac-7fab-4d74-ac5d-10ea5d09716b-sys\") pod \"node-exporter-8lmr8\" (UID: \"14610eac-7fab-4d74-ac5d-10ea5d09716b\") " pod="openshift-monitoring/node-exporter-8lmr8"
Jul 22 01:48:57.389255 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:57.389212    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/14610eac-7fab-4d74-ac5d-10ea5d09716b-node-exporter-tls\") pod \"node-exporter-8lmr8\" (UID: \"14610eac-7fab-4d74-ac5d-10ea5d09716b\") " pod="openshift-monitoring/node-exporter-8lmr8"
Jul 22 01:48:57.389255 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:57.389239    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/14610eac-7fab-4d74-ac5d-10ea5d09716b-node-exporter-accelerators-collector-config\") pod \"node-exporter-8lmr8\" (UID: \"14610eac-7fab-4d74-ac5d-10ea5d09716b\") " pod="openshift-monitoring/node-exporter-8lmr8"
Jul 22 01:48:57.389478 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:57.389255    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14610eac-7fab-4d74-ac5d-10ea5d09716b-sys\") pod \"node-exporter-8lmr8\" (UID: \"14610eac-7fab-4d74-ac5d-10ea5d09716b\") " pod="openshift-monitoring/node-exporter-8lmr8"
Jul 22 01:48:57.389478 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:57.389318    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14610eac-7fab-4d74-ac5d-10ea5d09716b-metrics-client-ca\") pod \"node-exporter-8lmr8\" (UID: \"14610eac-7fab-4d74-ac5d-10ea5d09716b\") " pod="openshift-monitoring/node-exporter-8lmr8"
Jul 22 01:48:57.389478 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:57.389406    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/14610eac-7fab-4d74-ac5d-10ea5d09716b-root\") pod \"node-exporter-8lmr8\" (UID: \"14610eac-7fab-4d74-ac5d-10ea5d09716b\") " pod="openshift-monitoring/node-exporter-8lmr8"
Jul 22 01:48:57.389478 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:57.389420    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/14610eac-7fab-4d74-ac5d-10ea5d09716b-node-exporter-textfile\") pod \"node-exporter-8lmr8\" (UID: \"14610eac-7fab-4d74-ac5d-10ea5d09716b\") " pod="openshift-monitoring/node-exporter-8lmr8"
Jul 22 01:48:57.389478 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:57.389442    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/14610eac-7fab-4d74-ac5d-10ea5d09716b-node-exporter-wtmp\") pod \"node-exporter-8lmr8\" (UID: \"14610eac-7fab-4d74-ac5d-10ea5d09716b\") " pod="openshift-monitoring/node-exporter-8lmr8"
Jul 22 01:48:57.389705 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:57.389487    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/14610eac-7fab-4d74-ac5d-10ea5d09716b-root\") pod \"node-exporter-8lmr8\" (UID: \"14610eac-7fab-4d74-ac5d-10ea5d09716b\") " pod="openshift-monitoring/node-exporter-8lmr8"
Jul 22 01:48:57.389705 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:57.389566    2563 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 22 01:48:57.389705 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:57.389599    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/14610eac-7fab-4d74-ac5d-10ea5d09716b-node-exporter-wtmp\") pod \"node-exporter-8lmr8\" (UID: \"14610eac-7fab-4d74-ac5d-10ea5d09716b\") " pod="openshift-monitoring/node-exporter-8lmr8"
Jul 22 01:48:57.389705 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:48:57.389633    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14610eac-7fab-4d74-ac5d-10ea5d09716b-node-exporter-tls podName:14610eac-7fab-4d74-ac5d-10ea5d09716b nodeName:}" failed. No retries permitted until 2026-07-22 01:48:57.88961419 +0000 UTC m=+90.733443303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/14610eac-7fab-4d74-ac5d-10ea5d09716b-node-exporter-tls") pod "node-exporter-8lmr8" (UID: "14610eac-7fab-4d74-ac5d-10ea5d09716b") : secret "node-exporter-tls" not found
Jul 22 01:48:57.389955 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:57.389934    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/14610eac-7fab-4d74-ac5d-10ea5d09716b-node-exporter-accelerators-collector-config\") pod \"node-exporter-8lmr8\" (UID: \"14610eac-7fab-4d74-ac5d-10ea5d09716b\") " pod="openshift-monitoring/node-exporter-8lmr8"
Jul 22 01:48:57.389991 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:57.389938    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14610eac-7fab-4d74-ac5d-10ea5d09716b-metrics-client-ca\") pod \"node-exporter-8lmr8\" (UID: \"14610eac-7fab-4d74-ac5d-10ea5d09716b\") " pod="openshift-monitoring/node-exporter-8lmr8"
Jul 22 01:48:57.391480 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:57.391464    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14610eac-7fab-4d74-ac5d-10ea5d09716b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8lmr8\" (UID: \"14610eac-7fab-4d74-ac5d-10ea5d09716b\") " pod="openshift-monitoring/node-exporter-8lmr8"
Jul 22 01:48:57.397318 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:57.397296    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qf7l\" (UniqueName: \"kubernetes.io/projected/14610eac-7fab-4d74-ac5d-10ea5d09716b-kube-api-access-4qf7l\") pod \"node-exporter-8lmr8\" (UID: \"14610eac-7fab-4d74-ac5d-10ea5d09716b\") " pod="openshift-monitoring/node-exporter-8lmr8"
Jul 22 01:48:57.894290 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:57.894247    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/14610eac-7fab-4d74-ac5d-10ea5d09716b-node-exporter-tls\") pod \"node-exporter-8lmr8\" (UID: \"14610eac-7fab-4d74-ac5d-10ea5d09716b\") " pod="openshift-monitoring/node-exporter-8lmr8"
Jul 22 01:48:57.896350 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:57.896322    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/14610eac-7fab-4d74-ac5d-10ea5d09716b-node-exporter-tls\") pod \"node-exporter-8lmr8\" (UID: \"14610eac-7fab-4d74-ac5d-10ea5d09716b\") " pod="openshift-monitoring/node-exporter-8lmr8"
Jul 22 01:48:58.076557 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:58.076522    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8lmr8"
Jul 22 01:48:58.090426 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:48:58.090400    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14610eac_7fab_4d74_ac5d_10ea5d09716b.slice/crio-884547506a946a86dcd93b823a4d6cb84b689b4fdba5992b22c3f89530877ef0 WatchSource:0}: Error finding container 884547506a946a86dcd93b823a4d6cb84b689b4fdba5992b22c3f89530877ef0: Status 404 returned error can't find the container with id 884547506a946a86dcd93b823a4d6cb84b689b4fdba5992b22c3f89530877ef0
Jul 22 01:48:58.127689 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:58.127659    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8lmr8" event={"ID":"14610eac-7fab-4d74-ac5d-10ea5d09716b","Type":"ContainerStarted","Data":"884547506a946a86dcd93b823a4d6cb84b689b4fdba5992b22c3f89530877ef0"}
Jul 22 01:48:59.132148 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:59.132119    2563 generic.go:356] "Generic (PLEG): container finished" podID="14610eac-7fab-4d74-ac5d-10ea5d09716b" containerID="5b3c5186f86afcf92ded7a2d90e70438913343ee67b4313bcf5289a90ec71bea" exitCode=0
Jul 22 01:48:59.132534 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:48:59.132167    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8lmr8" event={"ID":"14610eac-7fab-4d74-ac5d-10ea5d09716b","Type":"ContainerDied","Data":"5b3c5186f86afcf92ded7a2d90e70438913343ee67b4313bcf5289a90ec71bea"}
Jul 22 01:49:00.139784 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:49:00.139743    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8lmr8" event={"ID":"14610eac-7fab-4d74-ac5d-10ea5d09716b","Type":"ContainerStarted","Data":"0ab9474fd924fa304437cf3aa54d5f1cc2ac7993a08795bcd0ce0ef5210bfd17"}
Jul 22 01:49:00.140237 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:49:00.139810    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8lmr8" event={"ID":"14610eac-7fab-4d74-ac5d-10ea5d09716b","Type":"ContainerStarted","Data":"2086770c7e0d7d65a0cfc7bc48cbd69cd3edf7ddc5a08de87dc9cff3f649acaa"}
Jul 22 01:49:16.083928 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:49:16.083895    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fltbj"
Jul 22 01:49:16.099497 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:49:16.099451    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8lmr8" podStartSLOduration=18.15963746 podStartE2EDuration="19.099436824s" totalImagesPullingTime="875.422533ms" totalInitContainerRuntime="64.376831ms" isStatefulPod=true podCreationTimestamp="2026-07-22 01:48:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:49:00.159175451 +0000 UTC m=+93.003004584" watchObservedRunningTime="2026-07-22 01:49:16.099436824 +0000 UTC m=+108.943265994"
Jul 22 01:52:22.081615 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:22.081579    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2xnk"]
Jul 22 01:52:22.085012 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:22.084990    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2xnk"
Jul 22 01:52:22.092824 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:22.089070    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 22 01:52:22.092824 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:22.089280    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-gmb4g\""
Jul 22 01:52:22.092824 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:22.089947    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 22 01:52:22.095770 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:22.095741    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2xnk"]
Jul 22 01:52:22.154069 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:22.154026    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq77r\" (UniqueName: \"kubernetes.io/projected/50eda2d3-7463-40fa-a18e-ccfcc2f42b96-kube-api-access-vq77r\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2xnk\" (UID: \"50eda2d3-7463-40fa-a18e-ccfcc2f42b96\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2xnk"
Jul 22 01:52:22.154260 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:22.154092    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50eda2d3-7463-40fa-a18e-ccfcc2f42b96-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2xnk\" (UID: \"50eda2d3-7463-40fa-a18e-ccfcc2f42b96\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2xnk"
Jul 22 01:52:22.154260 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:22.154150    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50eda2d3-7463-40fa-a18e-ccfcc2f42b96-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2xnk\" (UID: \"50eda2d3-7463-40fa-a18e-ccfcc2f42b96\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2xnk"
Jul 22 01:52:22.255451 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:22.255410    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50eda2d3-7463-40fa-a18e-ccfcc2f42b96-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2xnk\" (UID: \"50eda2d3-7463-40fa-a18e-ccfcc2f42b96\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2xnk"
Jul 22 01:52:22.255629 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:22.255466    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50eda2d3-7463-40fa-a18e-ccfcc2f42b96-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2xnk\" (UID: \"50eda2d3-7463-40fa-a18e-ccfcc2f42b96\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2xnk"
Jul 22 01:52:22.255629 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:22.255507    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq77r\" (UniqueName: \"kubernetes.io/projected/50eda2d3-7463-40fa-a18e-ccfcc2f42b96-kube-api-access-vq77r\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2xnk\" (UID: \"50eda2d3-7463-40fa-a18e-ccfcc2f42b96\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2xnk"
Jul 22 01:52:22.255819 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:22.255793    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50eda2d3-7463-40fa-a18e-ccfcc2f42b96-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2xnk\" (UID: \"50eda2d3-7463-40fa-a18e-ccfcc2f42b96\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2xnk"
Jul 22 01:52:22.255883 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:22.255839    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50eda2d3-7463-40fa-a18e-ccfcc2f42b96-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2xnk\" (UID: \"50eda2d3-7463-40fa-a18e-ccfcc2f42b96\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2xnk"
Jul 22 01:52:22.265190 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:22.265165    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq77r\" (UniqueName: \"kubernetes.io/projected/50eda2d3-7463-40fa-a18e-ccfcc2f42b96-kube-api-access-vq77r\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2xnk\" (UID: \"50eda2d3-7463-40fa-a18e-ccfcc2f42b96\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2xnk"
Jul 22 01:52:22.399673 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:22.399597    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2xnk"
Jul 22 01:52:22.523907 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:22.523874    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2xnk"]
Jul 22 01:52:22.526904 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:52:22.526878    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50eda2d3_7463_40fa_a18e_ccfcc2f42b96.slice/crio-de7cb435d11c2563b64e5dfed4ae4a781fa0352932745f99e7ed89dee0d13e6f WatchSource:0}: Error finding container de7cb435d11c2563b64e5dfed4ae4a781fa0352932745f99e7ed89dee0d13e6f: Status 404 returned error can't find the container with id de7cb435d11c2563b64e5dfed4ae4a781fa0352932745f99e7ed89dee0d13e6f
Jul 22 01:52:22.791431 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:22.791396    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2xnk" event={"ID":"50eda2d3-7463-40fa-a18e-ccfcc2f42b96","Type":"ContainerStarted","Data":"de7cb435d11c2563b64e5dfed4ae4a781fa0352932745f99e7ed89dee0d13e6f"}
Jul 22 01:52:28.265613 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:28.265587    2563 kubelet.go:1648] "Image garbage collection succeeded"
Jul 22 01:52:28.816388 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:28.816344    2563 generic.go:356] "Generic (PLEG): container finished" podID="50eda2d3-7463-40fa-a18e-ccfcc2f42b96" containerID="d944a51eba70727183f3787483bcc02060dfb455af333e0f9aa239d33297652e" exitCode=0
Jul 22 01:52:28.816562 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:28.816428    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2xnk" event={"ID":"50eda2d3-7463-40fa-a18e-ccfcc2f42b96","Type":"ContainerDied","Data":"d944a51eba70727183f3787483bcc02060dfb455af333e0f9aa239d33297652e"}
Jul 22 01:52:28.817515 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:28.817499    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 22 01:52:31.829573 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:31.829542    2563 generic.go:356] "Generic (PLEG): container finished" podID="50eda2d3-7463-40fa-a18e-ccfcc2f42b96" containerID="b08385dd34838ddc1de7666f40a3994b7363e9346b0b4ac2b7cc15a00a456eab" exitCode=0
Jul 22 01:52:31.830051 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:31.829602    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2xnk" event={"ID":"50eda2d3-7463-40fa-a18e-ccfcc2f42b96","Type":"ContainerDied","Data":"b08385dd34838ddc1de7666f40a3994b7363e9346b0b4ac2b7cc15a00a456eab"}
Jul 22 01:52:37.854815 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:37.854779    2563 generic.go:356] "Generic (PLEG): container finished" podID="50eda2d3-7463-40fa-a18e-ccfcc2f42b96" containerID="5efaa4d0f6bad06615740b1844b875c619dd47b4224fb4f58450852decd7b6c1" exitCode=0
Jul 22 01:52:37.855226 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:37.854843    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2xnk" event={"ID":"50eda2d3-7463-40fa-a18e-ccfcc2f42b96","Type":"ContainerDied","Data":"5efaa4d0f6bad06615740b1844b875c619dd47b4224fb4f58450852decd7b6c1"}
Jul 22 01:52:38.976728 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:38.976698    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2xnk"
Jul 22 01:52:39.086577 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:39.086534    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50eda2d3-7463-40fa-a18e-ccfcc2f42b96-bundle\") pod \"50eda2d3-7463-40fa-a18e-ccfcc2f42b96\" (UID: \"50eda2d3-7463-40fa-a18e-ccfcc2f42b96\") "
Jul 22 01:52:39.086756 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:39.086628    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vq77r\" (UniqueName: \"kubernetes.io/projected/50eda2d3-7463-40fa-a18e-ccfcc2f42b96-kube-api-access-vq77r\") pod \"50eda2d3-7463-40fa-a18e-ccfcc2f42b96\" (UID: \"50eda2d3-7463-40fa-a18e-ccfcc2f42b96\") "
Jul 22 01:52:39.086756 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:39.086664    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50eda2d3-7463-40fa-a18e-ccfcc2f42b96-util\") pod \"50eda2d3-7463-40fa-a18e-ccfcc2f42b96\" (UID: \"50eda2d3-7463-40fa-a18e-ccfcc2f42b96\") "
Jul 22 01:52:39.087081 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:39.087057    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50eda2d3-7463-40fa-a18e-ccfcc2f42b96-bundle" (OuterVolumeSpecName: "bundle") pod "50eda2d3-7463-40fa-a18e-ccfcc2f42b96" (UID: "50eda2d3-7463-40fa-a18e-ccfcc2f42b96"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 01:52:39.088911 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:39.088879    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50eda2d3-7463-40fa-a18e-ccfcc2f42b96-kube-api-access-vq77r" (OuterVolumeSpecName: "kube-api-access-vq77r") pod "50eda2d3-7463-40fa-a18e-ccfcc2f42b96" (UID: "50eda2d3-7463-40fa-a18e-ccfcc2f42b96"). InnerVolumeSpecName "kube-api-access-vq77r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 01:52:39.091463 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:39.091427    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50eda2d3-7463-40fa-a18e-ccfcc2f42b96-util" (OuterVolumeSpecName: "util") pod "50eda2d3-7463-40fa-a18e-ccfcc2f42b96" (UID: "50eda2d3-7463-40fa-a18e-ccfcc2f42b96"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 01:52:39.187386 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:39.187344    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50eda2d3-7463-40fa-a18e-ccfcc2f42b96-bundle\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:52:39.187520 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:39.187392    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq77r\" (UniqueName: \"kubernetes.io/projected/50eda2d3-7463-40fa-a18e-ccfcc2f42b96-kube-api-access-vq77r\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:52:39.187520 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:39.187406    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50eda2d3-7463-40fa-a18e-ccfcc2f42b96-util\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:52:39.863348 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:39.863302    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2xnk" event={"ID":"50eda2d3-7463-40fa-a18e-ccfcc2f42b96","Type":"ContainerDied","Data":"de7cb435d11c2563b64e5dfed4ae4a781fa0352932745f99e7ed89dee0d13e6f"}
Jul 22 01:52:39.863348 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:39.863334    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl2xnk"
Jul 22 01:52:39.863556 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:39.863360    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de7cb435d11c2563b64e5dfed4ae4a781fa0352932745f99e7ed89dee0d13e6f"
Jul 22 01:52:45.882340 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:45.882309    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hwnwh"]
Jul 22 01:52:45.882804 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:45.882668    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50eda2d3-7463-40fa-a18e-ccfcc2f42b96" containerName="util"
Jul 22 01:52:45.882804 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:45.882685    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="50eda2d3-7463-40fa-a18e-ccfcc2f42b96" containerName="util"
Jul 22 01:52:45.882804 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:45.882700    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50eda2d3-7463-40fa-a18e-ccfcc2f42b96" containerName="pull"
Jul 22 01:52:45.882804 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:45.882708    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="50eda2d3-7463-40fa-a18e-ccfcc2f42b96" containerName="pull"
Jul 22 01:52:45.882804 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:45.882718    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50eda2d3-7463-40fa-a18e-ccfcc2f42b96" containerName="extract"
Jul 22 01:52:45.882804 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:45.882726    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="50eda2d3-7463-40fa-a18e-ccfcc2f42b96" containerName="extract"
Jul 22 01:52:45.882804 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:45.882797    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="50eda2d3-7463-40fa-a18e-ccfcc2f42b96" containerName="extract"
Jul 22 01:52:45.919941 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:45.919903    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hwnwh"]
Jul 22 01:52:45.920119 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:45.919996    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hwnwh"
Jul 22 01:52:45.922393 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:45.922360    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 22 01:52:45.922500 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:45.922453    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 22 01:52:45.923246 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:45.923230    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-8wz45\""
Jul 22 01:52:46.038017 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:46.037982    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c950b15-1953-4303-a2eb-d4ba5156b935-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hwnwh\" (UID: \"8c950b15-1953-4303-a2eb-d4ba5156b935\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hwnwh"
Jul 22 01:52:46.038168 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:46.038061    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhbwq\" (UniqueName: \"kubernetes.io/projected/8c950b15-1953-4303-a2eb-d4ba5156b935-kube-api-access-dhbwq\") pod \"openshift-lws-operator-fd8ccff4c-hwnwh\" (UID: \"8c950b15-1953-4303-a2eb-d4ba5156b935\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hwnwh"
Jul 22 01:52:46.139283 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:46.139190    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhbwq\" (UniqueName: \"kubernetes.io/projected/8c950b15-1953-4303-a2eb-d4ba5156b935-kube-api-access-dhbwq\") pod \"openshift-lws-operator-fd8ccff4c-hwnwh\" (UID: \"8c950b15-1953-4303-a2eb-d4ba5156b935\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hwnwh"
Jul 22 01:52:46.139283 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:46.139240    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c950b15-1953-4303-a2eb-d4ba5156b935-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hwnwh\" (UID: \"8c950b15-1953-4303-a2eb-d4ba5156b935\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hwnwh"
Jul 22 01:52:46.139626 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:46.139609    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c950b15-1953-4303-a2eb-d4ba5156b935-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hwnwh\" (UID: \"8c950b15-1953-4303-a2eb-d4ba5156b935\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hwnwh"
Jul 22 01:52:46.146842 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:46.146819    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhbwq\" (UniqueName: \"kubernetes.io/projected/8c950b15-1953-4303-a2eb-d4ba5156b935-kube-api-access-dhbwq\") pod \"openshift-lws-operator-fd8ccff4c-hwnwh\" (UID: \"8c950b15-1953-4303-a2eb-d4ba5156b935\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hwnwh"
Jul 22 01:52:46.228948 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:46.228908    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hwnwh"
Jul 22 01:52:46.377599 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:46.377563    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hwnwh"]
Jul 22 01:52:46.381398 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:52:46.381344    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c950b15_1953_4303_a2eb_d4ba5156b935.slice/crio-8d4e7cd7f4613f58959097544d6ab751a6777b6ea5a3f832e6c3a5469e2f980b WatchSource:0}: Error finding container 8d4e7cd7f4613f58959097544d6ab751a6777b6ea5a3f832e6c3a5469e2f980b: Status 404 returned error can't find the container with id 8d4e7cd7f4613f58959097544d6ab751a6777b6ea5a3f832e6c3a5469e2f980b
Jul 22 01:52:46.888469 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:46.888432    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hwnwh" event={"ID":"8c950b15-1953-4303-a2eb-d4ba5156b935","Type":"ContainerStarted","Data":"8d4e7cd7f4613f58959097544d6ab751a6777b6ea5a3f832e6c3a5469e2f980b"}
Jul 22 01:52:48.897667 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:48.897574    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hwnwh" event={"ID":"8c950b15-1953-4303-a2eb-d4ba5156b935","Type":"ContainerStarted","Data":"fa45c622defc4f639c5377b715a88ca0223b5b8b3aa9c2be7322a10ed8fa81f5"}
Jul 22 01:52:48.913295 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:52:48.913232    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hwnwh" podStartSLOduration=1.663694271 podStartE2EDuration="3.913214247s" totalImagesPullingTime="2.249519976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:52:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:52:48.912195554 +0000 UTC m=+321.756024688" watchObservedRunningTime="2026-07-22 01:52:48.913214247 +0000 UTC m=+321.757043378"
Jul 22 01:53:04.736693 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:04.736662    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76db4bc599-kvtx4"]
Jul 22 01:53:04.739686 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:04.739667    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-76db4bc599-kvtx4"
Jul 22 01:53:04.742878 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:04.742842    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 22 01:53:04.742878 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:04.742873    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 22 01:53:04.743087 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:04.742907    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 22 01:53:04.743087 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:04.742951    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-5gsh8\""
Jul 22 01:53:04.743363 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:04.743129    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 22 01:53:04.762373 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:04.762346    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76db4bc599-kvtx4"]
Jul 22 01:53:04.880155 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:04.880113    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f8dd657c-fe0f-4329-87bd-5381cbb98728-manifests\") pod \"opendatahub-operator-controller-manager-76db4bc599-kvtx4\" (UID: \"f8dd657c-fe0f-4329-87bd-5381cbb98728\") " pod="opendatahub/opendatahub-operator-controller-manager-76db4bc599-kvtx4"
Jul 22 01:53:04.880155 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:04.880158    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4jpc\" (UniqueName: \"kubernetes.io/projected/f8dd657c-fe0f-4329-87bd-5381cbb98728-kube-api-access-p4jpc\") pod \"opendatahub-operator-controller-manager-76db4bc599-kvtx4\" (UID: \"f8dd657c-fe0f-4329-87bd-5381cbb98728\") " pod="opendatahub/opendatahub-operator-controller-manager-76db4bc599-kvtx4"
Jul 22 01:53:04.880371 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:04.880202    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8dd657c-fe0f-4329-87bd-5381cbb98728-tmp\") pod \"opendatahub-operator-controller-manager-76db4bc599-kvtx4\" (UID: \"f8dd657c-fe0f-4329-87bd-5381cbb98728\") " pod="opendatahub/opendatahub-operator-controller-manager-76db4bc599-kvtx4"
Jul 22 01:53:04.880371 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:04.880243    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f8dd657c-fe0f-4329-87bd-5381cbb98728-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76db4bc599-kvtx4\" (UID: \"f8dd657c-fe0f-4329-87bd-5381cbb98728\") " pod="opendatahub/opendatahub-operator-controller-manager-76db4bc599-kvtx4"
Jul 22 01:53:04.880371 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:04.880316    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f8dd657c-fe0f-4329-87bd-5381cbb98728-webhook-cert\") pod \"opendatahub-operator-controller-manager-76db4bc599-kvtx4\" (UID: \"f8dd657c-fe0f-4329-87bd-5381cbb98728\") " pod="opendatahub/opendatahub-operator-controller-manager-76db4bc599-kvtx4"
Jul 22 01:53:04.981700 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:04.981660    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8dd657c-fe0f-4329-87bd-5381cbb98728-tmp\") pod \"opendatahub-operator-controller-manager-76db4bc599-kvtx4\" (UID: \"f8dd657c-fe0f-4329-87bd-5381cbb98728\") " pod="opendatahub/opendatahub-operator-controller-manager-76db4bc599-kvtx4"
Jul 22 01:53:04.981700 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:04.981703    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f8dd657c-fe0f-4329-87bd-5381cbb98728-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76db4bc599-kvtx4\" (UID: \"f8dd657c-fe0f-4329-87bd-5381cbb98728\") " pod="opendatahub/opendatahub-operator-controller-manager-76db4bc599-kvtx4"
Jul 22 01:53:04.981951 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:04.981834    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f8dd657c-fe0f-4329-87bd-5381cbb98728-webhook-cert\") pod \"opendatahub-operator-controller-manager-76db4bc599-kvtx4\" (UID: \"f8dd657c-fe0f-4329-87bd-5381cbb98728\") " pod="opendatahub/opendatahub-operator-controller-manager-76db4bc599-kvtx4"
Jul 22 01:53:04.981951 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:04.981912    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f8dd657c-fe0f-4329-87bd-5381cbb98728-manifests\") pod \"opendatahub-operator-controller-manager-76db4bc599-kvtx4\" (UID: \"f8dd657c-fe0f-4329-87bd-5381cbb98728\") " pod="opendatahub/opendatahub-operator-controller-manager-76db4bc599-kvtx4"
Jul 22 01:53:04.981951 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:04.981929    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4jpc\" (UniqueName: \"kubernetes.io/projected/f8dd657c-fe0f-4329-87bd-5381cbb98728-kube-api-access-p4jpc\") pod \"opendatahub-operator-controller-manager-76db4bc599-kvtx4\" (UID: \"f8dd657c-fe0f-4329-87bd-5381cbb98728\") " pod="opendatahub/opendatahub-operator-controller-manager-76db4bc599-kvtx4"
Jul 22 01:53:04.982307 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:04.982277    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f8dd657c-fe0f-4329-87bd-5381cbb98728-manifests\") pod \"opendatahub-operator-controller-manager-76db4bc599-kvtx4\" (UID: \"f8dd657c-fe0f-4329-87bd-5381cbb98728\") " pod="opendatahub/opendatahub-operator-controller-manager-76db4bc599-kvtx4"
Jul 22 01:53:04.984050 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:04.984018    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8dd657c-fe0f-4329-87bd-5381cbb98728-tmp\") pod \"opendatahub-operator-controller-manager-76db4bc599-kvtx4\" (UID: \"f8dd657c-fe0f-4329-87bd-5381cbb98728\") " pod="opendatahub/opendatahub-operator-controller-manager-76db4bc599-kvtx4"
Jul 22 01:53:04.984213 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:04.984193    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f8dd657c-fe0f-4329-87bd-5381cbb98728-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76db4bc599-kvtx4\" (UID: \"f8dd657c-fe0f-4329-87bd-5381cbb98728\") " pod="opendatahub/opendatahub-operator-controller-manager-76db4bc599-kvtx4"
Jul 22 01:53:04.984269 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:04.984245    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f8dd657c-fe0f-4329-87bd-5381cbb98728-webhook-cert\") pod \"opendatahub-operator-controller-manager-76db4bc599-kvtx4\" (UID: \"f8dd657c-fe0f-4329-87bd-5381cbb98728\") " pod="opendatahub/opendatahub-operator-controller-manager-76db4bc599-kvtx4"
Jul 22 01:53:04.992981 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:04.992924    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4jpc\" (UniqueName: \"kubernetes.io/projected/f8dd657c-fe0f-4329-87bd-5381cbb98728-kube-api-access-p4jpc\") pod \"opendatahub-operator-controller-manager-76db4bc599-kvtx4\" (UID: \"f8dd657c-fe0f-4329-87bd-5381cbb98728\") " pod="opendatahub/opendatahub-operator-controller-manager-76db4bc599-kvtx4"
Jul 22 01:53:05.050123 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:05.050088    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-76db4bc599-kvtx4"
Jul 22 01:53:05.185093 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:05.185030    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76db4bc599-kvtx4"]
Jul 22 01:53:05.187974 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:53:05.187948    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8dd657c_fe0f_4329_87bd_5381cbb98728.slice/crio-fae3869600fcb5a8a4af5af32ec569b54dc1f39d38b5aaefac7899da293a9c4e WatchSource:0}: Error finding container fae3869600fcb5a8a4af5af32ec569b54dc1f39d38b5aaefac7899da293a9c4e: Status 404 returned error can't find the container with id fae3869600fcb5a8a4af5af32ec569b54dc1f39d38b5aaefac7899da293a9c4e
Jul 22 01:53:05.958107 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:05.958067    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76db4bc599-kvtx4" event={"ID":"f8dd657c-fe0f-4329-87bd-5381cbb98728","Type":"ContainerStarted","Data":"fae3869600fcb5a8a4af5af32ec569b54dc1f39d38b5aaefac7899da293a9c4e"}
Jul 22 01:53:08.970091 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:08.970053    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76db4bc599-kvtx4" event={"ID":"f8dd657c-fe0f-4329-87bd-5381cbb98728","Type":"ContainerStarted","Data":"147a4141b676d9cdde35e7e607ad6435e4247dfbd21defa685459dcb659c9706"}
Jul 22 01:53:09.980167 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:09.980127    2563 generic.go:356] "Generic (PLEG): container finished" podID="f8dd657c-fe0f-4329-87bd-5381cbb98728" containerID="147a4141b676d9cdde35e7e607ad6435e4247dfbd21defa685459dcb659c9706" exitCode=0
Jul 22 01:53:09.980601 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:09.980290    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76db4bc599-kvtx4" event={"ID":"f8dd657c-fe0f-4329-87bd-5381cbb98728","Type":"ContainerDied","Data":"147a4141b676d9cdde35e7e607ad6435e4247dfbd21defa685459dcb659c9706"}
Jul 22 01:53:10.986221 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:10.986188    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76db4bc599-kvtx4" event={"ID":"f8dd657c-fe0f-4329-87bd-5381cbb98728","Type":"ContainerStarted","Data":"e22233e1c415526adaaf89c0d3ace5128385bb1c5915f322a59fce21a4cfde5b"}
Jul 22 01:53:10.986618 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:10.986303    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-76db4bc599-kvtx4"
Jul 22 01:53:11.008177 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:11.008123    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-76db4bc599-kvtx4" podStartSLOduration=2.733407465 podStartE2EDuration="7.008104673s" totalImagesPullingTime="3.262565003s" totalInitContainerRuntime="1.012132205s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:53:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:53:11.005855476 +0000 UTC m=+343.849684609" watchObservedRunningTime="2026-07-22 01:53:11.008104673 +0000 UTC m=+343.851933804"
Jul 22 01:53:21.992410 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:21.992312    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-76db4bc599-kvtx4"
Jul 22 01:53:24.879609 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:24.879576    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-75f64fb6dd-xrvcm"]
Jul 22 01:53:24.888668 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:24.888648    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-75f64fb6dd-xrvcm"
Jul 22 01:53:24.891534 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:24.891507    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 22 01:53:24.891651 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:24.891507    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-f8xnk\""
Jul 22 01:53:24.892253 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:24.892232    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 22 01:53:24.892253 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:24.892234    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 22 01:53:24.892488 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:24.892462    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 22 01:53:24.896008 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:24.895781    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-75f64fb6dd-xrvcm"]
Jul 22 01:53:24.932735 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:24.932709    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/179aab03-1b60-45e8-a9a0-1dce658c6971-tls-certs\") pod \"kube-auth-proxy-75f64fb6dd-xrvcm\" (UID: \"179aab03-1b60-45e8-a9a0-1dce658c6971\") " pod="openshift-ingress/kube-auth-proxy-75f64fb6dd-xrvcm"
Jul 22 01:53:24.932848 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:24.932772    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/179aab03-1b60-45e8-a9a0-1dce658c6971-tmp\") pod \"kube-auth-proxy-75f64fb6dd-xrvcm\" (UID: \"179aab03-1b60-45e8-a9a0-1dce658c6971\") " pod="openshift-ingress/kube-auth-proxy-75f64fb6dd-xrvcm"
Jul 22 01:53:24.932848 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:24.932827    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlc8m\" (UniqueName: \"kubernetes.io/projected/179aab03-1b60-45e8-a9a0-1dce658c6971-kube-api-access-qlc8m\") pod \"kube-auth-proxy-75f64fb6dd-xrvcm\" (UID: \"179aab03-1b60-45e8-a9a0-1dce658c6971\") " pod="openshift-ingress/kube-auth-proxy-75f64fb6dd-xrvcm"
Jul 22 01:53:25.033058 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:25.033035    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/179aab03-1b60-45e8-a9a0-1dce658c6971-tmp\") pod \"kube-auth-proxy-75f64fb6dd-xrvcm\" (UID: \"179aab03-1b60-45e8-a9a0-1dce658c6971\") " pod="openshift-ingress/kube-auth-proxy-75f64fb6dd-xrvcm"
Jul 22 01:53:25.033164 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:25.033073    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlc8m\" (UniqueName: \"kubernetes.io/projected/179aab03-1b60-45e8-a9a0-1dce658c6971-kube-api-access-qlc8m\") pod \"kube-auth-proxy-75f64fb6dd-xrvcm\" (UID: \"179aab03-1b60-45e8-a9a0-1dce658c6971\") " pod="openshift-ingress/kube-auth-proxy-75f64fb6dd-xrvcm"
Jul 22 01:53:25.033164 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:25.033098    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/179aab03-1b60-45e8-a9a0-1dce658c6971-tls-certs\") pod \"kube-auth-proxy-75f64fb6dd-xrvcm\" (UID: \"179aab03-1b60-45e8-a9a0-1dce658c6971\") " pod="openshift-ingress/kube-auth-proxy-75f64fb6dd-xrvcm"
Jul 22 01:53:25.035381 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:25.035359    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/179aab03-1b60-45e8-a9a0-1dce658c6971-tmp\") pod \"kube-auth-proxy-75f64fb6dd-xrvcm\" (UID: \"179aab03-1b60-45e8-a9a0-1dce658c6971\") " pod="openshift-ingress/kube-auth-proxy-75f64fb6dd-xrvcm"
Jul 22 01:53:25.035568 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:25.035544    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/179aab03-1b60-45e8-a9a0-1dce658c6971-tls-certs\") pod \"kube-auth-proxy-75f64fb6dd-xrvcm\" (UID: \"179aab03-1b60-45e8-a9a0-1dce658c6971\") " pod="openshift-ingress/kube-auth-proxy-75f64fb6dd-xrvcm"
Jul 22 01:53:25.042931 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:25.042913    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlc8m\" (UniqueName: \"kubernetes.io/projected/179aab03-1b60-45e8-a9a0-1dce658c6971-kube-api-access-qlc8m\") pod \"kube-auth-proxy-75f64fb6dd-xrvcm\" (UID: \"179aab03-1b60-45e8-a9a0-1dce658c6971\") " pod="openshift-ingress/kube-auth-proxy-75f64fb6dd-xrvcm"
Jul 22 01:53:25.080831 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:25.080807    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-lgqv9"]
Jul 22 01:53:25.084244 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:25.084229    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-lgqv9"
Jul 22 01:53:25.086873 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:25.086854    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 22 01:53:25.086974 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:25.086920    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-4jd64\""
Jul 22 01:53:25.094164 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:25.094144    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-lgqv9"]
Jul 22 01:53:25.134230 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:25.134171    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/250b2d23-5084-4267-acf0-857731a2fe8c-cert\") pod \"odh-model-controller-7998c67dc4-lgqv9\" (UID: \"250b2d23-5084-4267-acf0-857731a2fe8c\") " pod="opendatahub/odh-model-controller-7998c67dc4-lgqv9"
Jul 22 01:53:25.134318 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:25.134228    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtvng\" (UniqueName: \"kubernetes.io/projected/250b2d23-5084-4267-acf0-857731a2fe8c-kube-api-access-rtvng\") pod \"odh-model-controller-7998c67dc4-lgqv9\" (UID: \"250b2d23-5084-4267-acf0-857731a2fe8c\") " pod="opendatahub/odh-model-controller-7998c67dc4-lgqv9"
Jul 22 01:53:25.199091 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:25.199063    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-75f64fb6dd-xrvcm"
Jul 22 01:53:25.234963 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:25.234933    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/250b2d23-5084-4267-acf0-857731a2fe8c-cert\") pod \"odh-model-controller-7998c67dc4-lgqv9\" (UID: \"250b2d23-5084-4267-acf0-857731a2fe8c\") " pod="opendatahub/odh-model-controller-7998c67dc4-lgqv9"
Jul 22 01:53:25.235111 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:25.234988    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtvng\" (UniqueName: \"kubernetes.io/projected/250b2d23-5084-4267-acf0-857731a2fe8c-kube-api-access-rtvng\") pod \"odh-model-controller-7998c67dc4-lgqv9\" (UID: \"250b2d23-5084-4267-acf0-857731a2fe8c\") " pod="opendatahub/odh-model-controller-7998c67dc4-lgqv9"
Jul 22 01:53:25.235111 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:53:25.235086    2563 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 22 01:53:25.235218 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:53:25.235147    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/250b2d23-5084-4267-acf0-857731a2fe8c-cert podName:250b2d23-5084-4267-acf0-857731a2fe8c nodeName:}" failed. No retries permitted until 2026-07-22 01:53:25.73512908 +0000 UTC m=+358.578958190 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/250b2d23-5084-4267-acf0-857731a2fe8c-cert") pod "odh-model-controller-7998c67dc4-lgqv9" (UID: "250b2d23-5084-4267-acf0-857731a2fe8c") : secret "odh-model-controller-webhook-cert" not found
Jul 22 01:53:25.247178 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:25.246934    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtvng\" (UniqueName: \"kubernetes.io/projected/250b2d23-5084-4267-acf0-857731a2fe8c-kube-api-access-rtvng\") pod \"odh-model-controller-7998c67dc4-lgqv9\" (UID: \"250b2d23-5084-4267-acf0-857731a2fe8c\") " pod="opendatahub/odh-model-controller-7998c67dc4-lgqv9"
Jul 22 01:53:25.330991 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:25.330965    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-75f64fb6dd-xrvcm"]
Jul 22 01:53:25.332622 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:53:25.332590    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod179aab03_1b60_45e8_a9a0_1dce658c6971.slice/crio-080f500029f0c444c715179852dc12b707e84e92d39846ddf2e15ded1dbc2744 WatchSource:0}: Error finding container 080f500029f0c444c715179852dc12b707e84e92d39846ddf2e15ded1dbc2744: Status 404 returned error can't find the container with id 080f500029f0c444c715179852dc12b707e84e92d39846ddf2e15ded1dbc2744
Jul 22 01:53:25.739363 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:25.739328    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/250b2d23-5084-4267-acf0-857731a2fe8c-cert\") pod \"odh-model-controller-7998c67dc4-lgqv9\" (UID: \"250b2d23-5084-4267-acf0-857731a2fe8c\") " pod="opendatahub/odh-model-controller-7998c67dc4-lgqv9"
Jul 22 01:53:25.741768 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:25.741745    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/250b2d23-5084-4267-acf0-857731a2fe8c-cert\") pod \"odh-model-controller-7998c67dc4-lgqv9\" (UID: \"250b2d23-5084-4267-acf0-857731a2fe8c\") " pod="opendatahub/odh-model-controller-7998c67dc4-lgqv9"
Jul 22 01:53:25.994329 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:25.994246    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-lgqv9"
Jul 22 01:53:26.012005 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:26.011979    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-ks2l4"]
Jul 22 01:53:26.015678 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:26.015660    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ks2l4"
Jul 22 01:53:26.017701 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:26.017680    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 22 01:53:26.018096 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:26.018059    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 22 01:53:26.018255 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:26.018111    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 22 01:53:26.018255 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:26.018141    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-54tjd\""
Jul 22 01:53:26.018255 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:26.018196    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 22 01:53:26.028116 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:26.028092    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-ks2l4"]
Jul 22 01:53:26.042435 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:26.042404    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e497d2fd-b855-4792-b91c-724e72ba1234-istio-token\") pod \"istiod-openshift-gateway-7574459656-ks2l4\" (UID: \"e497d2fd-b855-4792-b91c-724e72ba1234\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ks2l4"
Jul 22 01:53:26.042551 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:26.042447    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e497d2fd-b855-4792-b91c-724e72ba1234-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-ks2l4\" (UID: \"e497d2fd-b855-4792-b91c-724e72ba1234\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ks2l4"
Jul 22 01:53:26.042551 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:26.042482    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcqcq\" (UniqueName: \"kubernetes.io/projected/e497d2fd-b855-4792-b91c-724e72ba1234-kube-api-access-xcqcq\") pod \"istiod-openshift-gateway-7574459656-ks2l4\" (UID: \"e497d2fd-b855-4792-b91c-724e72ba1234\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ks2l4"
Jul 22 01:53:26.042551 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:26.042513    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e497d2fd-b855-4792-b91c-724e72ba1234-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-ks2l4\" (UID: \"e497d2fd-b855-4792-b91c-724e72ba1234\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ks2l4"
Jul 22 01:53:26.042667 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:26.042556    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e497d2fd-b855-4792-b91c-724e72ba1234-cacerts\") pod \"istiod-openshift-gateway-7574459656-ks2l4\" (UID: \"e497d2fd-b855-4792-b91c-724e72ba1234\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ks2l4"
Jul 22 01:53:26.042667 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:26.042591    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e497d2fd-b855-4792-b91c-724e72ba1234-local-certs\") pod \"istiod-openshift-gateway-7574459656-ks2l4\" (UID: \"e497d2fd-b855-4792-b91c-724e72ba1234\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ks2l4"
Jul 22 01:53:26.042667 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:26.042654    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e497d2fd-b855-4792-b91c-724e72ba1234-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-ks2l4\" (UID: \"e497d2fd-b855-4792-b91c-724e72ba1234\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ks2l4"
Jul 22 01:53:26.043214 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:26.043163    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-75f64fb6dd-xrvcm" event={"ID":"179aab03-1b60-45e8-a9a0-1dce658c6971","Type":"ContainerStarted","Data":"080f500029f0c444c715179852dc12b707e84e92d39846ddf2e15ded1dbc2744"}
Jul 22 01:53:26.132320 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:26.132278    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-lgqv9"]
Jul 22 01:53:26.143231 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:26.143202    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e497d2fd-b855-4792-b91c-724e72ba1234-cacerts\") pod \"istiod-openshift-gateway-7574459656-ks2l4\" (UID: \"e497d2fd-b855-4792-b91c-724e72ba1234\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ks2l4"
Jul 22 01:53:26.143330 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:26.143248    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e497d2fd-b855-4792-b91c-724e72ba1234-local-certs\") pod \"istiod-openshift-gateway-7574459656-ks2l4\" (UID: \"e497d2fd-b855-4792-b91c-724e72ba1234\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ks2l4"
Jul 22 01:53:26.143330 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:26.143296    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e497d2fd-b855-4792-b91c-724e72ba1234-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-ks2l4\" (UID: \"e497d2fd-b855-4792-b91c-724e72ba1234\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ks2l4"
Jul 22 01:53:26.143330 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:26.143320    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e497d2fd-b855-4792-b91c-724e72ba1234-istio-token\") pod \"istiod-openshift-gateway-7574459656-ks2l4\" (UID: \"e497d2fd-b855-4792-b91c-724e72ba1234\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ks2l4"
Jul 22 01:53:26.143618 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:26.143337    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e497d2fd-b855-4792-b91c-724e72ba1234-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-ks2l4\" (UID: \"e497d2fd-b855-4792-b91c-724e72ba1234\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ks2l4"
Jul 22 01:53:26.143618 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:26.143367    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcqcq\" (UniqueName: \"kubernetes.io/projected/e497d2fd-b855-4792-b91c-724e72ba1234-kube-api-access-xcqcq\") pod \"istiod-openshift-gateway-7574459656-ks2l4\" (UID: \"e497d2fd-b855-4792-b91c-724e72ba1234\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ks2l4"
Jul 22 01:53:26.143618 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:26.143422    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e497d2fd-b855-4792-b91c-724e72ba1234-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-ks2l4\" (UID: \"e497d2fd-b855-4792-b91c-724e72ba1234\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ks2l4"
Jul 22 01:53:26.144317 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:26.144290    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e497d2fd-b855-4792-b91c-724e72ba1234-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-ks2l4\" (UID: \"e497d2fd-b855-4792-b91c-724e72ba1234\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ks2l4"
Jul 22 01:53:26.145729 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:26.145707    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e497d2fd-b855-4792-b91c-724e72ba1234-local-certs\") pod \"istiod-openshift-gateway-7574459656-ks2l4\" (UID: \"e497d2fd-b855-4792-b91c-724e72ba1234\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ks2l4"
Jul 22 01:53:26.145809 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:26.145745    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e497d2fd-b855-4792-b91c-724e72ba1234-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-ks2l4\" (UID: \"e497d2fd-b855-4792-b91c-724e72ba1234\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ks2l4"
Jul 22 01:53:26.145809 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:26.145751    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e497d2fd-b855-4792-b91c-724e72ba1234-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-ks2l4\" (UID: \"e497d2fd-b855-4792-b91c-724e72ba1234\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ks2l4"
Jul 22 01:53:26.145883 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:26.145851    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e497d2fd-b855-4792-b91c-724e72ba1234-cacerts\") pod \"istiod-openshift-gateway-7574459656-ks2l4\" (UID: \"e497d2fd-b855-4792-b91c-724e72ba1234\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ks2l4"
Jul 22 01:53:26.156089 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:26.156069    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcqcq\" (UniqueName: \"kubernetes.io/projected/e497d2fd-b855-4792-b91c-724e72ba1234-kube-api-access-xcqcq\") pod \"istiod-openshift-gateway-7574459656-ks2l4\" (UID: \"e497d2fd-b855-4792-b91c-724e72ba1234\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ks2l4"
Jul 22 01:53:26.156845 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:26.156823    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e497d2fd-b855-4792-b91c-724e72ba1234-istio-token\") pod \"istiod-openshift-gateway-7574459656-ks2l4\" (UID: \"e497d2fd-b855-4792-b91c-724e72ba1234\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ks2l4"
Jul 22 01:53:26.326124 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:26.326043    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ks2l4"
Jul 22 01:53:26.474796 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:26.474764    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-ks2l4"]
Jul 22 01:53:26.477871 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:53:26.477841    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode497d2fd_b855_4792_b91c_724e72ba1234.slice/crio-c271b1b9476a6c7cb23e910896e8e5ddad72e7eb931d3c3cec7642a195f414a9 WatchSource:0}: Error finding container c271b1b9476a6c7cb23e910896e8e5ddad72e7eb931d3c3cec7642a195f414a9: Status 404 returned error can't find the container with id c271b1b9476a6c7cb23e910896e8e5ddad72e7eb931d3c3cec7642a195f414a9
Jul 22 01:53:27.051412 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:27.051271    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-lgqv9" event={"ID":"250b2d23-5084-4267-acf0-857731a2fe8c","Type":"ContainerStarted","Data":"913d17638b81ea65d825bdcc376238ce6ee7e899e8abcbb620cba540d8ca7dfc"}
Jul 22 01:53:27.053750 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:27.053666    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ks2l4" event={"ID":"e497d2fd-b855-4792-b91c-724e72ba1234","Type":"ContainerStarted","Data":"c271b1b9476a6c7cb23e910896e8e5ddad72e7eb931d3c3cec7642a195f414a9"}
Jul 22 01:53:30.457462 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:30.457429    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 22 01:53:30.468309 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:30.468277    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 22 01:53:30.468372 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:30.468359    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 22 01:53:31.073993 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:31.073959    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-75f64fb6dd-xrvcm" event={"ID":"179aab03-1b60-45e8-a9a0-1dce658c6971","Type":"ContainerStarted","Data":"033d1c058ff5d3d36d3419559b6b01826347ab9704dc65f8c1c05838dfa95fad"}
Jul 22 01:53:31.075608 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:31.075576    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-lgqv9" event={"ID":"250b2d23-5084-4267-acf0-857731a2fe8c","Type":"ContainerStarted","Data":"47573458ff19fed24ae57d5a37d2e0ef14e96fbfdb2ef04f45aad91499e08f7a"}
Jul 22 01:53:31.075737 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:31.075635    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-lgqv9"
Jul 22 01:53:31.077290 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:31.077262    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ks2l4" event={"ID":"e497d2fd-b855-4792-b91c-724e72ba1234","Type":"ContainerStarted","Data":"9c9fd4ad96eea153ca44023c59068750bb3e715d1dd25e2bbc5a7e205ceb07fd"}
Jul 22 01:53:31.077417 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:31.077311    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ks2l4"
Jul 22 01:53:31.079127 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:31.079100    2563 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-ks2l4 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 22 01:53:31.079237 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:31.079166    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ks2l4" podUID="e497d2fd-b855-4792-b91c-724e72ba1234" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 22 01:53:31.095577 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:31.092658    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-75f64fb6dd-xrvcm" podStartSLOduration=1.972311427 podStartE2EDuration="7.092638159s" totalImagesPullingTime="5.120326732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:53:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:53:31.090241453 +0000 UTC m=+363.934070587" watchObservedRunningTime="2026-07-22 01:53:31.092638159 +0000 UTC m=+363.936467295"
Jul 22 01:53:31.121121 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:31.121031    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ks2l4" podStartSLOduration=2.133175428 podStartE2EDuration="6.121011823s" totalImagesPullingTime="3.987836395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:53:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:53:31.119516943 +0000 UTC m=+363.963346077" watchObservedRunningTime="2026-07-22 01:53:31.121011823 +0000 UTC m=+363.964840958"
Jul 22 01:53:31.136691 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:31.136627    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-lgqv9" podStartSLOduration=1.8054802460000001 podStartE2EDuration="6.136604174s" totalImagesPullingTime="4.331123928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:53:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:53:31.135877142 +0000 UTC m=+363.979706283" watchObservedRunningTime="2026-07-22 01:53:31.136604174 +0000 UTC m=+363.980433307"
Jul 22 01:53:32.082712 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:32.082686    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ks2l4"
Jul 22 01:53:35.970711 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:35.970677    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-62wkf"]
Jul 22 01:53:35.974264 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:35.974241    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-62wkf"
Jul 22 01:53:35.976919 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:35.976878    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 22 01:53:35.977048 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:35.977005    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-ctxgx\""
Jul 22 01:53:35.984482 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:35.984455    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-9qxz4"]
Jul 22 01:53:35.988403 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:35.987942    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9qxz4"
Jul 22 01:53:35.990190 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:35.990170    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-62wkf"]
Jul 22 01:53:35.990743 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:35.990728    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 22 01:53:35.990835 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:35.990792    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-c4gsm\""
Jul 22 01:53:35.998471 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:35.998439    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-9qxz4"]
Jul 22 01:53:36.029161 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:36.029123    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s77rn\" (UniqueName: \"kubernetes.io/projected/92e08f68-a6f8-482b-ba21-4dab98b2a400-kube-api-access-s77rn\") pod \"llmisvc-controller-manager-777c6bd7d5-9qxz4\" (UID: \"92e08f68-a6f8-482b-ba21-4dab98b2a400\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9qxz4"
Jul 22 01:53:36.029391 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:36.029184    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/958bb2e8-a856-4e85-9482-36370ce716c6-cert\") pod \"kserve-controller-manager-d6f4b65b8-62wkf\" (UID: \"958bb2e8-a856-4e85-9482-36370ce716c6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-62wkf"
Jul 22 01:53:36.029391 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:36.029263    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfrhs\" (UniqueName: \"kubernetes.io/projected/958bb2e8-a856-4e85-9482-36370ce716c6-kube-api-access-vfrhs\") pod \"kserve-controller-manager-d6f4b65b8-62wkf\" (UID: \"958bb2e8-a856-4e85-9482-36370ce716c6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-62wkf"
Jul 22 01:53:36.029391 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:36.029292    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92e08f68-a6f8-482b-ba21-4dab98b2a400-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-9qxz4\" (UID: \"92e08f68-a6f8-482b-ba21-4dab98b2a400\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9qxz4"
Jul 22 01:53:36.130510 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:36.130471    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/958bb2e8-a856-4e85-9482-36370ce716c6-cert\") pod \"kserve-controller-manager-d6f4b65b8-62wkf\" (UID: \"958bb2e8-a856-4e85-9482-36370ce716c6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-62wkf"
Jul 22 01:53:36.130706 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:36.130552    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfrhs\" (UniqueName: \"kubernetes.io/projected/958bb2e8-a856-4e85-9482-36370ce716c6-kube-api-access-vfrhs\") pod \"kserve-controller-manager-d6f4b65b8-62wkf\" (UID: \"958bb2e8-a856-4e85-9482-36370ce716c6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-62wkf"
Jul 22 01:53:36.130706 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:36.130581    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92e08f68-a6f8-482b-ba21-4dab98b2a400-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-9qxz4\" (UID: \"92e08f68-a6f8-482b-ba21-4dab98b2a400\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9qxz4"
Jul 22 01:53:36.130706 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:36.130641    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s77rn\" (UniqueName: \"kubernetes.io/projected/92e08f68-a6f8-482b-ba21-4dab98b2a400-kube-api-access-s77rn\") pod \"llmisvc-controller-manager-777c6bd7d5-9qxz4\" (UID: \"92e08f68-a6f8-482b-ba21-4dab98b2a400\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9qxz4"
Jul 22 01:53:36.130706 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:53:36.130644    2563 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 22 01:53:36.130899 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:53:36.130751    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/958bb2e8-a856-4e85-9482-36370ce716c6-cert podName:958bb2e8-a856-4e85-9482-36370ce716c6 nodeName:}" failed. No retries permitted until 2026-07-22 01:53:36.630728245 +0000 UTC m=+369.474557371 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/958bb2e8-a856-4e85-9482-36370ce716c6-cert") pod "kserve-controller-manager-d6f4b65b8-62wkf" (UID: "958bb2e8-a856-4e85-9482-36370ce716c6") : secret "kserve-webhook-server-cert" not found
Jul 22 01:53:36.130899 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:53:36.130763    2563 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 22 01:53:36.130899 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:53:36.130813    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e08f68-a6f8-482b-ba21-4dab98b2a400-cert podName:92e08f68-a6f8-482b-ba21-4dab98b2a400 nodeName:}" failed. No retries permitted until 2026-07-22 01:53:36.630801635 +0000 UTC m=+369.474630755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/92e08f68-a6f8-482b-ba21-4dab98b2a400-cert") pod "llmisvc-controller-manager-777c6bd7d5-9qxz4" (UID: "92e08f68-a6f8-482b-ba21-4dab98b2a400") : secret "llmisvc-webhook-server-cert" not found
Jul 22 01:53:36.144367 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:36.144331    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfrhs\" (UniqueName: \"kubernetes.io/projected/958bb2e8-a856-4e85-9482-36370ce716c6-kube-api-access-vfrhs\") pod \"kserve-controller-manager-d6f4b65b8-62wkf\" (UID: \"958bb2e8-a856-4e85-9482-36370ce716c6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-62wkf"
Jul 22 01:53:36.145145 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:36.145120    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s77rn\" (UniqueName: \"kubernetes.io/projected/92e08f68-a6f8-482b-ba21-4dab98b2a400-kube-api-access-s77rn\") pod \"llmisvc-controller-manager-777c6bd7d5-9qxz4\" (UID: \"92e08f68-a6f8-482b-ba21-4dab98b2a400\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9qxz4"
Jul 22 01:53:36.635657 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:36.635619    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92e08f68-a6f8-482b-ba21-4dab98b2a400-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-9qxz4\" (UID: \"92e08f68-a6f8-482b-ba21-4dab98b2a400\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9qxz4"
Jul 22 01:53:36.635841 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:36.635689    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/958bb2e8-a856-4e85-9482-36370ce716c6-cert\") pod \"kserve-controller-manager-d6f4b65b8-62wkf\" (UID: \"958bb2e8-a856-4e85-9482-36370ce716c6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-62wkf"
Jul 22 01:53:36.638267 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:36.638240    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/958bb2e8-a856-4e85-9482-36370ce716c6-cert\") pod \"kserve-controller-manager-d6f4b65b8-62wkf\" (UID: \"958bb2e8-a856-4e85-9482-36370ce716c6\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-62wkf"
Jul 22 01:53:36.638352 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:36.638240    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92e08f68-a6f8-482b-ba21-4dab98b2a400-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-9qxz4\" (UID: \"92e08f68-a6f8-482b-ba21-4dab98b2a400\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9qxz4"
Jul 22 01:53:36.887137 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:36.887047    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-62wkf"
Jul 22 01:53:36.902445 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:36.902410    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9qxz4"
Jul 22 01:53:37.032137 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:37.032112    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-62wkf"]
Jul 22 01:53:37.034865 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:53:37.034832    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod958bb2e8_a856_4e85_9482_36370ce716c6.slice/crio-80ec97bc34566af240d2b4067ed304605a670bbcadb500392459af193f34dd13 WatchSource:0}: Error finding container 80ec97bc34566af240d2b4067ed304605a670bbcadb500392459af193f34dd13: Status 404 returned error can't find the container with id 80ec97bc34566af240d2b4067ed304605a670bbcadb500392459af193f34dd13
Jul 22 01:53:37.061659 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:37.061632    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-9qxz4"]
Jul 22 01:53:37.064721 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:53:37.064699    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod92e08f68_a6f8_482b_ba21_4dab98b2a400.slice/crio-9eb4faddd5573e705c266b4d195bcdaaa5b1a1bb1e98800d436c37834b9a18b5 WatchSource:0}: Error finding container 9eb4faddd5573e705c266b4d195bcdaaa5b1a1bb1e98800d436c37834b9a18b5: Status 404 returned error can't find the container with id 9eb4faddd5573e705c266b4d195bcdaaa5b1a1bb1e98800d436c37834b9a18b5
Jul 22 01:53:37.103214 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:37.103174    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9qxz4" event={"ID":"92e08f68-a6f8-482b-ba21-4dab98b2a400","Type":"ContainerStarted","Data":"9eb4faddd5573e705c266b4d195bcdaaa5b1a1bb1e98800d436c37834b9a18b5"}
Jul 22 01:53:37.104443 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:37.104406    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-62wkf" event={"ID":"958bb2e8-a856-4e85-9482-36370ce716c6","Type":"ContainerStarted","Data":"80ec97bc34566af240d2b4067ed304605a670bbcadb500392459af193f34dd13"}
Jul 22 01:53:40.122239 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:40.122200    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9qxz4" event={"ID":"92e08f68-a6f8-482b-ba21-4dab98b2a400","Type":"ContainerStarted","Data":"14ffef5c066d2899fd16b9ae61827071f147ef998e68ae3cfa7715e4deb01484"}
Jul 22 01:53:40.122718 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:40.122272    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9qxz4"
Jul 22 01:53:40.123698 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:40.123672    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-62wkf" event={"ID":"958bb2e8-a856-4e85-9482-36370ce716c6","Type":"ContainerStarted","Data":"6a9fa7b3f4a249bab0d39c77e1adb258470c5c6eb9eecae1d2687b075a985450"}
Jul 22 01:53:40.123809 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:40.123793    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-62wkf"
Jul 22 01:53:40.139242 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:40.139199    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9qxz4" podStartSLOduration=2.289904833 podStartE2EDuration="5.139185893s" totalImagesPullingTime="2.84928106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:53:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:53:40.138090793 +0000 UTC m=+372.981919927" watchObservedRunningTime="2026-07-22 01:53:40.139185893 +0000 UTC m=+372.983015025"
Jul 22 01:53:40.154489 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:40.154450    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-62wkf" podStartSLOduration=2.891833857 podStartE2EDuration="5.154439451s" totalImagesPullingTime="2.262605594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:53:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:53:40.153502746 +0000 UTC m=+372.997331890" watchObservedRunningTime="2026-07-22 01:53:40.154439451 +0000 UTC m=+372.998268617"
Jul 22 01:53:40.444949 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:40.444917    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-ks2l4"]
Jul 22 01:53:40.445126 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:40.445103    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ks2l4" podUID="e497d2fd-b855-4792-b91c-724e72ba1234" containerName="discovery" containerID="cri-o://9c9fd4ad96eea153ca44023c59068750bb3e715d1dd25e2bbc5a7e205ceb07fd" gracePeriod=30
Jul 22 01:53:40.691914 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:40.691888    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ks2l4"
Jul 22 01:53:40.773951 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:40.773858    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e497d2fd-b855-4792-b91c-724e72ba1234-istio-csr-dns-cert\") pod \"e497d2fd-b855-4792-b91c-724e72ba1234\" (UID: \"e497d2fd-b855-4792-b91c-724e72ba1234\") "
Jul 22 01:53:40.773951 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:40.773914    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xcqcq\" (UniqueName: \"kubernetes.io/projected/e497d2fd-b855-4792-b91c-724e72ba1234-kube-api-access-xcqcq\") pod \"e497d2fd-b855-4792-b91c-724e72ba1234\" (UID: \"e497d2fd-b855-4792-b91c-724e72ba1234\") "
Jul 22 01:53:40.773951 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:40.773952    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e497d2fd-b855-4792-b91c-724e72ba1234-local-certs\") pod \"e497d2fd-b855-4792-b91c-724e72ba1234\" (UID: \"e497d2fd-b855-4792-b91c-724e72ba1234\") "
Jul 22 01:53:40.774213 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:40.773974    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e497d2fd-b855-4792-b91c-724e72ba1234-istio-csr-ca-configmap\") pod \"e497d2fd-b855-4792-b91c-724e72ba1234\" (UID: \"e497d2fd-b855-4792-b91c-724e72ba1234\") "
Jul 22 01:53:40.774213 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:40.774010    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e497d2fd-b855-4792-b91c-724e72ba1234-istio-token\") pod \"e497d2fd-b855-4792-b91c-724e72ba1234\" (UID: \"e497d2fd-b855-4792-b91c-724e72ba1234\") "
Jul 22 01:53:40.774213 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:40.774037    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e497d2fd-b855-4792-b91c-724e72ba1234-cacerts\") pod \"e497d2fd-b855-4792-b91c-724e72ba1234\" (UID: \"e497d2fd-b855-4792-b91c-724e72ba1234\") "
Jul 22 01:53:40.774213 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:40.774090    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e497d2fd-b855-4792-b91c-724e72ba1234-istio-kubeconfig\") pod \"e497d2fd-b855-4792-b91c-724e72ba1234\" (UID: \"e497d2fd-b855-4792-b91c-724e72ba1234\") "
Jul 22 01:53:40.774502 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:40.774469    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e497d2fd-b855-4792-b91c-724e72ba1234-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "e497d2fd-b855-4792-b91c-724e72ba1234" (UID: "e497d2fd-b855-4792-b91c-724e72ba1234"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 01:53:40.776372 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:40.776311    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e497d2fd-b855-4792-b91c-724e72ba1234-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "e497d2fd-b855-4792-b91c-724e72ba1234" (UID: "e497d2fd-b855-4792-b91c-724e72ba1234"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 01:53:40.776526 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:40.776468    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e497d2fd-b855-4792-b91c-724e72ba1234-kube-api-access-xcqcq" (OuterVolumeSpecName: "kube-api-access-xcqcq") pod "e497d2fd-b855-4792-b91c-724e72ba1234" (UID: "e497d2fd-b855-4792-b91c-724e72ba1234"). InnerVolumeSpecName "kube-api-access-xcqcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 01:53:40.776609 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:40.776586    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e497d2fd-b855-4792-b91c-724e72ba1234-cacerts" (OuterVolumeSpecName: "cacerts") pod "e497d2fd-b855-4792-b91c-724e72ba1234" (UID: "e497d2fd-b855-4792-b91c-724e72ba1234"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 01:53:40.776660 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:40.776617    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e497d2fd-b855-4792-b91c-724e72ba1234-local-certs" (OuterVolumeSpecName: "local-certs") pod "e497d2fd-b855-4792-b91c-724e72ba1234" (UID: "e497d2fd-b855-4792-b91c-724e72ba1234"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 01:53:40.776805 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:40.776780    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e497d2fd-b855-4792-b91c-724e72ba1234-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "e497d2fd-b855-4792-b91c-724e72ba1234" (UID: "e497d2fd-b855-4792-b91c-724e72ba1234"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 01:53:40.776844 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:40.776782    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e497d2fd-b855-4792-b91c-724e72ba1234-istio-token" (OuterVolumeSpecName: "istio-token") pod "e497d2fd-b855-4792-b91c-724e72ba1234" (UID: "e497d2fd-b855-4792-b91c-724e72ba1234"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 01:53:40.875612 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:40.875572    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e497d2fd-b855-4792-b91c-724e72ba1234-istio-token\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:53:40.875612 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:40.875603    2563 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e497d2fd-b855-4792-b91c-724e72ba1234-cacerts\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:53:40.875612 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:40.875612    2563 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e497d2fd-b855-4792-b91c-724e72ba1234-istio-kubeconfig\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:53:40.875612 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:40.875621    2563 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e497d2fd-b855-4792-b91c-724e72ba1234-istio-csr-dns-cert\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:53:40.875874 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:40.875632    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcqcq\" (UniqueName: \"kubernetes.io/projected/e497d2fd-b855-4792-b91c-724e72ba1234-kube-api-access-xcqcq\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:53:40.875874 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:40.875640    2563 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e497d2fd-b855-4792-b91c-724e72ba1234-local-certs\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:53:40.875874 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:40.875649    2563 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e497d2fd-b855-4792-b91c-724e72ba1234-istio-csr-ca-configmap\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:53:41.133678 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:41.133589    2563 generic.go:356] "Generic (PLEG): container finished" podID="e497d2fd-b855-4792-b91c-724e72ba1234" containerID="9c9fd4ad96eea153ca44023c59068750bb3e715d1dd25e2bbc5a7e205ceb07fd" exitCode=0
Jul 22 01:53:41.133678 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:41.133644    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ks2l4"
Jul 22 01:53:41.134135 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:41.133681    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ks2l4" event={"ID":"e497d2fd-b855-4792-b91c-724e72ba1234","Type":"ContainerDied","Data":"9c9fd4ad96eea153ca44023c59068750bb3e715d1dd25e2bbc5a7e205ceb07fd"}
Jul 22 01:53:41.134135 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:41.133747    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ks2l4" event={"ID":"e497d2fd-b855-4792-b91c-724e72ba1234","Type":"ContainerDied","Data":"c271b1b9476a6c7cb23e910896e8e5ddad72e7eb931d3c3cec7642a195f414a9"}
Jul 22 01:53:41.134135 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:41.133770    2563 scope.go:117] "RemoveContainer" containerID="9c9fd4ad96eea153ca44023c59068750bb3e715d1dd25e2bbc5a7e205ceb07fd"
Jul 22 01:53:41.142094 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:41.142077    2563 scope.go:117] "RemoveContainer" containerID="9c9fd4ad96eea153ca44023c59068750bb3e715d1dd25e2bbc5a7e205ceb07fd"
Jul 22 01:53:41.142363 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:53:41.142345    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c9fd4ad96eea153ca44023c59068750bb3e715d1dd25e2bbc5a7e205ceb07fd\": container with ID starting with 9c9fd4ad96eea153ca44023c59068750bb3e715d1dd25e2bbc5a7e205ceb07fd not found: ID does not exist" containerID="9c9fd4ad96eea153ca44023c59068750bb3e715d1dd25e2bbc5a7e205ceb07fd"
Jul 22 01:53:41.142444 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:41.142372    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c9fd4ad96eea153ca44023c59068750bb3e715d1dd25e2bbc5a7e205ceb07fd"} err="failed to get container status \"9c9fd4ad96eea153ca44023c59068750bb3e715d1dd25e2bbc5a7e205ceb07fd\": rpc error: code = NotFound desc = could not find container \"9c9fd4ad96eea153ca44023c59068750bb3e715d1dd25e2bbc5a7e205ceb07fd\": container with ID starting with 9c9fd4ad96eea153ca44023c59068750bb3e715d1dd25e2bbc5a7e205ceb07fd not found: ID does not exist"
Jul 22 01:53:41.154237 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:41.154206    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-ks2l4"]
Jul 22 01:53:41.159318 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:41.159293    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-ks2l4"]
Jul 22 01:53:41.662552 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:41.662513    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e497d2fd-b855-4792-b91c-724e72ba1234" path="/var/lib/kubelet/pods/e497d2fd-b855-4792-b91c-724e72ba1234/volumes"
Jul 22 01:53:42.084624 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:53:42.084596    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-lgqv9"
Jul 22 01:54:11.136367 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:11.136332    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9qxz4"
Jul 22 01:54:11.139466 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:11.139445    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-62wkf"
Jul 22 01:54:25.985743 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:25.985708    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4b8qh"]
Jul 22 01:54:25.986111 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:25.985973    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e497d2fd-b855-4792-b91c-724e72ba1234" containerName="discovery"
Jul 22 01:54:25.986111 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:25.985983    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e497d2fd-b855-4792-b91c-724e72ba1234" containerName="discovery"
Jul 22 01:54:25.986111 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:25.986040    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e497d2fd-b855-4792-b91c-724e72ba1234" containerName="discovery"
Jul 22 01:54:25.988848 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:25.988829    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4b8qh"
Jul 22 01:54:25.991470 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:25.991446    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-glq4q\""
Jul 22 01:54:25.991470 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:25.991462    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 22 01:54:25.992131 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:25.992112    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 22 01:54:25.998025 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:25.997393    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4b8qh"]
Jul 22 01:54:26.125544 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:26.125505    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhnxj\" (UniqueName: \"kubernetes.io/projected/60dda1f5-0654-40ad-8db5-574b1ff068de-kube-api-access-vhnxj\") pod \"kuadrant-operator-catalog-4b8qh\" (UID: \"60dda1f5-0654-40ad-8db5-574b1ff068de\") " pod="kuadrant-system/kuadrant-operator-catalog-4b8qh"
Jul 22 01:54:26.226464 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:26.226422    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhnxj\" (UniqueName: \"kubernetes.io/projected/60dda1f5-0654-40ad-8db5-574b1ff068de-kube-api-access-vhnxj\") pod \"kuadrant-operator-catalog-4b8qh\" (UID: \"60dda1f5-0654-40ad-8db5-574b1ff068de\") " pod="kuadrant-system/kuadrant-operator-catalog-4b8qh"
Jul 22 01:54:26.235550 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:26.235518    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhnxj\" (UniqueName: \"kubernetes.io/projected/60dda1f5-0654-40ad-8db5-574b1ff068de-kube-api-access-vhnxj\") pod \"kuadrant-operator-catalog-4b8qh\" (UID: \"60dda1f5-0654-40ad-8db5-574b1ff068de\") " pod="kuadrant-system/kuadrant-operator-catalog-4b8qh"
Jul 22 01:54:26.299430 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:26.299325    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4b8qh"
Jul 22 01:54:26.354918 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:26.354888    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4b8qh"]
Jul 22 01:54:26.427633 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:26.427604    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4b8qh"]
Jul 22 01:54:26.430560 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:54:26.430532    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60dda1f5_0654_40ad_8db5_574b1ff068de.slice/crio-855c4ed78aa194fad64346d632cba15b4b9a9180f4640f78f38617a9d92d7816 WatchSource:0}: Error finding container 855c4ed78aa194fad64346d632cba15b4b9a9180f4640f78f38617a9d92d7816: Status 404 returned error can't find the container with id 855c4ed78aa194fad64346d632cba15b4b9a9180f4640f78f38617a9d92d7816
Jul 22 01:54:26.562810 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:26.562728    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bhvjd"]
Jul 22 01:54:26.566908 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:26.566891    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-bhvjd"
Jul 22 01:54:26.574099 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:26.574059    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bhvjd"]
Jul 22 01:54:26.729454 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:26.729421    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58w6g\" (UniqueName: \"kubernetes.io/projected/869bc209-0252-4361-a934-c57c0284885e-kube-api-access-58w6g\") pod \"kuadrant-operator-catalog-bhvjd\" (UID: \"869bc209-0252-4361-a934-c57c0284885e\") " pod="kuadrant-system/kuadrant-operator-catalog-bhvjd"
Jul 22 01:54:26.831032 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:26.830929    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58w6g\" (UniqueName: \"kubernetes.io/projected/869bc209-0252-4361-a934-c57c0284885e-kube-api-access-58w6g\") pod \"kuadrant-operator-catalog-bhvjd\" (UID: \"869bc209-0252-4361-a934-c57c0284885e\") " pod="kuadrant-system/kuadrant-operator-catalog-bhvjd"
Jul 22 01:54:26.838763 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:26.838734    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58w6g\" (UniqueName: \"kubernetes.io/projected/869bc209-0252-4361-a934-c57c0284885e-kube-api-access-58w6g\") pod \"kuadrant-operator-catalog-bhvjd\" (UID: \"869bc209-0252-4361-a934-c57c0284885e\") " pod="kuadrant-system/kuadrant-operator-catalog-bhvjd"
Jul 22 01:54:26.876881 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:26.876842    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-bhvjd"
Jul 22 01:54:27.014223 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:27.014184    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bhvjd"]
Jul 22 01:54:27.016914 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:54:27.016881    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod869bc209_0252_4361_a934_c57c0284885e.slice/crio-b1126c6d7a59ddb1f845fccddd56aa7342d94d52a4332fca1df3ba3966fe7921 WatchSource:0}: Error finding container b1126c6d7a59ddb1f845fccddd56aa7342d94d52a4332fca1df3ba3966fe7921: Status 404 returned error can't find the container with id b1126c6d7a59ddb1f845fccddd56aa7342d94d52a4332fca1df3ba3966fe7921
Jul 22 01:54:27.322335 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:27.322300    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-bhvjd" event={"ID":"869bc209-0252-4361-a934-c57c0284885e","Type":"ContainerStarted","Data":"b1126c6d7a59ddb1f845fccddd56aa7342d94d52a4332fca1df3ba3966fe7921"}
Jul 22 01:54:27.323353 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:27.323327    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4b8qh" event={"ID":"60dda1f5-0654-40ad-8db5-574b1ff068de","Type":"ContainerStarted","Data":"855c4ed78aa194fad64346d632cba15b4b9a9180f4640f78f38617a9d92d7816"}
Jul 22 01:54:29.333353 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:29.333312    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4b8qh" event={"ID":"60dda1f5-0654-40ad-8db5-574b1ff068de","Type":"ContainerStarted","Data":"a863bd6390c730d6b689bda66dc70dbb70b41cf4fb11561fa109ef37aeb39d18"}
Jul 22 01:54:29.333863 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:29.333357    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-4b8qh" podUID="60dda1f5-0654-40ad-8db5-574b1ff068de" containerName="registry-server" containerID="cri-o://a863bd6390c730d6b689bda66dc70dbb70b41cf4fb11561fa109ef37aeb39d18" gracePeriod=2
Jul 22 01:54:29.334984 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:29.334955    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-bhvjd" event={"ID":"869bc209-0252-4361-a934-c57c0284885e","Type":"ContainerStarted","Data":"5791a599286f15e1c7fbf9924b4a01597c1bf621e27786160f302ef7331bd56a"}
Jul 22 01:54:29.349148 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:29.349097    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-4b8qh" podStartSLOduration=2.241941406 podStartE2EDuration="4.349081086s" totalImagesPullingTime="2.10713968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:54:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:54:29.347134497 +0000 UTC m=+422.190963630" watchObservedRunningTime="2026-07-22 01:54:29.349081086 +0000 UTC m=+422.192910233"
Jul 22 01:54:29.362238 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:29.362192    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-bhvjd" podStartSLOduration=1.838111496 podStartE2EDuration="3.362176322s" totalImagesPullingTime="1.524064826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:54:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:54:29.360983322 +0000 UTC m=+422.204812455" watchObservedRunningTime="2026-07-22 01:54:29.362176322 +0000 UTC m=+422.206005453"
Jul 22 01:54:29.581524 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:29.581499    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4b8qh"
Jul 22 01:54:29.756871 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:29.756830    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhnxj\" (UniqueName: \"kubernetes.io/projected/60dda1f5-0654-40ad-8db5-574b1ff068de-kube-api-access-vhnxj\") pod \"60dda1f5-0654-40ad-8db5-574b1ff068de\" (UID: \"60dda1f5-0654-40ad-8db5-574b1ff068de\") "
Jul 22 01:54:29.759125 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:29.759099    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60dda1f5-0654-40ad-8db5-574b1ff068de-kube-api-access-vhnxj" (OuterVolumeSpecName: "kube-api-access-vhnxj") pod "60dda1f5-0654-40ad-8db5-574b1ff068de" (UID: "60dda1f5-0654-40ad-8db5-574b1ff068de"). InnerVolumeSpecName "kube-api-access-vhnxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 01:54:29.857572 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:29.857529    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhnxj\" (UniqueName: \"kubernetes.io/projected/60dda1f5-0654-40ad-8db5-574b1ff068de-kube-api-access-vhnxj\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:54:30.340827 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:30.340788    2563 generic.go:356] "Generic (PLEG): container finished" podID="60dda1f5-0654-40ad-8db5-574b1ff068de" containerID="a863bd6390c730d6b689bda66dc70dbb70b41cf4fb11561fa109ef37aeb39d18" exitCode=0
Jul 22 01:54:30.341310 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:30.340832    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4b8qh"
Jul 22 01:54:30.341310 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:30.340876    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4b8qh" event={"ID":"60dda1f5-0654-40ad-8db5-574b1ff068de","Type":"ContainerDied","Data":"a863bd6390c730d6b689bda66dc70dbb70b41cf4fb11561fa109ef37aeb39d18"}
Jul 22 01:54:30.341310 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:30.340928    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4b8qh" event={"ID":"60dda1f5-0654-40ad-8db5-574b1ff068de","Type":"ContainerDied","Data":"855c4ed78aa194fad64346d632cba15b4b9a9180f4640f78f38617a9d92d7816"}
Jul 22 01:54:30.341310 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:30.340949    2563 scope.go:117] "RemoveContainer" containerID="a863bd6390c730d6b689bda66dc70dbb70b41cf4fb11561fa109ef37aeb39d18"
Jul 22 01:54:30.350402 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:30.350355    2563 scope.go:117] "RemoveContainer" containerID="a863bd6390c730d6b689bda66dc70dbb70b41cf4fb11561fa109ef37aeb39d18"
Jul 22 01:54:30.350701 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:54:30.350685    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a863bd6390c730d6b689bda66dc70dbb70b41cf4fb11561fa109ef37aeb39d18\": container with ID starting with a863bd6390c730d6b689bda66dc70dbb70b41cf4fb11561fa109ef37aeb39d18 not found: ID does not exist" containerID="a863bd6390c730d6b689bda66dc70dbb70b41cf4fb11561fa109ef37aeb39d18"
Jul 22 01:54:30.350752 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:30.350710    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a863bd6390c730d6b689bda66dc70dbb70b41cf4fb11561fa109ef37aeb39d18"} err="failed to get container status \"a863bd6390c730d6b689bda66dc70dbb70b41cf4fb11561fa109ef37aeb39d18\": rpc error: code = NotFound desc = could not find container \"a863bd6390c730d6b689bda66dc70dbb70b41cf4fb11561fa109ef37aeb39d18\": container with ID starting with a863bd6390c730d6b689bda66dc70dbb70b41cf4fb11561fa109ef37aeb39d18 not found: ID does not exist"
Jul 22 01:54:30.361106 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:30.361078    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4b8qh"]
Jul 22 01:54:30.366660 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:30.366638    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4b8qh"]
Jul 22 01:54:31.664837 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:31.664795    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60dda1f5-0654-40ad-8db5-574b1ff068de" path="/var/lib/kubelet/pods/60dda1f5-0654-40ad-8db5-574b1ff068de/volumes"
Jul 22 01:54:36.877680 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:36.877638    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-bhvjd"
Jul 22 01:54:36.878091 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:36.877715    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-bhvjd"
Jul 22 01:54:36.900353 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:36.900328    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-bhvjd"
Jul 22 01:54:37.394245 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:37.394211    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-bhvjd"
Jul 22 01:54:41.597074 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:41.597036    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfjcd"]
Jul 22 01:54:41.597509 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:41.597341    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60dda1f5-0654-40ad-8db5-574b1ff068de" containerName="registry-server"
Jul 22 01:54:41.597509 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:41.597353    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="60dda1f5-0654-40ad-8db5-574b1ff068de" containerName="registry-server"
Jul 22 01:54:41.597509 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:41.597437    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="60dda1f5-0654-40ad-8db5-574b1ff068de" containerName="registry-server"
Jul 22 01:54:41.601882 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:41.601862    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfjcd"
Jul 22 01:54:41.604004 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:41.603982    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7zd2t\""
Jul 22 01:54:41.607543 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:41.607511    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfjcd"]
Jul 22 01:54:41.649863 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:41.649820    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62e08628-fa98-48a4-99cc-3dc660dbc84e-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfjcd\" (UID: \"62e08628-fa98-48a4-99cc-3dc660dbc84e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfjcd"
Jul 22 01:54:41.649863 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:41.649862    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztqdc\" (UniqueName: \"kubernetes.io/projected/62e08628-fa98-48a4-99cc-3dc660dbc84e-kube-api-access-ztqdc\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfjcd\" (UID: \"62e08628-fa98-48a4-99cc-3dc660dbc84e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfjcd"
Jul 22 01:54:41.650091 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:41.649895    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62e08628-fa98-48a4-99cc-3dc660dbc84e-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfjcd\" (UID: \"62e08628-fa98-48a4-99cc-3dc660dbc84e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfjcd"
Jul 22 01:54:41.751038 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:41.750991    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62e08628-fa98-48a4-99cc-3dc660dbc84e-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfjcd\" (UID: \"62e08628-fa98-48a4-99cc-3dc660dbc84e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfjcd"
Jul 22 01:54:41.751233 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:41.751077    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62e08628-fa98-48a4-99cc-3dc660dbc84e-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfjcd\" (UID: \"62e08628-fa98-48a4-99cc-3dc660dbc84e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfjcd"
Jul 22 01:54:41.751233 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:41.751095    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztqdc\" (UniqueName: \"kubernetes.io/projected/62e08628-fa98-48a4-99cc-3dc660dbc84e-kube-api-access-ztqdc\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfjcd\" (UID: \"62e08628-fa98-48a4-99cc-3dc660dbc84e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfjcd"
Jul 22 01:54:41.751457 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:41.751439    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62e08628-fa98-48a4-99cc-3dc660dbc84e-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfjcd\" (UID: \"62e08628-fa98-48a4-99cc-3dc660dbc84e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfjcd"
Jul 22 01:54:41.751507 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:41.751459    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62e08628-fa98-48a4-99cc-3dc660dbc84e-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfjcd\" (UID: \"62e08628-fa98-48a4-99cc-3dc660dbc84e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfjcd"
Jul 22 01:54:41.759172 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:41.759140    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztqdc\" (UniqueName: \"kubernetes.io/projected/62e08628-fa98-48a4-99cc-3dc660dbc84e-kube-api-access-ztqdc\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfjcd\" (UID: \"62e08628-fa98-48a4-99cc-3dc660dbc84e\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfjcd"
Jul 22 01:54:41.912167 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:41.912062    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfjcd"
Jul 22 01:54:42.042999 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:42.042911    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfjcd"]
Jul 22 01:54:42.045046 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:54:42.045018    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62e08628_fa98_48a4_99cc_3dc660dbc84e.slice/crio-9f7f01530c3b06e76caaa296cfa964058cd800fcf3dfcccbdcfaf2992c126c4b WatchSource:0}: Error finding container 9f7f01530c3b06e76caaa296cfa964058cd800fcf3dfcccbdcfaf2992c126c4b: Status 404 returned error can't find the container with id 9f7f01530c3b06e76caaa296cfa964058cd800fcf3dfcccbdcfaf2992c126c4b
Jul 22 01:54:42.393921 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:42.393890    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vjn4s"]
Jul 22 01:54:42.396014 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:42.395989    2563 generic.go:356] "Generic (PLEG): container finished" podID="62e08628-fa98-48a4-99cc-3dc660dbc84e" containerID="0d950b1c7428ebe60b8f5f9d3dc79e015734ef6f04845a5ffeffa11a7d9dab80" exitCode=0
Jul 22 01:54:42.397366 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:42.397343    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfjcd" event={"ID":"62e08628-fa98-48a4-99cc-3dc660dbc84e","Type":"ContainerDied","Data":"0d950b1c7428ebe60b8f5f9d3dc79e015734ef6f04845a5ffeffa11a7d9dab80"}
Jul 22 01:54:42.397465 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:42.397386    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfjcd" event={"ID":"62e08628-fa98-48a4-99cc-3dc660dbc84e","Type":"ContainerStarted","Data":"9f7f01530c3b06e76caaa296cfa964058cd800fcf3dfcccbdcfaf2992c126c4b"}
Jul 22 01:54:42.397504 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:42.397460    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vjn4s"
Jul 22 01:54:42.405978 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:42.405954    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vjn4s"]
Jul 22 01:54:42.455950 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:42.455921    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43423d8a-faea-4db0-84ae-8af34f0eb834-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vjn4s\" (UID: \"43423d8a-faea-4db0-84ae-8af34f0eb834\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vjn4s"
Jul 22 01:54:42.456131 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:42.455967    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43423d8a-faea-4db0-84ae-8af34f0eb834-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vjn4s\" (UID: \"43423d8a-faea-4db0-84ae-8af34f0eb834\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vjn4s"
Jul 22 01:54:42.456131 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:42.456035    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5gp9\" (UniqueName: \"kubernetes.io/projected/43423d8a-faea-4db0-84ae-8af34f0eb834-kube-api-access-s5gp9\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vjn4s\" (UID: \"43423d8a-faea-4db0-84ae-8af34f0eb834\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vjn4s"
Jul 22 01:54:42.557396 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:42.557347    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5gp9\" (UniqueName: \"kubernetes.io/projected/43423d8a-faea-4db0-84ae-8af34f0eb834-kube-api-access-s5gp9\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vjn4s\" (UID: \"43423d8a-faea-4db0-84ae-8af34f0eb834\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vjn4s"
Jul 22 01:54:42.557587 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:42.557448    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43423d8a-faea-4db0-84ae-8af34f0eb834-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vjn4s\" (UID: \"43423d8a-faea-4db0-84ae-8af34f0eb834\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vjn4s"
Jul 22 01:54:42.557587 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:42.557474    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43423d8a-faea-4db0-84ae-8af34f0eb834-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vjn4s\" (UID: \"43423d8a-faea-4db0-84ae-8af34f0eb834\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vjn4s"
Jul 22 01:54:42.557844 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:42.557824    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43423d8a-faea-4db0-84ae-8af34f0eb834-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vjn4s\" (UID: \"43423d8a-faea-4db0-84ae-8af34f0eb834\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vjn4s"
Jul 22 01:54:42.557926 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:42.557868    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43423d8a-faea-4db0-84ae-8af34f0eb834-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vjn4s\" (UID: \"43423d8a-faea-4db0-84ae-8af34f0eb834\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vjn4s"
Jul 22 01:54:42.565368 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:42.565341    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5gp9\" (UniqueName: \"kubernetes.io/projected/43423d8a-faea-4db0-84ae-8af34f0eb834-kube-api-access-s5gp9\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vjn4s\" (UID: \"43423d8a-faea-4db0-84ae-8af34f0eb834\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vjn4s"
Jul 22 01:54:42.707590 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:42.707555    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vjn4s"
Jul 22 01:54:42.837452 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:42.837421    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vjn4s"]
Jul 22 01:54:42.840424 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:54:42.840399    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43423d8a_faea_4db0_84ae_8af34f0eb834.slice/crio-0474457f281fc459528f9e232b24468ea21b9193c1955008669f4f3d9ffea1c2 WatchSource:0}: Error finding container 0474457f281fc459528f9e232b24468ea21b9193c1955008669f4f3d9ffea1c2: Status 404 returned error can't find the container with id 0474457f281fc459528f9e232b24468ea21b9193c1955008669f4f3d9ffea1c2
Jul 22 01:54:42.996784 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:42.996712    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gzzv6"]
Jul 22 01:54:43.000610 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:43.000584    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gzzv6"
Jul 22 01:54:43.007966 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:43.007938    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gzzv6"]
Jul 22 01:54:43.062747 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:43.062704    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f575f0b-aaa2-413d-87f2-ce9d69096fb8-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gzzv6\" (UID: \"5f575f0b-aaa2-413d-87f2-ce9d69096fb8\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gzzv6"
Jul 22 01:54:43.062909 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:43.062795    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mqzm\" (UniqueName: \"kubernetes.io/projected/5f575f0b-aaa2-413d-87f2-ce9d69096fb8-kube-api-access-8mqzm\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gzzv6\" (UID: \"5f575f0b-aaa2-413d-87f2-ce9d69096fb8\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gzzv6"
Jul 22 01:54:43.062909 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:43.062845    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f575f0b-aaa2-413d-87f2-ce9d69096fb8-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gzzv6\" (UID: \"5f575f0b-aaa2-413d-87f2-ce9d69096fb8\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gzzv6"
Jul 22 01:54:43.163694 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:43.163655    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f575f0b-aaa2-413d-87f2-ce9d69096fb8-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gzzv6\" (UID: \"5f575f0b-aaa2-413d-87f2-ce9d69096fb8\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gzzv6"
Jul 22 01:54:43.163694 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:43.163699    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f575f0b-aaa2-413d-87f2-ce9d69096fb8-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gzzv6\" (UID: \"5f575f0b-aaa2-413d-87f2-ce9d69096fb8\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gzzv6"
Jul 22 01:54:43.163893 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:43.163742    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mqzm\" (UniqueName: \"kubernetes.io/projected/5f575f0b-aaa2-413d-87f2-ce9d69096fb8-kube-api-access-8mqzm\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gzzv6\" (UID: \"5f575f0b-aaa2-413d-87f2-ce9d69096fb8\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gzzv6"
Jul 22 01:54:43.164128 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:43.164102    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f575f0b-aaa2-413d-87f2-ce9d69096fb8-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gzzv6\" (UID: \"5f575f0b-aaa2-413d-87f2-ce9d69096fb8\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gzzv6"
Jul 22 01:54:43.164170 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:43.164110    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f575f0b-aaa2-413d-87f2-ce9d69096fb8-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gzzv6\" (UID: \"5f575f0b-aaa2-413d-87f2-ce9d69096fb8\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gzzv6"
Jul 22 01:54:43.172436 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:43.172410    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mqzm\" (UniqueName: \"kubernetes.io/projected/5f575f0b-aaa2-413d-87f2-ce9d69096fb8-kube-api-access-8mqzm\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gzzv6\" (UID: \"5f575f0b-aaa2-413d-87f2-ce9d69096fb8\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gzzv6"
Jul 22 01:54:43.312251 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:43.312221    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gzzv6"
Jul 22 01:54:43.402283 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:43.402252    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bmkm4"]
Jul 22 01:54:43.403130 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:43.403104    2563 generic.go:356] "Generic (PLEG): container finished" podID="43423d8a-faea-4db0-84ae-8af34f0eb834" containerID="f10809d070c5df7ed1f83aa6f4014b31ac634bc8262c6b02e8ea12761ae887da" exitCode=0
Jul 22 01:54:43.405520 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:43.405495    2563 generic.go:356] "Generic (PLEG): container finished" podID="62e08628-fa98-48a4-99cc-3dc660dbc84e" containerID="ea2f29cbdd0c873ed23a81645c0b3e47930239f2396d99351c27849ca07eda95" exitCode=0
Jul 22 01:54:43.407398 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:43.407360    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vjn4s" event={"ID":"43423d8a-faea-4db0-84ae-8af34f0eb834","Type":"ContainerDied","Data":"f10809d070c5df7ed1f83aa6f4014b31ac634bc8262c6b02e8ea12761ae887da"}
Jul 22 01:54:43.407533 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:43.407487    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vjn4s" event={"ID":"43423d8a-faea-4db0-84ae-8af34f0eb834","Type":"ContainerStarted","Data":"0474457f281fc459528f9e232b24468ea21b9193c1955008669f4f3d9ffea1c2"}
Jul 22 01:54:43.407533 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:43.407518    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfjcd" event={"ID":"62e08628-fa98-48a4-99cc-3dc660dbc84e","Type":"ContainerDied","Data":"ea2f29cbdd0c873ed23a81645c0b3e47930239f2396d99351c27849ca07eda95"}
Jul 22 01:54:43.407684 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:43.407495    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bmkm4"
Jul 22 01:54:43.417077 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:43.416631    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bmkm4"]
Jul 22 01:54:43.450202 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:43.450180    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gzzv6"]
Jul 22 01:54:43.452933 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:54:43.452909    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f575f0b_aaa2_413d_87f2_ce9d69096fb8.slice/crio-c47ab7906acc2609a13e1b0b1eb50435629846eb43c9fe29372746005ac6d272 WatchSource:0}: Error finding container c47ab7906acc2609a13e1b0b1eb50435629846eb43c9fe29372746005ac6d272: Status 404 returned error can't find the container with id c47ab7906acc2609a13e1b0b1eb50435629846eb43c9fe29372746005ac6d272
Jul 22 01:54:43.466208 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:43.466181    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67lb7\" (UniqueName: \"kubernetes.io/projected/24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9-kube-api-access-67lb7\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bmkm4\" (UID: \"24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bmkm4"
Jul 22 01:54:43.466293 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:43.466263    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bmkm4\" (UID: \"24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bmkm4"
Jul 22 01:54:43.466355 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:43.466316    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bmkm4\" (UID: \"24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bmkm4"
Jul 22 01:54:43.567620 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:43.567586    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bmkm4\" (UID: \"24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bmkm4"
Jul 22 01:54:43.567768 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:43.567651    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bmkm4\" (UID: \"24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bmkm4"
Jul 22 01:54:43.567768 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:43.567682    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67lb7\" (UniqueName: \"kubernetes.io/projected/24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9-kube-api-access-67lb7\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bmkm4\" (UID: \"24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bmkm4"
Jul 22 01:54:43.567935 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:43.567918    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bmkm4\" (UID: \"24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bmkm4"
Jul 22 01:54:43.567981 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:43.567949    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bmkm4\" (UID: \"24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bmkm4"
Jul 22 01:54:43.581371 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:43.581344    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67lb7\" (UniqueName: \"kubernetes.io/projected/24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9-kube-api-access-67lb7\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bmkm4\" (UID: \"24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bmkm4"
Jul 22 01:54:43.721042 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:43.720995    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bmkm4"
Jul 22 01:54:43.852336 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:43.852277    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bmkm4"]
Jul 22 01:54:43.854986 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:54:43.854954    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24fb3a78_b34a_466a_bb1f_ec9bbf1d5bb9.slice/crio-a6f76bdb545448287406d3478abfce16e717a6615e1487d7a9a34fd613850efc WatchSource:0}: Error finding container a6f76bdb545448287406d3478abfce16e717a6615e1487d7a9a34fd613850efc: Status 404 returned error can't find the container with id a6f76bdb545448287406d3478abfce16e717a6615e1487d7a9a34fd613850efc
Jul 22 01:54:44.411694 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:44.411607    2563 generic.go:356] "Generic (PLEG): container finished" podID="43423d8a-faea-4db0-84ae-8af34f0eb834" containerID="690f726988136cf4cdda0c02acb3ed14c15904ee788a7dcdbce08dc83b88a202" exitCode=0
Jul 22 01:54:44.411845 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:44.411693    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vjn4s" event={"ID":"43423d8a-faea-4db0-84ae-8af34f0eb834","Type":"ContainerDied","Data":"690f726988136cf4cdda0c02acb3ed14c15904ee788a7dcdbce08dc83b88a202"}
Jul 22 01:54:44.413912 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:44.413883    2563 generic.go:356] "Generic (PLEG): container finished" podID="62e08628-fa98-48a4-99cc-3dc660dbc84e" containerID="d8b08e43824ad24b00da8020df97c95050718de3574582d2bbae4f85d6c66b56" exitCode=0
Jul 22 01:54:44.414000 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:44.413967    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfjcd" event={"ID":"62e08628-fa98-48a4-99cc-3dc660dbc84e","Type":"ContainerDied","Data":"d8b08e43824ad24b00da8020df97c95050718de3574582d2bbae4f85d6c66b56"}
Jul 22 01:54:44.415400 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:44.415352    2563 generic.go:356] "Generic (PLEG): container finished" podID="5f575f0b-aaa2-413d-87f2-ce9d69096fb8" containerID="84c6ab85f2b7cfe277d9917a980f2a5347a93ed5d665a156af315087baac769f" exitCode=0
Jul 22 01:54:44.415495 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:44.415414    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gzzv6" event={"ID":"5f575f0b-aaa2-413d-87f2-ce9d69096fb8","Type":"ContainerDied","Data":"84c6ab85f2b7cfe277d9917a980f2a5347a93ed5d665a156af315087baac769f"}
Jul 22 01:54:44.415495 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:44.415456    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gzzv6" event={"ID":"5f575f0b-aaa2-413d-87f2-ce9d69096fb8","Type":"ContainerStarted","Data":"c47ab7906acc2609a13e1b0b1eb50435629846eb43c9fe29372746005ac6d272"}
Jul 22 01:54:44.417034 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:44.417010    2563 generic.go:356] "Generic (PLEG): container finished" podID="24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9" containerID="8f21eb9125699d5780d258ca16fc0b4bb4da98af1ca6bc033bca66b749e39c46" exitCode=0
Jul 22 01:54:44.417121 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:44.417092    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bmkm4" event={"ID":"24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9","Type":"ContainerDied","Data":"8f21eb9125699d5780d258ca16fc0b4bb4da98af1ca6bc033bca66b749e39c46"}
Jul 22 01:54:44.417183 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:44.417138    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bmkm4" event={"ID":"24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9","Type":"ContainerStarted","Data":"a6f76bdb545448287406d3478abfce16e717a6615e1487d7a9a34fd613850efc"}
Jul 22 01:54:45.423476 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:45.423447    2563 generic.go:356] "Generic (PLEG): container finished" podID="43423d8a-faea-4db0-84ae-8af34f0eb834" containerID="ca20de39aa6129191fe39231de72636e55f20c7e26810b628328530138cdfa81" exitCode=0
Jul 22 01:54:45.423916 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:45.423537    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vjn4s" event={"ID":"43423d8a-faea-4db0-84ae-8af34f0eb834","Type":"ContainerDied","Data":"ca20de39aa6129191fe39231de72636e55f20c7e26810b628328530138cdfa81"}
Jul 22 01:54:45.425281 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:45.425260    2563 generic.go:356] "Generic (PLEG): container finished" podID="5f575f0b-aaa2-413d-87f2-ce9d69096fb8" containerID="5cf75ba37f0d4c1aa1d035077307b7948841dbb8acbf16ba319e7726041a51c9" exitCode=0
Jul 22 01:54:45.425421 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:45.425342    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gzzv6" event={"ID":"5f575f0b-aaa2-413d-87f2-ce9d69096fb8","Type":"ContainerDied","Data":"5cf75ba37f0d4c1aa1d035077307b7948841dbb8acbf16ba319e7726041a51c9"}
Jul 22 01:54:45.427168 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:45.427144    2563 generic.go:356] "Generic (PLEG): container finished" podID="24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9" containerID="ad5185e4da51cf7689064f11b6015b8f9a00ae568567290df22a77c7e8e83bab" exitCode=0
Jul 22 01:54:45.427260 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:45.427225    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bmkm4" event={"ID":"24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9","Type":"ContainerDied","Data":"ad5185e4da51cf7689064f11b6015b8f9a00ae568567290df22a77c7e8e83bab"}
Jul 22 01:54:45.556720 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:45.556692    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfjcd"
Jul 22 01:54:45.586013 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:45.585984    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62e08628-fa98-48a4-99cc-3dc660dbc84e-bundle\") pod \"62e08628-fa98-48a4-99cc-3dc660dbc84e\" (UID: \"62e08628-fa98-48a4-99cc-3dc660dbc84e\") "
Jul 22 01:54:45.586160 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:45.586050    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62e08628-fa98-48a4-99cc-3dc660dbc84e-util\") pod \"62e08628-fa98-48a4-99cc-3dc660dbc84e\" (UID: \"62e08628-fa98-48a4-99cc-3dc660dbc84e\") "
Jul 22 01:54:45.586160 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:45.586077    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztqdc\" (UniqueName: \"kubernetes.io/projected/62e08628-fa98-48a4-99cc-3dc660dbc84e-kube-api-access-ztqdc\") pod \"62e08628-fa98-48a4-99cc-3dc660dbc84e\" (UID: \"62e08628-fa98-48a4-99cc-3dc660dbc84e\") "
Jul 22 01:54:45.586718 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:45.586686    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62e08628-fa98-48a4-99cc-3dc660dbc84e-bundle" (OuterVolumeSpecName: "bundle") pod "62e08628-fa98-48a4-99cc-3dc660dbc84e" (UID: "62e08628-fa98-48a4-99cc-3dc660dbc84e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 01:54:45.588475 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:45.588439    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62e08628-fa98-48a4-99cc-3dc660dbc84e-kube-api-access-ztqdc" (OuterVolumeSpecName: "kube-api-access-ztqdc") pod "62e08628-fa98-48a4-99cc-3dc660dbc84e" (UID: "62e08628-fa98-48a4-99cc-3dc660dbc84e"). InnerVolumeSpecName "kube-api-access-ztqdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 01:54:45.592149 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:45.592123    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62e08628-fa98-48a4-99cc-3dc660dbc84e-util" (OuterVolumeSpecName: "util") pod "62e08628-fa98-48a4-99cc-3dc660dbc84e" (UID: "62e08628-fa98-48a4-99cc-3dc660dbc84e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 01:54:45.687139 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:45.687113    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62e08628-fa98-48a4-99cc-3dc660dbc84e-util\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:54:45.687139 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:45.687139    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztqdc\" (UniqueName: \"kubernetes.io/projected/62e08628-fa98-48a4-99cc-3dc660dbc84e-kube-api-access-ztqdc\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:54:45.687296 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:45.687151    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62e08628-fa98-48a4-99cc-3dc660dbc84e-bundle\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:54:46.433649 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:46.433612    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfjcd"
Jul 22 01:54:46.434045 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:46.433616    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfjcd" event={"ID":"62e08628-fa98-48a4-99cc-3dc660dbc84e","Type":"ContainerDied","Data":"9f7f01530c3b06e76caaa296cfa964058cd800fcf3dfcccbdcfaf2992c126c4b"}
Jul 22 01:54:46.434045 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:46.433734    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f7f01530c3b06e76caaa296cfa964058cd800fcf3dfcccbdcfaf2992c126c4b"
Jul 22 01:54:46.435674 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:46.435613    2563 generic.go:356] "Generic (PLEG): container finished" podID="5f575f0b-aaa2-413d-87f2-ce9d69096fb8" containerID="9cf46d6233fb807886a79bff27446958dbed8f1892a260d5e4f979caed1ced37" exitCode=0
Jul 22 01:54:46.435853 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:46.435697    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gzzv6" event={"ID":"5f575f0b-aaa2-413d-87f2-ce9d69096fb8","Type":"ContainerDied","Data":"9cf46d6233fb807886a79bff27446958dbed8f1892a260d5e4f979caed1ced37"}
Jul 22 01:54:46.437735 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:46.437716    2563 generic.go:356] "Generic (PLEG): container finished" podID="24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9" containerID="6e5677f9128870b7a2b896a40856f9067d39dc7fe22855029ea5b3f143d463fb" exitCode=0
Jul 22 01:54:46.437836 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:46.437777    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bmkm4" event={"ID":"24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9","Type":"ContainerDied","Data":"6e5677f9128870b7a2b896a40856f9067d39dc7fe22855029ea5b3f143d463fb"}
Jul 22 01:54:46.559760 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:46.559735    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vjn4s"
Jul 22 01:54:46.595864 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:46.595832    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5gp9\" (UniqueName: \"kubernetes.io/projected/43423d8a-faea-4db0-84ae-8af34f0eb834-kube-api-access-s5gp9\") pod \"43423d8a-faea-4db0-84ae-8af34f0eb834\" (UID: \"43423d8a-faea-4db0-84ae-8af34f0eb834\") "
Jul 22 01:54:46.596045 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:46.595877    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43423d8a-faea-4db0-84ae-8af34f0eb834-bundle\") pod \"43423d8a-faea-4db0-84ae-8af34f0eb834\" (UID: \"43423d8a-faea-4db0-84ae-8af34f0eb834\") "
Jul 22 01:54:46.596045 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:46.595923    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43423d8a-faea-4db0-84ae-8af34f0eb834-util\") pod \"43423d8a-faea-4db0-84ae-8af34f0eb834\" (UID: \"43423d8a-faea-4db0-84ae-8af34f0eb834\") "
Jul 22 01:54:46.596677 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:46.596648    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43423d8a-faea-4db0-84ae-8af34f0eb834-bundle" (OuterVolumeSpecName: "bundle") pod "43423d8a-faea-4db0-84ae-8af34f0eb834" (UID: "43423d8a-faea-4db0-84ae-8af34f0eb834"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 01:54:46.597996 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:46.597962    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43423d8a-faea-4db0-84ae-8af34f0eb834-kube-api-access-s5gp9" (OuterVolumeSpecName: "kube-api-access-s5gp9") pod "43423d8a-faea-4db0-84ae-8af34f0eb834" (UID: "43423d8a-faea-4db0-84ae-8af34f0eb834"). InnerVolumeSpecName "kube-api-access-s5gp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 01:54:46.601647 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:46.601621    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43423d8a-faea-4db0-84ae-8af34f0eb834-util" (OuterVolumeSpecName: "util") pod "43423d8a-faea-4db0-84ae-8af34f0eb834" (UID: "43423d8a-faea-4db0-84ae-8af34f0eb834"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 01:54:46.697027 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:46.696932    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5gp9\" (UniqueName: \"kubernetes.io/projected/43423d8a-faea-4db0-84ae-8af34f0eb834-kube-api-access-s5gp9\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:54:46.697027 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:46.696968    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43423d8a-faea-4db0-84ae-8af34f0eb834-bundle\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:54:46.697027 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:46.696979    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43423d8a-faea-4db0-84ae-8af34f0eb834-util\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:54:47.443664 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:47.443632    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vjn4s"
Jul 22 01:54:47.443664 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:47.443651    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vjn4s" event={"ID":"43423d8a-faea-4db0-84ae-8af34f0eb834","Type":"ContainerDied","Data":"0474457f281fc459528f9e232b24468ea21b9193c1955008669f4f3d9ffea1c2"}
Jul 22 01:54:47.444193 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:47.443698    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0474457f281fc459528f9e232b24468ea21b9193c1955008669f4f3d9ffea1c2"
Jul 22 01:54:47.584747 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:47.584725    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bmkm4"
Jul 22 01:54:47.610582 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:47.610563    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gzzv6"
Jul 22 01:54:47.706144 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:47.706066    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67lb7\" (UniqueName: \"kubernetes.io/projected/24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9-kube-api-access-67lb7\") pod \"24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9\" (UID: \"24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9\") "
Jul 22 01:54:47.706144 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:47.706112    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mqzm\" (UniqueName: \"kubernetes.io/projected/5f575f0b-aaa2-413d-87f2-ce9d69096fb8-kube-api-access-8mqzm\") pod \"5f575f0b-aaa2-413d-87f2-ce9d69096fb8\" (UID: \"5f575f0b-aaa2-413d-87f2-ce9d69096fb8\") "
Jul 22 01:54:47.706144 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:47.706138    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f575f0b-aaa2-413d-87f2-ce9d69096fb8-bundle\") pod \"5f575f0b-aaa2-413d-87f2-ce9d69096fb8\" (UID: \"5f575f0b-aaa2-413d-87f2-ce9d69096fb8\") "
Jul 22 01:54:47.706513 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:47.706166    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f575f0b-aaa2-413d-87f2-ce9d69096fb8-util\") pod \"5f575f0b-aaa2-413d-87f2-ce9d69096fb8\" (UID: \"5f575f0b-aaa2-413d-87f2-ce9d69096fb8\") "
Jul 22 01:54:47.706513 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:47.706196    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9-util\") pod \"24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9\" (UID: \"24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9\") "
Jul 22 01:54:47.706513 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:47.706225    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9-bundle\") pod \"24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9\" (UID: \"24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9\") "
Jul 22 01:54:47.706791 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:47.706763    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f575f0b-aaa2-413d-87f2-ce9d69096fb8-bundle" (OuterVolumeSpecName: "bundle") pod "5f575f0b-aaa2-413d-87f2-ce9d69096fb8" (UID: "5f575f0b-aaa2-413d-87f2-ce9d69096fb8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 01:54:47.706886 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:47.706802    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9-bundle" (OuterVolumeSpecName: "bundle") pod "24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9" (UID: "24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 01:54:47.708249 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:47.708211    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f575f0b-aaa2-413d-87f2-ce9d69096fb8-kube-api-access-8mqzm" (OuterVolumeSpecName: "kube-api-access-8mqzm") pod "5f575f0b-aaa2-413d-87f2-ce9d69096fb8" (UID: "5f575f0b-aaa2-413d-87f2-ce9d69096fb8"). InnerVolumeSpecName "kube-api-access-8mqzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 01:54:47.708341 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:47.708302    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9-kube-api-access-67lb7" (OuterVolumeSpecName: "kube-api-access-67lb7") pod "24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9" (UID: "24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9"). InnerVolumeSpecName "kube-api-access-67lb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 01:54:47.714925 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:47.714901    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9-util" (OuterVolumeSpecName: "util") pod "24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9" (UID: "24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 01:54:47.715310 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:47.715294    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f575f0b-aaa2-413d-87f2-ce9d69096fb8-util" (OuterVolumeSpecName: "util") pod "5f575f0b-aaa2-413d-87f2-ce9d69096fb8" (UID: "5f575f0b-aaa2-413d-87f2-ce9d69096fb8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 01:54:47.807392 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:47.807343    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67lb7\" (UniqueName: \"kubernetes.io/projected/24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9-kube-api-access-67lb7\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:54:47.807392 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:47.807369    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mqzm\" (UniqueName: \"kubernetes.io/projected/5f575f0b-aaa2-413d-87f2-ce9d69096fb8-kube-api-access-8mqzm\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:54:47.807392 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:47.807398    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f575f0b-aaa2-413d-87f2-ce9d69096fb8-bundle\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:54:47.807612 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:47.807407    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f575f0b-aaa2-413d-87f2-ce9d69096fb8-util\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:54:47.807612 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:47.807415    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9-util\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:54:47.807612 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:47.807423    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9-bundle\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:54:48.449621 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:48.449589    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gzzv6" event={"ID":"5f575f0b-aaa2-413d-87f2-ce9d69096fb8","Type":"ContainerDied","Data":"c47ab7906acc2609a13e1b0b1eb50435629846eb43c9fe29372746005ac6d272"}
Jul 22 01:54:48.450038 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:48.449643    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c47ab7906acc2609a13e1b0b1eb50435629846eb43c9fe29372746005ac6d272"
Jul 22 01:54:48.450038 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:48.449598    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gzzv6"
Jul 22 01:54:48.451535 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:48.451511    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bmkm4"
Jul 22 01:54:48.451535 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:48.451526    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bmkm4" event={"ID":"24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9","Type":"ContainerDied","Data":"a6f76bdb545448287406d3478abfce16e717a6615e1487d7a9a34fd613850efc"}
Jul 22 01:54:48.451688 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:48.451561    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6f76bdb545448287406d3478abfce16e717a6615e1487d7a9a34fd613850efc"
Jul 22 01:54:54.240115 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:54.240011    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t7dxw"]
Jul 22 01:54:54.240658 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:54.240625    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62e08628-fa98-48a4-99cc-3dc660dbc84e" containerName="pull"
Jul 22 01:54:54.240658 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:54.240645    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="62e08628-fa98-48a4-99cc-3dc660dbc84e" containerName="pull"
Jul 22 01:54:54.240779 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:54.240660    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f575f0b-aaa2-413d-87f2-ce9d69096fb8" containerName="extract"
Jul 22 01:54:54.240779 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:54.240675    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f575f0b-aaa2-413d-87f2-ce9d69096fb8" containerName="extract"
Jul 22 01:54:54.240779 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:54.240687    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62e08628-fa98-48a4-99cc-3dc660dbc84e" containerName="util"
Jul 22 01:54:54.240779 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:54.240696    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="62e08628-fa98-48a4-99cc-3dc660dbc84e" containerName="util"
Jul 22 01:54:54.240779 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:54.240704    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62e08628-fa98-48a4-99cc-3dc660dbc84e" containerName="extract"
Jul 22 01:54:54.240779 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:54.240712    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="62e08628-fa98-48a4-99cc-3dc660dbc84e" containerName="extract"
Jul 22 01:54:54.240779 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:54.240727    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9" containerName="util"
Jul 22 01:54:54.240779 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:54.240734    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9" containerName="util"
Jul 22 01:54:54.240779 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:54.240764    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f575f0b-aaa2-413d-87f2-ce9d69096fb8" containerName="pull"
Jul 22 01:54:54.240779 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:54.240772    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f575f0b-aaa2-413d-87f2-ce9d69096fb8" containerName="pull"
Jul 22 01:54:54.240779 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:54.240781    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9" containerName="pull"
Jul 22 01:54:54.241217 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:54.240790    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9" containerName="pull"
Jul 22 01:54:54.241217 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:54.240815    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f575f0b-aaa2-413d-87f2-ce9d69096fb8" containerName="util"
Jul 22 01:54:54.241217 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:54.240823    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f575f0b-aaa2-413d-87f2-ce9d69096fb8" containerName="util"
Jul 22 01:54:54.241217 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:54.240839    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43423d8a-faea-4db0-84ae-8af34f0eb834" containerName="pull"
Jul 22 01:54:54.241217 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:54.240847    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="43423d8a-faea-4db0-84ae-8af34f0eb834" containerName="pull"
Jul 22 01:54:54.241217 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:54.240856    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43423d8a-faea-4db0-84ae-8af34f0eb834" containerName="extract"
Jul 22 01:54:54.241217 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:54.240864    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="43423d8a-faea-4db0-84ae-8af34f0eb834" containerName="extract"
Jul 22 01:54:54.241217 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:54.240873    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43423d8a-faea-4db0-84ae-8af34f0eb834" containerName="util"
Jul 22 01:54:54.241217 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:54.240886    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="43423d8a-faea-4db0-84ae-8af34f0eb834" containerName="util"
Jul 22 01:54:54.241217 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:54.240895    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9" containerName="extract"
Jul 22 01:54:54.241217 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:54.240912    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9" containerName="extract"
Jul 22 01:54:54.241217 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:54.241042    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="62e08628-fa98-48a4-99cc-3dc660dbc84e" containerName="extract"
Jul 22 01:54:54.241217 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:54.241060    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f575f0b-aaa2-413d-87f2-ce9d69096fb8" containerName="extract"
Jul 22 01:54:54.241217 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:54.241072    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9" containerName="extract"
Jul 22 01:54:54.241217 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:54.241082    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="43423d8a-faea-4db0-84ae-8af34f0eb834" containerName="extract"
Jul 22 01:54:54.244721 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:54.244697    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t7dxw"
Jul 22 01:54:54.247076 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:54.247048    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-lqnv4\""
Jul 22 01:54:54.255445 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:54.255421    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t7dxw"]
Jul 22 01:54:54.354823 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:54.354785    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/236549be-02d1-4e47-bda6-63bc133cac4b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-t7dxw\" (UID: \"236549be-02d1-4e47-bda6-63bc133cac4b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t7dxw"
Jul 22 01:54:54.354998 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:54.354877    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg9xv\" (UniqueName: \"kubernetes.io/projected/236549be-02d1-4e47-bda6-63bc133cac4b-kube-api-access-wg9xv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-t7dxw\" (UID: \"236549be-02d1-4e47-bda6-63bc133cac4b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t7dxw"
Jul 22 01:54:54.455670 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:54.455626    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/236549be-02d1-4e47-bda6-63bc133cac4b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-t7dxw\" (UID: \"236549be-02d1-4e47-bda6-63bc133cac4b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t7dxw"
Jul 22 01:54:54.455860 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:54.455711    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg9xv\" (UniqueName: \"kubernetes.io/projected/236549be-02d1-4e47-bda6-63bc133cac4b-kube-api-access-wg9xv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-t7dxw\" (UID: \"236549be-02d1-4e47-bda6-63bc133cac4b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t7dxw"
Jul 22 01:54:54.456023 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:54.455999    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/236549be-02d1-4e47-bda6-63bc133cac4b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-t7dxw\" (UID: \"236549be-02d1-4e47-bda6-63bc133cac4b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t7dxw"
Jul 22 01:54:54.467894 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:54.467867    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg9xv\" (UniqueName: \"kubernetes.io/projected/236549be-02d1-4e47-bda6-63bc133cac4b-kube-api-access-wg9xv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-t7dxw\" (UID: \"236549be-02d1-4e47-bda6-63bc133cac4b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t7dxw"
Jul 22 01:54:54.555240 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:54.555153    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t7dxw"
Jul 22 01:54:54.696309 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:54.696285    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t7dxw"]
Jul 22 01:54:54.698679 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:54:54.698644    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod236549be_02d1_4e47_bda6_63bc133cac4b.slice/crio-6af46517f501e04ea81543019b11962d81afe8d9bf9c420432bcbe5fbb2f0945 WatchSource:0}: Error finding container 6af46517f501e04ea81543019b11962d81afe8d9bf9c420432bcbe5fbb2f0945: Status 404 returned error can't find the container with id 6af46517f501e04ea81543019b11962d81afe8d9bf9c420432bcbe5fbb2f0945
Jul 22 01:54:55.483783 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:55.483744    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t7dxw" event={"ID":"236549be-02d1-4e47-bda6-63bc133cac4b","Type":"ContainerStarted","Data":"6af46517f501e04ea81543019b11962d81afe8d9bf9c420432bcbe5fbb2f0945"}
Jul 22 01:54:56.194428 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:56.194390    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xfl5z"]
Jul 22 01:54:56.197675 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:56.197654    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xfl5z"
Jul 22 01:54:56.199857 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:56.199834    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-t5h59\""
Jul 22 01:54:56.210168 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:56.210134    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xfl5z"]
Jul 22 01:54:56.271519 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:56.271484    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr2cr\" (UniqueName: \"kubernetes.io/projected/b59f1105-495c-4c70-b009-32627cfd1acf-kube-api-access-xr2cr\") pod \"limitador-operator-controller-manager-859d7b78bd-xfl5z\" (UID: \"b59f1105-495c-4c70-b009-32627cfd1acf\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xfl5z"
Jul 22 01:54:56.372786 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:56.372751    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr2cr\" (UniqueName: \"kubernetes.io/projected/b59f1105-495c-4c70-b009-32627cfd1acf-kube-api-access-xr2cr\") pod \"limitador-operator-controller-manager-859d7b78bd-xfl5z\" (UID: \"b59f1105-495c-4c70-b009-32627cfd1acf\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xfl5z"
Jul 22 01:54:56.383999 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:56.383954    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr2cr\" (UniqueName: \"kubernetes.io/projected/b59f1105-495c-4c70-b009-32627cfd1acf-kube-api-access-xr2cr\") pod \"limitador-operator-controller-manager-859d7b78bd-xfl5z\" (UID: \"b59f1105-495c-4c70-b009-32627cfd1acf\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xfl5z"
Jul 22 01:54:56.508577 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:56.508487    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xfl5z"
Jul 22 01:54:56.654620 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:56.654588    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xfl5z"]
Jul 22 01:54:56.658008 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:54:56.657978    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb59f1105_495c_4c70_b009_32627cfd1acf.slice/crio-fdf0de227c14bb494c2d5a125350f1a55c8ecc390919558b3e91e7d652351309 WatchSource:0}: Error finding container fdf0de227c14bb494c2d5a125350f1a55c8ecc390919558b3e91e7d652351309: Status 404 returned error can't find the container with id fdf0de227c14bb494c2d5a125350f1a55c8ecc390919558b3e91e7d652351309
Jul 22 01:54:57.495876 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:57.495815    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xfl5z" event={"ID":"b59f1105-495c-4c70-b009-32627cfd1acf","Type":"ContainerStarted","Data":"fdf0de227c14bb494c2d5a125350f1a55c8ecc390919558b3e91e7d652351309"}
Jul 22 01:54:58.757427 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:58.757320    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-pvwj2"]
Jul 22 01:54:58.761011 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:58.760987    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-pvwj2"
Jul 22 01:54:58.763453 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:58.763428    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-ll5b7\""
Jul 22 01:54:58.770121 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:58.770095    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-pvwj2"]
Jul 22 01:54:58.893853 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:58.893805    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nqhd\" (UniqueName: \"kubernetes.io/projected/7497ee29-5208-4496-97c4-e4b083d32712-kube-api-access-5nqhd\") pod \"authorino-operator-75cc8d5cb7-pvwj2\" (UID: \"7497ee29-5208-4496-97c4-e4b083d32712\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-pvwj2"
Jul 22 01:54:58.995324 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:58.995283    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nqhd\" (UniqueName: \"kubernetes.io/projected/7497ee29-5208-4496-97c4-e4b083d32712-kube-api-access-5nqhd\") pod \"authorino-operator-75cc8d5cb7-pvwj2\" (UID: \"7497ee29-5208-4496-97c4-e4b083d32712\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-pvwj2"
Jul 22 01:54:59.004330 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:59.004295    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nqhd\" (UniqueName: \"kubernetes.io/projected/7497ee29-5208-4496-97c4-e4b083d32712-kube-api-access-5nqhd\") pod \"authorino-operator-75cc8d5cb7-pvwj2\" (UID: \"7497ee29-5208-4496-97c4-e4b083d32712\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-pvwj2"
Jul 22 01:54:59.073318 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:59.073229    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-pvwj2"
Jul 22 01:54:59.507803 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:59.507754    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xfl5z" event={"ID":"b59f1105-495c-4c70-b009-32627cfd1acf","Type":"ContainerStarted","Data":"0465a8dd9512d94b188658b7acc6ab7e1a3e72a4bc2ce948ad0edabc6020524c"}
Jul 22 01:54:59.508016 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:59.507839    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xfl5z"
Jul 22 01:54:59.525399 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:59.525323    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xfl5z" podStartSLOduration=1.703489963 podStartE2EDuration="3.525300602s" totalImagesPullingTime="1.821810639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:54:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:54:59.524019773 +0000 UTC m=+452.367848906" watchObservedRunningTime="2026-07-22 01:54:59.525300602 +0000 UTC m=+452.369129725"
Jul 22 01:54:59.852289 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:54:59.852265    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-pvwj2"]
Jul 22 01:54:59.855195 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:54:59.855169    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7497ee29_5208_4496_97c4_e4b083d32712.slice/crio-5dce17782216c2ccaad7045dad148ede006fb16544b759de6b99d1b88456d076 WatchSource:0}: Error finding container 5dce17782216c2ccaad7045dad148ede006fb16544b759de6b99d1b88456d076: Status 404 returned error can't find the container with id 5dce17782216c2ccaad7045dad148ede006fb16544b759de6b99d1b88456d076
Jul 22 01:55:00.514395 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:00.514344    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t7dxw" event={"ID":"236549be-02d1-4e47-bda6-63bc133cac4b","Type":"ContainerStarted","Data":"21ba712bc0c15364792d55573cb412767491d48b31e6abb9dc72a25aa933c00f"}
Jul 22 01:55:00.514584 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:00.514522    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t7dxw"
Jul 22 01:55:00.515631 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:00.515608    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-pvwj2" event={"ID":"7497ee29-5208-4496-97c4-e4b083d32712","Type":"ContainerStarted","Data":"5dce17782216c2ccaad7045dad148ede006fb16544b759de6b99d1b88456d076"}
Jul 22 01:55:00.537941 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:00.537899    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t7dxw" podStartSLOduration=1.442013639 podStartE2EDuration="6.537885749s" totalImagesPullingTime="5.09587211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:54:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:55:00.536985991 +0000 UTC m=+453.380815120" watchObservedRunningTime="2026-07-22 01:55:00.537885749 +0000 UTC m=+453.381715008"
Jul 22 01:55:02.087635 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:02.087597    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-kzbvr"]
Jul 22 01:55:02.095863 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:02.095827    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-kzbvr"
Jul 22 01:55:02.098396 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:02.098350    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 22 01:55:02.098547 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:02.098350    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-mr67f\""
Jul 22 01:55:02.103372 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:02.103339    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-kzbvr"]
Jul 22 01:55:02.226010 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:02.225982    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqvmm\" (UniqueName: \"kubernetes.io/projected/2c568a93-ab9d-453e-82dc-a943b46669ff-kube-api-access-jqvmm\") pod \"dns-operator-controller-manager-864dd78c4c-kzbvr\" (UID: \"2c568a93-ab9d-453e-82dc-a943b46669ff\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-kzbvr"
Jul 22 01:55:02.327450 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:02.327414    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqvmm\" (UniqueName: \"kubernetes.io/projected/2c568a93-ab9d-453e-82dc-a943b46669ff-kube-api-access-jqvmm\") pod \"dns-operator-controller-manager-864dd78c4c-kzbvr\" (UID: \"2c568a93-ab9d-453e-82dc-a943b46669ff\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-kzbvr"
Jul 22 01:55:02.338836 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:02.338783    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqvmm\" (UniqueName: \"kubernetes.io/projected/2c568a93-ab9d-453e-82dc-a943b46669ff-kube-api-access-jqvmm\") pod \"dns-operator-controller-manager-864dd78c4c-kzbvr\" (UID: \"2c568a93-ab9d-453e-82dc-a943b46669ff\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-kzbvr"
Jul 22 01:55:02.411012 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:02.410980    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-kzbvr"
Jul 22 01:55:02.527193 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:02.527162    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-pvwj2" event={"ID":"7497ee29-5208-4496-97c4-e4b083d32712","Type":"ContainerStarted","Data":"e1f81e01e272feb372140d6a91590d8fb7030d2b51dc83d7a6fe9cdc06799613"}
Jul 22 01:55:02.527359 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:02.527299    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-pvwj2"
Jul 22 01:55:02.552192 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:02.552130    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-pvwj2" podStartSLOduration=2.199575786 podStartE2EDuration="4.552116697s" totalImagesPullingTime="2.352540911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:54:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:55:02.549618488 +0000 UTC m=+455.393447621" watchObservedRunningTime="2026-07-22 01:55:02.552116697 +0000 UTC m=+455.395945887"
Jul 22 01:55:02.552970 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:55:02.552946    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c568a93_ab9d_453e_82dc_a943b46669ff.slice/crio-efd7cdfc35994e2c4f9f64545fa945e2a72de745ad172ca8ef71a80366ce3be2 WatchSource:0}: Error finding container efd7cdfc35994e2c4f9f64545fa945e2a72de745ad172ca8ef71a80366ce3be2: Status 404 returned error can't find the container with id efd7cdfc35994e2c4f9f64545fa945e2a72de745ad172ca8ef71a80366ce3be2
Jul 22 01:55:02.554866 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:02.554847    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-kzbvr"]
Jul 22 01:55:03.535122 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:03.535082    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-kzbvr" event={"ID":"2c568a93-ab9d-453e-82dc-a943b46669ff","Type":"ContainerStarted","Data":"efd7cdfc35994e2c4f9f64545fa945e2a72de745ad172ca8ef71a80366ce3be2"}
Jul 22 01:55:04.541800 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:04.541766    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-kzbvr" event={"ID":"2c568a93-ab9d-453e-82dc-a943b46669ff","Type":"ContainerStarted","Data":"ab432462e7d35c2718f21385bfc89efa8c56c6b2a36b0759246c9895883bef84"}
Jul 22 01:55:04.542160 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:04.541923    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-kzbvr"
Jul 22 01:55:04.560354 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:04.560309    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-kzbvr" podStartSLOduration=0.721419874 podStartE2EDuration="2.560295177s" totalImagesPullingTime="1.838875303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:55:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:55:04.558762126 +0000 UTC m=+457.402591259" watchObservedRunningTime="2026-07-22 01:55:04.560295177 +0000 UTC m=+457.404124308"
Jul 22 01:55:10.517764 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:10.517730    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xfl5z"
Jul 22 01:55:11.523325 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:11.523290    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t7dxw"
Jul 22 01:55:13.322448 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.322407    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dkq9r"]
Jul 22 01:55:13.325880 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.325862    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dkq9r"
Jul 22 01:55:13.338158 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.338129    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dkq9r"]
Jul 22 01:55:13.412930 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.412895    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d1c501a-fd7b-464c-956c-2853fa27bcc2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-dkq9r\" (UID: \"9d1c501a-fd7b-464c-956c-2853fa27bcc2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dkq9r"
Jul 22 01:55:13.413050 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.412938    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gvch\" (UniqueName: \"kubernetes.io/projected/9d1c501a-fd7b-464c-956c-2853fa27bcc2-kube-api-access-2gvch\") pod \"kuadrant-operator-controller-manager-86878c769f-dkq9r\" (UID: \"9d1c501a-fd7b-464c-956c-2853fa27bcc2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dkq9r"
Jul 22 01:55:13.441552 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.441521    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t7dxw"]
Jul 22 01:55:13.441755 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.441729    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t7dxw" podUID="236549be-02d1-4e47-bda6-63bc133cac4b" containerName="manager" containerID="cri-o://21ba712bc0c15364792d55573cb412767491d48b31e6abb9dc72a25aa933c00f" gracePeriod=2
Jul 22 01:55:13.449282 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.449256    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t7dxw"]
Jul 22 01:55:13.464024 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.464000    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dkq9r"]
Jul 22 01:55:13.464202 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:55:13.464184    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-2gvch], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dkq9r" podUID="9d1c501a-fd7b-464c-956c-2853fa27bcc2"
Jul 22 01:55:13.469728 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.469702    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tlnzk"]
Jul 22 01:55:13.470136 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.470118    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="236549be-02d1-4e47-bda6-63bc133cac4b" containerName="manager"
Jul 22 01:55:13.470136 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.470138    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="236549be-02d1-4e47-bda6-63bc133cac4b" containerName="manager"
Jul 22 01:55:13.470282 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.470234    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="236549be-02d1-4e47-bda6-63bc133cac4b" containerName="manager"
Jul 22 01:55:13.473409 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.473363    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dkq9r"]
Jul 22 01:55:13.473496 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.473477    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tlnzk"
Jul 22 01:55:13.491909 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.491887    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tlnzk"]
Jul 22 01:55:13.503779 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.503754    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xfl5z"]
Jul 22 01:55:13.503910 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.503888    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xfl5z" podUID="b59f1105-495c-4c70-b009-32627cfd1acf" containerName="manager" containerID="cri-o://0465a8dd9512d94b188658b7acc6ab7e1a3e72a4bc2ce948ad0edabc6020524c" gracePeriod=2
Jul 22 01:55:13.508561 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.508542    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9tqq5"]
Jul 22 01:55:13.511731 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.511714    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9tqq5"
Jul 22 01:55:13.513282 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.513262    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d1c501a-fd7b-464c-956c-2853fa27bcc2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-dkq9r\" (UID: \"9d1c501a-fd7b-464c-956c-2853fa27bcc2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dkq9r"
Jul 22 01:55:13.513357 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.513296    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gvch\" (UniqueName: \"kubernetes.io/projected/9d1c501a-fd7b-464c-956c-2853fa27bcc2-kube-api-access-2gvch\") pod \"kuadrant-operator-controller-manager-86878c769f-dkq9r\" (UID: \"9d1c501a-fd7b-464c-956c-2853fa27bcc2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dkq9r"
Jul 22 01:55:13.513737 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.513702    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d1c501a-fd7b-464c-956c-2853fa27bcc2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-dkq9r\" (UID: \"9d1c501a-fd7b-464c-956c-2853fa27bcc2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dkq9r"
Jul 22 01:55:13.517914 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.517891    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xfl5z"]
Jul 22 01:55:13.518524 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:55:13.518506    2563 projected.go:196] Error preparing data for projected volume kube-api-access-2gvch for pod kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dkq9r: failed to fetch token: pod "kuadrant-operator-controller-manager-86878c769f-dkq9r" not found
Jul 22 01:55:13.518597 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:55:13.518579    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d1c501a-fd7b-464c-956c-2853fa27bcc2-kube-api-access-2gvch podName:9d1c501a-fd7b-464c-956c-2853fa27bcc2 nodeName:}" failed. No retries permitted until 2026-07-22 01:55:14.018558257 +0000 UTC m=+466.862387373 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2gvch" (UniqueName: "kubernetes.io/projected/9d1c501a-fd7b-464c-956c-2853fa27bcc2-kube-api-access-2gvch") pod "kuadrant-operator-controller-manager-86878c769f-dkq9r" (UID: "9d1c501a-fd7b-464c-956c-2853fa27bcc2") : failed to fetch token: pod "kuadrant-operator-controller-manager-86878c769f-dkq9r" not found
Jul 22 01:55:13.523309 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:55:13.523283    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-t7dxw\" is forbidden: User \"system:node:ip-10-0-135-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-230.ec2.internal' and this object" podUID="236549be-02d1-4e47-bda6-63bc133cac4b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t7dxw"
Jul 22 01:55:13.525268 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:55:13.525242    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xfl5z\" is forbidden: User \"system:node:ip-10-0-135-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-230.ec2.internal' and this object" podUID="b59f1105-495c-4c70-b009-32627cfd1acf" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xfl5z"
Jul 22 01:55:13.530884 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.530864    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9tqq5"]
Jul 22 01:55:13.537571 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.537547    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-pvwj2"
Jul 22 01:55:13.539926 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.539903    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2ghjg"]
Jul 22 01:55:13.540353 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.540334    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b59f1105-495c-4c70-b009-32627cfd1acf" containerName="manager"
Jul 22 01:55:13.540427 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.540358    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b59f1105-495c-4c70-b009-32627cfd1acf" containerName="manager"
Jul 22 01:55:13.540480 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.540467    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b59f1105-495c-4c70-b009-32627cfd1acf" containerName="manager"
Jul 22 01:55:13.543790 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.543770    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2ghjg"
Jul 22 01:55:13.550096 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:55:13.550067    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-t7dxw\" is forbidden: User \"system:node:ip-10-0-135-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-230.ec2.internal' and this object" podUID="236549be-02d1-4e47-bda6-63bc133cac4b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t7dxw"
Jul 22 01:55:13.557514 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.557493    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2ghjg"]
Jul 22 01:55:13.575791 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:55:13.575738    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-t7dxw\" is forbidden: User \"system:node:ip-10-0-135-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-230.ec2.internal' and this object" podUID="236549be-02d1-4e47-bda6-63bc133cac4b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t7dxw"
Jul 22 01:55:13.577551 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:55:13.577518    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xfl5z\" is forbidden: User \"system:node:ip-10-0-135-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-230.ec2.internal' and this object" podUID="b59f1105-495c-4c70-b009-32627cfd1acf" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xfl5z"
Jul 22 01:55:13.590043 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.590024    2563 generic.go:356] "Generic (PLEG): container finished" podID="236549be-02d1-4e47-bda6-63bc133cac4b" containerID="21ba712bc0c15364792d55573cb412767491d48b31e6abb9dc72a25aa933c00f" exitCode=0
Jul 22 01:55:13.590126 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.590095    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dkq9r"
Jul 22 01:55:13.595919 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.595902    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dkq9r"
Jul 22 01:55:13.596492 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:55:13.596471    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xfl5z\" is forbidden: User \"system:node:ip-10-0-135-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-230.ec2.internal' and this object" podUID="b59f1105-495c-4c70-b009-32627cfd1acf" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xfl5z"
Jul 22 01:55:13.598280 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:55:13.598260    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-dkq9r\" is forbidden: User \"system:node:ip-10-0-135-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-230.ec2.internal' and this object" podUID="9d1c501a-fd7b-464c-956c-2853fa27bcc2" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dkq9r"
Jul 22 01:55:13.600039 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:55:13.600017    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-t7dxw\" is forbidden: User \"system:node:ip-10-0-135-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-230.ec2.internal' and this object" podUID="236549be-02d1-4e47-bda6-63bc133cac4b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t7dxw"
Jul 22 01:55:13.601847 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:55:13.601820    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xfl5z\" is forbidden: User \"system:node:ip-10-0-135-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-230.ec2.internal' and this object" podUID="b59f1105-495c-4c70-b009-32627cfd1acf" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xfl5z"
Jul 22 01:55:13.603550 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:55:13.603522    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-dkq9r\" is forbidden: User \"system:node:ip-10-0-135-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-230.ec2.internal' and this object" podUID="9d1c501a-fd7b-464c-956c-2853fa27bcc2" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dkq9r"
Jul 22 01:55:13.605306 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:55:13.605285    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-t7dxw\" is forbidden: User \"system:node:ip-10-0-135-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-230.ec2.internal' and this object" podUID="236549be-02d1-4e47-bda6-63bc133cac4b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t7dxw"
Jul 22 01:55:13.614277 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.614245    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/05054cf6-5841-44f2-b541-0c486c70f2bb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-9tqq5\" (UID: \"05054cf6-5841-44f2-b541-0c486c70f2bb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9tqq5"
Jul 22 01:55:13.614397 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.614299    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8q5x\" (UniqueName: \"kubernetes.io/projected/be5832ee-cedb-4e5e-8262-1e63d96e3c22-kube-api-access-x8q5x\") pod \"limitador-operator-controller-manager-859d7b78bd-2ghjg\" (UID: \"be5832ee-cedb-4e5e-8262-1e63d96e3c22\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2ghjg"
Jul 22 01:55:13.614470 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.614437    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp6l5\" (UniqueName: \"kubernetes.io/projected/05054cf6-5841-44f2-b541-0c486c70f2bb-kube-api-access-rp6l5\") pod \"kuadrant-operator-controller-manager-86878c769f-9tqq5\" (UID: \"05054cf6-5841-44f2-b541-0c486c70f2bb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9tqq5"
Jul 22 01:55:13.614524 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.614468    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9fm7\" (UniqueName: \"kubernetes.io/projected/eff4b50b-23ed-4ab2-a3a7-5e1bf9ad1d08-kube-api-access-w9fm7\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tlnzk\" (UID: \"eff4b50b-23ed-4ab2-a3a7-5e1bf9ad1d08\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tlnzk"
Jul 22 01:55:13.614692 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.614671    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eff4b50b-23ed-4ab2-a3a7-5e1bf9ad1d08-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tlnzk\" (UID: \"eff4b50b-23ed-4ab2-a3a7-5e1bf9ad1d08\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tlnzk"
Jul 22 01:55:13.715134 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.715103    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d1c501a-fd7b-464c-956c-2853fa27bcc2-extensions-socket-volume\") pod \"9d1c501a-fd7b-464c-956c-2853fa27bcc2\" (UID: \"9d1c501a-fd7b-464c-956c-2853fa27bcc2\") "
Jul 22 01:55:13.715283 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.715262    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eff4b50b-23ed-4ab2-a3a7-5e1bf9ad1d08-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tlnzk\" (UID: \"eff4b50b-23ed-4ab2-a3a7-5e1bf9ad1d08\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tlnzk"
Jul 22 01:55:13.715355 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.715326    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/05054cf6-5841-44f2-b541-0c486c70f2bb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-9tqq5\" (UID: \"05054cf6-5841-44f2-b541-0c486c70f2bb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9tqq5"
Jul 22 01:55:13.715442 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.715357    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d1c501a-fd7b-464c-956c-2853fa27bcc2-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9d1c501a-fd7b-464c-956c-2853fa27bcc2" (UID: "9d1c501a-fd7b-464c-956c-2853fa27bcc2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 01:55:13.715442 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.715412    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8q5x\" (UniqueName: \"kubernetes.io/projected/be5832ee-cedb-4e5e-8262-1e63d96e3c22-kube-api-access-x8q5x\") pod \"limitador-operator-controller-manager-859d7b78bd-2ghjg\" (UID: \"be5832ee-cedb-4e5e-8262-1e63d96e3c22\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2ghjg"
Jul 22 01:55:13.715553 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.715505    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp6l5\" (UniqueName: \"kubernetes.io/projected/05054cf6-5841-44f2-b541-0c486c70f2bb-kube-api-access-rp6l5\") pod \"kuadrant-operator-controller-manager-86878c769f-9tqq5\" (UID: \"05054cf6-5841-44f2-b541-0c486c70f2bb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9tqq5"
Jul 22 01:55:13.715553 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.715536    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9fm7\" (UniqueName: \"kubernetes.io/projected/eff4b50b-23ed-4ab2-a3a7-5e1bf9ad1d08-kube-api-access-w9fm7\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tlnzk\" (UID: \"eff4b50b-23ed-4ab2-a3a7-5e1bf9ad1d08\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tlnzk"
Jul 22 01:55:13.715660 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.715591    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d1c501a-fd7b-464c-956c-2853fa27bcc2-extensions-socket-volume\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:55:13.715660 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.715609    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gvch\" (UniqueName: \"kubernetes.io/projected/9d1c501a-fd7b-464c-956c-2853fa27bcc2-kube-api-access-2gvch\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:55:13.715660 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.715642    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eff4b50b-23ed-4ab2-a3a7-5e1bf9ad1d08-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tlnzk\" (UID: \"eff4b50b-23ed-4ab2-a3a7-5e1bf9ad1d08\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tlnzk"
Jul 22 01:55:13.715808 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.715751    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/05054cf6-5841-44f2-b541-0c486c70f2bb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-9tqq5\" (UID: \"05054cf6-5841-44f2-b541-0c486c70f2bb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9tqq5"
Jul 22 01:55:13.724140 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.724119    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp6l5\" (UniqueName: \"kubernetes.io/projected/05054cf6-5841-44f2-b541-0c486c70f2bb-kube-api-access-rp6l5\") pod \"kuadrant-operator-controller-manager-86878c769f-9tqq5\" (UID: \"05054cf6-5841-44f2-b541-0c486c70f2bb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9tqq5"
Jul 22 01:55:13.724367 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.724328    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9fm7\" (UniqueName: \"kubernetes.io/projected/eff4b50b-23ed-4ab2-a3a7-5e1bf9ad1d08-kube-api-access-w9fm7\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tlnzk\" (UID: \"eff4b50b-23ed-4ab2-a3a7-5e1bf9ad1d08\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tlnzk"
Jul 22 01:55:13.724689 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.724673    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8q5x\" (UniqueName: \"kubernetes.io/projected/be5832ee-cedb-4e5e-8262-1e63d96e3c22-kube-api-access-x8q5x\") pod \"limitador-operator-controller-manager-859d7b78bd-2ghjg\" (UID: \"be5832ee-cedb-4e5e-8262-1e63d96e3c22\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2ghjg"
Jul 22 01:55:13.751488 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.751472    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t7dxw"
Jul 22 01:55:13.753489 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.753475    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xfl5z"
Jul 22 01:55:13.816492 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.816467    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/236549be-02d1-4e47-bda6-63bc133cac4b-extensions-socket-volume\") pod \"236549be-02d1-4e47-bda6-63bc133cac4b\" (UID: \"236549be-02d1-4e47-bda6-63bc133cac4b\") "
Jul 22 01:55:13.816564 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.816511    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wg9xv\" (UniqueName: \"kubernetes.io/projected/236549be-02d1-4e47-bda6-63bc133cac4b-kube-api-access-wg9xv\") pod \"236549be-02d1-4e47-bda6-63bc133cac4b\" (UID: \"236549be-02d1-4e47-bda6-63bc133cac4b\") "
Jul 22 01:55:13.816564 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.816560    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xr2cr\" (UniqueName: \"kubernetes.io/projected/b59f1105-495c-4c70-b009-32627cfd1acf-kube-api-access-xr2cr\") pod \"b59f1105-495c-4c70-b009-32627cfd1acf\" (UID: \"b59f1105-495c-4c70-b009-32627cfd1acf\") "
Jul 22 01:55:13.816972 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.816947    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/236549be-02d1-4e47-bda6-63bc133cac4b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "236549be-02d1-4e47-bda6-63bc133cac4b" (UID: "236549be-02d1-4e47-bda6-63bc133cac4b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 01:55:13.818329 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.818308    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/236549be-02d1-4e47-bda6-63bc133cac4b-kube-api-access-wg9xv" (OuterVolumeSpecName: "kube-api-access-wg9xv") pod "236549be-02d1-4e47-bda6-63bc133cac4b" (UID: "236549be-02d1-4e47-bda6-63bc133cac4b"). InnerVolumeSpecName "kube-api-access-wg9xv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 01:55:13.818598 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.818570    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b59f1105-495c-4c70-b009-32627cfd1acf-kube-api-access-xr2cr" (OuterVolumeSpecName: "kube-api-access-xr2cr") pod "b59f1105-495c-4c70-b009-32627cfd1acf" (UID: "b59f1105-495c-4c70-b009-32627cfd1acf"). InnerVolumeSpecName "kube-api-access-xr2cr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 01:55:13.876845 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.876795    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tlnzk"
Jul 22 01:55:13.882691 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.882673    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9tqq5"
Jul 22 01:55:13.888501 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.888483    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2ghjg"
Jul 22 01:55:13.917420 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.917355    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/236549be-02d1-4e47-bda6-63bc133cac4b-extensions-socket-volume\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:55:13.917420 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.917413    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg9xv\" (UniqueName: \"kubernetes.io/projected/236549be-02d1-4e47-bda6-63bc133cac4b-kube-api-access-wg9xv\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:55:13.917420 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:13.917436    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr2cr\" (UniqueName: \"kubernetes.io/projected/b59f1105-495c-4c70-b009-32627cfd1acf-kube-api-access-xr2cr\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:55:14.033190 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:14.033169    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tlnzk"]
Jul 22 01:55:14.035720 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:55:14.035688    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeff4b50b_23ed_4ab2_a3a7_5e1bf9ad1d08.slice/crio-a93fced306033da635fcddce4b0c3da779557542a78b1cb1b401de928d0b3875 WatchSource:0}: Error finding container a93fced306033da635fcddce4b0c3da779557542a78b1cb1b401de928d0b3875: Status 404 returned error can't find the container with id a93fced306033da635fcddce4b0c3da779557542a78b1cb1b401de928d0b3875
Jul 22 01:55:14.048032 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:14.048006    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9tqq5"]
Jul 22 01:55:14.050928 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:55:14.050902    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05054cf6_5841_44f2_b541_0c486c70f2bb.slice/crio-045c5c87771302ae91350001f0377573bd176f66378c9efc6d05a2c7267fe909 WatchSource:0}: Error finding container 045c5c87771302ae91350001f0377573bd176f66378c9efc6d05a2c7267fe909: Status 404 returned error can't find the container with id 045c5c87771302ae91350001f0377573bd176f66378c9efc6d05a2c7267fe909
Jul 22 01:55:14.068228 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:14.068166    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2ghjg"]
Jul 22 01:55:14.070770 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:55:14.070744    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe5832ee_cedb_4e5e_8262_1e63d96e3c22.slice/crio-ee825c8e0b5cc604d94ff761153f1a0d1e33800422edb135309188297b44f6c1 WatchSource:0}: Error finding container ee825c8e0b5cc604d94ff761153f1a0d1e33800422edb135309188297b44f6c1: Status 404 returned error can't find the container with id ee825c8e0b5cc604d94ff761153f1a0d1e33800422edb135309188297b44f6c1
Jul 22 01:55:14.598948 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:14.598918    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t7dxw"
Jul 22 01:55:14.599363 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:14.598964    2563 scope.go:117] "RemoveContainer" containerID="21ba712bc0c15364792d55573cb412767491d48b31e6abb9dc72a25aa933c00f"
Jul 22 01:55:14.600716 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:14.600682    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9tqq5" event={"ID":"05054cf6-5841-44f2-b541-0c486c70f2bb","Type":"ContainerStarted","Data":"620f4362941bd47cc357404e61d893b4f4fca83baec05f96396b2003eff259b4"}
Jul 22 01:55:14.600842 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:14.600738    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9tqq5" event={"ID":"05054cf6-5841-44f2-b541-0c486c70f2bb","Type":"ContainerStarted","Data":"045c5c87771302ae91350001f0377573bd176f66378c9efc6d05a2c7267fe909"}
Jul 22 01:55:14.600842 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:14.600769    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9tqq5"
Jul 22 01:55:14.602064 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:14.602039    2563 generic.go:356] "Generic (PLEG): container finished" podID="b59f1105-495c-4c70-b009-32627cfd1acf" containerID="0465a8dd9512d94b188658b7acc6ab7e1a3e72a4bc2ce948ad0edabc6020524c" exitCode=0
Jul 22 01:55:14.602191 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:14.602079    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xfl5z"
Jul 22 01:55:14.606697 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:14.606674    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2ghjg" event={"ID":"be5832ee-cedb-4e5e-8262-1e63d96e3c22","Type":"ContainerStarted","Data":"a98d2731a54a36b58ab4a60bb8335974eb1dcdb9e9d87e19987372ea8feeb24e"}
Jul 22 01:55:14.606810 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:14.606718    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2ghjg" event={"ID":"be5832ee-cedb-4e5e-8262-1e63d96e3c22","Type":"ContainerStarted","Data":"ee825c8e0b5cc604d94ff761153f1a0d1e33800422edb135309188297b44f6c1"}
Jul 22 01:55:14.606810 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:14.606746    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2ghjg"
Jul 22 01:55:14.609322 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:14.609228    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tlnzk" event={"ID":"eff4b50b-23ed-4ab2-a3a7-5e1bf9ad1d08","Type":"ContainerStarted","Data":"bbe8d3cc2c53d6235ff2d3d96d7933a90673a6ced038fcb302dafa47dbfb8839"}
Jul 22 01:55:14.609322 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:14.609245    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dkq9r"
Jul 22 01:55:14.609322 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:14.609269    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tlnzk" event={"ID":"eff4b50b-23ed-4ab2-a3a7-5e1bf9ad1d08","Type":"ContainerStarted","Data":"a93fced306033da635fcddce4b0c3da779557542a78b1cb1b401de928d0b3875"}
Jul 22 01:55:14.609542 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:14.609342    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tlnzk"
Jul 22 01:55:14.609542 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:14.609357    2563 scope.go:117] "RemoveContainer" containerID="0465a8dd9512d94b188658b7acc6ab7e1a3e72a4bc2ce948ad0edabc6020524c"
Jul 22 01:55:14.617422 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:14.617404    2563 scope.go:117] "RemoveContainer" containerID="0465a8dd9512d94b188658b7acc6ab7e1a3e72a4bc2ce948ad0edabc6020524c"
Jul 22 01:55:14.617672 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:55:14.617655    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0465a8dd9512d94b188658b7acc6ab7e1a3e72a4bc2ce948ad0edabc6020524c\": container with ID starting with 0465a8dd9512d94b188658b7acc6ab7e1a3e72a4bc2ce948ad0edabc6020524c not found: ID does not exist" containerID="0465a8dd9512d94b188658b7acc6ab7e1a3e72a4bc2ce948ad0edabc6020524c"
Jul 22 01:55:14.617721 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:14.617679    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0465a8dd9512d94b188658b7acc6ab7e1a3e72a4bc2ce948ad0edabc6020524c"} err="failed to get container status \"0465a8dd9512d94b188658b7acc6ab7e1a3e72a4bc2ce948ad0edabc6020524c\": rpc error: code = NotFound desc = could not find container \"0465a8dd9512d94b188658b7acc6ab7e1a3e72a4bc2ce948ad0edabc6020524c\": container with ID starting with 0465a8dd9512d94b188658b7acc6ab7e1a3e72a4bc2ce948ad0edabc6020524c not found: ID does not exist"
Jul 22 01:55:14.630185 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:14.630148    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9tqq5" podStartSLOduration=1.6301362 podStartE2EDuration="1.6301362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:55:14.629444848 +0000 UTC m=+467.473273982" watchObservedRunningTime="2026-07-22 01:55:14.6301362 +0000 UTC m=+467.473965328"
Jul 22 01:55:14.652755 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:14.652721    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2ghjg" podStartSLOduration=1.652709433 podStartE2EDuration="1.652709433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:55:14.651728771 +0000 UTC m=+467.495557903" watchObservedRunningTime="2026-07-22 01:55:14.652709433 +0000 UTC m=+467.496538565"
Jul 22 01:55:14.653478 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:55:14.653458    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-dkq9r\" is forbidden: User \"system:node:ip-10-0-135-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-230.ec2.internal' and this object" podUID="9d1c501a-fd7b-464c-956c-2853fa27bcc2" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dkq9r"
Jul 22 01:55:14.675449 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:14.675415    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tlnzk" podStartSLOduration=1.67540304 podStartE2EDuration="1.67540304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:55:14.674471073 +0000 UTC m=+467.518300206" watchObservedRunningTime="2026-07-22 01:55:14.67540304 +0000 UTC m=+467.519232169"
Jul 22 01:55:15.551584 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:15.551547    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-kzbvr"
Jul 22 01:55:15.572019 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:55:15.571983    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-dkq9r\" is forbidden: User \"system:node:ip-10-0-135-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-230.ec2.internal' and this object" podUID="9d1c501a-fd7b-464c-956c-2853fa27bcc2" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dkq9r"
Jul 22 01:55:15.663892 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:15.663863    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="236549be-02d1-4e47-bda6-63bc133cac4b" path="/var/lib/kubelet/pods/236549be-02d1-4e47-bda6-63bc133cac4b/volumes"
Jul 22 01:55:15.664201 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:15.664157    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d1c501a-fd7b-464c-956c-2853fa27bcc2" path="/var/lib/kubelet/pods/9d1c501a-fd7b-464c-956c-2853fa27bcc2/volumes"
Jul 22 01:55:15.664342 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:15.664331    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b59f1105-495c-4c70-b009-32627cfd1acf" path="/var/lib/kubelet/pods/b59f1105-495c-4c70-b009-32627cfd1acf/volumes"
Jul 22 01:55:25.620396 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:25.620342    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9tqq5"
Jul 22 01:55:25.620830 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:25.620451    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-2ghjg"
Jul 22 01:55:25.620830 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:25.620609    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tlnzk"
Jul 22 01:55:25.688251 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:25.688217    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tlnzk"]
Jul 22 01:55:25.688502 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:25.688477    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tlnzk" podUID="eff4b50b-23ed-4ab2-a3a7-5e1bf9ad1d08" containerName="manager" containerID="cri-o://bbe8d3cc2c53d6235ff2d3d96d7933a90673a6ced038fcb302dafa47dbfb8839" gracePeriod=10
Jul 22 01:55:25.933194 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:25.933172    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tlnzk"
Jul 22 01:55:26.108623 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:26.108593    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eff4b50b-23ed-4ab2-a3a7-5e1bf9ad1d08-extensions-socket-volume\") pod \"eff4b50b-23ed-4ab2-a3a7-5e1bf9ad1d08\" (UID: \"eff4b50b-23ed-4ab2-a3a7-5e1bf9ad1d08\") "
Jul 22 01:55:26.108780 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:26.108645    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9fm7\" (UniqueName: \"kubernetes.io/projected/eff4b50b-23ed-4ab2-a3a7-5e1bf9ad1d08-kube-api-access-w9fm7\") pod \"eff4b50b-23ed-4ab2-a3a7-5e1bf9ad1d08\" (UID: \"eff4b50b-23ed-4ab2-a3a7-5e1bf9ad1d08\") "
Jul 22 01:55:26.109008 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:26.108985    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff4b50b-23ed-4ab2-a3a7-5e1bf9ad1d08-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "eff4b50b-23ed-4ab2-a3a7-5e1bf9ad1d08" (UID: "eff4b50b-23ed-4ab2-a3a7-5e1bf9ad1d08"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 01:55:26.110578 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:26.110552    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eff4b50b-23ed-4ab2-a3a7-5e1bf9ad1d08-kube-api-access-w9fm7" (OuterVolumeSpecName: "kube-api-access-w9fm7") pod "eff4b50b-23ed-4ab2-a3a7-5e1bf9ad1d08" (UID: "eff4b50b-23ed-4ab2-a3a7-5e1bf9ad1d08"). InnerVolumeSpecName "kube-api-access-w9fm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 01:55:26.210065 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:26.210043    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eff4b50b-23ed-4ab2-a3a7-5e1bf9ad1d08-extensions-socket-volume\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:55:26.210065 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:26.210064    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9fm7\" (UniqueName: \"kubernetes.io/projected/eff4b50b-23ed-4ab2-a3a7-5e1bf9ad1d08-kube-api-access-w9fm7\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:55:26.675087 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:26.675062    2563 generic.go:356] "Generic (PLEG): container finished" podID="eff4b50b-23ed-4ab2-a3a7-5e1bf9ad1d08" containerID="bbe8d3cc2c53d6235ff2d3d96d7933a90673a6ced038fcb302dafa47dbfb8839" exitCode=0
Jul 22 01:55:26.675425 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:26.675105    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tlnzk"
Jul 22 01:55:26.675425 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:26.675130    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tlnzk" event={"ID":"eff4b50b-23ed-4ab2-a3a7-5e1bf9ad1d08","Type":"ContainerDied","Data":"bbe8d3cc2c53d6235ff2d3d96d7933a90673a6ced038fcb302dafa47dbfb8839"}
Jul 22 01:55:26.675425 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:26.675172    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tlnzk" event={"ID":"eff4b50b-23ed-4ab2-a3a7-5e1bf9ad1d08","Type":"ContainerDied","Data":"a93fced306033da635fcddce4b0c3da779557542a78b1cb1b401de928d0b3875"}
Jul 22 01:55:26.675425 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:26.675196    2563 scope.go:117] "RemoveContainer" containerID="bbe8d3cc2c53d6235ff2d3d96d7933a90673a6ced038fcb302dafa47dbfb8839"
Jul 22 01:55:26.684674 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:26.684653    2563 scope.go:117] "RemoveContainer" containerID="bbe8d3cc2c53d6235ff2d3d96d7933a90673a6ced038fcb302dafa47dbfb8839"
Jul 22 01:55:26.684961 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:55:26.684944    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbe8d3cc2c53d6235ff2d3d96d7933a90673a6ced038fcb302dafa47dbfb8839\": container with ID starting with bbe8d3cc2c53d6235ff2d3d96d7933a90673a6ced038fcb302dafa47dbfb8839 not found: ID does not exist" containerID="bbe8d3cc2c53d6235ff2d3d96d7933a90673a6ced038fcb302dafa47dbfb8839"
Jul 22 01:55:26.685009 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:26.684967    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbe8d3cc2c53d6235ff2d3d96d7933a90673a6ced038fcb302dafa47dbfb8839"} err="failed to get container status \"bbe8d3cc2c53d6235ff2d3d96d7933a90673a6ced038fcb302dafa47dbfb8839\": rpc error: code = NotFound desc = could not find container \"bbe8d3cc2c53d6235ff2d3d96d7933a90673a6ced038fcb302dafa47dbfb8839\": container with ID starting with bbe8d3cc2c53d6235ff2d3d96d7933a90673a6ced038fcb302dafa47dbfb8839 not found: ID does not exist"
Jul 22 01:55:26.702697 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:26.702671    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tlnzk"]
Jul 22 01:55:26.718021 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:26.718002    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tlnzk"]
Jul 22 01:55:27.666894 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:27.666859    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eff4b50b-23ed-4ab2-a3a7-5e1bf9ad1d08" path="/var/lib/kubelet/pods/eff4b50b-23ed-4ab2-a3a7-5e1bf9ad1d08/volumes"
Jul 22 01:55:47.339314 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:47.336943    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-7qd5c"]
Jul 22 01:55:47.339314 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:47.337712    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eff4b50b-23ed-4ab2-a3a7-5e1bf9ad1d08" containerName="manager"
Jul 22 01:55:47.339314 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:47.337740    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="eff4b50b-23ed-4ab2-a3a7-5e1bf9ad1d08" containerName="manager"
Jul 22 01:55:47.339314 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:47.337900    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="eff4b50b-23ed-4ab2-a3a7-5e1bf9ad1d08" containerName="manager"
Jul 22 01:55:47.346763 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:47.346739    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-7qd5c"
Jul 22 01:55:47.348303 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:47.348280    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-7qd5c"]
Jul 22 01:55:47.349904 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:47.349883    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7zd2t\""
Jul 22 01:55:47.353443 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:47.353420    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 22 01:55:47.364059 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:47.364037    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl8rj\" (UniqueName: \"kubernetes.io/projected/fefc2bab-4fc5-4d96-b6a6-67ef3143fd6e-kube-api-access-zl8rj\") pod \"limitador-limitador-64c94fd64f-7qd5c\" (UID: \"fefc2bab-4fc5-4d96-b6a6-67ef3143fd6e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-7qd5c"
Jul 22 01:55:47.364162 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:47.364072    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fefc2bab-4fc5-4d96-b6a6-67ef3143fd6e-config-file\") pod \"limitador-limitador-64c94fd64f-7qd5c\" (UID: \"fefc2bab-4fc5-4d96-b6a6-67ef3143fd6e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-7qd5c"
Jul 22 01:55:47.448829 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:47.448801    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-7qd5c"]
Jul 22 01:55:47.465171 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:47.465147    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl8rj\" (UniqueName: \"kubernetes.io/projected/fefc2bab-4fc5-4d96-b6a6-67ef3143fd6e-kube-api-access-zl8rj\") pod \"limitador-limitador-64c94fd64f-7qd5c\" (UID: \"fefc2bab-4fc5-4d96-b6a6-67ef3143fd6e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-7qd5c"
Jul 22 01:55:47.465296 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:47.465180    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fefc2bab-4fc5-4d96-b6a6-67ef3143fd6e-config-file\") pod \"limitador-limitador-64c94fd64f-7qd5c\" (UID: \"fefc2bab-4fc5-4d96-b6a6-67ef3143fd6e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-7qd5c"
Jul 22 01:55:47.465855 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:47.465831    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fefc2bab-4fc5-4d96-b6a6-67ef3143fd6e-config-file\") pod \"limitador-limitador-64c94fd64f-7qd5c\" (UID: \"fefc2bab-4fc5-4d96-b6a6-67ef3143fd6e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-7qd5c"
Jul 22 01:55:47.476328 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:47.476299    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl8rj\" (UniqueName: \"kubernetes.io/projected/fefc2bab-4fc5-4d96-b6a6-67ef3143fd6e-kube-api-access-zl8rj\") pod \"limitador-limitador-64c94fd64f-7qd5c\" (UID: \"fefc2bab-4fc5-4d96-b6a6-67ef3143fd6e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-7qd5c"
Jul 22 01:55:47.657637 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:47.657563    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-7qd5c"
Jul 22 01:55:47.799780 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:47.799759    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-7qd5c"]
Jul 22 01:55:47.801977 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:55:47.801949    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfefc2bab_4fc5_4d96_b6a6_67ef3143fd6e.slice/crio-a214d6a325ac665735bb3627065c5bc28f32f58c18f73fea064ae14a89169e4a WatchSource:0}: Error finding container a214d6a325ac665735bb3627065c5bc28f32f58c18f73fea064ae14a89169e4a: Status 404 returned error can't find the container with id a214d6a325ac665735bb3627065c5bc28f32f58c18f73fea064ae14a89169e4a
Jul 22 01:55:48.148362 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:48.148332    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-rxrp6"]
Jul 22 01:55:48.152954 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:48.152929    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-rxrp6"
Jul 22 01:55:48.155308 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:48.155288    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mzt77\""
Jul 22 01:55:48.158082 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:48.158062    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-rxrp6"]
Jul 22 01:55:48.169532 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:48.169508    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs6r9\" (UniqueName: \"kubernetes.io/projected/8ae3355a-8fad-4b5c-b061-f055a671fe95-kube-api-access-rs6r9\") pod \"authorino-58cd955675-rxrp6\" (UID: \"8ae3355a-8fad-4b5c-b061-f055a671fe95\") " pod="kuadrant-system/authorino-58cd955675-rxrp6"
Jul 22 01:55:48.269873 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:48.269851    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs6r9\" (UniqueName: \"kubernetes.io/projected/8ae3355a-8fad-4b5c-b061-f055a671fe95-kube-api-access-rs6r9\") pod \"authorino-58cd955675-rxrp6\" (UID: \"8ae3355a-8fad-4b5c-b061-f055a671fe95\") " pod="kuadrant-system/authorino-58cd955675-rxrp6"
Jul 22 01:55:48.277636 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:48.277611    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs6r9\" (UniqueName: \"kubernetes.io/projected/8ae3355a-8fad-4b5c-b061-f055a671fe95-kube-api-access-rs6r9\") pod \"authorino-58cd955675-rxrp6\" (UID: \"8ae3355a-8fad-4b5c-b061-f055a671fe95\") " pod="kuadrant-system/authorino-58cd955675-rxrp6"
Jul 22 01:55:48.318689 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:48.318661    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-5gkbf"]
Jul 22 01:55:48.321925 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:48.321905    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-5gkbf"
Jul 22 01:55:48.330159 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:48.330138    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-5gkbf"]
Jul 22 01:55:48.370499 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:48.370471    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54k4j\" (UniqueName: \"kubernetes.io/projected/932054b0-f79a-42c3-952a-4bb6967abb81-kube-api-access-54k4j\") pod \"authorino-75d556bd77-5gkbf\" (UID: \"932054b0-f79a-42c3-952a-4bb6967abb81\") " pod="kuadrant-system/authorino-75d556bd77-5gkbf"
Jul 22 01:55:48.463459 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:48.463427    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-rxrp6"
Jul 22 01:55:48.471562 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:48.471541    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54k4j\" (UniqueName: \"kubernetes.io/projected/932054b0-f79a-42c3-952a-4bb6967abb81-kube-api-access-54k4j\") pod \"authorino-75d556bd77-5gkbf\" (UID: \"932054b0-f79a-42c3-952a-4bb6967abb81\") " pod="kuadrant-system/authorino-75d556bd77-5gkbf"
Jul 22 01:55:48.479152 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:48.479130    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54k4j\" (UniqueName: \"kubernetes.io/projected/932054b0-f79a-42c3-952a-4bb6967abb81-kube-api-access-54k4j\") pod \"authorino-75d556bd77-5gkbf\" (UID: \"932054b0-f79a-42c3-952a-4bb6967abb81\") " pod="kuadrant-system/authorino-75d556bd77-5gkbf"
Jul 22 01:55:48.622135 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:48.622084    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-rxrp6"]
Jul 22 01:55:48.622841 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:55:48.622813    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ae3355a_8fad_4b5c_b061_f055a671fe95.slice/crio-343d8b791192b5c881bd15bbeaefcbb119117dedbe7514e377218c659543c140 WatchSource:0}: Error finding container 343d8b791192b5c881bd15bbeaefcbb119117dedbe7514e377218c659543c140: Status 404 returned error can't find the container with id 343d8b791192b5c881bd15bbeaefcbb119117dedbe7514e377218c659543c140
Jul 22 01:55:48.630478 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:48.630274    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-5gkbf"
Jul 22 01:55:48.794707 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:48.793908    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-rxrp6" event={"ID":"8ae3355a-8fad-4b5c-b061-f055a671fe95","Type":"ContainerStarted","Data":"343d8b791192b5c881bd15bbeaefcbb119117dedbe7514e377218c659543c140"}
Jul 22 01:55:48.797026 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:48.796938    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-7qd5c" event={"ID":"fefc2bab-4fc5-4d96-b6a6-67ef3143fd6e","Type":"ContainerStarted","Data":"a214d6a325ac665735bb3627065c5bc28f32f58c18f73fea064ae14a89169e4a"}
Jul 22 01:55:48.803879 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:48.803509    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-5gkbf"]
Jul 22 01:55:49.815294 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:49.815254    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-5gkbf" event={"ID":"932054b0-f79a-42c3-952a-4bb6967abb81","Type":"ContainerStarted","Data":"0c78e89628de7689df2ff6d79cec2f8c652a336194927bf7e2cbca022b776b02"}
Jul 22 01:55:52.835959 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:52.835926    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-5gkbf" event={"ID":"932054b0-f79a-42c3-952a-4bb6967abb81","Type":"ContainerStarted","Data":"106eaffd8c17779290efc681e293c3825b65cfa1359f1ec7510f27fa4b318fc0"}
Jul 22 01:55:52.837417 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:52.837372    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-rxrp6" event={"ID":"8ae3355a-8fad-4b5c-b061-f055a671fe95","Type":"ContainerStarted","Data":"37881ae31bca1fdb8d9d6a0751e30ccba18af25f0233bcfbf06a394f123a8859"}
Jul 22 01:55:52.838790 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:52.838764    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-7qd5c" event={"ID":"fefc2bab-4fc5-4d96-b6a6-67ef3143fd6e","Type":"ContainerStarted","Data":"c9b0630050f5d2729082226ddd72efb905cddb0cb91b015042b1e6643954b829"}
Jul 22 01:55:52.838900 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:52.838823    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-7qd5c"
Jul 22 01:55:52.850448 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:52.850404    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-5gkbf" podStartSLOduration=1.010939314 podStartE2EDuration="4.850373454s" totalImagesPullingTime="3.83943414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:55:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:55:52.849487633 +0000 UTC m=+505.693316768" watchObservedRunningTime="2026-07-22 01:55:52.850373454 +0000 UTC m=+505.694202587"
Jul 22 01:55:52.865285 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:52.865234    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-rxrp6" podStartSLOduration=0.849651503 podStartE2EDuration="4.865216555s" totalImagesPullingTime="4.015565052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:55:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:55:52.864132189 +0000 UTC m=+505.707961321" watchObservedRunningTime="2026-07-22 01:55:52.865216555 +0000 UTC m=+505.709045689"
Jul 22 01:55:52.883306 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:52.883230    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-7qd5c" podStartSLOduration=1.047094058 podStartE2EDuration="5.883213497s" totalImagesPullingTime="4.836119439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:55:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:55:52.880349401 +0000 UTC m=+505.724178535" watchObservedRunningTime="2026-07-22 01:55:52.883213497 +0000 UTC m=+505.727042629"
Jul 22 01:55:52.886754 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:52.886734    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-rxrp6"]
Jul 22 01:55:54.849323 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:54.849281    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-rxrp6" podUID="8ae3355a-8fad-4b5c-b061-f055a671fe95" containerName="authorino" containerID="cri-o://37881ae31bca1fdb8d9d6a0751e30ccba18af25f0233bcfbf06a394f123a8859" gracePeriod=30
Jul 22 01:55:55.096270 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:55.096248    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-rxrp6"
Jul 22 01:55:55.115469 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:55.115402    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rs6r9\" (UniqueName: \"kubernetes.io/projected/8ae3355a-8fad-4b5c-b061-f055a671fe95-kube-api-access-rs6r9\") pod \"8ae3355a-8fad-4b5c-b061-f055a671fe95\" (UID: \"8ae3355a-8fad-4b5c-b061-f055a671fe95\") "
Jul 22 01:55:55.117511 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:55.117489    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ae3355a-8fad-4b5c-b061-f055a671fe95-kube-api-access-rs6r9" (OuterVolumeSpecName: "kube-api-access-rs6r9") pod "8ae3355a-8fad-4b5c-b061-f055a671fe95" (UID: "8ae3355a-8fad-4b5c-b061-f055a671fe95"). InnerVolumeSpecName "kube-api-access-rs6r9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 01:55:55.216447 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:55.216413    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs6r9\" (UniqueName: \"kubernetes.io/projected/8ae3355a-8fad-4b5c-b061-f055a671fe95-kube-api-access-rs6r9\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:55:55.855769 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:55.855734    2563 generic.go:356] "Generic (PLEG): container finished" podID="8ae3355a-8fad-4b5c-b061-f055a671fe95" containerID="37881ae31bca1fdb8d9d6a0751e30ccba18af25f0233bcfbf06a394f123a8859" exitCode=0
Jul 22 01:55:55.856127 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:55.855789    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-rxrp6"
Jul 22 01:55:55.856127 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:55.855833    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-rxrp6" event={"ID":"8ae3355a-8fad-4b5c-b061-f055a671fe95","Type":"ContainerDied","Data":"37881ae31bca1fdb8d9d6a0751e30ccba18af25f0233bcfbf06a394f123a8859"}
Jul 22 01:55:55.856127 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:55.855875    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-rxrp6" event={"ID":"8ae3355a-8fad-4b5c-b061-f055a671fe95","Type":"ContainerDied","Data":"343d8b791192b5c881bd15bbeaefcbb119117dedbe7514e377218c659543c140"}
Jul 22 01:55:55.856127 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:55.855895    2563 scope.go:117] "RemoveContainer" containerID="37881ae31bca1fdb8d9d6a0751e30ccba18af25f0233bcfbf06a394f123a8859"
Jul 22 01:55:55.864941 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:55.864921    2563 scope.go:117] "RemoveContainer" containerID="37881ae31bca1fdb8d9d6a0751e30ccba18af25f0233bcfbf06a394f123a8859"
Jul 22 01:55:55.865240 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:55:55.865221    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37881ae31bca1fdb8d9d6a0751e30ccba18af25f0233bcfbf06a394f123a8859\": container with ID starting with 37881ae31bca1fdb8d9d6a0751e30ccba18af25f0233bcfbf06a394f123a8859 not found: ID does not exist" containerID="37881ae31bca1fdb8d9d6a0751e30ccba18af25f0233bcfbf06a394f123a8859"
Jul 22 01:55:55.865304 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:55.865250    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37881ae31bca1fdb8d9d6a0751e30ccba18af25f0233bcfbf06a394f123a8859"} err="failed to get container status \"37881ae31bca1fdb8d9d6a0751e30ccba18af25f0233bcfbf06a394f123a8859\": rpc error: code = NotFound desc = could not find container \"37881ae31bca1fdb8d9d6a0751e30ccba18af25f0233bcfbf06a394f123a8859\": container with ID starting with 37881ae31bca1fdb8d9d6a0751e30ccba18af25f0233bcfbf06a394f123a8859 not found: ID does not exist"
Jul 22 01:55:55.870463 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:55.870437    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-rxrp6"]
Jul 22 01:55:55.874131 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:55.874110    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-rxrp6"]
Jul 22 01:55:57.663997 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:55:57.663965    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ae3355a-8fad-4b5c-b061-f055a671fe95" path="/var/lib/kubelet/pods/8ae3355a-8fad-4b5c-b061-f055a671fe95/volumes"
Jul 22 01:56:02.343057 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:02.343015    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-7qd5c"]
Jul 22 01:56:02.343538 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:02.343279    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-7qd5c" podUID="fefc2bab-4fc5-4d96-b6a6-67ef3143fd6e" containerName="limitador" containerID="cri-o://c9b0630050f5d2729082226ddd72efb905cddb0cb91b015042b1e6643954b829" gracePeriod=30
Jul 22 01:56:02.344038 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:02.344016    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-7qd5c"
Jul 22 01:56:02.886646 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:02.886624    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-7qd5c"
Jul 22 01:56:02.891914 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:02.891886    2563 generic.go:356] "Generic (PLEG): container finished" podID="fefc2bab-4fc5-4d96-b6a6-67ef3143fd6e" containerID="c9b0630050f5d2729082226ddd72efb905cddb0cb91b015042b1e6643954b829" exitCode=0
Jul 22 01:56:02.892005 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:02.891922    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-7qd5c"
Jul 22 01:56:02.892005 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:02.891964    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-7qd5c" event={"ID":"fefc2bab-4fc5-4d96-b6a6-67ef3143fd6e","Type":"ContainerDied","Data":"c9b0630050f5d2729082226ddd72efb905cddb0cb91b015042b1e6643954b829"}
Jul 22 01:56:02.892099 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:02.892014    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-7qd5c" event={"ID":"fefc2bab-4fc5-4d96-b6a6-67ef3143fd6e","Type":"ContainerDied","Data":"a214d6a325ac665735bb3627065c5bc28f32f58c18f73fea064ae14a89169e4a"}
Jul 22 01:56:02.892099 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:02.892038    2563 scope.go:117] "RemoveContainer" containerID="c9b0630050f5d2729082226ddd72efb905cddb0cb91b015042b1e6643954b829"
Jul 22 01:56:02.899704 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:02.899688    2563 scope.go:117] "RemoveContainer" containerID="c9b0630050f5d2729082226ddd72efb905cddb0cb91b015042b1e6643954b829"
Jul 22 01:56:02.899964 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:56:02.899949    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9b0630050f5d2729082226ddd72efb905cddb0cb91b015042b1e6643954b829\": container with ID starting with c9b0630050f5d2729082226ddd72efb905cddb0cb91b015042b1e6643954b829 not found: ID does not exist" containerID="c9b0630050f5d2729082226ddd72efb905cddb0cb91b015042b1e6643954b829"
Jul 22 01:56:02.900000 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:02.899981    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9b0630050f5d2729082226ddd72efb905cddb0cb91b015042b1e6643954b829"} err="failed to get container status \"c9b0630050f5d2729082226ddd72efb905cddb0cb91b015042b1e6643954b829\": rpc error: code = NotFound desc = could not find container \"c9b0630050f5d2729082226ddd72efb905cddb0cb91b015042b1e6643954b829\": container with ID starting with c9b0630050f5d2729082226ddd72efb905cddb0cb91b015042b1e6643954b829 not found: ID does not exist"
Jul 22 01:56:02.969673 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:02.969650    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fefc2bab-4fc5-4d96-b6a6-67ef3143fd6e-config-file\") pod \"fefc2bab-4fc5-4d96-b6a6-67ef3143fd6e\" (UID: \"fefc2bab-4fc5-4d96-b6a6-67ef3143fd6e\") "
Jul 22 01:56:02.969783 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:02.969681    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zl8rj\" (UniqueName: \"kubernetes.io/projected/fefc2bab-4fc5-4d96-b6a6-67ef3143fd6e-kube-api-access-zl8rj\") pod \"fefc2bab-4fc5-4d96-b6a6-67ef3143fd6e\" (UID: \"fefc2bab-4fc5-4d96-b6a6-67ef3143fd6e\") "
Jul 22 01:56:02.970018 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:02.969996    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fefc2bab-4fc5-4d96-b6a6-67ef3143fd6e-config-file" (OuterVolumeSpecName: "config-file") pod "fefc2bab-4fc5-4d96-b6a6-67ef3143fd6e" (UID: "fefc2bab-4fc5-4d96-b6a6-67ef3143fd6e"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 01:56:02.971697 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:02.971677    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fefc2bab-4fc5-4d96-b6a6-67ef3143fd6e-kube-api-access-zl8rj" (OuterVolumeSpecName: "kube-api-access-zl8rj") pod "fefc2bab-4fc5-4d96-b6a6-67ef3143fd6e" (UID: "fefc2bab-4fc5-4d96-b6a6-67ef3143fd6e"). InnerVolumeSpecName "kube-api-access-zl8rj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 01:56:03.070893 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:03.070870    2563 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fefc2bab-4fc5-4d96-b6a6-67ef3143fd6e-config-file\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:56:03.070893 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:03.070890    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl8rj\" (UniqueName: \"kubernetes.io/projected/fefc2bab-4fc5-4d96-b6a6-67ef3143fd6e-kube-api-access-zl8rj\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:56:03.214304 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:03.214267    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-7qd5c"]
Jul 22 01:56:03.220346 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:03.220295    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-7qd5c"]
Jul 22 01:56:03.662791 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:03.662725    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fefc2bab-4fc5-4d96-b6a6-67ef3143fd6e" path="/var/lib/kubelet/pods/fefc2bab-4fc5-4d96-b6a6-67ef3143fd6e/volumes"
Jul 22 01:56:04.091605 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:04.091571    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-d96gh"]
Jul 22 01:56:04.092052 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:04.092032    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fefc2bab-4fc5-4d96-b6a6-67ef3143fd6e" containerName="limitador"
Jul 22 01:56:04.092093 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:04.092058    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fefc2bab-4fc5-4d96-b6a6-67ef3143fd6e" containerName="limitador"
Jul 22 01:56:04.092127 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:04.092098    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ae3355a-8fad-4b5c-b061-f055a671fe95" containerName="authorino"
Jul 22 01:56:04.092127 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:04.092108    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ae3355a-8fad-4b5c-b061-f055a671fe95" containerName="authorino"
Jul 22 01:56:04.092212 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:04.092200    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fefc2bab-4fc5-4d96-b6a6-67ef3143fd6e" containerName="limitador"
Jul 22 01:56:04.092247 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:04.092215    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ae3355a-8fad-4b5c-b061-f055a671fe95" containerName="authorino"
Jul 22 01:56:04.096650 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:04.096622    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-d96gh"
Jul 22 01:56:04.098781 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:04.098753    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 22 01:56:04.098892 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:04.098768    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-txxxw\""
Jul 22 01:56:04.098892 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:04.098806    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 22 01:56:04.099693 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:04.099670    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 22 01:56:04.102094 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:04.102073    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-d96gh"]
Jul 22 01:56:04.178149 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:04.178122    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/61a18758-f622-4cd7-a6ce-1d918b0d120b-data\") pod \"postgres-69fb4594cf-d96gh\" (UID: \"61a18758-f622-4cd7-a6ce-1d918b0d120b\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-d96gh"
Jul 22 01:56:04.178240 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:04.178200    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v962l\" (UniqueName: \"kubernetes.io/projected/61a18758-f622-4cd7-a6ce-1d918b0d120b-kube-api-access-v962l\") pod \"postgres-69fb4594cf-d96gh\" (UID: \"61a18758-f622-4cd7-a6ce-1d918b0d120b\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-d96gh"
Jul 22 01:56:04.279295 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:04.279271    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v962l\" (UniqueName: \"kubernetes.io/projected/61a18758-f622-4cd7-a6ce-1d918b0d120b-kube-api-access-v962l\") pod \"postgres-69fb4594cf-d96gh\" (UID: \"61a18758-f622-4cd7-a6ce-1d918b0d120b\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-d96gh"
Jul 22 01:56:04.279417 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:04.279311    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/61a18758-f622-4cd7-a6ce-1d918b0d120b-data\") pod \"postgres-69fb4594cf-d96gh\" (UID: \"61a18758-f622-4cd7-a6ce-1d918b0d120b\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-d96gh"
Jul 22 01:56:04.279644 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:04.279626    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/61a18758-f622-4cd7-a6ce-1d918b0d120b-data\") pod \"postgres-69fb4594cf-d96gh\" (UID: \"61a18758-f622-4cd7-a6ce-1d918b0d120b\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-d96gh"
Jul 22 01:56:04.287714 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:04.287696    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v962l\" (UniqueName: \"kubernetes.io/projected/61a18758-f622-4cd7-a6ce-1d918b0d120b-kube-api-access-v962l\") pod \"postgres-69fb4594cf-d96gh\" (UID: \"61a18758-f622-4cd7-a6ce-1d918b0d120b\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-d96gh"
Jul 22 01:56:04.409754 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:04.409708    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-d96gh"
Jul 22 01:56:04.535594 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:04.535569    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-d96gh"]
Jul 22 01:56:04.537697 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:56:04.537582    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a18758_f622_4cd7_a6ce_1d918b0d120b.slice/crio-4f34310ac5502021777d9aa5b1ff88f1566a891b19eb1becd89834a52cd729bd WatchSource:0}: Error finding container 4f34310ac5502021777d9aa5b1ff88f1566a891b19eb1becd89834a52cd729bd: Status 404 returned error can't find the container with id 4f34310ac5502021777d9aa5b1ff88f1566a891b19eb1becd89834a52cd729bd
Jul 22 01:56:04.908902 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:04.908871    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-d96gh" event={"ID":"61a18758-f622-4cd7-a6ce-1d918b0d120b","Type":"ContainerStarted","Data":"4f34310ac5502021777d9aa5b1ff88f1566a891b19eb1becd89834a52cd729bd"}
Jul 22 01:56:09.936892 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:09.936851    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-d96gh" event={"ID":"61a18758-f622-4cd7-a6ce-1d918b0d120b","Type":"ContainerStarted","Data":"bfd2ed8e38074791bbb9b6b3356f338dc9423488e7989251e341781cde6be82a"}
Jul 22 01:56:09.937247 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:09.936930    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-d96gh"
Jul 22 01:56:09.951870 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:09.951824    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-d96gh" podStartSLOduration=1.008319138 podStartE2EDuration="5.95180966s" totalImagesPullingTime="4.943490522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:56:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:56:09.950600826 +0000 UTC m=+522.794429948" watchObservedRunningTime="2026-07-22 01:56:09.95180966 +0000 UTC m=+522.795638792"
Jul 22 01:56:15.970212 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:15.970184    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-d96gh"
Jul 22 01:56:16.541054 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:16.541022    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-pqc4s"]
Jul 22 01:56:16.547633 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:16.547613    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-pqc4s"
Jul 22 01:56:16.551722 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:16.551696    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-pqc4s"]
Jul 22 01:56:16.569658 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:16.569637    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px6hf\" (UniqueName: \"kubernetes.io/projected/c076920c-e8d2-4dab-b5aa-38c4a4ff0741-kube-api-access-px6hf\") pod \"authorino-5d4b67c446-pqc4s\" (UID: \"c076920c-e8d2-4dab-b5aa-38c4a4ff0741\") " pod="kuadrant-system/authorino-5d4b67c446-pqc4s"
Jul 22 01:56:16.670732 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:16.670707    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px6hf\" (UniqueName: \"kubernetes.io/projected/c076920c-e8d2-4dab-b5aa-38c4a4ff0741-kube-api-access-px6hf\") pod \"authorino-5d4b67c446-pqc4s\" (UID: \"c076920c-e8d2-4dab-b5aa-38c4a4ff0741\") " pod="kuadrant-system/authorino-5d4b67c446-pqc4s"
Jul 22 01:56:16.679428 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:16.679406    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-px6hf\" (UniqueName: \"kubernetes.io/projected/c076920c-e8d2-4dab-b5aa-38c4a4ff0741-kube-api-access-px6hf\") pod \"authorino-5d4b67c446-pqc4s\" (UID: \"c076920c-e8d2-4dab-b5aa-38c4a4ff0741\") " pod="kuadrant-system/authorino-5d4b67c446-pqc4s"
Jul 22 01:56:16.794667 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:16.794611    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-pqc4s"]
Jul 22 01:56:16.794802 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:16.794790    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-pqc4s"
Jul 22 01:56:16.821731 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:16.821710    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7cc6d57897-n4czl"]
Jul 22 01:56:16.826974 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:16.826956    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cc6d57897-n4czl"
Jul 22 01:56:16.832772 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:16.832743    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cc6d57897-n4czl"]
Jul 22 01:56:16.871654 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:16.871628    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsv68\" (UniqueName: \"kubernetes.io/projected/de8d5a15-8261-4d9e-81dd-68509accd4ad-kube-api-access-xsv68\") pod \"authorino-7cc6d57897-n4czl\" (UID: \"de8d5a15-8261-4d9e-81dd-68509accd4ad\") " pod="kuadrant-system/authorino-7cc6d57897-n4czl"
Jul 22 01:56:16.916568 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:16.916543    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7cc6d57897-n4czl"]
Jul 22 01:56:16.916781 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:56:16.916764    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-xsv68], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-7cc6d57897-n4czl" podUID="de8d5a15-8261-4d9e-81dd-68509accd4ad"
Jul 22 01:56:16.930400 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:16.930358    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-pqc4s"]
Jul 22 01:56:16.932420 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:56:16.932393    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc076920c_e8d2_4dab_b5aa_38c4a4ff0741.slice/crio-53119443633bc40b2b70871c62017113a25861100c6d7f924c3d42df147127a4 WatchSource:0}: Error finding container 53119443633bc40b2b70871c62017113a25861100c6d7f924c3d42df147127a4: Status 404 returned error can't find the container with id 53119443633bc40b2b70871c62017113a25861100c6d7f924c3d42df147127a4
Jul 22 01:56:16.946224 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:16.946200    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-74d8ccbfbf-55l94"]
Jul 22 01:56:16.950213 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:16.950196    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74d8ccbfbf-55l94"
Jul 22 01:56:16.952305 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:16.952286    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 22 01:56:16.958842 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:16.958820    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-74d8ccbfbf-55l94"]
Jul 22 01:56:16.975154 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:16.972461    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x9fb\" (UniqueName: \"kubernetes.io/projected/3b9564a7-4d3c-4f10-a397-3f7b994442e8-kube-api-access-4x9fb\") pod \"authorino-74d8ccbfbf-55l94\" (UID: \"3b9564a7-4d3c-4f10-a397-3f7b994442e8\") " pod="kuadrant-system/authorino-74d8ccbfbf-55l94"
Jul 22 01:56:16.975154 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:16.972554    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsv68\" (UniqueName: \"kubernetes.io/projected/de8d5a15-8261-4d9e-81dd-68509accd4ad-kube-api-access-xsv68\") pod \"authorino-7cc6d57897-n4czl\" (UID: \"de8d5a15-8261-4d9e-81dd-68509accd4ad\") " pod="kuadrant-system/authorino-7cc6d57897-n4czl"
Jul 22 01:56:16.975154 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:16.972604    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3b9564a7-4d3c-4f10-a397-3f7b994442e8-tls-cert\") pod \"authorino-74d8ccbfbf-55l94\" (UID: \"3b9564a7-4d3c-4f10-a397-3f7b994442e8\") " pod="kuadrant-system/authorino-74d8ccbfbf-55l94"
Jul 22 01:56:16.977951 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:16.977925    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cc6d57897-n4czl"
Jul 22 01:56:16.977951 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:16.977936    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-pqc4s" event={"ID":"c076920c-e8d2-4dab-b5aa-38c4a4ff0741","Type":"ContainerStarted","Data":"53119443633bc40b2b70871c62017113a25861100c6d7f924c3d42df147127a4"}
Jul 22 01:56:16.980024 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:16.980007    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsv68\" (UniqueName: \"kubernetes.io/projected/de8d5a15-8261-4d9e-81dd-68509accd4ad-kube-api-access-xsv68\") pod \"authorino-7cc6d57897-n4czl\" (UID: \"de8d5a15-8261-4d9e-81dd-68509accd4ad\") " pod="kuadrant-system/authorino-7cc6d57897-n4czl"
Jul 22 01:56:16.990833 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:16.990817    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cc6d57897-n4czl"
Jul 22 01:56:17.073301 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:17.073253    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsv68\" (UniqueName: \"kubernetes.io/projected/de8d5a15-8261-4d9e-81dd-68509accd4ad-kube-api-access-xsv68\") pod \"de8d5a15-8261-4d9e-81dd-68509accd4ad\" (UID: \"de8d5a15-8261-4d9e-81dd-68509accd4ad\") "
Jul 22 01:56:17.073418 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:17.073327    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x9fb\" (UniqueName: \"kubernetes.io/projected/3b9564a7-4d3c-4f10-a397-3f7b994442e8-kube-api-access-4x9fb\") pod \"authorino-74d8ccbfbf-55l94\" (UID: \"3b9564a7-4d3c-4f10-a397-3f7b994442e8\") " pod="kuadrant-system/authorino-74d8ccbfbf-55l94"
Jul 22 01:56:17.073418 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:17.073410    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3b9564a7-4d3c-4f10-a397-3f7b994442e8-tls-cert\") pod \"authorino-74d8ccbfbf-55l94\" (UID: \"3b9564a7-4d3c-4f10-a397-3f7b994442e8\") " pod="kuadrant-system/authorino-74d8ccbfbf-55l94"
Jul 22 01:56:17.075157 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:17.075132    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de8d5a15-8261-4d9e-81dd-68509accd4ad-kube-api-access-xsv68" (OuterVolumeSpecName: "kube-api-access-xsv68") pod "de8d5a15-8261-4d9e-81dd-68509accd4ad" (UID: "de8d5a15-8261-4d9e-81dd-68509accd4ad"). InnerVolumeSpecName "kube-api-access-xsv68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 01:56:17.075630 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:17.075614    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3b9564a7-4d3c-4f10-a397-3f7b994442e8-tls-cert\") pod \"authorino-74d8ccbfbf-55l94\" (UID: \"3b9564a7-4d3c-4f10-a397-3f7b994442e8\") " pod="kuadrant-system/authorino-74d8ccbfbf-55l94"
Jul 22 01:56:17.080856 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:17.080834    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x9fb\" (UniqueName: \"kubernetes.io/projected/3b9564a7-4d3c-4f10-a397-3f7b994442e8-kube-api-access-4x9fb\") pod \"authorino-74d8ccbfbf-55l94\" (UID: \"3b9564a7-4d3c-4f10-a397-3f7b994442e8\") " pod="kuadrant-system/authorino-74d8ccbfbf-55l94"
Jul 22 01:56:17.174489 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:17.174470    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsv68\" (UniqueName: \"kubernetes.io/projected/de8d5a15-8261-4d9e-81dd-68509accd4ad-kube-api-access-xsv68\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:56:17.260356 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:17.260330    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74d8ccbfbf-55l94"
Jul 22 01:56:17.392794 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:17.392759    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-74d8ccbfbf-55l94"]
Jul 22 01:56:17.395171 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:56:17.395129    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b9564a7_4d3c_4f10_a397_3f7b994442e8.slice/crio-1545c1651c49fac023a5c59b328cbca2b6a9644b37aa2cd00f6e65d582c045ae WatchSource:0}: Error finding container 1545c1651c49fac023a5c59b328cbca2b6a9644b37aa2cd00f6e65d582c045ae: Status 404 returned error can't find the container with id 1545c1651c49fac023a5c59b328cbca2b6a9644b37aa2cd00f6e65d582c045ae
Jul 22 01:56:17.983614 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:17.983588    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74d8ccbfbf-55l94" event={"ID":"3b9564a7-4d3c-4f10-a397-3f7b994442e8","Type":"ContainerStarted","Data":"1545c1651c49fac023a5c59b328cbca2b6a9644b37aa2cd00f6e65d582c045ae"}
Jul 22 01:56:17.984925 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:17.984900    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cc6d57897-n4czl"
Jul 22 01:56:17.985015 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:17.984898    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-pqc4s" event={"ID":"c076920c-e8d2-4dab-b5aa-38c4a4ff0741","Type":"ContainerStarted","Data":"7376f9f6bbd7b82f2aa42c1543b1f17eb1120ca73809d246a9ca5321fd20cda3"}
Jul 22 01:56:17.985015 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:17.984963    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-pqc4s" podUID="c076920c-e8d2-4dab-b5aa-38c4a4ff0741" containerName="authorino" containerID="cri-o://7376f9f6bbd7b82f2aa42c1543b1f17eb1120ca73809d246a9ca5321fd20cda3" gracePeriod=30
Jul 22 01:56:18.000101 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:18.000060    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-pqc4s" podStartSLOduration=1.6141135420000001 podStartE2EDuration="2.000047267s" totalImagesPullingTime="385.933725ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:56:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:56:17.99884189 +0000 UTC m=+530.842671022" watchObservedRunningTime="2026-07-22 01:56:18.000047267 +0000 UTC m=+530.843876437"
Jul 22 01:56:18.020248 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:18.020224    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7cc6d57897-n4czl"]
Jul 22 01:56:18.026420 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:18.026396    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7cc6d57897-n4czl"]
Jul 22 01:56:18.236485 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:18.236434    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-pqc4s"
Jul 22 01:56:18.281561 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:18.281536    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-px6hf\" (UniqueName: \"kubernetes.io/projected/c076920c-e8d2-4dab-b5aa-38c4a4ff0741-kube-api-access-px6hf\") pod \"c076920c-e8d2-4dab-b5aa-38c4a4ff0741\" (UID: \"c076920c-e8d2-4dab-b5aa-38c4a4ff0741\") "
Jul 22 01:56:18.283475 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:18.283451    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c076920c-e8d2-4dab-b5aa-38c4a4ff0741-kube-api-access-px6hf" (OuterVolumeSpecName: "kube-api-access-px6hf") pod "c076920c-e8d2-4dab-b5aa-38c4a4ff0741" (UID: "c076920c-e8d2-4dab-b5aa-38c4a4ff0741"). InnerVolumeSpecName "kube-api-access-px6hf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 01:56:18.382493 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:18.382472    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px6hf\" (UniqueName: \"kubernetes.io/projected/c076920c-e8d2-4dab-b5aa-38c4a4ff0741-kube-api-access-px6hf\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:56:18.990700 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:18.990640    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74d8ccbfbf-55l94" event={"ID":"3b9564a7-4d3c-4f10-a397-3f7b994442e8","Type":"ContainerStarted","Data":"03727d231de795a1d36c3f3d4e228f94b278e3f09e47975cf2781b3c364d0ece"}
Jul 22 01:56:18.991843 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:18.991813    2563 generic.go:356] "Generic (PLEG): container finished" podID="c076920c-e8d2-4dab-b5aa-38c4a4ff0741" containerID="7376f9f6bbd7b82f2aa42c1543b1f17eb1120ca73809d246a9ca5321fd20cda3" exitCode=0
Jul 22 01:56:18.991843 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:18.991840    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-pqc4s"
Jul 22 01:56:18.991989 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:18.991859    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-pqc4s" event={"ID":"c076920c-e8d2-4dab-b5aa-38c4a4ff0741","Type":"ContainerDied","Data":"7376f9f6bbd7b82f2aa42c1543b1f17eb1120ca73809d246a9ca5321fd20cda3"}
Jul 22 01:56:18.991989 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:18.991883    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-pqc4s" event={"ID":"c076920c-e8d2-4dab-b5aa-38c4a4ff0741","Type":"ContainerDied","Data":"53119443633bc40b2b70871c62017113a25861100c6d7f924c3d42df147127a4"}
Jul 22 01:56:18.991989 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:18.991904    2563 scope.go:117] "RemoveContainer" containerID="7376f9f6bbd7b82f2aa42c1543b1f17eb1120ca73809d246a9ca5321fd20cda3"
Jul 22 01:56:19.002039 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:19.002024    2563 scope.go:117] "RemoveContainer" containerID="7376f9f6bbd7b82f2aa42c1543b1f17eb1120ca73809d246a9ca5321fd20cda3"
Jul 22 01:56:19.002323 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:56:19.002306    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7376f9f6bbd7b82f2aa42c1543b1f17eb1120ca73809d246a9ca5321fd20cda3\": container with ID starting with 7376f9f6bbd7b82f2aa42c1543b1f17eb1120ca73809d246a9ca5321fd20cda3 not found: ID does not exist" containerID="7376f9f6bbd7b82f2aa42c1543b1f17eb1120ca73809d246a9ca5321fd20cda3"
Jul 22 01:56:19.002372 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:19.002330    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7376f9f6bbd7b82f2aa42c1543b1f17eb1120ca73809d246a9ca5321fd20cda3"} err="failed to get container status \"7376f9f6bbd7b82f2aa42c1543b1f17eb1120ca73809d246a9ca5321fd20cda3\": rpc error: code = NotFound desc = could not find container \"7376f9f6bbd7b82f2aa42c1543b1f17eb1120ca73809d246a9ca5321fd20cda3\": container with ID starting with 7376f9f6bbd7b82f2aa42c1543b1f17eb1120ca73809d246a9ca5321fd20cda3 not found: ID does not exist"
Jul 22 01:56:19.008998 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:19.008960    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-74d8ccbfbf-55l94" podStartSLOduration=2.330597907 podStartE2EDuration="3.008949044s" totalImagesPullingTime="678.351137ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:56:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:56:19.007319241 +0000 UTC m=+531.851148372" watchObservedRunningTime="2026-07-22 01:56:19.008949044 +0000 UTC m=+531.852778176"
Jul 22 01:56:19.022297 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:19.022278    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-pqc4s"]
Jul 22 01:56:19.025664 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:19.025643    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-pqc4s"]
Jul 22 01:56:19.035071 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:19.035049    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-5gkbf"]
Jul 22 01:56:19.035212 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:19.035190    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-5gkbf" podUID="932054b0-f79a-42c3-952a-4bb6967abb81" containerName="authorino" containerID="cri-o://106eaffd8c17779290efc681e293c3825b65cfa1359f1ec7510f27fa4b318fc0" gracePeriod=30
Jul 22 01:56:19.323802 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:19.323779    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-5gkbf"
Jul 22 01:56:19.388402 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:19.388325    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54k4j\" (UniqueName: \"kubernetes.io/projected/932054b0-f79a-42c3-952a-4bb6967abb81-kube-api-access-54k4j\") pod \"932054b0-f79a-42c3-952a-4bb6967abb81\" (UID: \"932054b0-f79a-42c3-952a-4bb6967abb81\") "
Jul 22 01:56:19.390274 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:19.390252    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/932054b0-f79a-42c3-952a-4bb6967abb81-kube-api-access-54k4j" (OuterVolumeSpecName: "kube-api-access-54k4j") pod "932054b0-f79a-42c3-952a-4bb6967abb81" (UID: "932054b0-f79a-42c3-952a-4bb6967abb81"). InnerVolumeSpecName "kube-api-access-54k4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 01:56:19.488854 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:19.488834    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54k4j\" (UniqueName: \"kubernetes.io/projected/932054b0-f79a-42c3-952a-4bb6967abb81-kube-api-access-54k4j\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:56:19.663097 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:19.663040    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c076920c-e8d2-4dab-b5aa-38c4a4ff0741" path="/var/lib/kubelet/pods/c076920c-e8d2-4dab-b5aa-38c4a4ff0741/volumes"
Jul 22 01:56:19.663350 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:19.663337    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de8d5a15-8261-4d9e-81dd-68509accd4ad" path="/var/lib/kubelet/pods/de8d5a15-8261-4d9e-81dd-68509accd4ad/volumes"
Jul 22 01:56:19.997277 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:19.997249    2563 generic.go:356] "Generic (PLEG): container finished" podID="932054b0-f79a-42c3-952a-4bb6967abb81" containerID="106eaffd8c17779290efc681e293c3825b65cfa1359f1ec7510f27fa4b318fc0" exitCode=0
Jul 22 01:56:19.997710 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:19.997306    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-5gkbf"
Jul 22 01:56:19.997710 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:19.997331    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-5gkbf" event={"ID":"932054b0-f79a-42c3-952a-4bb6967abb81","Type":"ContainerDied","Data":"106eaffd8c17779290efc681e293c3825b65cfa1359f1ec7510f27fa4b318fc0"}
Jul 22 01:56:19.997710 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:19.997396    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-5gkbf" event={"ID":"932054b0-f79a-42c3-952a-4bb6967abb81","Type":"ContainerDied","Data":"0c78e89628de7689df2ff6d79cec2f8c652a336194927bf7e2cbca022b776b02"}
Jul 22 01:56:19.997710 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:19.997424    2563 scope.go:117] "RemoveContainer" containerID="106eaffd8c17779290efc681e293c3825b65cfa1359f1ec7510f27fa4b318fc0"
Jul 22 01:56:20.005931 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:20.005917    2563 scope.go:117] "RemoveContainer" containerID="106eaffd8c17779290efc681e293c3825b65cfa1359f1ec7510f27fa4b318fc0"
Jul 22 01:56:20.006161 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:56:20.006144    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"106eaffd8c17779290efc681e293c3825b65cfa1359f1ec7510f27fa4b318fc0\": container with ID starting with 106eaffd8c17779290efc681e293c3825b65cfa1359f1ec7510f27fa4b318fc0 not found: ID does not exist" containerID="106eaffd8c17779290efc681e293c3825b65cfa1359f1ec7510f27fa4b318fc0"
Jul 22 01:56:20.006210 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:20.006170    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"106eaffd8c17779290efc681e293c3825b65cfa1359f1ec7510f27fa4b318fc0"} err="failed to get container status \"106eaffd8c17779290efc681e293c3825b65cfa1359f1ec7510f27fa4b318fc0\": rpc error: code = NotFound desc = could not find container \"106eaffd8c17779290efc681e293c3825b65cfa1359f1ec7510f27fa4b318fc0\": container with ID starting with 106eaffd8c17779290efc681e293c3825b65cfa1359f1ec7510f27fa4b318fc0 not found: ID does not exist"
Jul 22 01:56:20.013730 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:20.013661    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-5gkbf"]
Jul 22 01:56:20.015469 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:20.015451    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-5gkbf"]
Jul 22 01:56:21.663756 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:21.663721    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="932054b0-f79a-42c3-952a-4bb6967abb81" path="/var/lib/kubelet/pods/932054b0-f79a-42c3-952a-4bb6967abb81/volumes"
Jul 22 01:56:59.064094 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:59.064063    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-8459478d64-8xtx5"]
Jul 22 01:56:59.064542 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:59.064372    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c076920c-e8d2-4dab-b5aa-38c4a4ff0741" containerName="authorino"
Jul 22 01:56:59.064542 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:59.064396    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c076920c-e8d2-4dab-b5aa-38c4a4ff0741" containerName="authorino"
Jul 22 01:56:59.064542 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:59.064414    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="932054b0-f79a-42c3-952a-4bb6967abb81" containerName="authorino"
Jul 22 01:56:59.064542 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:59.064420    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="932054b0-f79a-42c3-952a-4bb6967abb81" containerName="authorino"
Jul 22 01:56:59.064542 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:59.064472    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c076920c-e8d2-4dab-b5aa-38c4a4ff0741" containerName="authorino"
Jul 22 01:56:59.064542 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:59.064480    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="932054b0-f79a-42c3-952a-4bb6967abb81" containerName="authorino"
Jul 22 01:56:59.067313 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:59.067297    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8459478d64-8xtx5"
Jul 22 01:56:59.069625 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:59.069599    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-kgsb5\""
Jul 22 01:56:59.069749 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:59.069657    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 22 01:56:59.078690 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:59.078669    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8459478d64-8xtx5"]
Jul 22 01:56:59.168963 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:59.168915    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33bc4a81-9a52-468d-b37d-dfbfd3edb289-openshift-service-ca\") pod \"maas-api-8459478d64-8xtx5\" (UID: \"33bc4a81-9a52-468d-b37d-dfbfd3edb289\") " pod="odh-ai-gateway-infra/maas-api-8459478d64-8xtx5"
Jul 22 01:56:59.169059 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:59.169035    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w65qn\" (UniqueName: \"kubernetes.io/projected/33bc4a81-9a52-468d-b37d-dfbfd3edb289-kube-api-access-w65qn\") pod \"maas-api-8459478d64-8xtx5\" (UID: \"33bc4a81-9a52-468d-b37d-dfbfd3edb289\") " pod="odh-ai-gateway-infra/maas-api-8459478d64-8xtx5"
Jul 22 01:56:59.169111 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:59.169097    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33bc4a81-9a52-468d-b37d-dfbfd3edb289-maas-api-tls\") pod \"maas-api-8459478d64-8xtx5\" (UID: \"33bc4a81-9a52-468d-b37d-dfbfd3edb289\") " pod="odh-ai-gateway-infra/maas-api-8459478d64-8xtx5"
Jul 22 01:56:59.269584 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:59.269562    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w65qn\" (UniqueName: \"kubernetes.io/projected/33bc4a81-9a52-468d-b37d-dfbfd3edb289-kube-api-access-w65qn\") pod \"maas-api-8459478d64-8xtx5\" (UID: \"33bc4a81-9a52-468d-b37d-dfbfd3edb289\") " pod="odh-ai-gateway-infra/maas-api-8459478d64-8xtx5"
Jul 22 01:56:59.269671 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:59.269603    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33bc4a81-9a52-468d-b37d-dfbfd3edb289-maas-api-tls\") pod \"maas-api-8459478d64-8xtx5\" (UID: \"33bc4a81-9a52-468d-b37d-dfbfd3edb289\") " pod="odh-ai-gateway-infra/maas-api-8459478d64-8xtx5"
Jul 22 01:56:59.269671 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:59.269661    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33bc4a81-9a52-468d-b37d-dfbfd3edb289-openshift-service-ca\") pod \"maas-api-8459478d64-8xtx5\" (UID: \"33bc4a81-9a52-468d-b37d-dfbfd3edb289\") " pod="odh-ai-gateway-infra/maas-api-8459478d64-8xtx5"
Jul 22 01:56:59.270318 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:59.270301    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33bc4a81-9a52-468d-b37d-dfbfd3edb289-openshift-service-ca\") pod \"maas-api-8459478d64-8xtx5\" (UID: \"33bc4a81-9a52-468d-b37d-dfbfd3edb289\") " pod="odh-ai-gateway-infra/maas-api-8459478d64-8xtx5"
Jul 22 01:56:59.271926 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:59.271908    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33bc4a81-9a52-468d-b37d-dfbfd3edb289-maas-api-tls\") pod \"maas-api-8459478d64-8xtx5\" (UID: \"33bc4a81-9a52-468d-b37d-dfbfd3edb289\") " pod="odh-ai-gateway-infra/maas-api-8459478d64-8xtx5"
Jul 22 01:56:59.278343 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:59.278320    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w65qn\" (UniqueName: \"kubernetes.io/projected/33bc4a81-9a52-468d-b37d-dfbfd3edb289-kube-api-access-w65qn\") pod \"maas-api-8459478d64-8xtx5\" (UID: \"33bc4a81-9a52-468d-b37d-dfbfd3edb289\") " pod="odh-ai-gateway-infra/maas-api-8459478d64-8xtx5"
Jul 22 01:56:59.378256 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:59.378205    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8459478d64-8xtx5"
Jul 22 01:56:59.519032 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:59.519000    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8459478d64-8xtx5"]
Jul 22 01:56:59.521911 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:56:59.521879    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33bc4a81_9a52_468d_b37d_dfbfd3edb289.slice/crio-8d0505f9650a85f060d61cef446478f07277f3a8b80f508a09cad0a4aee2f409 WatchSource:0}: Error finding container 8d0505f9650a85f060d61cef446478f07277f3a8b80f508a09cad0a4aee2f409: Status 404 returned error can't find the container with id 8d0505f9650a85f060d61cef446478f07277f3a8b80f508a09cad0a4aee2f409
Jul 22 01:56:59.611125 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:59.611099    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-tcq4d"]
Jul 22 01:56:59.615603 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:59.615586    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-tcq4d"
Jul 22 01:56:59.618026 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:59.618006    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 22 01:56:59.618453 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:59.618368    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-sgtnk\""
Jul 22 01:56:59.629550 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:59.629498    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-tcq4d"]
Jul 22 01:56:59.773369 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:59.773329    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2snrn\" (UniqueName: \"kubernetes.io/projected/e71fb6d5-df07-41bc-96fe-95df778f814d-kube-api-access-2snrn\") pod \"payload-processing-5bf47cd988-tcq4d\" (UID: \"e71fb6d5-df07-41bc-96fe-95df778f814d\") " pod="openshift-ingress/payload-processing-5bf47cd988-tcq4d"
Jul 22 01:56:59.773520 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:59.773465    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e71fb6d5-df07-41bc-96fe-95df778f814d-plugins-config-volume\") pod \"payload-processing-5bf47cd988-tcq4d\" (UID: \"e71fb6d5-df07-41bc-96fe-95df778f814d\") " pod="openshift-ingress/payload-processing-5bf47cd988-tcq4d"
Jul 22 01:56:59.874235 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:59.874209    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e71fb6d5-df07-41bc-96fe-95df778f814d-plugins-config-volume\") pod \"payload-processing-5bf47cd988-tcq4d\" (UID: \"e71fb6d5-df07-41bc-96fe-95df778f814d\") " pod="openshift-ingress/payload-processing-5bf47cd988-tcq4d"
Jul 22 01:56:59.874440 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:59.874250    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2snrn\" (UniqueName: \"kubernetes.io/projected/e71fb6d5-df07-41bc-96fe-95df778f814d-kube-api-access-2snrn\") pod \"payload-processing-5bf47cd988-tcq4d\" (UID: \"e71fb6d5-df07-41bc-96fe-95df778f814d\") " pod="openshift-ingress/payload-processing-5bf47cd988-tcq4d"
Jul 22 01:56:59.874939 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:59.874916    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e71fb6d5-df07-41bc-96fe-95df778f814d-plugins-config-volume\") pod \"payload-processing-5bf47cd988-tcq4d\" (UID: \"e71fb6d5-df07-41bc-96fe-95df778f814d\") " pod="openshift-ingress/payload-processing-5bf47cd988-tcq4d"
Jul 22 01:56:59.882402 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:59.882343    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2snrn\" (UniqueName: \"kubernetes.io/projected/e71fb6d5-df07-41bc-96fe-95df778f814d-kube-api-access-2snrn\") pod \"payload-processing-5bf47cd988-tcq4d\" (UID: \"e71fb6d5-df07-41bc-96fe-95df778f814d\") " pod="openshift-ingress/payload-processing-5bf47cd988-tcq4d"
Jul 22 01:56:59.926372 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:56:59.926350    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-tcq4d"
Jul 22 01:57:00.054405 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:00.054349    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-tcq4d"]
Jul 22 01:57:00.056121 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:57:00.056097    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode71fb6d5_df07_41bc_96fe_95df778f814d.slice/crio-8ec048bdbd0121cf629ed86bfcf9bbde33ef7973395c3291beca9fc54daa4102 WatchSource:0}: Error finding container 8ec048bdbd0121cf629ed86bfcf9bbde33ef7973395c3291beca9fc54daa4102: Status 404 returned error can't find the container with id 8ec048bdbd0121cf629ed86bfcf9bbde33ef7973395c3291beca9fc54daa4102
Jul 22 01:57:00.200838 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:00.200765    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8459478d64-8xtx5" event={"ID":"33bc4a81-9a52-468d-b37d-dfbfd3edb289","Type":"ContainerStarted","Data":"8d0505f9650a85f060d61cef446478f07277f3a8b80f508a09cad0a4aee2f409"}
Jul 22 01:57:00.202567 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:00.202537    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-tcq4d" event={"ID":"e71fb6d5-df07-41bc-96fe-95df778f814d","Type":"ContainerStarted","Data":"8ec048bdbd0121cf629ed86bfcf9bbde33ef7973395c3291beca9fc54daa4102"}
Jul 22 01:57:03.221736 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:03.221702    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8459478d64-8xtx5" event={"ID":"33bc4a81-9a52-468d-b37d-dfbfd3edb289","Type":"ContainerStarted","Data":"48d3ea353095d7e5fa0e80acd64c09f033456c050f22b63eb1cbe7f19ebfdfa1"}
Jul 22 01:57:03.222167 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:03.221804    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-8459478d64-8xtx5"
Jul 22 01:57:03.223397 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:03.223365    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-tcq4d" event={"ID":"e71fb6d5-df07-41bc-96fe-95df778f814d","Type":"ContainerStarted","Data":"aa51e58233b0620a5b1f279dd105b033e904e9560c959d4b7712c3c1b797a83a"}
Jul 22 01:57:03.223502 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:03.223473    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-tcq4d"
Jul 22 01:57:03.237704 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:03.237657    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-8459478d64-8xtx5" podStartSLOduration=0.870732355 podStartE2EDuration="4.23764152s" totalImagesPullingTime="3.366909165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:56:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:57:03.236262156 +0000 UTC m=+576.080091286" watchObservedRunningTime="2026-07-22 01:57:03.23764152 +0000 UTC m=+576.081470656"
Jul 22 01:57:03.252191 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:03.252153    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-tcq4d" podStartSLOduration=1.37580441 podStartE2EDuration="4.252141603s" totalImagesPullingTime="2.876337193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:56:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:57:03.250624771 +0000 UTC m=+576.094453914" watchObservedRunningTime="2026-07-22 01:57:03.252141603 +0000 UTC m=+576.095970732"
Jul 22 01:57:09.234061 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:09.234034    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-8459478d64-8xtx5"
Jul 22 01:57:14.230067 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:14.230035    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-tcq4d"
Jul 22 01:57:19.738509 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:19.738474    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc"]
Jul 22 01:57:19.742932 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:19.742912    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc"
Jul 22 01:57:19.746316 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:19.746294    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 22 01:57:19.746786 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:19.746522    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 22 01:57:19.746786 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:19.746584    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 22 01:57:19.746786 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:19.746536    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2n9gm\""
Jul 22 01:57:19.750326 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:19.750290    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc"]
Jul 22 01:57:19.819294 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:19.819272    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fe97e5a-cb67-49a0-9f85-318daeb2cca3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc\" (UID: \"8fe97e5a-cb67-49a0-9f85-318daeb2cca3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc"
Jul 22 01:57:19.819422 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:19.819366    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9568\" (UniqueName: \"kubernetes.io/projected/8fe97e5a-cb67-49a0-9f85-318daeb2cca3-kube-api-access-f9568\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc\" (UID: \"8fe97e5a-cb67-49a0-9f85-318daeb2cca3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc"
Jul 22 01:57:19.819481 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:19.819447    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fe97e5a-cb67-49a0-9f85-318daeb2cca3-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc\" (UID: \"8fe97e5a-cb67-49a0-9f85-318daeb2cca3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc"
Jul 22 01:57:19.819530 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:19.819484    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fe97e5a-cb67-49a0-9f85-318daeb2cca3-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc\" (UID: \"8fe97e5a-cb67-49a0-9f85-318daeb2cca3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc"
Jul 22 01:57:19.819530 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:19.819508    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fe97e5a-cb67-49a0-9f85-318daeb2cca3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc\" (UID: \"8fe97e5a-cb67-49a0-9f85-318daeb2cca3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc"
Jul 22 01:57:19.819635 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:19.819543    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fe97e5a-cb67-49a0-9f85-318daeb2cca3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc\" (UID: \"8fe97e5a-cb67-49a0-9f85-318daeb2cca3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc"
Jul 22 01:57:19.920793 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:19.920774    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9568\" (UniqueName: \"kubernetes.io/projected/8fe97e5a-cb67-49a0-9f85-318daeb2cca3-kube-api-access-f9568\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc\" (UID: \"8fe97e5a-cb67-49a0-9f85-318daeb2cca3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc"
Jul 22 01:57:19.920908 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:19.920809    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fe97e5a-cb67-49a0-9f85-318daeb2cca3-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc\" (UID: \"8fe97e5a-cb67-49a0-9f85-318daeb2cca3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc"
Jul 22 01:57:19.920908 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:19.920837    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fe97e5a-cb67-49a0-9f85-318daeb2cca3-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc\" (UID: \"8fe97e5a-cb67-49a0-9f85-318daeb2cca3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc"
Jul 22 01:57:19.920908 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:19.920874    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fe97e5a-cb67-49a0-9f85-318daeb2cca3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc\" (UID: \"8fe97e5a-cb67-49a0-9f85-318daeb2cca3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc"
Jul 22 01:57:19.921065 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:19.920909    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fe97e5a-cb67-49a0-9f85-318daeb2cca3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc\" (UID: \"8fe97e5a-cb67-49a0-9f85-318daeb2cca3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc"
Jul 22 01:57:19.921065 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:19.920965    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fe97e5a-cb67-49a0-9f85-318daeb2cca3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc\" (UID: \"8fe97e5a-cb67-49a0-9f85-318daeb2cca3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc"
Jul 22 01:57:19.921178 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:19.921162    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fe97e5a-cb67-49a0-9f85-318daeb2cca3-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc\" (UID: \"8fe97e5a-cb67-49a0-9f85-318daeb2cca3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc"
Jul 22 01:57:19.921291 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:19.921271    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fe97e5a-cb67-49a0-9f85-318daeb2cca3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc\" (UID: \"8fe97e5a-cb67-49a0-9f85-318daeb2cca3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc"
Jul 22 01:57:19.921352 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:19.921313    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fe97e5a-cb67-49a0-9f85-318daeb2cca3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc\" (UID: \"8fe97e5a-cb67-49a0-9f85-318daeb2cca3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc"
Jul 22 01:57:19.923038 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:19.923016    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fe97e5a-cb67-49a0-9f85-318daeb2cca3-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc\" (UID: \"8fe97e5a-cb67-49a0-9f85-318daeb2cca3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc"
Jul 22 01:57:19.923543 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:19.923523    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fe97e5a-cb67-49a0-9f85-318daeb2cca3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc\" (UID: \"8fe97e5a-cb67-49a0-9f85-318daeb2cca3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc"
Jul 22 01:57:19.928021 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:19.927999    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9568\" (UniqueName: \"kubernetes.io/projected/8fe97e5a-cb67-49a0-9f85-318daeb2cca3-kube-api-access-f9568\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc\" (UID: \"8fe97e5a-cb67-49a0-9f85-318daeb2cca3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc"
Jul 22 01:57:20.055337 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:20.055282    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc"
Jul 22 01:57:20.210018 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:20.209992    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc"]
Jul 22 01:57:20.212454 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:57:20.212369    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fe97e5a_cb67_49a0_9f85_318daeb2cca3.slice/crio-21a6df8ed2be13318119be7d8b2ed6b55b1a16741143b364765cde6874a3e551 WatchSource:0}: Error finding container 21a6df8ed2be13318119be7d8b2ed6b55b1a16741143b364765cde6874a3e551: Status 404 returned error can't find the container with id 21a6df8ed2be13318119be7d8b2ed6b55b1a16741143b364765cde6874a3e551
Jul 22 01:57:20.305868 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:20.305787    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc" event={"ID":"8fe97e5a-cb67-49a0-9f85-318daeb2cca3","Type":"ContainerStarted","Data":"21a6df8ed2be13318119be7d8b2ed6b55b1a16741143b364765cde6874a3e551"}
Jul 22 01:57:22.317131 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:22.317096    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc" event={"ID":"8fe97e5a-cb67-49a0-9f85-318daeb2cca3","Type":"ContainerStarted","Data":"ea63fca2d493616365135fa8a89b7d60a06530afc447690e26a2f6e6fc024ffe"}
Jul 22 01:57:22.335718 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:22.335670    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc" podStartSLOduration=2.230993137 podStartE2EDuration="3.335654097s" totalImagesPullingTime="1.10466096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:57:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:57:22.333497733 +0000 UTC m=+595.177326865" watchObservedRunningTime="2026-07-22 01:57:22.335654097 +0000 UTC m=+595.179483229"
Jul 22 01:57:30.056053 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:30.056004    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc"
Jul 22 01:57:30.058582 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:30.056077    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc"
Jul 22 01:57:30.068268 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:30.068245    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc"
Jul 22 01:57:30.371003 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:57:30.370901    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc"
Jul 22 01:58:55.731154 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:58:55.731122    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-f6578bcf6-s5zfn"]
Jul 22 01:58:55.734503 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:58:55.734487    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f6578bcf6-s5zfn"
Jul 22 01:58:55.742090 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:58:55.742069    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f6578bcf6-s5zfn"]
Jul 22 01:58:55.851619 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:58:55.851593    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2948x\" (UniqueName: \"kubernetes.io/projected/f522f711-4278-4185-b9ce-95a84ae51f40-kube-api-access-2948x\") pod \"authorino-f6578bcf6-s5zfn\" (UID: \"f522f711-4278-4185-b9ce-95a84ae51f40\") " pod="kuadrant-system/authorino-f6578bcf6-s5zfn"
Jul 22 01:58:55.851760 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:58:55.851642    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f522f711-4278-4185-b9ce-95a84ae51f40-tls-cert\") pod \"authorino-f6578bcf6-s5zfn\" (UID: \"f522f711-4278-4185-b9ce-95a84ae51f40\") " pod="kuadrant-system/authorino-f6578bcf6-s5zfn"
Jul 22 01:58:55.952153 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:58:55.952126    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f522f711-4278-4185-b9ce-95a84ae51f40-tls-cert\") pod \"authorino-f6578bcf6-s5zfn\" (UID: \"f522f711-4278-4185-b9ce-95a84ae51f40\") " pod="kuadrant-system/authorino-f6578bcf6-s5zfn"
Jul 22 01:58:55.952263 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:58:55.952226    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2948x\" (UniqueName: \"kubernetes.io/projected/f522f711-4278-4185-b9ce-95a84ae51f40-kube-api-access-2948x\") pod \"authorino-f6578bcf6-s5zfn\" (UID: \"f522f711-4278-4185-b9ce-95a84ae51f40\") " pod="kuadrant-system/authorino-f6578bcf6-s5zfn"
Jul 22 01:58:55.954360 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:58:55.954340    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f522f711-4278-4185-b9ce-95a84ae51f40-tls-cert\") pod \"authorino-f6578bcf6-s5zfn\" (UID: \"f522f711-4278-4185-b9ce-95a84ae51f40\") " pod="kuadrant-system/authorino-f6578bcf6-s5zfn"
Jul 22 01:58:55.960662 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:58:55.960641    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2948x\" (UniqueName: \"kubernetes.io/projected/f522f711-4278-4185-b9ce-95a84ae51f40-kube-api-access-2948x\") pod \"authorino-f6578bcf6-s5zfn\" (UID: \"f522f711-4278-4185-b9ce-95a84ae51f40\") " pod="kuadrant-system/authorino-f6578bcf6-s5zfn"
Jul 22 01:58:56.044842 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:58:56.044787    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f6578bcf6-s5zfn"
Jul 22 01:58:56.169362 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:58:56.169339    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f6578bcf6-s5zfn"]
Jul 22 01:58:56.171341 ip-10-0-135-230 kubenswrapper[2563]: W0722 01:58:56.171317    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf522f711_4278_4185_b9ce_95a84ae51f40.slice/crio-b18786060c1710e013382ba8321f03fb5d46b702784f95a1d0e0b9a5f3cc902a WatchSource:0}: Error finding container b18786060c1710e013382ba8321f03fb5d46b702784f95a1d0e0b9a5f3cc902a: Status 404 returned error can't find the container with id b18786060c1710e013382ba8321f03fb5d46b702784f95a1d0e0b9a5f3cc902a
Jul 22 01:58:56.172998 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:58:56.172981    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 22 01:58:56.787992 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:58:56.787967    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f6578bcf6-s5zfn" event={"ID":"f522f711-4278-4185-b9ce-95a84ae51f40","Type":"ContainerStarted","Data":"b18786060c1710e013382ba8321f03fb5d46b702784f95a1d0e0b9a5f3cc902a"}
Jul 22 01:58:57.795171 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:58:57.795132    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f6578bcf6-s5zfn" event={"ID":"f522f711-4278-4185-b9ce-95a84ae51f40","Type":"ContainerStarted","Data":"80790832e74ca8d27ed09bfe47510992574a50060f13138c532126a488f0cb97"}
Jul 22 01:58:57.811767 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:58:57.811711    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-f6578bcf6-s5zfn" podStartSLOduration=2.25072009 podStartE2EDuration="2.811689397s" totalImagesPullingTime="560.969307ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:58:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:58:57.811238677 +0000 UTC m=+690.655067811" watchObservedRunningTime="2026-07-22 01:58:57.811689397 +0000 UTC m=+690.655518531"
Jul 22 01:58:57.839557 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:58:57.839525    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-74d8ccbfbf-55l94"]
Jul 22 01:58:57.839701 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:58:57.839680    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-74d8ccbfbf-55l94" podUID="3b9564a7-4d3c-4f10-a397-3f7b994442e8" containerName="authorino" containerID="cri-o://03727d231de795a1d36c3f3d4e228f94b278e3f09e47975cf2781b3c364d0ece" gracePeriod=30
Jul 22 01:58:58.101247 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:58:58.101223    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74d8ccbfbf-55l94"
Jul 22 01:58:58.271365 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:58:58.271325    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4x9fb\" (UniqueName: \"kubernetes.io/projected/3b9564a7-4d3c-4f10-a397-3f7b994442e8-kube-api-access-4x9fb\") pod \"3b9564a7-4d3c-4f10-a397-3f7b994442e8\" (UID: \"3b9564a7-4d3c-4f10-a397-3f7b994442e8\") "
Jul 22 01:58:58.271549 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:58:58.271426    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3b9564a7-4d3c-4f10-a397-3f7b994442e8-tls-cert\") pod \"3b9564a7-4d3c-4f10-a397-3f7b994442e8\" (UID: \"3b9564a7-4d3c-4f10-a397-3f7b994442e8\") "
Jul 22 01:58:58.273514 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:58:58.273484    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b9564a7-4d3c-4f10-a397-3f7b994442e8-kube-api-access-4x9fb" (OuterVolumeSpecName: "kube-api-access-4x9fb") pod "3b9564a7-4d3c-4f10-a397-3f7b994442e8" (UID: "3b9564a7-4d3c-4f10-a397-3f7b994442e8"). InnerVolumeSpecName "kube-api-access-4x9fb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 01:58:58.280931 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:58:58.280896    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b9564a7-4d3c-4f10-a397-3f7b994442e8-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "3b9564a7-4d3c-4f10-a397-3f7b994442e8" (UID: "3b9564a7-4d3c-4f10-a397-3f7b994442e8"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 01:58:58.372304 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:58:58.372231    2563 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3b9564a7-4d3c-4f10-a397-3f7b994442e8-tls-cert\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:58:58.372304 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:58:58.372263    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x9fb\" (UniqueName: \"kubernetes.io/projected/3b9564a7-4d3c-4f10-a397-3f7b994442e8-kube-api-access-4x9fb\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 01:58:58.804417 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:58:58.804351    2563 generic.go:356] "Generic (PLEG): container finished" podID="3b9564a7-4d3c-4f10-a397-3f7b994442e8" containerID="03727d231de795a1d36c3f3d4e228f94b278e3f09e47975cf2781b3c364d0ece" exitCode=0
Jul 22 01:58:58.805136 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:58:58.805115    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74d8ccbfbf-55l94"
Jul 22 01:58:58.809447 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:58:58.809418    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74d8ccbfbf-55l94" event={"ID":"3b9564a7-4d3c-4f10-a397-3f7b994442e8","Type":"ContainerDied","Data":"03727d231de795a1d36c3f3d4e228f94b278e3f09e47975cf2781b3c364d0ece"}
Jul 22 01:58:58.809560 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:58:58.809470    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74d8ccbfbf-55l94" event={"ID":"3b9564a7-4d3c-4f10-a397-3f7b994442e8","Type":"ContainerDied","Data":"1545c1651c49fac023a5c59b328cbca2b6a9644b37aa2cd00f6e65d582c045ae"}
Jul 22 01:58:58.809560 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:58:58.809495    2563 scope.go:117] "RemoveContainer" containerID="03727d231de795a1d36c3f3d4e228f94b278e3f09e47975cf2781b3c364d0ece"
Jul 22 01:58:58.822273 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:58:58.822251    2563 scope.go:117] "RemoveContainer" containerID="03727d231de795a1d36c3f3d4e228f94b278e3f09e47975cf2781b3c364d0ece"
Jul 22 01:58:58.823998 ip-10-0-135-230 kubenswrapper[2563]: E0722 01:58:58.823971    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03727d231de795a1d36c3f3d4e228f94b278e3f09e47975cf2781b3c364d0ece\": container with ID starting with 03727d231de795a1d36c3f3d4e228f94b278e3f09e47975cf2781b3c364d0ece not found: ID does not exist" containerID="03727d231de795a1d36c3f3d4e228f94b278e3f09e47975cf2781b3c364d0ece"
Jul 22 01:58:58.824106 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:58:58.824007    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03727d231de795a1d36c3f3d4e228f94b278e3f09e47975cf2781b3c364d0ece"} err="failed to get container status \"03727d231de795a1d36c3f3d4e228f94b278e3f09e47975cf2781b3c364d0ece\": rpc error: code = NotFound desc = could not find container \"03727d231de795a1d36c3f3d4e228f94b278e3f09e47975cf2781b3c364d0ece\": container with ID starting with 03727d231de795a1d36c3f3d4e228f94b278e3f09e47975cf2781b3c364d0ece not found: ID does not exist"
Jul 22 01:58:58.830532 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:58:58.830510    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-74d8ccbfbf-55l94"]
Jul 22 01:58:58.836389 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:58:58.836355    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-74d8ccbfbf-55l94"]
Jul 22 01:58:59.662678 ip-10-0-135-230 kubenswrapper[2563]: I0722 01:58:59.662644    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b9564a7-4d3c-4f10-a397-3f7b994442e8" path="/var/lib/kubelet/pods/3b9564a7-4d3c-4f10-a397-3f7b994442e8/volumes"
Jul 22 02:05:46.560743 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:05:46.560711    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9tqq5"]
Jul 22 02:05:46.563296 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:05:46.560906    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9tqq5" podUID="05054cf6-5841-44f2-b541-0c486c70f2bb" containerName="manager" containerID="cri-o://620f4362941bd47cc357404e61d893b4f4fca83baec05f96396b2003eff259b4" gracePeriod=10
Jul 22 02:05:50.006856 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:05:50.006833    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9tqq5"
Jul 22 02:05:50.072998 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:05:50.072927    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/05054cf6-5841-44f2-b541-0c486c70f2bb-extensions-socket-volume\") pod \"05054cf6-5841-44f2-b541-0c486c70f2bb\" (UID: \"05054cf6-5841-44f2-b541-0c486c70f2bb\") "
Jul 22 02:05:50.072998 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:05:50.072982    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rp6l5\" (UniqueName: \"kubernetes.io/projected/05054cf6-5841-44f2-b541-0c486c70f2bb-kube-api-access-rp6l5\") pod \"05054cf6-5841-44f2-b541-0c486c70f2bb\" (UID: \"05054cf6-5841-44f2-b541-0c486c70f2bb\") "
Jul 22 02:05:50.073306 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:05:50.073284    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05054cf6-5841-44f2-b541-0c486c70f2bb-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "05054cf6-5841-44f2-b541-0c486c70f2bb" (UID: "05054cf6-5841-44f2-b541-0c486c70f2bb"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:05:50.074927 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:05:50.074899    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05054cf6-5841-44f2-b541-0c486c70f2bb-kube-api-access-rp6l5" (OuterVolumeSpecName: "kube-api-access-rp6l5") pod "05054cf6-5841-44f2-b541-0c486c70f2bb" (UID: "05054cf6-5841-44f2-b541-0c486c70f2bb"). InnerVolumeSpecName "kube-api-access-rp6l5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:05:50.174214 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:05:50.174193    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp6l5\" (UniqueName: \"kubernetes.io/projected/05054cf6-5841-44f2-b541-0c486c70f2bb-kube-api-access-rp6l5\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:05:50.174214 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:05:50.174213    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/05054cf6-5841-44f2-b541-0c486c70f2bb-extensions-socket-volume\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:05:50.830479 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:05:50.830446    2563 generic.go:356] "Generic (PLEG): container finished" podID="05054cf6-5841-44f2-b541-0c486c70f2bb" containerID="620f4362941bd47cc357404e61d893b4f4fca83baec05f96396b2003eff259b4" exitCode=0
Jul 22 02:05:50.830645 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:05:50.830491    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9tqq5"
Jul 22 02:05:50.830645 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:05:50.830540    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9tqq5" event={"ID":"05054cf6-5841-44f2-b541-0c486c70f2bb","Type":"ContainerDied","Data":"620f4362941bd47cc357404e61d893b4f4fca83baec05f96396b2003eff259b4"}
Jul 22 02:05:50.830645 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:05:50.830603    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9tqq5" event={"ID":"05054cf6-5841-44f2-b541-0c486c70f2bb","Type":"ContainerDied","Data":"045c5c87771302ae91350001f0377573bd176f66378c9efc6d05a2c7267fe909"}
Jul 22 02:05:50.830645 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:05:50.830637    2563 scope.go:117] "RemoveContainer" containerID="620f4362941bd47cc357404e61d893b4f4fca83baec05f96396b2003eff259b4"
Jul 22 02:05:50.841266 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:05:50.841248    2563 scope.go:117] "RemoveContainer" containerID="620f4362941bd47cc357404e61d893b4f4fca83baec05f96396b2003eff259b4"
Jul 22 02:05:50.841587 ip-10-0-135-230 kubenswrapper[2563]: E0722 02:05:50.841569    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"620f4362941bd47cc357404e61d893b4f4fca83baec05f96396b2003eff259b4\": container with ID starting with 620f4362941bd47cc357404e61d893b4f4fca83baec05f96396b2003eff259b4 not found: ID does not exist" containerID="620f4362941bd47cc357404e61d893b4f4fca83baec05f96396b2003eff259b4"
Jul 22 02:05:50.841663 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:05:50.841599    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"620f4362941bd47cc357404e61d893b4f4fca83baec05f96396b2003eff259b4"} err="failed to get container status \"620f4362941bd47cc357404e61d893b4f4fca83baec05f96396b2003eff259b4\": rpc error: code = NotFound desc = could not find container \"620f4362941bd47cc357404e61d893b4f4fca83baec05f96396b2003eff259b4\": container with ID starting with 620f4362941bd47cc357404e61d893b4f4fca83baec05f96396b2003eff259b4 not found: ID does not exist"
Jul 22 02:05:50.852683 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:05:50.852662    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9tqq5"]
Jul 22 02:05:50.856801 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:05:50.856779    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9tqq5"]
Jul 22 02:05:51.663818 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:05:51.663788    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05054cf6-5841-44f2-b541-0c486c70f2bb" path="/var/lib/kubelet/pods/05054cf6-5841-44f2-b541-0c486c70f2bb/volumes"
Jul 22 02:06:47.835548 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:06:47.835513    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dcm6p"]
Jul 22 02:06:47.835934 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:06:47.835845    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b9564a7-4d3c-4f10-a397-3f7b994442e8" containerName="authorino"
Jul 22 02:06:47.835934 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:06:47.835857    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b9564a7-4d3c-4f10-a397-3f7b994442e8" containerName="authorino"
Jul 22 02:06:47.835934 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:06:47.835865    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05054cf6-5841-44f2-b541-0c486c70f2bb" containerName="manager"
Jul 22 02:06:47.835934 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:06:47.835870    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="05054cf6-5841-44f2-b541-0c486c70f2bb" containerName="manager"
Jul 22 02:06:47.835934 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:06:47.835924    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="05054cf6-5841-44f2-b541-0c486c70f2bb" containerName="manager"
Jul 22 02:06:47.835934 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:06:47.835932    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b9564a7-4d3c-4f10-a397-3f7b994442e8" containerName="authorino"
Jul 22 02:06:47.839930 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:06:47.839913    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dcm6p"
Jul 22 02:06:47.842120 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:06:47.842100    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-lqnv4\""
Jul 22 02:06:47.851612 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:06:47.851591    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dcm6p"]
Jul 22 02:06:47.901136 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:06:47.901114    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k77t\" (UniqueName: \"kubernetes.io/projected/e61078f7-e62c-4c53-9186-80e33d10ab4f-kube-api-access-5k77t\") pod \"kuadrant-operator-controller-manager-86878c769f-dcm6p\" (UID: \"e61078f7-e62c-4c53-9186-80e33d10ab4f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dcm6p"
Jul 22 02:06:47.901265 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:06:47.901152    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e61078f7-e62c-4c53-9186-80e33d10ab4f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-dcm6p\" (UID: \"e61078f7-e62c-4c53-9186-80e33d10ab4f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dcm6p"
Jul 22 02:06:48.002120 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:06:48.002092    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e61078f7-e62c-4c53-9186-80e33d10ab4f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-dcm6p\" (UID: \"e61078f7-e62c-4c53-9186-80e33d10ab4f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dcm6p"
Jul 22 02:06:48.002256 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:06:48.002196    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k77t\" (UniqueName: \"kubernetes.io/projected/e61078f7-e62c-4c53-9186-80e33d10ab4f-kube-api-access-5k77t\") pod \"kuadrant-operator-controller-manager-86878c769f-dcm6p\" (UID: \"e61078f7-e62c-4c53-9186-80e33d10ab4f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dcm6p"
Jul 22 02:06:48.002482 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:06:48.002460    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e61078f7-e62c-4c53-9186-80e33d10ab4f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-dcm6p\" (UID: \"e61078f7-e62c-4c53-9186-80e33d10ab4f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dcm6p"
Jul 22 02:06:48.010750 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:06:48.010733    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k77t\" (UniqueName: \"kubernetes.io/projected/e61078f7-e62c-4c53-9186-80e33d10ab4f-kube-api-access-5k77t\") pod \"kuadrant-operator-controller-manager-86878c769f-dcm6p\" (UID: \"e61078f7-e62c-4c53-9186-80e33d10ab4f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dcm6p"
Jul 22 02:06:48.149873 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:06:48.149807    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dcm6p"
Jul 22 02:06:48.294184 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:06:48.294161    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dcm6p"]
Jul 22 02:06:48.295948 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:06:48.295911    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode61078f7_e62c_4c53_9186_80e33d10ab4f.slice/crio-f31a8aa20e3faac4d04b47ec2fcc5d0b88cb20da6ba1df7f018359b6413780bd WatchSource:0}: Error finding container f31a8aa20e3faac4d04b47ec2fcc5d0b88cb20da6ba1df7f018359b6413780bd: Status 404 returned error can't find the container with id f31a8aa20e3faac4d04b47ec2fcc5d0b88cb20da6ba1df7f018359b6413780bd
Jul 22 02:06:49.111366 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:06:49.111332    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dcm6p" event={"ID":"e61078f7-e62c-4c53-9186-80e33d10ab4f","Type":"ContainerStarted","Data":"8018e69894272280ac2d1e9bdba77ecb54dd406ddc90c2db63041ae5a6a3e722"}
Jul 22 02:06:49.111755 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:06:49.111409    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dcm6p" event={"ID":"e61078f7-e62c-4c53-9186-80e33d10ab4f","Type":"ContainerStarted","Data":"f31a8aa20e3faac4d04b47ec2fcc5d0b88cb20da6ba1df7f018359b6413780bd"}
Jul 22 02:06:49.111755 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:06:49.111525    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dcm6p"
Jul 22 02:06:49.135792 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:06:49.135721    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dcm6p" podStartSLOduration=2.135706044 podStartE2EDuration="2.135706044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:06:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:06:49.134700261 +0000 UTC m=+1161.978529392" watchObservedRunningTime="2026-07-22 02:06:49.135706044 +0000 UTC m=+1161.979535175"
Jul 22 02:07:00.118626 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:07:00.118596    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dcm6p"
Jul 22 02:09:43.571283 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:09:43.571254    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q"]
Jul 22 02:09:43.574766 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:09:43.574750    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q"
Jul 22 02:09:43.577536 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:09:43.577515    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-a65c4feb-kserve-self-signed-certs\""
Jul 22 02:09:43.594009 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:09:43.593988    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q"]
Jul 22 02:09:43.634335 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:09:43.634310    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c7be3c0-c389-4e70-ac50-4d2de850d3c7-tls-certs\") pod \"e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q\" (UID: \"8c7be3c0-c389-4e70-ac50-4d2de850d3c7\") " pod="llm/e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q"
Jul 22 02:09:43.634449 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:09:43.634334    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqf5q\" (UniqueName: \"kubernetes.io/projected/8c7be3c0-c389-4e70-ac50-4d2de850d3c7-kube-api-access-jqf5q\") pod \"e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q\" (UID: \"8c7be3c0-c389-4e70-ac50-4d2de850d3c7\") " pod="llm/e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q"
Jul 22 02:09:43.634449 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:09:43.634360    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c7be3c0-c389-4e70-ac50-4d2de850d3c7-model-cache\") pod \"e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q\" (UID: \"8c7be3c0-c389-4e70-ac50-4d2de850d3c7\") " pod="llm/e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q"
Jul 22 02:09:43.634449 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:09:43.634433    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c7be3c0-c389-4e70-ac50-4d2de850d3c7-tmp-dir\") pod \"e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q\" (UID: \"8c7be3c0-c389-4e70-ac50-4d2de850d3c7\") " pod="llm/e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q"
Jul 22 02:09:43.634558 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:09:43.634474    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c7be3c0-c389-4e70-ac50-4d2de850d3c7-dshm\") pod \"e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q\" (UID: \"8c7be3c0-c389-4e70-ac50-4d2de850d3c7\") " pod="llm/e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q"
Jul 22 02:09:43.634558 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:09:43.634544    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c7be3c0-c389-4e70-ac50-4d2de850d3c7-home\") pod \"e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q\" (UID: \"8c7be3c0-c389-4e70-ac50-4d2de850d3c7\") " pod="llm/e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q"
Jul 22 02:09:43.735791 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:09:43.735769    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c7be3c0-c389-4e70-ac50-4d2de850d3c7-tls-certs\") pod \"e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q\" (UID: \"8c7be3c0-c389-4e70-ac50-4d2de850d3c7\") " pod="llm/e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q"
Jul 22 02:09:43.735901 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:09:43.735797    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqf5q\" (UniqueName: \"kubernetes.io/projected/8c7be3c0-c389-4e70-ac50-4d2de850d3c7-kube-api-access-jqf5q\") pod \"e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q\" (UID: \"8c7be3c0-c389-4e70-ac50-4d2de850d3c7\") " pod="llm/e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q"
Jul 22 02:09:43.735901 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:09:43.735820    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c7be3c0-c389-4e70-ac50-4d2de850d3c7-model-cache\") pod \"e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q\" (UID: \"8c7be3c0-c389-4e70-ac50-4d2de850d3c7\") " pod="llm/e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q"
Jul 22 02:09:43.735988 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:09:43.735966    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c7be3c0-c389-4e70-ac50-4d2de850d3c7-tmp-dir\") pod \"e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q\" (UID: \"8c7be3c0-c389-4e70-ac50-4d2de850d3c7\") " pod="llm/e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q"
Jul 22 02:09:43.736033 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:09:43.736015    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c7be3c0-c389-4e70-ac50-4d2de850d3c7-dshm\") pod \"e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q\" (UID: \"8c7be3c0-c389-4e70-ac50-4d2de850d3c7\") " pod="llm/e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q"
Jul 22 02:09:43.736162 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:09:43.736108    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c7be3c0-c389-4e70-ac50-4d2de850d3c7-home\") pod \"e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q\" (UID: \"8c7be3c0-c389-4e70-ac50-4d2de850d3c7\") " pod="llm/e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q"
Jul 22 02:09:43.736280 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:09:43.736173    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c7be3c0-c389-4e70-ac50-4d2de850d3c7-model-cache\") pod \"e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q\" (UID: \"8c7be3c0-c389-4e70-ac50-4d2de850d3c7\") " pod="llm/e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q"
Jul 22 02:09:43.736367 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:09:43.736346    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c7be3c0-c389-4e70-ac50-4d2de850d3c7-tmp-dir\") pod \"e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q\" (UID: \"8c7be3c0-c389-4e70-ac50-4d2de850d3c7\") " pod="llm/e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q"
Jul 22 02:09:43.736467 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:09:43.736450    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c7be3c0-c389-4e70-ac50-4d2de850d3c7-home\") pod \"e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q\" (UID: \"8c7be3c0-c389-4e70-ac50-4d2de850d3c7\") " pod="llm/e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q"
Jul 22 02:09:43.738197 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:09:43.738177    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c7be3c0-c389-4e70-ac50-4d2de850d3c7-dshm\") pod \"e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q\" (UID: \"8c7be3c0-c389-4e70-ac50-4d2de850d3c7\") " pod="llm/e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q"
Jul 22 02:09:43.738455 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:09:43.738441    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c7be3c0-c389-4e70-ac50-4d2de850d3c7-tls-certs\") pod \"e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q\" (UID: \"8c7be3c0-c389-4e70-ac50-4d2de850d3c7\") " pod="llm/e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q"
Jul 22 02:09:43.744197 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:09:43.744175    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqf5q\" (UniqueName: \"kubernetes.io/projected/8c7be3c0-c389-4e70-ac50-4d2de850d3c7-kube-api-access-jqf5q\") pod \"e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q\" (UID: \"8c7be3c0-c389-4e70-ac50-4d2de850d3c7\") " pod="llm/e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q"
Jul 22 02:09:43.883455 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:09:43.883370    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q"
Jul 22 02:09:44.020834 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:09:44.020807    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q"]
Jul 22 02:09:44.022675 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:09:44.022649    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c7be3c0_c389_4e70_ac50_4d2de850d3c7.slice/crio-6c8b1751bed66fa0eb3ef5858c045aec5a9c03d8f982c320692eef59837447e2 WatchSource:0}: Error finding container 6c8b1751bed66fa0eb3ef5858c045aec5a9c03d8f982c320692eef59837447e2: Status 404 returned error can't find the container with id 6c8b1751bed66fa0eb3ef5858c045aec5a9c03d8f982c320692eef59837447e2
Jul 22 02:09:44.969944 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:09:44.969914    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q" event={"ID":"8c7be3c0-c389-4e70-ac50-4d2de850d3c7","Type":"ContainerStarted","Data":"f32f90bbb01a324de49a0de11f9ed88fcfce2128e83f19fb47830ebe5343210b"}
Jul 22 02:09:44.970298 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:09:44.969955    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q" event={"ID":"8c7be3c0-c389-4e70-ac50-4d2de850d3c7","Type":"ContainerStarted","Data":"6c8b1751bed66fa0eb3ef5858c045aec5a9c03d8f982c320692eef59837447e2"}
Jul 22 02:09:44.988707 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:09:44.988660    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q" podStartSLOduration=1.988646074 podStartE2EDuration="1.988646074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:09:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:09:44.987825766 +0000 UTC m=+1337.831654898" watchObservedRunningTime="2026-07-22 02:09:44.988646074 +0000 UTC m=+1337.832475205"
Jul 22 02:09:53.884033 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:09:53.883947    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q"
Jul 22 02:09:53.884033 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:09:53.883998    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q"
Jul 22 02:09:53.896489 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:09:53.896465    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q"
Jul 22 02:09:54.025606 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:09:54.025579    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q"
Jul 22 02:10:01.265571 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:01.265539    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc"]
Jul 22 02:10:01.269228 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:01.269207    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc"
Jul 22 02:10:01.272812 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:01.272795    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-a65c4feb-kserve-self-signed-certs\""
Jul 22 02:10:01.280187 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:01.280161    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc"]
Jul 22 02:10:01.367951 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:01.367923    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/437ccf43-dae1-4f4b-9615-dabb5b6bab73-dshm\") pod \"e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc\" (UID: \"437ccf43-dae1-4f4b-9615-dabb5b6bab73\") " pod="llm/e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc"
Jul 22 02:10:01.368065 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:01.367958    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/437ccf43-dae1-4f4b-9615-dabb5b6bab73-tmp-dir\") pod \"e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc\" (UID: \"437ccf43-dae1-4f4b-9615-dabb5b6bab73\") " pod="llm/e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc"
Jul 22 02:10:01.368065 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:01.367983    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/437ccf43-dae1-4f4b-9615-dabb5b6bab73-home\") pod \"e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc\" (UID: \"437ccf43-dae1-4f4b-9615-dabb5b6bab73\") " pod="llm/e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc"
Jul 22 02:10:01.368065 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:01.368045    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/437ccf43-dae1-4f4b-9615-dabb5b6bab73-model-cache\") pod \"e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc\" (UID: \"437ccf43-dae1-4f4b-9615-dabb5b6bab73\") " pod="llm/e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc"
Jul 22 02:10:01.368230 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:01.368147    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jpxx\" (UniqueName: \"kubernetes.io/projected/437ccf43-dae1-4f4b-9615-dabb5b6bab73-kube-api-access-2jpxx\") pod \"e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc\" (UID: \"437ccf43-dae1-4f4b-9615-dabb5b6bab73\") " pod="llm/e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc"
Jul 22 02:10:01.368230 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:01.368194    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/437ccf43-dae1-4f4b-9615-dabb5b6bab73-tls-certs\") pod \"e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc\" (UID: \"437ccf43-dae1-4f4b-9615-dabb5b6bab73\") " pod="llm/e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc"
Jul 22 02:10:01.469008 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:01.468978    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/437ccf43-dae1-4f4b-9615-dabb5b6bab73-dshm\") pod \"e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc\" (UID: \"437ccf43-dae1-4f4b-9615-dabb5b6bab73\") " pod="llm/e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc"
Jul 22 02:10:01.469111 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:01.469015    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/437ccf43-dae1-4f4b-9615-dabb5b6bab73-tmp-dir\") pod \"e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc\" (UID: \"437ccf43-dae1-4f4b-9615-dabb5b6bab73\") " pod="llm/e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc"
Jul 22 02:10:01.469111 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:01.469048    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/437ccf43-dae1-4f4b-9615-dabb5b6bab73-home\") pod \"e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc\" (UID: \"437ccf43-dae1-4f4b-9615-dabb5b6bab73\") " pod="llm/e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc"
Jul 22 02:10:01.469111 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:01.469078    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/437ccf43-dae1-4f4b-9615-dabb5b6bab73-model-cache\") pod \"e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc\" (UID: \"437ccf43-dae1-4f4b-9615-dabb5b6bab73\") " pod="llm/e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc"
Jul 22 02:10:01.469210 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:01.469123    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jpxx\" (UniqueName: \"kubernetes.io/projected/437ccf43-dae1-4f4b-9615-dabb5b6bab73-kube-api-access-2jpxx\") pod \"e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc\" (UID: \"437ccf43-dae1-4f4b-9615-dabb5b6bab73\") " pod="llm/e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc"
Jul 22 02:10:01.469210 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:01.469157    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/437ccf43-dae1-4f4b-9615-dabb5b6bab73-tls-certs\") pod \"e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc\" (UID: \"437ccf43-dae1-4f4b-9615-dabb5b6bab73\") " pod="llm/e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc"
Jul 22 02:10:01.469500 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:01.469476    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/437ccf43-dae1-4f4b-9615-dabb5b6bab73-tmp-dir\") pod \"e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc\" (UID: \"437ccf43-dae1-4f4b-9615-dabb5b6bab73\") " pod="llm/e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc"
Jul 22 02:10:01.469597 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:01.469499    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/437ccf43-dae1-4f4b-9615-dabb5b6bab73-home\") pod \"e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc\" (UID: \"437ccf43-dae1-4f4b-9615-dabb5b6bab73\") " pod="llm/e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc"
Jul 22 02:10:01.469597 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:01.469522    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/437ccf43-dae1-4f4b-9615-dabb5b6bab73-model-cache\") pod \"e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc\" (UID: \"437ccf43-dae1-4f4b-9615-dabb5b6bab73\") " pod="llm/e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc"
Jul 22 02:10:01.471341 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:01.471325    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/437ccf43-dae1-4f4b-9615-dabb5b6bab73-dshm\") pod \"e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc\" (UID: \"437ccf43-dae1-4f4b-9615-dabb5b6bab73\") " pod="llm/e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc"
Jul 22 02:10:01.471756 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:01.471737    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/437ccf43-dae1-4f4b-9615-dabb5b6bab73-tls-certs\") pod \"e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc\" (UID: \"437ccf43-dae1-4f4b-9615-dabb5b6bab73\") " pod="llm/e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc"
Jul 22 02:10:01.478786 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:01.478758    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jpxx\" (UniqueName: \"kubernetes.io/projected/437ccf43-dae1-4f4b-9615-dabb5b6bab73-kube-api-access-2jpxx\") pod \"e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc\" (UID: \"437ccf43-dae1-4f4b-9615-dabb5b6bab73\") " pod="llm/e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc"
Jul 22 02:10:01.580123 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:01.580066    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc"
Jul 22 02:10:01.712837 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:01.712814    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc"]
Jul 22 02:10:01.714850 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:10:01.714826    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod437ccf43_dae1_4f4b_9615_dabb5b6bab73.slice/crio-3ea1ee35edf46aefe60d360e9baa8974651227961aef0157d61b0ba8295a7d17 WatchSource:0}: Error finding container 3ea1ee35edf46aefe60d360e9baa8974651227961aef0157d61b0ba8295a7d17: Status 404 returned error can't find the container with id 3ea1ee35edf46aefe60d360e9baa8974651227961aef0157d61b0ba8295a7d17
Jul 22 02:10:02.053394 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:02.053344    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc" event={"ID":"437ccf43-dae1-4f4b-9615-dabb5b6bab73","Type":"ContainerStarted","Data":"577a80386794376d5368c49af616ba083179d415023a6facf55459dee91f347f"}
Jul 22 02:10:02.053568 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:02.053410    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc" event={"ID":"437ccf43-dae1-4f4b-9615-dabb5b6bab73","Type":"ContainerStarted","Data":"3ea1ee35edf46aefe60d360e9baa8974651227961aef0157d61b0ba8295a7d17"}
Jul 22 02:10:02.079401 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:02.075990    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc" podStartSLOduration=1.075969822 podStartE2EDuration="1.075969822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:10:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:10:02.072652968 +0000 UTC m=+1354.916482100" watchObservedRunningTime="2026-07-22 02:10:02.075969822 +0000 UTC m=+1354.919798958"
Jul 22 02:10:11.580603 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:11.580570    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc"
Jul 22 02:10:11.581052 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:11.580650    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc"
Jul 22 02:10:11.592777 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:11.592752    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc"
Jul 22 02:10:12.116804 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:12.116777    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc"
Jul 22 02:10:18.175266 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:18.175232    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q"]
Jul 22 02:10:19.634601 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:19.634570    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc"]
Jul 22 02:10:33.196942 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:33.196866    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q" podUID="8c7be3c0-c389-4e70-ac50-4d2de850d3c7" containerName="main" containerID="cri-o://f32f90bbb01a324de49a0de11f9ed88fcfce2128e83f19fb47830ebe5343210b" gracePeriod=45
Jul 22 02:10:33.448101 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:33.448047    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q"
Jul 22 02:10:33.517219 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:33.517193    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c7be3c0-c389-4e70-ac50-4d2de850d3c7-model-cache\") pod \"8c7be3c0-c389-4e70-ac50-4d2de850d3c7\" (UID: \"8c7be3c0-c389-4e70-ac50-4d2de850d3c7\") "
Jul 22 02:10:33.517334 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:33.517236    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jqf5q\" (UniqueName: \"kubernetes.io/projected/8c7be3c0-c389-4e70-ac50-4d2de850d3c7-kube-api-access-jqf5q\") pod \"8c7be3c0-c389-4e70-ac50-4d2de850d3c7\" (UID: \"8c7be3c0-c389-4e70-ac50-4d2de850d3c7\") "
Jul 22 02:10:33.517334 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:33.517292    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c7be3c0-c389-4e70-ac50-4d2de850d3c7-tmp-dir\") pod \"8c7be3c0-c389-4e70-ac50-4d2de850d3c7\" (UID: \"8c7be3c0-c389-4e70-ac50-4d2de850d3c7\") "
Jul 22 02:10:33.517430 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:33.517341    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c7be3c0-c389-4e70-ac50-4d2de850d3c7-dshm\") pod \"8c7be3c0-c389-4e70-ac50-4d2de850d3c7\" (UID: \"8c7be3c0-c389-4e70-ac50-4d2de850d3c7\") "
Jul 22 02:10:33.517473 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:33.517437    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c7be3c0-c389-4e70-ac50-4d2de850d3c7-tls-certs\") pod \"8c7be3c0-c389-4e70-ac50-4d2de850d3c7\" (UID: \"8c7be3c0-c389-4e70-ac50-4d2de850d3c7\") "
Jul 22 02:10:33.517525 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:33.517473    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c7be3c0-c389-4e70-ac50-4d2de850d3c7-home\") pod \"8c7be3c0-c389-4e70-ac50-4d2de850d3c7\" (UID: \"8c7be3c0-c389-4e70-ac50-4d2de850d3c7\") "
Jul 22 02:10:33.517578 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:33.517513    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c7be3c0-c389-4e70-ac50-4d2de850d3c7-model-cache" (OuterVolumeSpecName: "model-cache") pod "8c7be3c0-c389-4e70-ac50-4d2de850d3c7" (UID: "8c7be3c0-c389-4e70-ac50-4d2de850d3c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:10:33.517578 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:33.517548    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c7be3c0-c389-4e70-ac50-4d2de850d3c7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8c7be3c0-c389-4e70-ac50-4d2de850d3c7" (UID: "8c7be3c0-c389-4e70-ac50-4d2de850d3c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:10:33.517784 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:33.517760    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c7be3c0-c389-4e70-ac50-4d2de850d3c7-model-cache\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:10:33.517784 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:33.517788    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c7be3c0-c389-4e70-ac50-4d2de850d3c7-tmp-dir\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:10:33.517941 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:33.517791    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c7be3c0-c389-4e70-ac50-4d2de850d3c7-home" (OuterVolumeSpecName: "home") pod "8c7be3c0-c389-4e70-ac50-4d2de850d3c7" (UID: "8c7be3c0-c389-4e70-ac50-4d2de850d3c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:10:33.519491 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:33.519465    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c7be3c0-c389-4e70-ac50-4d2de850d3c7-dshm" (OuterVolumeSpecName: "dshm") pod "8c7be3c0-c389-4e70-ac50-4d2de850d3c7" (UID: "8c7be3c0-c389-4e70-ac50-4d2de850d3c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:10:33.519574 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:33.519531    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c7be3c0-c389-4e70-ac50-4d2de850d3c7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8c7be3c0-c389-4e70-ac50-4d2de850d3c7" (UID: "8c7be3c0-c389-4e70-ac50-4d2de850d3c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:10:33.519631 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:33.519613    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c7be3c0-c389-4e70-ac50-4d2de850d3c7-kube-api-access-jqf5q" (OuterVolumeSpecName: "kube-api-access-jqf5q") pod "8c7be3c0-c389-4e70-ac50-4d2de850d3c7" (UID: "8c7be3c0-c389-4e70-ac50-4d2de850d3c7"). InnerVolumeSpecName "kube-api-access-jqf5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:10:33.619079 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:33.619055    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqf5q\" (UniqueName: \"kubernetes.io/projected/8c7be3c0-c389-4e70-ac50-4d2de850d3c7-kube-api-access-jqf5q\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:10:33.619079 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:33.619073    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c7be3c0-c389-4e70-ac50-4d2de850d3c7-dshm\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:10:33.619211 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:33.619085    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c7be3c0-c389-4e70-ac50-4d2de850d3c7-tls-certs\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:10:33.619211 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:33.619094    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c7be3c0-c389-4e70-ac50-4d2de850d3c7-home\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:10:34.219424 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:34.219363    2563 generic.go:356] "Generic (PLEG): container finished" podID="8c7be3c0-c389-4e70-ac50-4d2de850d3c7" containerID="f32f90bbb01a324de49a0de11f9ed88fcfce2128e83f19fb47830ebe5343210b" exitCode=0
Jul 22 02:10:34.219819 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:34.219429    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q"
Jul 22 02:10:34.219819 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:34.219450    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q" event={"ID":"8c7be3c0-c389-4e70-ac50-4d2de850d3c7","Type":"ContainerDied","Data":"f32f90bbb01a324de49a0de11f9ed88fcfce2128e83f19fb47830ebe5343210b"}
Jul 22 02:10:34.219819 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:34.219496    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q" event={"ID":"8c7be3c0-c389-4e70-ac50-4d2de850d3c7","Type":"ContainerDied","Data":"6c8b1751bed66fa0eb3ef5858c045aec5a9c03d8f982c320692eef59837447e2"}
Jul 22 02:10:34.219819 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:34.219515    2563 scope.go:117] "RemoveContainer" containerID="f32f90bbb01a324de49a0de11f9ed88fcfce2128e83f19fb47830ebe5343210b"
Jul 22 02:10:34.230812 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:34.230789    2563 scope.go:117] "RemoveContainer" containerID="f32f90bbb01a324de49a0de11f9ed88fcfce2128e83f19fb47830ebe5343210b"
Jul 22 02:10:34.231341 ip-10-0-135-230 kubenswrapper[2563]: E0722 02:10:34.231315    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f32f90bbb01a324de49a0de11f9ed88fcfce2128e83f19fb47830ebe5343210b\": container with ID starting with f32f90bbb01a324de49a0de11f9ed88fcfce2128e83f19fb47830ebe5343210b not found: ID does not exist" containerID="f32f90bbb01a324de49a0de11f9ed88fcfce2128e83f19fb47830ebe5343210b"
Jul 22 02:10:34.231457 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:34.231352    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f32f90bbb01a324de49a0de11f9ed88fcfce2128e83f19fb47830ebe5343210b"} err="failed to get container status \"f32f90bbb01a324de49a0de11f9ed88fcfce2128e83f19fb47830ebe5343210b\": rpc error: code = NotFound desc = could not find container \"f32f90bbb01a324de49a0de11f9ed88fcfce2128e83f19fb47830ebe5343210b\": container with ID starting with f32f90bbb01a324de49a0de11f9ed88fcfce2128e83f19fb47830ebe5343210b not found: ID does not exist"
Jul 22 02:10:34.236104 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:34.236082    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q"]
Jul 22 02:10:34.239575 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:34.239555    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-a65c4feb-kserve-78c7fbc9dc-qvj9q"]
Jul 22 02:10:34.654865 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:34.654773    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc" podUID="437ccf43-dae1-4f4b-9615-dabb5b6bab73" containerName="main" containerID="cri-o://577a80386794376d5368c49af616ba083179d415023a6facf55459dee91f347f" gracePeriod=45
Jul 22 02:10:34.900428 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:34.900403    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc"
Jul 22 02:10:34.931009 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:34.930990    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/437ccf43-dae1-4f4b-9615-dabb5b6bab73-dshm\") pod \"437ccf43-dae1-4f4b-9615-dabb5b6bab73\" (UID: \"437ccf43-dae1-4f4b-9615-dabb5b6bab73\") "
Jul 22 02:10:34.931124 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:34.931039    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/437ccf43-dae1-4f4b-9615-dabb5b6bab73-model-cache\") pod \"437ccf43-dae1-4f4b-9615-dabb5b6bab73\" (UID: \"437ccf43-dae1-4f4b-9615-dabb5b6bab73\") "
Jul 22 02:10:34.931124 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:34.931071    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jpxx\" (UniqueName: \"kubernetes.io/projected/437ccf43-dae1-4f4b-9615-dabb5b6bab73-kube-api-access-2jpxx\") pod \"437ccf43-dae1-4f4b-9615-dabb5b6bab73\" (UID: \"437ccf43-dae1-4f4b-9615-dabb5b6bab73\") "
Jul 22 02:10:34.931124 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:34.931107    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/437ccf43-dae1-4f4b-9615-dabb5b6bab73-tmp-dir\") pod \"437ccf43-dae1-4f4b-9615-dabb5b6bab73\" (UID: \"437ccf43-dae1-4f4b-9615-dabb5b6bab73\") "
Jul 22 02:10:34.931302 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:34.931146    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/437ccf43-dae1-4f4b-9615-dabb5b6bab73-home\") pod \"437ccf43-dae1-4f4b-9615-dabb5b6bab73\" (UID: \"437ccf43-dae1-4f4b-9615-dabb5b6bab73\") "
Jul 22 02:10:34.931302 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:34.931210    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/437ccf43-dae1-4f4b-9615-dabb5b6bab73-tls-certs\") pod \"437ccf43-dae1-4f4b-9615-dabb5b6bab73\" (UID: \"437ccf43-dae1-4f4b-9615-dabb5b6bab73\") "
Jul 22 02:10:34.931434 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:34.931358    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/437ccf43-dae1-4f4b-9615-dabb5b6bab73-model-cache" (OuterVolumeSpecName: "model-cache") pod "437ccf43-dae1-4f4b-9615-dabb5b6bab73" (UID: "437ccf43-dae1-4f4b-9615-dabb5b6bab73"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:10:34.931505 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:34.931492    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/437ccf43-dae1-4f4b-9615-dabb5b6bab73-model-cache\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:10:34.931562 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:34.931506    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/437ccf43-dae1-4f4b-9615-dabb5b6bab73-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "437ccf43-dae1-4f4b-9615-dabb5b6bab73" (UID: "437ccf43-dae1-4f4b-9615-dabb5b6bab73"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:10:34.931562 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:34.931541    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/437ccf43-dae1-4f4b-9615-dabb5b6bab73-home" (OuterVolumeSpecName: "home") pod "437ccf43-dae1-4f4b-9615-dabb5b6bab73" (UID: "437ccf43-dae1-4f4b-9615-dabb5b6bab73"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:10:34.933174 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:34.933144    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/437ccf43-dae1-4f4b-9615-dabb5b6bab73-dshm" (OuterVolumeSpecName: "dshm") pod "437ccf43-dae1-4f4b-9615-dabb5b6bab73" (UID: "437ccf43-dae1-4f4b-9615-dabb5b6bab73"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:10:34.933273 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:34.933207    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/437ccf43-dae1-4f4b-9615-dabb5b6bab73-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "437ccf43-dae1-4f4b-9615-dabb5b6bab73" (UID: "437ccf43-dae1-4f4b-9615-dabb5b6bab73"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:10:34.933273 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:34.933248    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/437ccf43-dae1-4f4b-9615-dabb5b6bab73-kube-api-access-2jpxx" (OuterVolumeSpecName: "kube-api-access-2jpxx") pod "437ccf43-dae1-4f4b-9615-dabb5b6bab73" (UID: "437ccf43-dae1-4f4b-9615-dabb5b6bab73"). InnerVolumeSpecName "kube-api-access-2jpxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:10:35.031858 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:35.031833    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/437ccf43-dae1-4f4b-9615-dabb5b6bab73-tls-certs\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:10:35.031858 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:35.031855    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/437ccf43-dae1-4f4b-9615-dabb5b6bab73-dshm\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:10:35.031985 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:35.031865    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jpxx\" (UniqueName: \"kubernetes.io/projected/437ccf43-dae1-4f4b-9615-dabb5b6bab73-kube-api-access-2jpxx\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:10:35.031985 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:35.031874    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/437ccf43-dae1-4f4b-9615-dabb5b6bab73-tmp-dir\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:10:35.031985 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:35.031885    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/437ccf43-dae1-4f4b-9615-dabb5b6bab73-home\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:10:35.226860 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:35.226796    2563 generic.go:356] "Generic (PLEG): container finished" podID="437ccf43-dae1-4f4b-9615-dabb5b6bab73" containerID="577a80386794376d5368c49af616ba083179d415023a6facf55459dee91f347f" exitCode=0
Jul 22 02:10:35.226860 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:35.226832    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc" event={"ID":"437ccf43-dae1-4f4b-9615-dabb5b6bab73","Type":"ContainerDied","Data":"577a80386794376d5368c49af616ba083179d415023a6facf55459dee91f347f"}
Jul 22 02:10:35.226860 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:35.226842    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc"
Jul 22 02:10:35.227288 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:35.226870    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc" event={"ID":"437ccf43-dae1-4f4b-9615-dabb5b6bab73","Type":"ContainerDied","Data":"3ea1ee35edf46aefe60d360e9baa8974651227961aef0157d61b0ba8295a7d17"}
Jul 22 02:10:35.227288 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:35.226889    2563 scope.go:117] "RemoveContainer" containerID="577a80386794376d5368c49af616ba083179d415023a6facf55459dee91f347f"
Jul 22 02:10:35.236115 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:35.236099    2563 scope.go:117] "RemoveContainer" containerID="577a80386794376d5368c49af616ba083179d415023a6facf55459dee91f347f"
Jul 22 02:10:35.236347 ip-10-0-135-230 kubenswrapper[2563]: E0722 02:10:35.236331    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"577a80386794376d5368c49af616ba083179d415023a6facf55459dee91f347f\": container with ID starting with 577a80386794376d5368c49af616ba083179d415023a6facf55459dee91f347f not found: ID does not exist" containerID="577a80386794376d5368c49af616ba083179d415023a6facf55459dee91f347f"
Jul 22 02:10:35.236421 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:35.236356    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"577a80386794376d5368c49af616ba083179d415023a6facf55459dee91f347f"} err="failed to get container status \"577a80386794376d5368c49af616ba083179d415023a6facf55459dee91f347f\": rpc error: code = NotFound desc = could not find container \"577a80386794376d5368c49af616ba083179d415023a6facf55459dee91f347f\": container with ID starting with 577a80386794376d5368c49af616ba083179d415023a6facf55459dee91f347f not found: ID does not exist"
Jul 22 02:10:35.248288 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:35.248268    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc"]
Jul 22 02:10:35.251762 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:35.251742    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-a65c4feb-kserve-59b576959-qzwbc"]
Jul 22 02:10:35.663462 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:35.663367    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="437ccf43-dae1-4f4b-9615-dabb5b6bab73" path="/var/lib/kubelet/pods/437ccf43-dae1-4f4b-9615-dabb5b6bab73/volumes"
Jul 22 02:10:35.663775 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:10:35.663762    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c7be3c0-c389-4e70-ac50-4d2de850d3c7" path="/var/lib/kubelet/pods/8c7be3c0-c389-4e70-ac50-4d2de850d3c7/volumes"
Jul 22 02:14:01.358799 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:01.358762    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-45f6f391-699d4cc697-r2kxc"]
Jul 22 02:14:01.359237 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:01.359100    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c7be3c0-c389-4e70-ac50-4d2de850d3c7" containerName="main"
Jul 22 02:14:01.359237 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:01.359110    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c7be3c0-c389-4e70-ac50-4d2de850d3c7" containerName="main"
Jul 22 02:14:01.359237 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:01.359119    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="437ccf43-dae1-4f4b-9615-dabb5b6bab73" containerName="main"
Jul 22 02:14:01.359237 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:01.359124    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="437ccf43-dae1-4f4b-9615-dabb5b6bab73" containerName="main"
Jul 22 02:14:01.359237 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:01.359187    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="437ccf43-dae1-4f4b-9615-dabb5b6bab73" containerName="main"
Jul 22 02:14:01.359237 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:01.359196    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c7be3c0-c389-4e70-ac50-4d2de850d3c7" containerName="main"
Jul 22 02:14:01.362195 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:01.362178    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45f6f391-699d4cc697-r2kxc"
Jul 22 02:14:01.364475 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:01.364459    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-45f6f391-serving-cert\""
Jul 22 02:14:01.371670 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:01.371644    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-45f6f391-699d4cc697-r2kxc"]
Jul 22 02:14:01.410485 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:01.410459    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq457\" (UniqueName: \"kubernetes.io/projected/78fc7f44-6515-4de8-96a4-e2e6732366e8-kube-api-access-mq457\") pod \"maas-api-e2e-ait-45f6f391-699d4cc697-r2kxc\" (UID: \"78fc7f44-6515-4de8-96a4-e2e6732366e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-45f6f391-699d4cc697-r2kxc"
Jul 22 02:14:01.410610 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:01.410499    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78fc7f44-6515-4de8-96a4-e2e6732366e8-openshift-service-ca\") pod \"maas-api-e2e-ait-45f6f391-699d4cc697-r2kxc\" (UID: \"78fc7f44-6515-4de8-96a4-e2e6732366e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-45f6f391-699d4cc697-r2kxc"
Jul 22 02:14:01.410610 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:01.410519    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78fc7f44-6515-4de8-96a4-e2e6732366e8-maas-api-tls\") pod \"maas-api-e2e-ait-45f6f391-699d4cc697-r2kxc\" (UID: \"78fc7f44-6515-4de8-96a4-e2e6732366e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-45f6f391-699d4cc697-r2kxc"
Jul 22 02:14:01.511066 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:01.511040    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78fc7f44-6515-4de8-96a4-e2e6732366e8-maas-api-tls\") pod \"maas-api-e2e-ait-45f6f391-699d4cc697-r2kxc\" (UID: \"78fc7f44-6515-4de8-96a4-e2e6732366e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-45f6f391-699d4cc697-r2kxc"
Jul 22 02:14:01.511215 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:01.511199    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq457\" (UniqueName: \"kubernetes.io/projected/78fc7f44-6515-4de8-96a4-e2e6732366e8-kube-api-access-mq457\") pod \"maas-api-e2e-ait-45f6f391-699d4cc697-r2kxc\" (UID: \"78fc7f44-6515-4de8-96a4-e2e6732366e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-45f6f391-699d4cc697-r2kxc"
Jul 22 02:14:01.511275 ip-10-0-135-230 kubenswrapper[2563]: E0722 02:14:01.511211    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-45f6f391-serving-cert: secret "maas-api-e2e-ait-45f6f391-serving-cert" not found
Jul 22 02:14:01.511275 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:01.511232    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78fc7f44-6515-4de8-96a4-e2e6732366e8-openshift-service-ca\") pod \"maas-api-e2e-ait-45f6f391-699d4cc697-r2kxc\" (UID: \"78fc7f44-6515-4de8-96a4-e2e6732366e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-45f6f391-699d4cc697-r2kxc"
Jul 22 02:14:01.511357 ip-10-0-135-230 kubenswrapper[2563]: E0722 02:14:01.511293    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78fc7f44-6515-4de8-96a4-e2e6732366e8-maas-api-tls podName:78fc7f44-6515-4de8-96a4-e2e6732366e8 nodeName:}" failed. No retries permitted until 2026-07-22 02:14:02.011272763 +0000 UTC m=+1594.855101874 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/78fc7f44-6515-4de8-96a4-e2e6732366e8-maas-api-tls") pod "maas-api-e2e-ait-45f6f391-699d4cc697-r2kxc" (UID: "78fc7f44-6515-4de8-96a4-e2e6732366e8") : secret "maas-api-e2e-ait-45f6f391-serving-cert" not found
Jul 22 02:14:01.511920 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:01.511903    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78fc7f44-6515-4de8-96a4-e2e6732366e8-openshift-service-ca\") pod \"maas-api-e2e-ait-45f6f391-699d4cc697-r2kxc\" (UID: \"78fc7f44-6515-4de8-96a4-e2e6732366e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-45f6f391-699d4cc697-r2kxc"
Jul 22 02:14:01.526072 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:01.526042    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq457\" (UniqueName: \"kubernetes.io/projected/78fc7f44-6515-4de8-96a4-e2e6732366e8-kube-api-access-mq457\") pod \"maas-api-e2e-ait-45f6f391-699d4cc697-r2kxc\" (UID: \"78fc7f44-6515-4de8-96a4-e2e6732366e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-45f6f391-699d4cc697-r2kxc"
Jul 22 02:14:02.015784 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:02.015754    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78fc7f44-6515-4de8-96a4-e2e6732366e8-maas-api-tls\") pod \"maas-api-e2e-ait-45f6f391-699d4cc697-r2kxc\" (UID: \"78fc7f44-6515-4de8-96a4-e2e6732366e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-45f6f391-699d4cc697-r2kxc"
Jul 22 02:14:02.017989 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:02.017970    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78fc7f44-6515-4de8-96a4-e2e6732366e8-maas-api-tls\") pod \"maas-api-e2e-ait-45f6f391-699d4cc697-r2kxc\" (UID: \"78fc7f44-6515-4de8-96a4-e2e6732366e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-45f6f391-699d4cc697-r2kxc"
Jul 22 02:14:02.273449 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:02.273388    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45f6f391-699d4cc697-r2kxc"
Jul 22 02:14:02.610286 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:02.610258    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-45f6f391-699d4cc697-r2kxc"]
Jul 22 02:14:02.612104 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:14:02.612081    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78fc7f44_6515_4de8_96a4_e2e6732366e8.slice/crio-9612e57af9ea1b398ceb3ca48bef96470f50debfbe54fa6de81d60437857e0e3 WatchSource:0}: Error finding container 9612e57af9ea1b398ceb3ca48bef96470f50debfbe54fa6de81d60437857e0e3: Status 404 returned error can't find the container with id 9612e57af9ea1b398ceb3ca48bef96470f50debfbe54fa6de81d60437857e0e3
Jul 22 02:14:02.613791 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:02.613772    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 22 02:14:03.254735 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:03.254702    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45f6f391-699d4cc697-r2kxc" event={"ID":"78fc7f44-6515-4de8-96a4-e2e6732366e8","Type":"ContainerStarted","Data":"6ba1414ac9e7d52256388b57dee07729b14ecf697cbf3096d03d41681cb64d9e"}
Jul 22 02:14:03.254906 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:03.254762    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45f6f391-699d4cc697-r2kxc" event={"ID":"78fc7f44-6515-4de8-96a4-e2e6732366e8","Type":"ContainerStarted","Data":"9612e57af9ea1b398ceb3ca48bef96470f50debfbe54fa6de81d60437857e0e3"}
Jul 22 02:14:03.254906 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:03.254813    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45f6f391-699d4cc697-r2kxc"
Jul 22 02:14:03.274015 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:03.273973    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45f6f391-699d4cc697-r2kxc" podStartSLOduration=1.839129765 podStartE2EDuration="2.273958514s" totalImagesPullingTime="434.828749ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:14:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:14:03.273331448 +0000 UTC m=+1596.117160579" watchObservedRunningTime="2026-07-22 02:14:03.273958514 +0000 UTC m=+1596.117787645"
Jul 22 02:14:09.265251 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:09.265215    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45f6f391-699d4cc697-r2kxc"
Jul 22 02:14:37.284430 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:37.284296    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-45f6f391-699d4cc697-r2kxc"]
Jul 22 02:14:37.285092 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:37.285065    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45f6f391-699d4cc697-r2kxc" podUID="78fc7f44-6515-4de8-96a4-e2e6732366e8" containerName="maas-api" containerID="cri-o://6ba1414ac9e7d52256388b57dee07729b14ecf697cbf3096d03d41681cb64d9e" gracePeriod=30
Jul 22 02:14:37.418039 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:37.418010    2563 generic.go:356] "Generic (PLEG): container finished" podID="78fc7f44-6515-4de8-96a4-e2e6732366e8" containerID="6ba1414ac9e7d52256388b57dee07729b14ecf697cbf3096d03d41681cb64d9e" exitCode=0
Jul 22 02:14:37.418168 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:37.418084    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45f6f391-699d4cc697-r2kxc" event={"ID":"78fc7f44-6515-4de8-96a4-e2e6732366e8","Type":"ContainerDied","Data":"6ba1414ac9e7d52256388b57dee07729b14ecf697cbf3096d03d41681cb64d9e"}
Jul 22 02:14:37.532026 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:37.532002    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45f6f391-699d4cc697-r2kxc"
Jul 22 02:14:37.668454 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:37.668373    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78fc7f44-6515-4de8-96a4-e2e6732366e8-openshift-service-ca\") pod \"78fc7f44-6515-4de8-96a4-e2e6732366e8\" (UID: \"78fc7f44-6515-4de8-96a4-e2e6732366e8\") "
Jul 22 02:14:37.668454 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:37.668431    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mq457\" (UniqueName: \"kubernetes.io/projected/78fc7f44-6515-4de8-96a4-e2e6732366e8-kube-api-access-mq457\") pod \"78fc7f44-6515-4de8-96a4-e2e6732366e8\" (UID: \"78fc7f44-6515-4de8-96a4-e2e6732366e8\") "
Jul 22 02:14:37.668640 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:37.668454    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78fc7f44-6515-4de8-96a4-e2e6732366e8-maas-api-tls\") pod \"78fc7f44-6515-4de8-96a4-e2e6732366e8\" (UID: \"78fc7f44-6515-4de8-96a4-e2e6732366e8\") "
Jul 22 02:14:37.668640 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:37.668624    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78fc7f44-6515-4de8-96a4-e2e6732366e8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "78fc7f44-6515-4de8-96a4-e2e6732366e8" (UID: "78fc7f44-6515-4de8-96a4-e2e6732366e8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:14:37.670608 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:37.670586    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78fc7f44-6515-4de8-96a4-e2e6732366e8-kube-api-access-mq457" (OuterVolumeSpecName: "kube-api-access-mq457") pod "78fc7f44-6515-4de8-96a4-e2e6732366e8" (UID: "78fc7f44-6515-4de8-96a4-e2e6732366e8"). InnerVolumeSpecName "kube-api-access-mq457". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:14:37.670704 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:37.670664    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78fc7f44-6515-4de8-96a4-e2e6732366e8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "78fc7f44-6515-4de8-96a4-e2e6732366e8" (UID: "78fc7f44-6515-4de8-96a4-e2e6732366e8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:14:37.769814 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:37.769790    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78fc7f44-6515-4de8-96a4-e2e6732366e8-openshift-service-ca\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:14:37.769814 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:37.769811    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq457\" (UniqueName: \"kubernetes.io/projected/78fc7f44-6515-4de8-96a4-e2e6732366e8-kube-api-access-mq457\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:14:37.769940 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:37.769820    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78fc7f44-6515-4de8-96a4-e2e6732366e8-maas-api-tls\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:14:38.425090 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:38.425061    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45f6f391-699d4cc697-r2kxc"
Jul 22 02:14:38.425570 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:38.425100    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45f6f391-699d4cc697-r2kxc" event={"ID":"78fc7f44-6515-4de8-96a4-e2e6732366e8","Type":"ContainerDied","Data":"9612e57af9ea1b398ceb3ca48bef96470f50debfbe54fa6de81d60437857e0e3"}
Jul 22 02:14:38.425570 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:38.425156    2563 scope.go:117] "RemoveContainer" containerID="6ba1414ac9e7d52256388b57dee07729b14ecf697cbf3096d03d41681cb64d9e"
Jul 22 02:14:38.446848 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:38.446826    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-45f6f391-699d4cc697-r2kxc"]
Jul 22 02:14:38.450686 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:38.450663    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-45f6f391-699d4cc697-r2kxc"]
Jul 22 02:14:39.662507 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:39.662476    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78fc7f44-6515-4de8-96a4-e2e6732366e8" path="/var/lib/kubelet/pods/78fc7f44-6515-4de8-96a4-e2e6732366e8/volumes"
Jul 22 02:14:46.159211 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:46.159178    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-57750b46-9867c7698-dzpf4"]
Jul 22 02:14:46.159641 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:46.159524    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78fc7f44-6515-4de8-96a4-e2e6732366e8" containerName="maas-api"
Jul 22 02:14:46.159641 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:46.159536    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="78fc7f44-6515-4de8-96a4-e2e6732366e8" containerName="maas-api"
Jul 22 02:14:46.159641 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:46.159602    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="78fc7f44-6515-4de8-96a4-e2e6732366e8" containerName="maas-api"
Jul 22 02:14:46.163790 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:46.163770    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57750b46-9867c7698-dzpf4"
Jul 22 02:14:46.165914 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:46.165893    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-57750b46-serving-cert\""
Jul 22 02:14:46.171958 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:46.171936    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-57750b46-9867c7698-dzpf4"]
Jul 22 02:14:46.235793 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:46.235764    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76f913df-7011-4250-a89b-541931f99e0a-openshift-service-ca\") pod \"maas-api-e2e-migrate-57750b46-9867c7698-dzpf4\" (UID: \"76f913df-7011-4250-a89b-541931f99e0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57750b46-9867c7698-dzpf4"
Jul 22 02:14:46.235899 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:46.235798    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76f913df-7011-4250-a89b-541931f99e0a-maas-api-tls\") pod \"maas-api-e2e-migrate-57750b46-9867c7698-dzpf4\" (UID: \"76f913df-7011-4250-a89b-541931f99e0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57750b46-9867c7698-dzpf4"
Jul 22 02:14:46.235940 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:46.235896    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwqpz\" (UniqueName: \"kubernetes.io/projected/76f913df-7011-4250-a89b-541931f99e0a-kube-api-access-lwqpz\") pod \"maas-api-e2e-migrate-57750b46-9867c7698-dzpf4\" (UID: \"76f913df-7011-4250-a89b-541931f99e0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57750b46-9867c7698-dzpf4"
Jul 22 02:14:46.336565 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:46.336540    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76f913df-7011-4250-a89b-541931f99e0a-openshift-service-ca\") pod \"maas-api-e2e-migrate-57750b46-9867c7698-dzpf4\" (UID: \"76f913df-7011-4250-a89b-541931f99e0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57750b46-9867c7698-dzpf4"
Jul 22 02:14:46.336696 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:46.336572    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76f913df-7011-4250-a89b-541931f99e0a-maas-api-tls\") pod \"maas-api-e2e-migrate-57750b46-9867c7698-dzpf4\" (UID: \"76f913df-7011-4250-a89b-541931f99e0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57750b46-9867c7698-dzpf4"
Jul 22 02:14:46.336696 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:46.336610    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwqpz\" (UniqueName: \"kubernetes.io/projected/76f913df-7011-4250-a89b-541931f99e0a-kube-api-access-lwqpz\") pod \"maas-api-e2e-migrate-57750b46-9867c7698-dzpf4\" (UID: \"76f913df-7011-4250-a89b-541931f99e0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57750b46-9867c7698-dzpf4"
Jul 22 02:14:46.337221 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:46.337198    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76f913df-7011-4250-a89b-541931f99e0a-openshift-service-ca\") pod \"maas-api-e2e-migrate-57750b46-9867c7698-dzpf4\" (UID: \"76f913df-7011-4250-a89b-541931f99e0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57750b46-9867c7698-dzpf4"
Jul 22 02:14:46.338981 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:46.338961    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76f913df-7011-4250-a89b-541931f99e0a-maas-api-tls\") pod \"maas-api-e2e-migrate-57750b46-9867c7698-dzpf4\" (UID: \"76f913df-7011-4250-a89b-541931f99e0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57750b46-9867c7698-dzpf4"
Jul 22 02:14:46.344960 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:46.344926    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwqpz\" (UniqueName: \"kubernetes.io/projected/76f913df-7011-4250-a89b-541931f99e0a-kube-api-access-lwqpz\") pod \"maas-api-e2e-migrate-57750b46-9867c7698-dzpf4\" (UID: \"76f913df-7011-4250-a89b-541931f99e0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57750b46-9867c7698-dzpf4"
Jul 22 02:14:46.474866 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:46.474851    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57750b46-9867c7698-dzpf4"
Jul 22 02:14:46.606195 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:46.606125    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-57750b46-9867c7698-dzpf4"]
Jul 22 02:14:46.608347 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:14:46.608323    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76f913df_7011_4250_a89b_541931f99e0a.slice/crio-4e5863378e9cc9b3e6a753849a91b85cb6db06871927bf73a7755f6f7768d76a WatchSource:0}: Error finding container 4e5863378e9cc9b3e6a753849a91b85cb6db06871927bf73a7755f6f7768d76a: Status 404 returned error can't find the container with id 4e5863378e9cc9b3e6a753849a91b85cb6db06871927bf73a7755f6f7768d76a
Jul 22 02:14:47.470488 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:47.470456    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57750b46-9867c7698-dzpf4" event={"ID":"76f913df-7011-4250-a89b-541931f99e0a","Type":"ContainerStarted","Data":"5ee1ec119b6ab442c760050c0dba85a43e16d42c55444bda02c3247f7f308293"}
Jul 22 02:14:47.470820 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:47.470505    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57750b46-9867c7698-dzpf4" event={"ID":"76f913df-7011-4250-a89b-541931f99e0a","Type":"ContainerStarted","Data":"4e5863378e9cc9b3e6a753849a91b85cb6db06871927bf73a7755f6f7768d76a"}
Jul 22 02:14:47.470820 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:47.470549    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57750b46-9867c7698-dzpf4"
Jul 22 02:14:47.486680 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:47.486643    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57750b46-9867c7698-dzpf4" podStartSLOduration=1.099326664 podStartE2EDuration="1.4866303s" totalImagesPullingTime="387.303636ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:14:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:14:47.484694784 +0000 UTC m=+1640.328523914" watchObservedRunningTime="2026-07-22 02:14:47.4866303 +0000 UTC m=+1640.330459432"
Jul 22 02:14:53.480392 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:14:53.480356    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57750b46-9867c7698-dzpf4"
Jul 22 02:15:00.131432 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:00.131402    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744775-pt2wf"]
Jul 22 02:15:00.134693 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:00.134678    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-57750b46-29744775-plt5l"]
Jul 22 02:15:00.134806 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:00.134790    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744775-pt2wf"
Jul 22 02:15:00.137405 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:00.137371    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wrtw7\""
Jul 22 02:15:00.137796 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:00.137779    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-57750b46-29744775-plt5l"
Jul 22 02:15:00.144046 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:00.144025    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744775-pt2wf"]
Jul 22 02:15:00.146146 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:00.146126    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-57750b46-29744775-plt5l"]
Jul 22 02:15:00.147352 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:00.147335    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744775-pt2wf"
Jul 22 02:15:00.154667 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:00.154653    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-57750b46-29744775-plt5l"
Jul 22 02:15:00.287641 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:00.287605    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744775-pt2wf"]
Jul 22 02:15:00.289747 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:15:00.289722    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2db07ffd_f96e_4d29_a4d1_846ee8796eed.slice/crio-c5429512eaaabd7217557c311a98f822aba0158153e603e7411761fe6685cc60 WatchSource:0}: Error finding container c5429512eaaabd7217557c311a98f822aba0158153e603e7411761fe6685cc60: Status 404 returned error can't find the container with id c5429512eaaabd7217557c311a98f822aba0158153e603e7411761fe6685cc60
Jul 22 02:15:00.516990 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:00.516950    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-57750b46-29744775-plt5l"]
Jul 22 02:15:00.518442 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:15:00.518419    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod141083e1_43d0_4c28_908f_9d73306f33c7.slice/crio-56f2153b830abdfdfb961e4c05323441da81eee749221166798dc005ee24be40 WatchSource:0}: Error finding container 56f2153b830abdfdfb961e4c05323441da81eee749221166798dc005ee24be40: Status 404 returned error can't find the container with id 56f2153b830abdfdfb961e4c05323441da81eee749221166798dc005ee24be40
Jul 22 02:15:00.536433 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:00.536411    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-57750b46-29744775-plt5l" event={"ID":"141083e1-43d0-4c28-908f-9d73306f33c7","Type":"ContainerStarted","Data":"56f2153b830abdfdfb961e4c05323441da81eee749221166798dc005ee24be40"}
Jul 22 02:15:00.537774 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:00.537754    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744775-pt2wf" event={"ID":"2db07ffd-f96e-4d29-a4d1-846ee8796eed","Type":"ContainerStarted","Data":"c5429512eaaabd7217557c311a98f822aba0158153e603e7411761fe6685cc60"}
Jul 22 02:15:03.554984 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:03.554956    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744775-pt2wf" event={"ID":"2db07ffd-f96e-4d29-a4d1-846ee8796eed","Type":"ContainerStarted","Data":"a322942c2451b42cc670167e5363362c44b165f4fc213af0d212b440290d1d89"}
Jul 22 02:15:03.570842 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:03.570797    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744775-pt2wf" podStartSLOduration=0.942948645 podStartE2EDuration="3.570784524s" totalImagesPullingTime="2.627835879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:15:03.568856974 +0000 UTC m=+1656.412686107" watchObservedRunningTime="2026-07-22 02:15:03.570784524 +0000 UTC m=+1656.414613656"
Jul 22 02:15:04.560741 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:04.560707    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-57750b46-29744775-plt5l" event={"ID":"141083e1-43d0-4c28-908f-9d73306f33c7","Type":"ContainerStarted","Data":"0a06e7e9f25e1cce50f92dcf40426007f50d102bbc60adc3a02d254b7a290f80"}
Jul 22 02:15:04.562269 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:04.562244    2563 generic.go:356] "Generic (PLEG): container finished" podID="2db07ffd-f96e-4d29-a4d1-846ee8796eed" containerID="a322942c2451b42cc670167e5363362c44b165f4fc213af0d212b440290d1d89" exitCode=0
Jul 22 02:15:04.562404 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:04.562319    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744775-pt2wf" event={"ID":"2db07ffd-f96e-4d29-a4d1-846ee8796eed","Type":"ContainerDied","Data":"a322942c2451b42cc670167e5363362c44b165f4fc213af0d212b440290d1d89"}
Jul 22 02:15:04.575545 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:04.575507    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-57750b46-29744775-plt5l" podStartSLOduration=1.104442954 podStartE2EDuration="4.575494142s" totalImagesPullingTime="3.471051188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:15:04.573860517 +0000 UTC m=+1657.417689649" watchObservedRunningTime="2026-07-22 02:15:04.575494142 +0000 UTC m=+1657.419323272"
Jul 22 02:15:05.569939 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:05.569907    2563 generic.go:356] "Generic (PLEG): container finished" podID="141083e1-43d0-4c28-908f-9d73306f33c7" containerID="0a06e7e9f25e1cce50f92dcf40426007f50d102bbc60adc3a02d254b7a290f80" exitCode=0
Jul 22 02:15:05.570405 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:05.569979    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-57750b46-29744775-plt5l" event={"ID":"141083e1-43d0-4c28-908f-9d73306f33c7","Type":"ContainerDied","Data":"0a06e7e9f25e1cce50f92dcf40426007f50d102bbc60adc3a02d254b7a290f80"}
Jul 22 02:15:05.700593 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:05.700572    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744775-pt2wf"
Jul 22 02:15:06.576432 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:06.576407    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744775-pt2wf"
Jul 22 02:15:06.576784 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:06.576429    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744775-pt2wf" event={"ID":"2db07ffd-f96e-4d29-a4d1-846ee8796eed","Type":"ContainerDied","Data":"c5429512eaaabd7217557c311a98f822aba0158153e603e7411761fe6685cc60"}
Jul 22 02:15:06.576784 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:06.576473    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5429512eaaabd7217557c311a98f822aba0158153e603e7411761fe6685cc60"
Jul 22 02:15:06.708949 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:06.708922    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-57750b46-29744775-plt5l"
Jul 22 02:15:07.582449 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:07.582426    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-57750b46-29744775-plt5l"
Jul 22 02:15:07.582896 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:07.582418    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-57750b46-29744775-plt5l" event={"ID":"141083e1-43d0-4c28-908f-9d73306f33c7","Type":"ContainerDied","Data":"56f2153b830abdfdfb961e4c05323441da81eee749221166798dc005ee24be40"}
Jul 22 02:15:07.582896 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:07.582551    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56f2153b830abdfdfb961e4c05323441da81eee749221166798dc005ee24be40"
Jul 22 02:15:11.067574 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:11.067541    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57750b46-5gq86"]
Jul 22 02:15:11.067913 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:11.067873    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2db07ffd-f96e-4d29-a4d1-846ee8796eed" containerName="cleanup"
Jul 22 02:15:11.067913 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:11.067885    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2db07ffd-f96e-4d29-a4d1-846ee8796eed" containerName="cleanup"
Jul 22 02:15:11.067913 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:11.067895    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="141083e1-43d0-4c28-908f-9d73306f33c7" containerName="cleanup"
Jul 22 02:15:11.067913 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:11.067900    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="141083e1-43d0-4c28-908f-9d73306f33c7" containerName="cleanup"
Jul 22 02:15:11.068040 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:11.067957    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="141083e1-43d0-4c28-908f-9d73306f33c7" containerName="cleanup"
Jul 22 02:15:11.068040 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:11.067966    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2db07ffd-f96e-4d29-a4d1-846ee8796eed" containerName="cleanup"
Jul 22 02:15:11.070945 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:11.070930    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57750b46-5gq86"
Jul 22 02:15:11.073258 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:11.073237    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wrtw7\""
Jul 22 02:15:11.078271 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:11.078077    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57750b46-5gq86"]
Jul 22 02:15:11.231682 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:11.231653    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10ffaab3-4751-401c-9fdf-5920d82cc532-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-57750b46-5gq86\" (UID: \"10ffaab3-4751-401c-9fdf-5920d82cc532\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57750b46-5gq86"
Jul 22 02:15:11.332675 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:11.332619    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10ffaab3-4751-401c-9fdf-5920d82cc532-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-57750b46-5gq86\" (UID: \"10ffaab3-4751-401c-9fdf-5920d82cc532\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57750b46-5gq86"
Jul 22 02:15:11.333165 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:11.333148    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10ffaab3-4751-401c-9fdf-5920d82cc532-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-57750b46-5gq86\" (UID: \"10ffaab3-4751-401c-9fdf-5920d82cc532\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57750b46-5gq86"
Jul 22 02:15:11.382194 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:11.382173    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57750b46-5gq86"
Jul 22 02:15:11.508294 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:11.508262    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57750b46-5gq86"]
Jul 22 02:15:11.511532 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:15:11.511504    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10ffaab3_4751_401c_9fdf_5920d82cc532.slice/crio-2d038fb1e22aa73f02cb69ef8d9ff889e9d4db9b919c2be148814f156588f692 WatchSource:0}: Error finding container 2d038fb1e22aa73f02cb69ef8d9ff889e9d4db9b919c2be148814f156588f692: Status 404 returned error can't find the container with id 2d038fb1e22aa73f02cb69ef8d9ff889e9d4db9b919c2be148814f156588f692
Jul 22 02:15:11.602873 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:11.602818    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57750b46-5gq86" event={"ID":"10ffaab3-4751-401c-9fdf-5920d82cc532","Type":"ContainerStarted","Data":"2d038fb1e22aa73f02cb69ef8d9ff889e9d4db9b919c2be148814f156588f692"}
Jul 22 02:15:12.609114 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:12.609043    2563 generic.go:356] "Generic (PLEG): container finished" podID="10ffaab3-4751-401c-9fdf-5920d82cc532" containerID="25f77f234e28ce445cb62d41b287dd4983af527ccf3d09dd0161eb4df8a7c2f3" exitCode=0
Jul 22 02:15:12.609447 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:12.609123    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57750b46-5gq86" event={"ID":"10ffaab3-4751-401c-9fdf-5920d82cc532","Type":"ContainerDied","Data":"25f77f234e28ce445cb62d41b287dd4983af527ccf3d09dd0161eb4df8a7c2f3"}
Jul 22 02:15:13.740164 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:13.740138    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57750b46-5gq86"
Jul 22 02:15:13.851975 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:13.851954    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10ffaab3-4751-401c-9fdf-5920d82cc532-maas-api-service-ca\") pod \"10ffaab3-4751-401c-9fdf-5920d82cc532\" (UID: \"10ffaab3-4751-401c-9fdf-5920d82cc532\") "
Jul 22 02:15:13.852303 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:13.852281    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10ffaab3-4751-401c-9fdf-5920d82cc532-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "10ffaab3-4751-401c-9fdf-5920d82cc532" (UID: "10ffaab3-4751-401c-9fdf-5920d82cc532"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:15:13.952512 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:13.952488    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10ffaab3-4751-401c-9fdf-5920d82cc532-maas-api-service-ca\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:15:14.619799 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:14.619771    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57750b46-5gq86"
Jul 22 02:15:14.619958 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:14.619804    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57750b46-5gq86" event={"ID":"10ffaab3-4751-401c-9fdf-5920d82cc532","Type":"ContainerDied","Data":"2d038fb1e22aa73f02cb69ef8d9ff889e9d4db9b919c2be148814f156588f692"}
Jul 22 02:15:14.619958 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:14.619854    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d038fb1e22aa73f02cb69ef8d9ff889e9d4db9b919c2be148814f156588f692"
Jul 22 02:15:21.589773 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:21.589734    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-57750b46-9867c7698-dzpf4"]
Jul 22 02:15:21.590368 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:21.589968    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57750b46-9867c7698-dzpf4" podUID="76f913df-7011-4250-a89b-541931f99e0a" containerName="maas-api" containerID="cri-o://5ee1ec119b6ab442c760050c0dba85a43e16d42c55444bda02c3247f7f308293" gracePeriod=30
Jul 22 02:15:21.619732 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:21.619705    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-57750b46-29744775-plt5l"]
Jul 22 02:15:21.622075 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:21.622052    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-57750b46-29744775-plt5l"]
Jul 22 02:15:21.663121 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:21.663086    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="141083e1-43d0-4c28-908f-9d73306f33c7" path="/var/lib/kubelet/pods/141083e1-43d0-4c28-908f-9d73306f33c7/volumes"
Jul 22 02:15:21.840966 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:21.840898    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57750b46-9867c7698-dzpf4"
Jul 22 02:15:21.914687 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:21.914650    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76f913df-7011-4250-a89b-541931f99e0a-maas-api-tls\") pod \"76f913df-7011-4250-a89b-541931f99e0a\" (UID: \"76f913df-7011-4250-a89b-541931f99e0a\") "
Jul 22 02:15:21.914881 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:21.914710    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76f913df-7011-4250-a89b-541931f99e0a-openshift-service-ca\") pod \"76f913df-7011-4250-a89b-541931f99e0a\" (UID: \"76f913df-7011-4250-a89b-541931f99e0a\") "
Jul 22 02:15:21.914881 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:21.914751    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwqpz\" (UniqueName: \"kubernetes.io/projected/76f913df-7011-4250-a89b-541931f99e0a-kube-api-access-lwqpz\") pod \"76f913df-7011-4250-a89b-541931f99e0a\" (UID: \"76f913df-7011-4250-a89b-541931f99e0a\") "
Jul 22 02:15:21.915077 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:21.915051    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76f913df-7011-4250-a89b-541931f99e0a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "76f913df-7011-4250-a89b-541931f99e0a" (UID: "76f913df-7011-4250-a89b-541931f99e0a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:15:21.916856 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:21.916821    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76f913df-7011-4250-a89b-541931f99e0a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "76f913df-7011-4250-a89b-541931f99e0a" (UID: "76f913df-7011-4250-a89b-541931f99e0a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:15:21.916987 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:21.916859    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76f913df-7011-4250-a89b-541931f99e0a-kube-api-access-lwqpz" (OuterVolumeSpecName: "kube-api-access-lwqpz") pod "76f913df-7011-4250-a89b-541931f99e0a" (UID: "76f913df-7011-4250-a89b-541931f99e0a"). InnerVolumeSpecName "kube-api-access-lwqpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:15:22.016044 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:22.016004    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76f913df-7011-4250-a89b-541931f99e0a-openshift-service-ca\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:15:22.016044 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:22.016049    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwqpz\" (UniqueName: \"kubernetes.io/projected/76f913df-7011-4250-a89b-541931f99e0a-kube-api-access-lwqpz\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:15:22.016257 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:22.016077    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76f913df-7011-4250-a89b-541931f99e0a-maas-api-tls\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:15:22.660423 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:22.660393    2563 generic.go:356] "Generic (PLEG): container finished" podID="76f913df-7011-4250-a89b-541931f99e0a" containerID="5ee1ec119b6ab442c760050c0dba85a43e16d42c55444bda02c3247f7f308293" exitCode=0
Jul 22 02:15:22.660836 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:22.660430    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57750b46-9867c7698-dzpf4"
Jul 22 02:15:22.660836 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:22.660472    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57750b46-9867c7698-dzpf4" event={"ID":"76f913df-7011-4250-a89b-541931f99e0a","Type":"ContainerDied","Data":"5ee1ec119b6ab442c760050c0dba85a43e16d42c55444bda02c3247f7f308293"}
Jul 22 02:15:22.660836 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:22.660514    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-57750b46-9867c7698-dzpf4" event={"ID":"76f913df-7011-4250-a89b-541931f99e0a","Type":"ContainerDied","Data":"4e5863378e9cc9b3e6a753849a91b85cb6db06871927bf73a7755f6f7768d76a"}
Jul 22 02:15:22.660836 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:22.660535    2563 scope.go:117] "RemoveContainer" containerID="5ee1ec119b6ab442c760050c0dba85a43e16d42c55444bda02c3247f7f308293"
Jul 22 02:15:22.670134 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:22.670119    2563 scope.go:117] "RemoveContainer" containerID="5ee1ec119b6ab442c760050c0dba85a43e16d42c55444bda02c3247f7f308293"
Jul 22 02:15:22.670369 ip-10-0-135-230 kubenswrapper[2563]: E0722 02:15:22.670351    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ee1ec119b6ab442c760050c0dba85a43e16d42c55444bda02c3247f7f308293\": container with ID starting with 5ee1ec119b6ab442c760050c0dba85a43e16d42c55444bda02c3247f7f308293 not found: ID does not exist" containerID="5ee1ec119b6ab442c760050c0dba85a43e16d42c55444bda02c3247f7f308293"
Jul 22 02:15:22.670474 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:22.670396    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ee1ec119b6ab442c760050c0dba85a43e16d42c55444bda02c3247f7f308293"} err="failed to get container status \"5ee1ec119b6ab442c760050c0dba85a43e16d42c55444bda02c3247f7f308293\": rpc error: code = NotFound desc = could not find container \"5ee1ec119b6ab442c760050c0dba85a43e16d42c55444bda02c3247f7f308293\": container with ID starting with 5ee1ec119b6ab442c760050c0dba85a43e16d42c55444bda02c3247f7f308293 not found: ID does not exist"
Jul 22 02:15:22.682099 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:22.682077    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-57750b46-9867c7698-dzpf4"]
Jul 22 02:15:22.684273 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:22.684253    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-57750b46-9867c7698-dzpf4"]
Jul 22 02:15:23.663310 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:23.663268    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76f913df-7011-4250-a89b-541931f99e0a" path="/var/lib/kubelet/pods/76f913df-7011-4250-a89b-541931f99e0a/volumes"
Jul 22 02:15:26.148699 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:26.148637    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57750b46-5gq86"]
Jul 22 02:15:26.151354 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:26.151330    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-57750b46-5gq86"]
Jul 22 02:15:27.663294 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:27.663264    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10ffaab3-4751-401c-9fdf-5920d82cc532" path="/var/lib/kubelet/pods/10ffaab3-4751-401c-9fdf-5920d82cc532/volumes"
Jul 22 02:15:30.194153 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:30.188456    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-353f6df7-75db95f8dc-dkxxj"]
Jul 22 02:15:30.194153 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:30.189241    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10ffaab3-4751-401c-9fdf-5920d82cc532" containerName="revoke-keys"
Jul 22 02:15:30.194153 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:30.189260    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="10ffaab3-4751-401c-9fdf-5920d82cc532" containerName="revoke-keys"
Jul 22 02:15:30.194153 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:30.189275    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76f913df-7011-4250-a89b-541931f99e0a" containerName="maas-api"
Jul 22 02:15:30.194153 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:30.189291    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="76f913df-7011-4250-a89b-541931f99e0a" containerName="maas-api"
Jul 22 02:15:30.194153 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:30.189470    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="10ffaab3-4751-401c-9fdf-5920d82cc532" containerName="revoke-keys"
Jul 22 02:15:30.194153 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:30.189487    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="76f913df-7011-4250-a89b-541931f99e0a" containerName="maas-api"
Jul 22 02:15:30.198493 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:30.198473    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-353f6df7-75db95f8dc-dkxxj"
Jul 22 02:15:30.200972 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:30.200949    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-353f6df7-75db95f8dc-dkxxj"]
Jul 22 02:15:30.202501 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:30.202482    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-353f6df7-serving-cert\""
Jul 22 02:15:30.276639 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:30.276615    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29bfa3f3-4dd7-404b-8bf9-e124da954d6e-openshift-service-ca\") pod \"maas-api-e2e-ait-353f6df7-75db95f8dc-dkxxj\" (UID: \"29bfa3f3-4dd7-404b-8bf9-e124da954d6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-353f6df7-75db95f8dc-dkxxj"
Jul 22 02:15:30.276762 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:30.276679    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29bfa3f3-4dd7-404b-8bf9-e124da954d6e-maas-api-tls\") pod \"maas-api-e2e-ait-353f6df7-75db95f8dc-dkxxj\" (UID: \"29bfa3f3-4dd7-404b-8bf9-e124da954d6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-353f6df7-75db95f8dc-dkxxj"
Jul 22 02:15:30.276762 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:30.276707    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vthzp\" (UniqueName: \"kubernetes.io/projected/29bfa3f3-4dd7-404b-8bf9-e124da954d6e-kube-api-access-vthzp\") pod \"maas-api-e2e-ait-353f6df7-75db95f8dc-dkxxj\" (UID: \"29bfa3f3-4dd7-404b-8bf9-e124da954d6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-353f6df7-75db95f8dc-dkxxj"
Jul 22 02:15:30.377077 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:30.377049    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29bfa3f3-4dd7-404b-8bf9-e124da954d6e-maas-api-tls\") pod \"maas-api-e2e-ait-353f6df7-75db95f8dc-dkxxj\" (UID: \"29bfa3f3-4dd7-404b-8bf9-e124da954d6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-353f6df7-75db95f8dc-dkxxj"
Jul 22 02:15:30.377233 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:30.377082    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vthzp\" (UniqueName: \"kubernetes.io/projected/29bfa3f3-4dd7-404b-8bf9-e124da954d6e-kube-api-access-vthzp\") pod \"maas-api-e2e-ait-353f6df7-75db95f8dc-dkxxj\" (UID: \"29bfa3f3-4dd7-404b-8bf9-e124da954d6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-353f6df7-75db95f8dc-dkxxj"
Jul 22 02:15:30.377233 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:30.377121    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29bfa3f3-4dd7-404b-8bf9-e124da954d6e-openshift-service-ca\") pod \"maas-api-e2e-ait-353f6df7-75db95f8dc-dkxxj\" (UID: \"29bfa3f3-4dd7-404b-8bf9-e124da954d6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-353f6df7-75db95f8dc-dkxxj"
Jul 22 02:15:30.377715 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:30.377696    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29bfa3f3-4dd7-404b-8bf9-e124da954d6e-openshift-service-ca\") pod \"maas-api-e2e-ait-353f6df7-75db95f8dc-dkxxj\" (UID: \"29bfa3f3-4dd7-404b-8bf9-e124da954d6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-353f6df7-75db95f8dc-dkxxj"
Jul 22 02:15:30.379431 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:30.379414    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29bfa3f3-4dd7-404b-8bf9-e124da954d6e-maas-api-tls\") pod \"maas-api-e2e-ait-353f6df7-75db95f8dc-dkxxj\" (UID: \"29bfa3f3-4dd7-404b-8bf9-e124da954d6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-353f6df7-75db95f8dc-dkxxj"
Jul 22 02:15:30.391695 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:30.391675    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vthzp\" (UniqueName: \"kubernetes.io/projected/29bfa3f3-4dd7-404b-8bf9-e124da954d6e-kube-api-access-vthzp\") pod \"maas-api-e2e-ait-353f6df7-75db95f8dc-dkxxj\" (UID: \"29bfa3f3-4dd7-404b-8bf9-e124da954d6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-353f6df7-75db95f8dc-dkxxj"
Jul 22 02:15:30.511316 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:30.511233    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-353f6df7-75db95f8dc-dkxxj"
Jul 22 02:15:30.645936 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:30.645910    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-353f6df7-75db95f8dc-dkxxj"]
Jul 22 02:15:30.648640 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:15:30.648616    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29bfa3f3_4dd7_404b_8bf9_e124da954d6e.slice/crio-a61b47d45ac27cacaff305056aa3c8ce21af2908937f676d73067e2ff702a544 WatchSource:0}: Error finding container a61b47d45ac27cacaff305056aa3c8ce21af2908937f676d73067e2ff702a544: Status 404 returned error can't find the container with id a61b47d45ac27cacaff305056aa3c8ce21af2908937f676d73067e2ff702a544
Jul 22 02:15:30.701702 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:30.701673    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-353f6df7-75db95f8dc-dkxxj" event={"ID":"29bfa3f3-4dd7-404b-8bf9-e124da954d6e","Type":"ContainerStarted","Data":"a61b47d45ac27cacaff305056aa3c8ce21af2908937f676d73067e2ff702a544"}
Jul 22 02:15:31.710708 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:31.710676    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-353f6df7-75db95f8dc-dkxxj" event={"ID":"29bfa3f3-4dd7-404b-8bf9-e124da954d6e","Type":"ContainerStarted","Data":"c2234de381a1fd5514d3d836baa5b2192435ac9df5500ad7d2f452a45f5431dc"}
Jul 22 02:15:31.711121 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:31.710736    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-353f6df7-75db95f8dc-dkxxj"
Jul 22 02:15:31.730344 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:31.730295    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-353f6df7-75db95f8dc-dkxxj" podStartSLOduration=1.290676887 podStartE2EDuration="1.730279489s" totalImagesPullingTime="439.602602ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:15:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:15:31.729116427 +0000 UTC m=+1684.572945569" watchObservedRunningTime="2026-07-22 02:15:31.730279489 +0000 UTC m=+1684.574108622"
Jul 22 02:15:37.721321 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:37.721291    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-353f6df7-75db95f8dc-dkxxj"
Jul 22 02:15:56.067635 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:56.067600    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-353f6df7-mq2tf"]
Jul 22 02:15:56.070874 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:56.070859    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-353f6df7-mq2tf"
Jul 22 02:15:56.073570 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:56.073550    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wrtw7\""
Jul 22 02:15:56.078829 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:56.078572    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-353f6df7-mq2tf"]
Jul 22 02:15:56.177355 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:56.177330    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb2ddf70-3ad8-4cf7-9a60-4e6262c67397-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-353f6df7-mq2tf\" (UID: \"bb2ddf70-3ad8-4cf7-9a60-4e6262c67397\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-353f6df7-mq2tf"
Jul 22 02:15:56.278045 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:56.278013    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb2ddf70-3ad8-4cf7-9a60-4e6262c67397-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-353f6df7-mq2tf\" (UID: \"bb2ddf70-3ad8-4cf7-9a60-4e6262c67397\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-353f6df7-mq2tf"
Jul 22 02:15:56.278626 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:56.278607    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb2ddf70-3ad8-4cf7-9a60-4e6262c67397-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-353f6df7-mq2tf\" (UID: \"bb2ddf70-3ad8-4cf7-9a60-4e6262c67397\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-353f6df7-mq2tf"
Jul 22 02:15:56.383774 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:56.383721    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-353f6df7-mq2tf"
Jul 22 02:15:56.514500 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:56.514475    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-353f6df7-mq2tf"]
Jul 22 02:15:56.515753 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:15:56.515731    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2ddf70_3ad8_4cf7_9a60_4e6262c67397.slice/crio-5774a935367c74dd0f7abcc816c15490c27811abee7f830eba4d1bdb6aada1bd WatchSource:0}: Error finding container 5774a935367c74dd0f7abcc816c15490c27811abee7f830eba4d1bdb6aada1bd: Status 404 returned error can't find the container with id 5774a935367c74dd0f7abcc816c15490c27811abee7f830eba4d1bdb6aada1bd
Jul 22 02:15:56.845780 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:56.845748    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-353f6df7-mq2tf" event={"ID":"bb2ddf70-3ad8-4cf7-9a60-4e6262c67397","Type":"ContainerStarted","Data":"76aff8eadb996ba334d99cff9f90439541bb10f98930f18527075dc36c6e49dc"}
Jul 22 02:15:56.845880 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:56.845802    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-353f6df7-mq2tf" event={"ID":"bb2ddf70-3ad8-4cf7-9a60-4e6262c67397","Type":"ContainerStarted","Data":"5774a935367c74dd0f7abcc816c15490c27811abee7f830eba4d1bdb6aada1bd"}
Jul 22 02:15:56.861066 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:56.861028    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-353f6df7-mq2tf" podStartSLOduration=0.861014354 podStartE2EDuration="861.014354ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:15:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:15:56.859416327 +0000 UTC m=+1709.703245460" watchObservedRunningTime="2026-07-22 02:15:56.861014354 +0000 UTC m=+1709.704843486"
Jul 22 02:15:57.852290 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:57.852258    2563 generic.go:356] "Generic (PLEG): container finished" podID="bb2ddf70-3ad8-4cf7-9a60-4e6262c67397" containerID="76aff8eadb996ba334d99cff9f90439541bb10f98930f18527075dc36c6e49dc" exitCode=0
Jul 22 02:15:57.852690 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:57.852329    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-353f6df7-mq2tf" event={"ID":"bb2ddf70-3ad8-4cf7-9a60-4e6262c67397","Type":"ContainerDied","Data":"76aff8eadb996ba334d99cff9f90439541bb10f98930f18527075dc36c6e49dc"}
Jul 22 02:15:58.983140 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:58.983119    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-353f6df7-mq2tf"
Jul 22 02:15:59.101273 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:59.101250    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb2ddf70-3ad8-4cf7-9a60-4e6262c67397-maas-api-service-ca\") pod \"bb2ddf70-3ad8-4cf7-9a60-4e6262c67397\" (UID: \"bb2ddf70-3ad8-4cf7-9a60-4e6262c67397\") "
Jul 22 02:15:59.101598 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:59.101578    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb2ddf70-3ad8-4cf7-9a60-4e6262c67397-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bb2ddf70-3ad8-4cf7-9a60-4e6262c67397" (UID: "bb2ddf70-3ad8-4cf7-9a60-4e6262c67397"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:15:59.202720 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:59.202694    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb2ddf70-3ad8-4cf7-9a60-4e6262c67397-maas-api-service-ca\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:15:59.863129 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:59.863102    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-353f6df7-mq2tf"
Jul 22 02:15:59.863261 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:59.863101    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-353f6df7-mq2tf" event={"ID":"bb2ddf70-3ad8-4cf7-9a60-4e6262c67397","Type":"ContainerDied","Data":"5774a935367c74dd0f7abcc816c15490c27811abee7f830eba4d1bdb6aada1bd"}
Jul 22 02:15:59.863261 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:15:59.863218    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5774a935367c74dd0f7abcc816c15490c27811abee7f830eba4d1bdb6aada1bd"
Jul 22 02:16:06.528892 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:06.528860    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-353f6df7-75db95f8dc-dkxxj"]
Jul 22 02:16:06.529368 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:06.529210    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-353f6df7-75db95f8dc-dkxxj" podUID="29bfa3f3-4dd7-404b-8bf9-e124da954d6e" containerName="maas-api" containerID="cri-o://c2234de381a1fd5514d3d836baa5b2192435ac9df5500ad7d2f452a45f5431dc" gracePeriod=30
Jul 22 02:16:06.779014 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:06.778955    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-353f6df7-75db95f8dc-dkxxj"
Jul 22 02:16:06.861274 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:06.861233    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vthzp\" (UniqueName: \"kubernetes.io/projected/29bfa3f3-4dd7-404b-8bf9-e124da954d6e-kube-api-access-vthzp\") pod \"29bfa3f3-4dd7-404b-8bf9-e124da954d6e\" (UID: \"29bfa3f3-4dd7-404b-8bf9-e124da954d6e\") "
Jul 22 02:16:06.861457 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:06.861300    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29bfa3f3-4dd7-404b-8bf9-e124da954d6e-maas-api-tls\") pod \"29bfa3f3-4dd7-404b-8bf9-e124da954d6e\" (UID: \"29bfa3f3-4dd7-404b-8bf9-e124da954d6e\") "
Jul 22 02:16:06.861457 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:06.861364    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29bfa3f3-4dd7-404b-8bf9-e124da954d6e-openshift-service-ca\") pod \"29bfa3f3-4dd7-404b-8bf9-e124da954d6e\" (UID: \"29bfa3f3-4dd7-404b-8bf9-e124da954d6e\") "
Jul 22 02:16:06.861775 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:06.861744    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29bfa3f3-4dd7-404b-8bf9-e124da954d6e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "29bfa3f3-4dd7-404b-8bf9-e124da954d6e" (UID: "29bfa3f3-4dd7-404b-8bf9-e124da954d6e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:16:06.863299 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:06.863274    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29bfa3f3-4dd7-404b-8bf9-e124da954d6e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "29bfa3f3-4dd7-404b-8bf9-e124da954d6e" (UID: "29bfa3f3-4dd7-404b-8bf9-e124da954d6e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:16:06.863388 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:06.863286    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29bfa3f3-4dd7-404b-8bf9-e124da954d6e-kube-api-access-vthzp" (OuterVolumeSpecName: "kube-api-access-vthzp") pod "29bfa3f3-4dd7-404b-8bf9-e124da954d6e" (UID: "29bfa3f3-4dd7-404b-8bf9-e124da954d6e"). InnerVolumeSpecName "kube-api-access-vthzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:16:06.901599 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:06.901566    2563 generic.go:356] "Generic (PLEG): container finished" podID="29bfa3f3-4dd7-404b-8bf9-e124da954d6e" containerID="c2234de381a1fd5514d3d836baa5b2192435ac9df5500ad7d2f452a45f5431dc" exitCode=0
Jul 22 02:16:06.901733 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:06.901603    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-353f6df7-75db95f8dc-dkxxj"
Jul 22 02:16:06.901733 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:06.901643    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-353f6df7-75db95f8dc-dkxxj" event={"ID":"29bfa3f3-4dd7-404b-8bf9-e124da954d6e","Type":"ContainerDied","Data":"c2234de381a1fd5514d3d836baa5b2192435ac9df5500ad7d2f452a45f5431dc"}
Jul 22 02:16:06.901733 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:06.901701    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-353f6df7-75db95f8dc-dkxxj" event={"ID":"29bfa3f3-4dd7-404b-8bf9-e124da954d6e","Type":"ContainerDied","Data":"a61b47d45ac27cacaff305056aa3c8ce21af2908937f676d73067e2ff702a544"}
Jul 22 02:16:06.901733 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:06.901728    2563 scope.go:117] "RemoveContainer" containerID="c2234de381a1fd5514d3d836baa5b2192435ac9df5500ad7d2f452a45f5431dc"
Jul 22 02:16:06.910713 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:06.910695    2563 scope.go:117] "RemoveContainer" containerID="c2234de381a1fd5514d3d836baa5b2192435ac9df5500ad7d2f452a45f5431dc"
Jul 22 02:16:06.910973 ip-10-0-135-230 kubenswrapper[2563]: E0722 02:16:06.910948    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2234de381a1fd5514d3d836baa5b2192435ac9df5500ad7d2f452a45f5431dc\": container with ID starting with c2234de381a1fd5514d3d836baa5b2192435ac9df5500ad7d2f452a45f5431dc not found: ID does not exist" containerID="c2234de381a1fd5514d3d836baa5b2192435ac9df5500ad7d2f452a45f5431dc"
Jul 22 02:16:06.911023 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:06.910984    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2234de381a1fd5514d3d836baa5b2192435ac9df5500ad7d2f452a45f5431dc"} err="failed to get container status \"c2234de381a1fd5514d3d836baa5b2192435ac9df5500ad7d2f452a45f5431dc\": rpc error: code = NotFound desc = could not find container \"c2234de381a1fd5514d3d836baa5b2192435ac9df5500ad7d2f452a45f5431dc\": container with ID starting with c2234de381a1fd5514d3d836baa5b2192435ac9df5500ad7d2f452a45f5431dc not found: ID does not exist"
Jul 22 02:16:06.924139 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:06.924117    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-353f6df7-75db95f8dc-dkxxj"]
Jul 22 02:16:06.928147 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:06.928123    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-353f6df7-75db95f8dc-dkxxj"]
Jul 22 02:16:06.962393 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:06.962346    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29bfa3f3-4dd7-404b-8bf9-e124da954d6e-openshift-service-ca\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:16:06.962393 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:06.962371    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vthzp\" (UniqueName: \"kubernetes.io/projected/29bfa3f3-4dd7-404b-8bf9-e124da954d6e-kube-api-access-vthzp\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:16:06.962573 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:06.962408    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29bfa3f3-4dd7-404b-8bf9-e124da954d6e-maas-api-tls\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:16:07.664121 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:07.664091    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29bfa3f3-4dd7-404b-8bf9-e124da954d6e" path="/var/lib/kubelet/pods/29bfa3f3-4dd7-404b-8bf9-e124da954d6e/volumes"
Jul 22 02:16:11.143442 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:11.143411    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-353f6df7-mq2tf"]
Jul 22 02:16:11.145806 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:11.145780    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-353f6df7-mq2tf"]
Jul 22 02:16:11.663353 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:11.663320    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb2ddf70-3ad8-4cf7-9a60-4e6262c67397" path="/var/lib/kubelet/pods/bb2ddf70-3ad8-4cf7-9a60-4e6262c67397/volumes"
Jul 22 02:16:16.065058 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:16.065020    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-404405cc-7f8d65bdf7-xtnn2"]
Jul 22 02:16:16.065460 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:16.065365    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29bfa3f3-4dd7-404b-8bf9-e124da954d6e" containerName="maas-api"
Jul 22 02:16:16.065460 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:16.065387    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="29bfa3f3-4dd7-404b-8bf9-e124da954d6e" containerName="maas-api"
Jul 22 02:16:16.065460 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:16.065396    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb2ddf70-3ad8-4cf7-9a60-4e6262c67397" containerName="revoke-keys"
Jul 22 02:16:16.065460 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:16.065402    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb2ddf70-3ad8-4cf7-9a60-4e6262c67397" containerName="revoke-keys"
Jul 22 02:16:16.065638 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:16.065466    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="29bfa3f3-4dd7-404b-8bf9-e124da954d6e" containerName="maas-api"
Jul 22 02:16:16.065638 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:16.065477    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb2ddf70-3ad8-4cf7-9a60-4e6262c67397" containerName="revoke-keys"
Jul 22 02:16:16.068344 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:16.068324    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-404405cc-7f8d65bdf7-xtnn2"
Jul 22 02:16:16.073225 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:16.073207    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-404405cc-serving-cert\""
Jul 22 02:16:16.091580 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:16.091553    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-404405cc-7f8d65bdf7-xtnn2"]
Jul 22 02:16:16.238214 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:16.238177    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3cea1d3-db85-4149-bde8-21974e5d4ff6-openshift-service-ca\") pod \"maas-api-e2e-derive-404405cc-7f8d65bdf7-xtnn2\" (UID: \"b3cea1d3-db85-4149-bde8-21974e5d4ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-404405cc-7f8d65bdf7-xtnn2"
Jul 22 02:16:16.238438 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:16.238234    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3cea1d3-db85-4149-bde8-21974e5d4ff6-maas-api-tls\") pod \"maas-api-e2e-derive-404405cc-7f8d65bdf7-xtnn2\" (UID: \"b3cea1d3-db85-4149-bde8-21974e5d4ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-404405cc-7f8d65bdf7-xtnn2"
Jul 22 02:16:16.238438 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:16.238256    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lmxw\" (UniqueName: \"kubernetes.io/projected/b3cea1d3-db85-4149-bde8-21974e5d4ff6-kube-api-access-2lmxw\") pod \"maas-api-e2e-derive-404405cc-7f8d65bdf7-xtnn2\" (UID: \"b3cea1d3-db85-4149-bde8-21974e5d4ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-404405cc-7f8d65bdf7-xtnn2"
Jul 22 02:16:16.339284 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:16.339193    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3cea1d3-db85-4149-bde8-21974e5d4ff6-openshift-service-ca\") pod \"maas-api-e2e-derive-404405cc-7f8d65bdf7-xtnn2\" (UID: \"b3cea1d3-db85-4149-bde8-21974e5d4ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-404405cc-7f8d65bdf7-xtnn2"
Jul 22 02:16:16.339284 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:16.339250    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3cea1d3-db85-4149-bde8-21974e5d4ff6-maas-api-tls\") pod \"maas-api-e2e-derive-404405cc-7f8d65bdf7-xtnn2\" (UID: \"b3cea1d3-db85-4149-bde8-21974e5d4ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-404405cc-7f8d65bdf7-xtnn2"
Jul 22 02:16:16.339284 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:16.339276    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lmxw\" (UniqueName: \"kubernetes.io/projected/b3cea1d3-db85-4149-bde8-21974e5d4ff6-kube-api-access-2lmxw\") pod \"maas-api-e2e-derive-404405cc-7f8d65bdf7-xtnn2\" (UID: \"b3cea1d3-db85-4149-bde8-21974e5d4ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-404405cc-7f8d65bdf7-xtnn2"
Jul 22 02:16:16.339869 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:16.339847    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3cea1d3-db85-4149-bde8-21974e5d4ff6-openshift-service-ca\") pod \"maas-api-e2e-derive-404405cc-7f8d65bdf7-xtnn2\" (UID: \"b3cea1d3-db85-4149-bde8-21974e5d4ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-404405cc-7f8d65bdf7-xtnn2"
Jul 22 02:16:16.341851 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:16.341828    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3cea1d3-db85-4149-bde8-21974e5d4ff6-maas-api-tls\") pod \"maas-api-e2e-derive-404405cc-7f8d65bdf7-xtnn2\" (UID: \"b3cea1d3-db85-4149-bde8-21974e5d4ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-404405cc-7f8d65bdf7-xtnn2"
Jul 22 02:16:16.349135 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:16.349110    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lmxw\" (UniqueName: \"kubernetes.io/projected/b3cea1d3-db85-4149-bde8-21974e5d4ff6-kube-api-access-2lmxw\") pod \"maas-api-e2e-derive-404405cc-7f8d65bdf7-xtnn2\" (UID: \"b3cea1d3-db85-4149-bde8-21974e5d4ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-404405cc-7f8d65bdf7-xtnn2"
Jul 22 02:16:16.378845 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:16.378814    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-404405cc-7f8d65bdf7-xtnn2"
Jul 22 02:16:16.512874 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:16.512843    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-404405cc-7f8d65bdf7-xtnn2"]
Jul 22 02:16:16.515610 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:16:16.515581    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3cea1d3_db85_4149_bde8_21974e5d4ff6.slice/crio-ce40b361b3e7ea9b46dc212add744ae478e73ca9beda5a9a6b3bfd966fea4b0f WatchSource:0}: Error finding container ce40b361b3e7ea9b46dc212add744ae478e73ca9beda5a9a6b3bfd966fea4b0f: Status 404 returned error can't find the container with id ce40b361b3e7ea9b46dc212add744ae478e73ca9beda5a9a6b3bfd966fea4b0f
Jul 22 02:16:16.951772 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:16.951736    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-404405cc-7f8d65bdf7-xtnn2" event={"ID":"b3cea1d3-db85-4149-bde8-21974e5d4ff6","Type":"ContainerStarted","Data":"2fa295531144b8573410a15e178cf4c389996dac864928c33976e5a4b56beb1e"}
Jul 22 02:16:16.951971 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:16.951800    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-404405cc-7f8d65bdf7-xtnn2" event={"ID":"b3cea1d3-db85-4149-bde8-21974e5d4ff6","Type":"ContainerStarted","Data":"ce40b361b3e7ea9b46dc212add744ae478e73ca9beda5a9a6b3bfd966fea4b0f"}
Jul 22 02:16:16.951971 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:16.951826    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-404405cc-7f8d65bdf7-xtnn2"
Jul 22 02:16:16.969502 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:16.969450    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-404405cc-7f8d65bdf7-xtnn2" podStartSLOduration=0.66278955 podStartE2EDuration="969.431812ms" totalImagesPullingTime="306.642262ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:16:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:16:16.96792745 +0000 UTC m=+1729.811756581" watchObservedRunningTime="2026-07-22 02:16:16.969431812 +0000 UTC m=+1729.813260943"
Jul 22 02:16:22.962635 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:22.962607    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-404405cc-7f8d65bdf7-xtnn2"
Jul 22 02:16:40.833004 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:40.832970    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-404405cc-6v49r"]
Jul 22 02:16:40.836327 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:40.836311    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-404405cc-6v49r"
Jul 22 02:16:40.838745 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:40.838722    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wrtw7\""
Jul 22 02:16:40.844099 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:40.843657    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-404405cc-6v49r"]
Jul 22 02:16:40.930639 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:40.930613    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed76a07c-add4-4730-b1f5-f972a75dd94f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-404405cc-6v49r\" (UID: \"ed76a07c-add4-4730-b1f5-f972a75dd94f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-404405cc-6v49r"
Jul 22 02:16:41.031287 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:41.031259    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed76a07c-add4-4730-b1f5-f972a75dd94f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-404405cc-6v49r\" (UID: \"ed76a07c-add4-4730-b1f5-f972a75dd94f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-404405cc-6v49r"
Jul 22 02:16:41.031808 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:41.031792    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed76a07c-add4-4730-b1f5-f972a75dd94f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-404405cc-6v49r\" (UID: \"ed76a07c-add4-4730-b1f5-f972a75dd94f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-404405cc-6v49r"
Jul 22 02:16:41.147330 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:41.147272    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-404405cc-6v49r"
Jul 22 02:16:41.276204 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:41.276181    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-404405cc-6v49r"]
Jul 22 02:16:41.278519 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:16:41.278494    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded76a07c_add4_4730_b1f5_f972a75dd94f.slice/crio-888b6834417df61c7cd9d80f01209bdbfeb0c6d0bd09328f0b171ccec2d7f76b WatchSource:0}: Error finding container 888b6834417df61c7cd9d80f01209bdbfeb0c6d0bd09328f0b171ccec2d7f76b: Status 404 returned error can't find the container with id 888b6834417df61c7cd9d80f01209bdbfeb0c6d0bd09328f0b171ccec2d7f76b
Jul 22 02:16:42.078762 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:42.078730    2563 generic.go:356] "Generic (PLEG): container finished" podID="ed76a07c-add4-4730-b1f5-f972a75dd94f" containerID="e5bec0542a0462926dfe4714a63bf904705a87da629024b97d15c6a26020d743" exitCode=0
Jul 22 02:16:42.079075 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:42.078785    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-404405cc-6v49r" event={"ID":"ed76a07c-add4-4730-b1f5-f972a75dd94f","Type":"ContainerDied","Data":"e5bec0542a0462926dfe4714a63bf904705a87da629024b97d15c6a26020d743"}
Jul 22 02:16:42.079075 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:42.078850    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-404405cc-6v49r" event={"ID":"ed76a07c-add4-4730-b1f5-f972a75dd94f","Type":"ContainerStarted","Data":"888b6834417df61c7cd9d80f01209bdbfeb0c6d0bd09328f0b171ccec2d7f76b"}
Jul 22 02:16:43.213134 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:43.213110    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-404405cc-6v49r"
Jul 22 02:16:43.246872 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:43.246847    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed76a07c-add4-4730-b1f5-f972a75dd94f-maas-api-service-ca\") pod \"ed76a07c-add4-4730-b1f5-f972a75dd94f\" (UID: \"ed76a07c-add4-4730-b1f5-f972a75dd94f\") "
Jul 22 02:16:43.247180 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:43.247158    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed76a07c-add4-4730-b1f5-f972a75dd94f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ed76a07c-add4-4730-b1f5-f972a75dd94f" (UID: "ed76a07c-add4-4730-b1f5-f972a75dd94f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:16:43.347274 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:43.347225    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed76a07c-add4-4730-b1f5-f972a75dd94f-maas-api-service-ca\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:16:44.090038 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:44.090017    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-404405cc-6v49r"
Jul 22 02:16:44.090151 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:44.090042    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-404405cc-6v49r" event={"ID":"ed76a07c-add4-4730-b1f5-f972a75dd94f","Type":"ContainerDied","Data":"888b6834417df61c7cd9d80f01209bdbfeb0c6d0bd09328f0b171ccec2d7f76b"}
Jul 22 02:16:44.090151 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:44.090094    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="888b6834417df61c7cd9d80f01209bdbfeb0c6d0bd09328f0b171ccec2d7f76b"
Jul 22 02:16:51.377511 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:51.377481    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-404405cc-7f8d65bdf7-xtnn2"]
Jul 22 02:16:51.377944 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:51.377685    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-404405cc-7f8d65bdf7-xtnn2" podUID="b3cea1d3-db85-4149-bde8-21974e5d4ff6" containerName="maas-api" containerID="cri-o://2fa295531144b8573410a15e178cf4c389996dac864928c33976e5a4b56beb1e" gracePeriod=30
Jul 22 02:16:51.620711 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:51.620688    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-404405cc-7f8d65bdf7-xtnn2"
Jul 22 02:16:51.708632 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:51.708608    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2lmxw\" (UniqueName: \"kubernetes.io/projected/b3cea1d3-db85-4149-bde8-21974e5d4ff6-kube-api-access-2lmxw\") pod \"b3cea1d3-db85-4149-bde8-21974e5d4ff6\" (UID: \"b3cea1d3-db85-4149-bde8-21974e5d4ff6\") "
Jul 22 02:16:51.708768 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:51.708657    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3cea1d3-db85-4149-bde8-21974e5d4ff6-openshift-service-ca\") pod \"b3cea1d3-db85-4149-bde8-21974e5d4ff6\" (UID: \"b3cea1d3-db85-4149-bde8-21974e5d4ff6\") "
Jul 22 02:16:51.708768 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:51.708697    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3cea1d3-db85-4149-bde8-21974e5d4ff6-maas-api-tls\") pod \"b3cea1d3-db85-4149-bde8-21974e5d4ff6\" (UID: \"b3cea1d3-db85-4149-bde8-21974e5d4ff6\") "
Jul 22 02:16:51.709066 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:51.709035    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3cea1d3-db85-4149-bde8-21974e5d4ff6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b3cea1d3-db85-4149-bde8-21974e5d4ff6" (UID: "b3cea1d3-db85-4149-bde8-21974e5d4ff6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:16:51.710640 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:51.710612    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3cea1d3-db85-4149-bde8-21974e5d4ff6-kube-api-access-2lmxw" (OuterVolumeSpecName: "kube-api-access-2lmxw") pod "b3cea1d3-db85-4149-bde8-21974e5d4ff6" (UID: "b3cea1d3-db85-4149-bde8-21974e5d4ff6"). InnerVolumeSpecName "kube-api-access-2lmxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:16:51.710735 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:51.710688    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3cea1d3-db85-4149-bde8-21974e5d4ff6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b3cea1d3-db85-4149-bde8-21974e5d4ff6" (UID: "b3cea1d3-db85-4149-bde8-21974e5d4ff6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:16:51.809636 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:51.809612    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lmxw\" (UniqueName: \"kubernetes.io/projected/b3cea1d3-db85-4149-bde8-21974e5d4ff6-kube-api-access-2lmxw\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:16:51.809636 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:51.809631    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3cea1d3-db85-4149-bde8-21974e5d4ff6-openshift-service-ca\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:16:51.809768 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:51.809641    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3cea1d3-db85-4149-bde8-21974e5d4ff6-maas-api-tls\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:16:52.129583 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:52.129516    2563 generic.go:356] "Generic (PLEG): container finished" podID="b3cea1d3-db85-4149-bde8-21974e5d4ff6" containerID="2fa295531144b8573410a15e178cf4c389996dac864928c33976e5a4b56beb1e" exitCode=0
Jul 22 02:16:52.129583 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:52.129560    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-404405cc-7f8d65bdf7-xtnn2"
Jul 22 02:16:52.129774 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:52.129598    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-404405cc-7f8d65bdf7-xtnn2" event={"ID":"b3cea1d3-db85-4149-bde8-21974e5d4ff6","Type":"ContainerDied","Data":"2fa295531144b8573410a15e178cf4c389996dac864928c33976e5a4b56beb1e"}
Jul 22 02:16:52.129774 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:52.129643    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-404405cc-7f8d65bdf7-xtnn2" event={"ID":"b3cea1d3-db85-4149-bde8-21974e5d4ff6","Type":"ContainerDied","Data":"ce40b361b3e7ea9b46dc212add744ae478e73ca9beda5a9a6b3bfd966fea4b0f"}
Jul 22 02:16:52.129774 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:52.129663    2563 scope.go:117] "RemoveContainer" containerID="2fa295531144b8573410a15e178cf4c389996dac864928c33976e5a4b56beb1e"
Jul 22 02:16:52.138938 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:52.138924    2563 scope.go:117] "RemoveContainer" containerID="2fa295531144b8573410a15e178cf4c389996dac864928c33976e5a4b56beb1e"
Jul 22 02:16:52.139195 ip-10-0-135-230 kubenswrapper[2563]: E0722 02:16:52.139178    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fa295531144b8573410a15e178cf4c389996dac864928c33976e5a4b56beb1e\": container with ID starting with 2fa295531144b8573410a15e178cf4c389996dac864928c33976e5a4b56beb1e not found: ID does not exist" containerID="2fa295531144b8573410a15e178cf4c389996dac864928c33976e5a4b56beb1e"
Jul 22 02:16:52.139241 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:52.139204    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fa295531144b8573410a15e178cf4c389996dac864928c33976e5a4b56beb1e"} err="failed to get container status \"2fa295531144b8573410a15e178cf4c389996dac864928c33976e5a4b56beb1e\": rpc error: code = NotFound desc = could not find container \"2fa295531144b8573410a15e178cf4c389996dac864928c33976e5a4b56beb1e\": container with ID starting with 2fa295531144b8573410a15e178cf4c389996dac864928c33976e5a4b56beb1e not found: ID does not exist"
Jul 22 02:16:52.150915 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:52.150896    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-404405cc-7f8d65bdf7-xtnn2"]
Jul 22 02:16:52.154972 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:52.154952    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-404405cc-7f8d65bdf7-xtnn2"]
Jul 22 02:16:53.663241 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:53.663209    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3cea1d3-db85-4149-bde8-21974e5d4ff6" path="/var/lib/kubelet/pods/b3cea1d3-db85-4149-bde8-21974e5d4ff6/volumes"
Jul 22 02:16:55.996018 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:55.995987    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-404405cc-6v49r"]
Jul 22 02:16:55.999107 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:55.999082    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-404405cc-6v49r"]
Jul 22 02:16:57.663402 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:16:57.663340    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed76a07c-add4-4730-b1f5-f972a75dd94f" path="/var/lib/kubelet/pods/ed76a07c-add4-4730-b1f5-f972a75dd94f/volumes"
Jul 22 02:18:44.534984 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:44.534950    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f050ef22-689954fd58-h7pqm"]
Jul 22 02:18:44.535561 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:44.535452    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed76a07c-add4-4730-b1f5-f972a75dd94f" containerName="revoke-keys"
Jul 22 02:18:44.535561 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:44.535472    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed76a07c-add4-4730-b1f5-f972a75dd94f" containerName="revoke-keys"
Jul 22 02:18:44.535561 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:44.535504    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3cea1d3-db85-4149-bde8-21974e5d4ff6" containerName="maas-api"
Jul 22 02:18:44.535561 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:44.535513    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3cea1d3-db85-4149-bde8-21974e5d4ff6" containerName="maas-api"
Jul 22 02:18:44.535809 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:44.535596    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed76a07c-add4-4730-b1f5-f972a75dd94f" containerName="revoke-keys"
Jul 22 02:18:44.535809 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:44.535613    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3cea1d3-db85-4149-bde8-21974e5d4ff6" containerName="maas-api"
Jul 22 02:18:44.538668 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:44.538650    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f050ef22-689954fd58-h7pqm"
Jul 22 02:18:44.541436 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:44.541417    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f050ef22-serving-cert\""
Jul 22 02:18:44.549547 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:44.549523    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f050ef22-689954fd58-h7pqm"]
Jul 22 02:18:44.572748 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:44.572719    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0018aa11-cc38-46d2-95ca-5d3ffaa53cd3-maas-api-tls\") pod \"maas-api-e2e-mt-f050ef22-689954fd58-h7pqm\" (UID: \"0018aa11-cc38-46d2-95ca-5d3ffaa53cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f050ef22-689954fd58-h7pqm"
Jul 22 02:18:44.572911 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:44.572756    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsh8c\" (UniqueName: \"kubernetes.io/projected/0018aa11-cc38-46d2-95ca-5d3ffaa53cd3-kube-api-access-rsh8c\") pod \"maas-api-e2e-mt-f050ef22-689954fd58-h7pqm\" (UID: \"0018aa11-cc38-46d2-95ca-5d3ffaa53cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f050ef22-689954fd58-h7pqm"
Jul 22 02:18:44.572911 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:44.572798    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0018aa11-cc38-46d2-95ca-5d3ffaa53cd3-openshift-service-ca\") pod \"maas-api-e2e-mt-f050ef22-689954fd58-h7pqm\" (UID: \"0018aa11-cc38-46d2-95ca-5d3ffaa53cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f050ef22-689954fd58-h7pqm"
Jul 22 02:18:44.673131 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:44.673101    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0018aa11-cc38-46d2-95ca-5d3ffaa53cd3-openshift-service-ca\") pod \"maas-api-e2e-mt-f050ef22-689954fd58-h7pqm\" (UID: \"0018aa11-cc38-46d2-95ca-5d3ffaa53cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f050ef22-689954fd58-h7pqm"
Jul 22 02:18:44.673293 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:44.673180    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0018aa11-cc38-46d2-95ca-5d3ffaa53cd3-maas-api-tls\") pod \"maas-api-e2e-mt-f050ef22-689954fd58-h7pqm\" (UID: \"0018aa11-cc38-46d2-95ca-5d3ffaa53cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f050ef22-689954fd58-h7pqm"
Jul 22 02:18:44.673293 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:44.673216    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsh8c\" (UniqueName: \"kubernetes.io/projected/0018aa11-cc38-46d2-95ca-5d3ffaa53cd3-kube-api-access-rsh8c\") pod \"maas-api-e2e-mt-f050ef22-689954fd58-h7pqm\" (UID: \"0018aa11-cc38-46d2-95ca-5d3ffaa53cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f050ef22-689954fd58-h7pqm"
Jul 22 02:18:44.673421 ip-10-0-135-230 kubenswrapper[2563]: E0722 02:18:44.673348    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-f050ef22-serving-cert: secret "maas-api-e2e-mt-f050ef22-serving-cert" not found
Jul 22 02:18:44.673488 ip-10-0-135-230 kubenswrapper[2563]: E0722 02:18:44.673450    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0018aa11-cc38-46d2-95ca-5d3ffaa53cd3-maas-api-tls podName:0018aa11-cc38-46d2-95ca-5d3ffaa53cd3 nodeName:}" failed. No retries permitted until 2026-07-22 02:18:45.173426375 +0000 UTC m=+1878.017255489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0018aa11-cc38-46d2-95ca-5d3ffaa53cd3-maas-api-tls") pod "maas-api-e2e-mt-f050ef22-689954fd58-h7pqm" (UID: "0018aa11-cc38-46d2-95ca-5d3ffaa53cd3") : secret "maas-api-e2e-mt-f050ef22-serving-cert" not found
Jul 22 02:18:44.673860 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:44.673842    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0018aa11-cc38-46d2-95ca-5d3ffaa53cd3-openshift-service-ca\") pod \"maas-api-e2e-mt-f050ef22-689954fd58-h7pqm\" (UID: \"0018aa11-cc38-46d2-95ca-5d3ffaa53cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f050ef22-689954fd58-h7pqm"
Jul 22 02:18:44.686729 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:44.686709    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsh8c\" (UniqueName: \"kubernetes.io/projected/0018aa11-cc38-46d2-95ca-5d3ffaa53cd3-kube-api-access-rsh8c\") pod \"maas-api-e2e-mt-f050ef22-689954fd58-h7pqm\" (UID: \"0018aa11-cc38-46d2-95ca-5d3ffaa53cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f050ef22-689954fd58-h7pqm"
Jul 22 02:18:45.175828 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:45.175798    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0018aa11-cc38-46d2-95ca-5d3ffaa53cd3-maas-api-tls\") pod \"maas-api-e2e-mt-f050ef22-689954fd58-h7pqm\" (UID: \"0018aa11-cc38-46d2-95ca-5d3ffaa53cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f050ef22-689954fd58-h7pqm"
Jul 22 02:18:45.178177 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:45.178154    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0018aa11-cc38-46d2-95ca-5d3ffaa53cd3-maas-api-tls\") pod \"maas-api-e2e-mt-f050ef22-689954fd58-h7pqm\" (UID: \"0018aa11-cc38-46d2-95ca-5d3ffaa53cd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f050ef22-689954fd58-h7pqm"
Jul 22 02:18:45.450619 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:45.450552    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f050ef22-689954fd58-h7pqm"
Jul 22 02:18:45.581671 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:45.581645    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f050ef22-689954fd58-h7pqm"]
Jul 22 02:18:45.584067 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:18:45.584026    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0018aa11_cc38_46d2_95ca_5d3ffaa53cd3.slice/crio-460eb7abd63b900b3f4021f6d0f4b09a5ae6c7ab14b293acbc538ddb911c47b5 WatchSource:0}: Error finding container 460eb7abd63b900b3f4021f6d0f4b09a5ae6c7ab14b293acbc538ddb911c47b5: Status 404 returned error can't find the container with id 460eb7abd63b900b3f4021f6d0f4b09a5ae6c7ab14b293acbc538ddb911c47b5
Jul 22 02:18:45.748669 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:45.748599    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f050ef22-689954fd58-h7pqm" event={"ID":"0018aa11-cc38-46d2-95ca-5d3ffaa53cd3","Type":"ContainerStarted","Data":"460eb7abd63b900b3f4021f6d0f4b09a5ae6c7ab14b293acbc538ddb911c47b5"}
Jul 22 02:18:46.755172 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:46.755129    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f050ef22-689954fd58-h7pqm" event={"ID":"0018aa11-cc38-46d2-95ca-5d3ffaa53cd3","Type":"ContainerStarted","Data":"4a79db98e6909eba9f6d4096e6ee09c83fbaeac9a28e59c1dffff0fedb1c594f"}
Jul 22 02:18:46.755558 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:46.755200    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f050ef22-689954fd58-h7pqm"
Jul 22 02:18:46.784141 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:46.784096    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f050ef22-689954fd58-h7pqm" podStartSLOduration=2.342640503 podStartE2EDuration="2.784081217s" totalImagesPullingTime="441.440714ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:18:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:18:46.779139472 +0000 UTC m=+1879.622968605" watchObservedRunningTime="2026-07-22 02:18:46.784081217 +0000 UTC m=+1879.627910402"
Jul 22 02:18:52.765587 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:52.765514    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f050ef22-689954fd58-h7pqm"
Jul 22 02:18:56.461654 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:56.461622    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f050ef22-h5vxw"]
Jul 22 02:18:56.465184 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:56.465167    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f050ef22-h5vxw"
Jul 22 02:18:56.467625 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:56.467609    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wrtw7\""
Jul 22 02:18:56.474879 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:56.474855    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f050ef22-h5vxw"]
Jul 22 02:18:56.558765 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:56.558741    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c56bb2-9cd8-4628-acdd-91f85281c0cd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f050ef22-h5vxw\" (UID: \"d9c56bb2-9cd8-4628-acdd-91f85281c0cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f050ef22-h5vxw"
Jul 22 02:18:56.659838 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:56.659811    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c56bb2-9cd8-4628-acdd-91f85281c0cd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f050ef22-h5vxw\" (UID: \"d9c56bb2-9cd8-4628-acdd-91f85281c0cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f050ef22-h5vxw"
Jul 22 02:18:56.660358 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:56.660341    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c56bb2-9cd8-4628-acdd-91f85281c0cd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f050ef22-h5vxw\" (UID: \"d9c56bb2-9cd8-4628-acdd-91f85281c0cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f050ef22-h5vxw"
Jul 22 02:18:56.775567 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:56.775509    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f050ef22-h5vxw"
Jul 22 02:18:56.903278 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:56.903249    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f050ef22-h5vxw"]
Jul 22 02:18:56.904881 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:18:56.904855    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9c56bb2_9cd8_4628_acdd_91f85281c0cd.slice/crio-a98905de12dd48dd91da5fdce283d8ff3c42d9201f28859e2df69d880770817f WatchSource:0}: Error finding container a98905de12dd48dd91da5fdce283d8ff3c42d9201f28859e2df69d880770817f: Status 404 returned error can't find the container with id a98905de12dd48dd91da5fdce283d8ff3c42d9201f28859e2df69d880770817f
Jul 22 02:18:57.814506 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:57.814479    2563 generic.go:356] "Generic (PLEG): container finished" podID="d9c56bb2-9cd8-4628-acdd-91f85281c0cd" containerID="a56994ca1ba0bfc0b7fc7a6ad5e2f4a0a8a3092790e874810d5033d6146237aa" exitCode=0
Jul 22 02:18:57.814793 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:57.814565    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f050ef22-h5vxw" event={"ID":"d9c56bb2-9cd8-4628-acdd-91f85281c0cd","Type":"ContainerDied","Data":"a56994ca1ba0bfc0b7fc7a6ad5e2f4a0a8a3092790e874810d5033d6146237aa"}
Jul 22 02:18:57.814793 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:57.814611    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f050ef22-h5vxw" event={"ID":"d9c56bb2-9cd8-4628-acdd-91f85281c0cd","Type":"ContainerStarted","Data":"a98905de12dd48dd91da5fdce283d8ff3c42d9201f28859e2df69d880770817f"}
Jul 22 02:18:58.947408 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:58.947371    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f050ef22-h5vxw"
Jul 22 02:18:58.978725 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:58.978701    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c56bb2-9cd8-4628-acdd-91f85281c0cd-maas-api-service-ca\") pod \"d9c56bb2-9cd8-4628-acdd-91f85281c0cd\" (UID: \"d9c56bb2-9cd8-4628-acdd-91f85281c0cd\") "
Jul 22 02:18:58.979010 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:58.978985    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9c56bb2-9cd8-4628-acdd-91f85281c0cd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d9c56bb2-9cd8-4628-acdd-91f85281c0cd" (UID: "d9c56bb2-9cd8-4628-acdd-91f85281c0cd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:18:59.079487 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:59.079441    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c56bb2-9cd8-4628-acdd-91f85281c0cd-maas-api-service-ca\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:18:59.824998 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:59.824956    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f050ef22-h5vxw"
Jul 22 02:18:59.825115 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:59.825015    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f050ef22-h5vxw" event={"ID":"d9c56bb2-9cd8-4628-acdd-91f85281c0cd","Type":"ContainerDied","Data":"a98905de12dd48dd91da5fdce283d8ff3c42d9201f28859e2df69d880770817f"}
Jul 22 02:18:59.825115 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:18:59.825066    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a98905de12dd48dd91da5fdce283d8ff3c42d9201f28859e2df69d880770817f"
Jul 22 02:19:06.997007 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:06.996974    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f050ef22-689954fd58-h7pqm"]
Jul 22 02:19:06.997464 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:06.997184    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f050ef22-689954fd58-h7pqm" podUID="0018aa11-cc38-46d2-95ca-5d3ffaa53cd3" containerName="maas-api" containerID="cri-o://4a79db98e6909eba9f6d4096e6ee09c83fbaeac9a28e59c1dffff0fedb1c594f" gracePeriod=30
Jul 22 02:19:07.239897 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:07.239876    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f050ef22-689954fd58-h7pqm"
Jul 22 02:19:07.343477 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:07.343370    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0018aa11-cc38-46d2-95ca-5d3ffaa53cd3-maas-api-tls\") pod \"0018aa11-cc38-46d2-95ca-5d3ffaa53cd3\" (UID: \"0018aa11-cc38-46d2-95ca-5d3ffaa53cd3\") "
Jul 22 02:19:07.343477 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:07.343443    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rsh8c\" (UniqueName: \"kubernetes.io/projected/0018aa11-cc38-46d2-95ca-5d3ffaa53cd3-kube-api-access-rsh8c\") pod \"0018aa11-cc38-46d2-95ca-5d3ffaa53cd3\" (UID: \"0018aa11-cc38-46d2-95ca-5d3ffaa53cd3\") "
Jul 22 02:19:07.343664 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:07.343505    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0018aa11-cc38-46d2-95ca-5d3ffaa53cd3-openshift-service-ca\") pod \"0018aa11-cc38-46d2-95ca-5d3ffaa53cd3\" (UID: \"0018aa11-cc38-46d2-95ca-5d3ffaa53cd3\") "
Jul 22 02:19:07.343882 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:07.343852    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0018aa11-cc38-46d2-95ca-5d3ffaa53cd3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0018aa11-cc38-46d2-95ca-5d3ffaa53cd3" (UID: "0018aa11-cc38-46d2-95ca-5d3ffaa53cd3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:19:07.345354 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:07.345330    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0018aa11-cc38-46d2-95ca-5d3ffaa53cd3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0018aa11-cc38-46d2-95ca-5d3ffaa53cd3" (UID: "0018aa11-cc38-46d2-95ca-5d3ffaa53cd3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:19:07.345480 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:07.345458    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0018aa11-cc38-46d2-95ca-5d3ffaa53cd3-kube-api-access-rsh8c" (OuterVolumeSpecName: "kube-api-access-rsh8c") pod "0018aa11-cc38-46d2-95ca-5d3ffaa53cd3" (UID: "0018aa11-cc38-46d2-95ca-5d3ffaa53cd3"). InnerVolumeSpecName "kube-api-access-rsh8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:19:07.444879 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:07.444853    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0018aa11-cc38-46d2-95ca-5d3ffaa53cd3-openshift-service-ca\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:19:07.444879 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:07.444876    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0018aa11-cc38-46d2-95ca-5d3ffaa53cd3-maas-api-tls\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:19:07.445040 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:07.444887    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsh8c\" (UniqueName: \"kubernetes.io/projected/0018aa11-cc38-46d2-95ca-5d3ffaa53cd3-kube-api-access-rsh8c\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:19:07.864009 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:07.863973    2563 generic.go:356] "Generic (PLEG): container finished" podID="0018aa11-cc38-46d2-95ca-5d3ffaa53cd3" containerID="4a79db98e6909eba9f6d4096e6ee09c83fbaeac9a28e59c1dffff0fedb1c594f" exitCode=0
Jul 22 02:19:07.864159 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:07.864020    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f050ef22-689954fd58-h7pqm"
Jul 22 02:19:07.864159 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:07.864054    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f050ef22-689954fd58-h7pqm" event={"ID":"0018aa11-cc38-46d2-95ca-5d3ffaa53cd3","Type":"ContainerDied","Data":"4a79db98e6909eba9f6d4096e6ee09c83fbaeac9a28e59c1dffff0fedb1c594f"}
Jul 22 02:19:07.864159 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:07.864105    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f050ef22-689954fd58-h7pqm" event={"ID":"0018aa11-cc38-46d2-95ca-5d3ffaa53cd3","Type":"ContainerDied","Data":"460eb7abd63b900b3f4021f6d0f4b09a5ae6c7ab14b293acbc538ddb911c47b5"}
Jul 22 02:19:07.864159 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:07.864126    2563 scope.go:117] "RemoveContainer" containerID="4a79db98e6909eba9f6d4096e6ee09c83fbaeac9a28e59c1dffff0fedb1c594f"
Jul 22 02:19:07.872922 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:07.872898    2563 scope.go:117] "RemoveContainer" containerID="4a79db98e6909eba9f6d4096e6ee09c83fbaeac9a28e59c1dffff0fedb1c594f"
Jul 22 02:19:07.873201 ip-10-0-135-230 kubenswrapper[2563]: E0722 02:19:07.873173    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a79db98e6909eba9f6d4096e6ee09c83fbaeac9a28e59c1dffff0fedb1c594f\": container with ID starting with 4a79db98e6909eba9f6d4096e6ee09c83fbaeac9a28e59c1dffff0fedb1c594f not found: ID does not exist" containerID="4a79db98e6909eba9f6d4096e6ee09c83fbaeac9a28e59c1dffff0fedb1c594f"
Jul 22 02:19:07.873249 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:07.873209    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a79db98e6909eba9f6d4096e6ee09c83fbaeac9a28e59c1dffff0fedb1c594f"} err="failed to get container status \"4a79db98e6909eba9f6d4096e6ee09c83fbaeac9a28e59c1dffff0fedb1c594f\": rpc error: code = NotFound desc = could not find container \"4a79db98e6909eba9f6d4096e6ee09c83fbaeac9a28e59c1dffff0fedb1c594f\": container with ID starting with 4a79db98e6909eba9f6d4096e6ee09c83fbaeac9a28e59c1dffff0fedb1c594f not found: ID does not exist"
Jul 22 02:19:07.880611 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:07.880589    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f050ef22-689954fd58-h7pqm"]
Jul 22 02:19:07.886094 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:07.886069    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f050ef22-689954fd58-h7pqm"]
Jul 22 02:19:09.663200 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:09.663170    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0018aa11-cc38-46d2-95ca-5d3ffaa53cd3" path="/var/lib/kubelet/pods/0018aa11-cc38-46d2-95ca-5d3ffaa53cd3/volumes"
Jul 22 02:19:11.539564 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:11.539533    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f050ef22-h5vxw"]
Jul 22 02:19:11.542826 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:11.542801    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f050ef22-h5vxw"]
Jul 22 02:19:11.663678 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:11.663648    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9c56bb2-9cd8-4628-acdd-91f85281c0cd" path="/var/lib/kubelet/pods/d9c56bb2-9cd8-4628-acdd-91f85281c0cd/volumes"
Jul 22 02:19:41.134793 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:41.134762    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4f16202b-5d85df9b5-854l4"]
Jul 22 02:19:41.135369 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:41.135347    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9c56bb2-9cd8-4628-acdd-91f85281c0cd" containerName="revoke-keys"
Jul 22 02:19:41.135369 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:41.135371    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9c56bb2-9cd8-4628-acdd-91f85281c0cd" containerName="revoke-keys"
Jul 22 02:19:41.135543 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:41.135410    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0018aa11-cc38-46d2-95ca-5d3ffaa53cd3" containerName="maas-api"
Jul 22 02:19:41.135543 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:41.135418    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0018aa11-cc38-46d2-95ca-5d3ffaa53cd3" containerName="maas-api"
Jul 22 02:19:41.135543 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:41.135504    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0018aa11-cc38-46d2-95ca-5d3ffaa53cd3" containerName="maas-api"
Jul 22 02:19:41.135543 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:41.135515    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9c56bb2-9cd8-4628-acdd-91f85281c0cd" containerName="revoke-keys"
Jul 22 02:19:41.138501 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:41.138478    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4f16202b-5d85df9b5-854l4"
Jul 22 02:19:41.140699 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:41.140674    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-4f16202b-serving-cert\""
Jul 22 02:19:41.149056 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:41.149033    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4f16202b-5d85df9b5-854l4"]
Jul 22 02:19:41.323634 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:41.323595    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26fd9d67-dec9-4ade-b368-47b05b152a1e-openshift-service-ca\") pod \"maas-api-e2e-mt-4f16202b-5d85df9b5-854l4\" (UID: \"26fd9d67-dec9-4ade-b368-47b05b152a1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4f16202b-5d85df9b5-854l4"
Jul 22 02:19:41.323634 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:41.323632    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9957f\" (UniqueName: \"kubernetes.io/projected/26fd9d67-dec9-4ade-b368-47b05b152a1e-kube-api-access-9957f\") pod \"maas-api-e2e-mt-4f16202b-5d85df9b5-854l4\" (UID: \"26fd9d67-dec9-4ade-b368-47b05b152a1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4f16202b-5d85df9b5-854l4"
Jul 22 02:19:41.323867 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:41.323654    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26fd9d67-dec9-4ade-b368-47b05b152a1e-maas-api-tls\") pod \"maas-api-e2e-mt-4f16202b-5d85df9b5-854l4\" (UID: \"26fd9d67-dec9-4ade-b368-47b05b152a1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4f16202b-5d85df9b5-854l4"
Jul 22 02:19:41.424858 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:41.424835    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26fd9d67-dec9-4ade-b368-47b05b152a1e-openshift-service-ca\") pod \"maas-api-e2e-mt-4f16202b-5d85df9b5-854l4\" (UID: \"26fd9d67-dec9-4ade-b368-47b05b152a1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4f16202b-5d85df9b5-854l4"
Jul 22 02:19:41.425007 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:41.424867    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9957f\" (UniqueName: \"kubernetes.io/projected/26fd9d67-dec9-4ade-b368-47b05b152a1e-kube-api-access-9957f\") pod \"maas-api-e2e-mt-4f16202b-5d85df9b5-854l4\" (UID: \"26fd9d67-dec9-4ade-b368-47b05b152a1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4f16202b-5d85df9b5-854l4"
Jul 22 02:19:41.425007 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:41.424891    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26fd9d67-dec9-4ade-b368-47b05b152a1e-maas-api-tls\") pod \"maas-api-e2e-mt-4f16202b-5d85df9b5-854l4\" (UID: \"26fd9d67-dec9-4ade-b368-47b05b152a1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4f16202b-5d85df9b5-854l4"
Jul 22 02:19:41.425007 ip-10-0-135-230 kubenswrapper[2563]: E0722 02:19:41.424999    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-4f16202b-serving-cert: secret "maas-api-e2e-mt-4f16202b-serving-cert" not found
Jul 22 02:19:41.425130 ip-10-0-135-230 kubenswrapper[2563]: E0722 02:19:41.425070    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26fd9d67-dec9-4ade-b368-47b05b152a1e-maas-api-tls podName:26fd9d67-dec9-4ade-b368-47b05b152a1e nodeName:}" failed. No retries permitted until 2026-07-22 02:19:41.925051241 +0000 UTC m=+1934.768880350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/26fd9d67-dec9-4ade-b368-47b05b152a1e-maas-api-tls") pod "maas-api-e2e-mt-4f16202b-5d85df9b5-854l4" (UID: "26fd9d67-dec9-4ade-b368-47b05b152a1e") : secret "maas-api-e2e-mt-4f16202b-serving-cert" not found
Jul 22 02:19:41.425663 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:41.425640    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26fd9d67-dec9-4ade-b368-47b05b152a1e-openshift-service-ca\") pod \"maas-api-e2e-mt-4f16202b-5d85df9b5-854l4\" (UID: \"26fd9d67-dec9-4ade-b368-47b05b152a1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4f16202b-5d85df9b5-854l4"
Jul 22 02:19:41.433509 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:41.433484    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9957f\" (UniqueName: \"kubernetes.io/projected/26fd9d67-dec9-4ade-b368-47b05b152a1e-kube-api-access-9957f\") pod \"maas-api-e2e-mt-4f16202b-5d85df9b5-854l4\" (UID: \"26fd9d67-dec9-4ade-b368-47b05b152a1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4f16202b-5d85df9b5-854l4"
Jul 22 02:19:41.929489 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:41.929456    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26fd9d67-dec9-4ade-b368-47b05b152a1e-maas-api-tls\") pod \"maas-api-e2e-mt-4f16202b-5d85df9b5-854l4\" (UID: \"26fd9d67-dec9-4ade-b368-47b05b152a1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4f16202b-5d85df9b5-854l4"
Jul 22 02:19:41.931811 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:41.931787    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26fd9d67-dec9-4ade-b368-47b05b152a1e-maas-api-tls\") pod \"maas-api-e2e-mt-4f16202b-5d85df9b5-854l4\" (UID: \"26fd9d67-dec9-4ade-b368-47b05b152a1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4f16202b-5d85df9b5-854l4"
Jul 22 02:19:42.049239 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:42.049216    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4f16202b-5d85df9b5-854l4"
Jul 22 02:19:42.177010 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:42.176986    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4f16202b-5d85df9b5-854l4"]
Jul 22 02:19:42.178706 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:19:42.178669    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26fd9d67_dec9_4ade_b368_47b05b152a1e.slice/crio-df15c5f75c3682250fff13c1ee3443e8a68fe8d4eba04b93410de82fc40dcabf WatchSource:0}: Error finding container df15c5f75c3682250fff13c1ee3443e8a68fe8d4eba04b93410de82fc40dcabf: Status 404 returned error can't find the container with id df15c5f75c3682250fff13c1ee3443e8a68fe8d4eba04b93410de82fc40dcabf
Jul 22 02:19:42.180027 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:42.179984    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 22 02:19:43.037770 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:43.037725    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4f16202b-5d85df9b5-854l4" event={"ID":"26fd9d67-dec9-4ade-b368-47b05b152a1e","Type":"ContainerStarted","Data":"30578ea8ab428f5a52cdbadaf2eb015bae3fb18e581cf4881fc6c91549107295"}
Jul 22 02:19:43.037947 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:43.037897    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4f16202b-5d85df9b5-854l4"
Jul 22 02:19:43.037947 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:43.037939    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4f16202b-5d85df9b5-854l4" event={"ID":"26fd9d67-dec9-4ade-b368-47b05b152a1e","Type":"ContainerStarted","Data":"df15c5f75c3682250fff13c1ee3443e8a68fe8d4eba04b93410de82fc40dcabf"}
Jul 22 02:19:43.054450 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:43.054404    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4f16202b-5d85df9b5-854l4" podStartSLOduration=1.577853743 podStartE2EDuration="2.054369002s" totalImagesPullingTime="476.515259ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:19:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:19:43.053494953 +0000 UTC m=+1935.897324084" watchObservedRunningTime="2026-07-22 02:19:43.054369002 +0000 UTC m=+1935.898198131"
Jul 22 02:19:49.048946 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:49.048900    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4f16202b-5d85df9b5-854l4"
Jul 22 02:19:55.961009 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:55.960980    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-4f16202b/e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv"]
Jul 22 02:19:55.964689 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:55.964666    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4f16202b/e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv"
Jul 22 02:19:55.967541 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:55.967518    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-4f16202b\"/\"openshift-service-ca.crt\""
Jul 22 02:19:55.968731 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:55.968710    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-4f16202b\"/\"default-dockercfg-mmz8r\""
Jul 22 02:19:55.968943 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:55.968765    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-4f16202b\"/\"kube-root-ca.crt\""
Jul 22 02:19:55.969081 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:55.968767    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-4f16202b\"/\"e2e-lifecycle-model-4f16202b-kserve-self-signed-certs\""
Jul 22 02:19:55.972899 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:55.972878    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-4f16202b/e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv"]
Jul 22 02:19:56.032387 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:56.032342    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85441672-38c0-4c4f-8ebf-c2d1caac406a-tls-certs\") pod \"e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv\" (UID: \"85441672-38c0-4c4f-8ebf-c2d1caac406a\") " pod="ai-tenant-e2e-mt-4f16202b/e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv"
Jul 22 02:19:56.032556 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:56.032459    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85441672-38c0-4c4f-8ebf-c2d1caac406a-tmp-dir\") pod \"e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv\" (UID: \"85441672-38c0-4c4f-8ebf-c2d1caac406a\") " pod="ai-tenant-e2e-mt-4f16202b/e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv"
Jul 22 02:19:56.032556 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:56.032491    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dhzk\" (UniqueName: \"kubernetes.io/projected/85441672-38c0-4c4f-8ebf-c2d1caac406a-kube-api-access-4dhzk\") pod \"e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv\" (UID: \"85441672-38c0-4c4f-8ebf-c2d1caac406a\") " pod="ai-tenant-e2e-mt-4f16202b/e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv"
Jul 22 02:19:56.032556 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:56.032518    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85441672-38c0-4c4f-8ebf-c2d1caac406a-model-cache\") pod \"e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv\" (UID: \"85441672-38c0-4c4f-8ebf-c2d1caac406a\") " pod="ai-tenant-e2e-mt-4f16202b/e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv"
Jul 22 02:19:56.032556 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:56.032542    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85441672-38c0-4c4f-8ebf-c2d1caac406a-dshm\") pod \"e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv\" (UID: \"85441672-38c0-4c4f-8ebf-c2d1caac406a\") " pod="ai-tenant-e2e-mt-4f16202b/e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv"
Jul 22 02:19:56.032746 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:56.032660    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85441672-38c0-4c4f-8ebf-c2d1caac406a-home\") pod \"e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv\" (UID: \"85441672-38c0-4c4f-8ebf-c2d1caac406a\") " pod="ai-tenant-e2e-mt-4f16202b/e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv"
Jul 22 02:19:56.134004 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:56.133976    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85441672-38c0-4c4f-8ebf-c2d1caac406a-tmp-dir\") pod \"e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv\" (UID: \"85441672-38c0-4c4f-8ebf-c2d1caac406a\") " pod="ai-tenant-e2e-mt-4f16202b/e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv"
Jul 22 02:19:56.134116 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:56.134020    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dhzk\" (UniqueName: \"kubernetes.io/projected/85441672-38c0-4c4f-8ebf-c2d1caac406a-kube-api-access-4dhzk\") pod \"e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv\" (UID: \"85441672-38c0-4c4f-8ebf-c2d1caac406a\") " pod="ai-tenant-e2e-mt-4f16202b/e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv"
Jul 22 02:19:56.134116 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:56.134051    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85441672-38c0-4c4f-8ebf-c2d1caac406a-model-cache\") pod \"e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv\" (UID: \"85441672-38c0-4c4f-8ebf-c2d1caac406a\") " pod="ai-tenant-e2e-mt-4f16202b/e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv"
Jul 22 02:19:56.134116 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:56.134076    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85441672-38c0-4c4f-8ebf-c2d1caac406a-dshm\") pod \"e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv\" (UID: \"85441672-38c0-4c4f-8ebf-c2d1caac406a\") " pod="ai-tenant-e2e-mt-4f16202b/e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv"
Jul 22 02:19:56.134371 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:56.134115    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85441672-38c0-4c4f-8ebf-c2d1caac406a-home\") pod \"e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv\" (UID: \"85441672-38c0-4c4f-8ebf-c2d1caac406a\") " pod="ai-tenant-e2e-mt-4f16202b/e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv"
Jul 22 02:19:56.134371 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:56.134173    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85441672-38c0-4c4f-8ebf-c2d1caac406a-tls-certs\") pod \"e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv\" (UID: \"85441672-38c0-4c4f-8ebf-c2d1caac406a\") " pod="ai-tenant-e2e-mt-4f16202b/e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv"
Jul 22 02:19:56.134532 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:56.134412    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85441672-38c0-4c4f-8ebf-c2d1caac406a-tmp-dir\") pod \"e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv\" (UID: \"85441672-38c0-4c4f-8ebf-c2d1caac406a\") " pod="ai-tenant-e2e-mt-4f16202b/e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv"
Jul 22 02:19:56.134532 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:56.134498    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85441672-38c0-4c4f-8ebf-c2d1caac406a-model-cache\") pod \"e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv\" (UID: \"85441672-38c0-4c4f-8ebf-c2d1caac406a\") " pod="ai-tenant-e2e-mt-4f16202b/e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv"
Jul 22 02:19:56.134632 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:56.134541    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85441672-38c0-4c4f-8ebf-c2d1caac406a-home\") pod \"e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv\" (UID: \"85441672-38c0-4c4f-8ebf-c2d1caac406a\") " pod="ai-tenant-e2e-mt-4f16202b/e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv"
Jul 22 02:19:56.136869 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:56.136842    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85441672-38c0-4c4f-8ebf-c2d1caac406a-dshm\") pod \"e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv\" (UID: \"85441672-38c0-4c4f-8ebf-c2d1caac406a\") " pod="ai-tenant-e2e-mt-4f16202b/e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv"
Jul 22 02:19:56.137094 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:56.137080    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85441672-38c0-4c4f-8ebf-c2d1caac406a-tls-certs\") pod \"e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv\" (UID: \"85441672-38c0-4c4f-8ebf-c2d1caac406a\") " pod="ai-tenant-e2e-mt-4f16202b/e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv"
Jul 22 02:19:56.142054 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:56.142034    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dhzk\" (UniqueName: \"kubernetes.io/projected/85441672-38c0-4c4f-8ebf-c2d1caac406a-kube-api-access-4dhzk\") pod \"e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv\" (UID: \"85441672-38c0-4c4f-8ebf-c2d1caac406a\") " pod="ai-tenant-e2e-mt-4f16202b/e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv"
Jul 22 02:19:56.275271 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:56.275220    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4f16202b/e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv"
Jul 22 02:19:56.402183 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:56.402098    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-4f16202b/e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv"]
Jul 22 02:19:56.404584 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:19:56.404554    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85441672_38c0_4c4f_8ebf_c2d1caac406a.slice/crio-b5ed27572f1945aafe63d3c2949e0fce1127235391dcec24a2806039ec2497a7 WatchSource:0}: Error finding container b5ed27572f1945aafe63d3c2949e0fce1127235391dcec24a2806039ec2497a7: Status 404 returned error can't find the container with id b5ed27572f1945aafe63d3c2949e0fce1127235391dcec24a2806039ec2497a7
Jul 22 02:19:57.109037 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:57.109002    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4f16202b/e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv" event={"ID":"85441672-38c0-4c4f-8ebf-c2d1caac406a","Type":"ContainerStarted","Data":"a967d4c2beafc25367089228cf2bb9f919164f88f68deabb4ec3b0065a5dfd89"}
Jul 22 02:19:57.109415 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:57.109044    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4f16202b/e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv" event={"ID":"85441672-38c0-4c4f-8ebf-c2d1caac406a","Type":"ContainerStarted","Data":"b5ed27572f1945aafe63d3c2949e0fce1127235391dcec24a2806039ec2497a7"}
Jul 22 02:19:57.127437 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:19:57.127390    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-4f16202b/e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv" podStartSLOduration=2.127363656 podStartE2EDuration="2.127363656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:19:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:19:57.124948447 +0000 UTC m=+1949.968777579" watchObservedRunningTime="2026-07-22 02:19:57.127363656 +0000 UTC m=+1949.971192786"
Jul 22 02:20:06.275931 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:06.275901    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-4f16202b/e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv"
Jul 22 02:20:06.276529 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:06.275954    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-4f16202b/e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv"
Jul 22 02:20:06.288120 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:06.288100    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-4f16202b/e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv"
Jul 22 02:20:07.166570 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:07.166542    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-4f16202b/e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv"
Jul 22 02:20:19.131081 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:19.131043    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4f16202b-hh8r8"]
Jul 22 02:20:19.134466 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:19.134448    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4f16202b-hh8r8"
Jul 22 02:20:19.136734 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:19.136716    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wrtw7\""
Jul 22 02:20:19.141586 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:19.141560    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4f16202b-hh8r8"]
Jul 22 02:20:19.218615 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:19.218590    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/656077ce-1466-49d1-985e-04302113079a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4f16202b-hh8r8\" (UID: \"656077ce-1466-49d1-985e-04302113079a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4f16202b-hh8r8"
Jul 22 02:20:19.319553 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:19.319528    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/656077ce-1466-49d1-985e-04302113079a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4f16202b-hh8r8\" (UID: \"656077ce-1466-49d1-985e-04302113079a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4f16202b-hh8r8"
Jul 22 02:20:19.320136 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:19.320115    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/656077ce-1466-49d1-985e-04302113079a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4f16202b-hh8r8\" (UID: \"656077ce-1466-49d1-985e-04302113079a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4f16202b-hh8r8"
Jul 22 02:20:19.445837 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:19.445817    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4f16202b-hh8r8"
Jul 22 02:20:19.573928 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:19.573906    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4f16202b-hh8r8"]
Jul 22 02:20:19.575757 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:20:19.575732    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod656077ce_1466_49d1_985e_04302113079a.slice/crio-68a82fa46a14c651fa0939100048259cd919f48890543495bdc9a955be5525cd WatchSource:0}: Error finding container 68a82fa46a14c651fa0939100048259cd919f48890543495bdc9a955be5525cd: Status 404 returned error can't find the container with id 68a82fa46a14c651fa0939100048259cd919f48890543495bdc9a955be5525cd
Jul 22 02:20:20.222805 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:20.222770    2563 generic.go:356] "Generic (PLEG): container finished" podID="656077ce-1466-49d1-985e-04302113079a" containerID="f93b5cc1dae4ed3035763c6fbb07130b16f6be3c5a4d6311ac89ea2a73ac3d39" exitCode=0
Jul 22 02:20:20.223137 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:20.222846    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4f16202b-hh8r8" event={"ID":"656077ce-1466-49d1-985e-04302113079a","Type":"ContainerDied","Data":"f93b5cc1dae4ed3035763c6fbb07130b16f6be3c5a4d6311ac89ea2a73ac3d39"}
Jul 22 02:20:20.223137 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:20.222926    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4f16202b-hh8r8" event={"ID":"656077ce-1466-49d1-985e-04302113079a","Type":"ContainerStarted","Data":"68a82fa46a14c651fa0939100048259cd919f48890543495bdc9a955be5525cd"}
Jul 22 02:20:21.358134 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:21.358112    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4f16202b-hh8r8"
Jul 22 02:20:21.437800 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:21.437770    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/656077ce-1466-49d1-985e-04302113079a-maas-api-service-ca\") pod \"656077ce-1466-49d1-985e-04302113079a\" (UID: \"656077ce-1466-49d1-985e-04302113079a\") "
Jul 22 02:20:21.438116 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:21.438092    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/656077ce-1466-49d1-985e-04302113079a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "656077ce-1466-49d1-985e-04302113079a" (UID: "656077ce-1466-49d1-985e-04302113079a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:20:21.538997 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:21.538930    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/656077ce-1466-49d1-985e-04302113079a-maas-api-service-ca\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:20:22.234619 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:22.234592    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4f16202b-hh8r8"
Jul 22 02:20:22.234619 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:22.234598    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4f16202b-hh8r8" event={"ID":"656077ce-1466-49d1-985e-04302113079a","Type":"ContainerDied","Data":"68a82fa46a14c651fa0939100048259cd919f48890543495bdc9a955be5525cd"}
Jul 22 02:20:22.234824 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:22.234644    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68a82fa46a14c651fa0939100048259cd919f48890543495bdc9a955be5525cd"
Jul 22 02:20:34.588772 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:34.588733    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4f16202b-5d85df9b5-854l4"]
Jul 22 02:20:34.589215 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:34.588993    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4f16202b-5d85df9b5-854l4" podUID="26fd9d67-dec9-4ade-b368-47b05b152a1e" containerName="maas-api" containerID="cri-o://30578ea8ab428f5a52cdbadaf2eb015bae3fb18e581cf4881fc6c91549107295" gracePeriod=30
Jul 22 02:20:34.834261 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:34.834235    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4f16202b-5d85df9b5-854l4"
Jul 22 02:20:34.943827 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:34.943803    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26fd9d67-dec9-4ade-b368-47b05b152a1e-openshift-service-ca\") pod \"26fd9d67-dec9-4ade-b368-47b05b152a1e\" (UID: \"26fd9d67-dec9-4ade-b368-47b05b152a1e\") "
Jul 22 02:20:34.943952 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:34.943839    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9957f\" (UniqueName: \"kubernetes.io/projected/26fd9d67-dec9-4ade-b368-47b05b152a1e-kube-api-access-9957f\") pod \"26fd9d67-dec9-4ade-b368-47b05b152a1e\" (UID: \"26fd9d67-dec9-4ade-b368-47b05b152a1e\") "
Jul 22 02:20:34.943952 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:34.943899    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26fd9d67-dec9-4ade-b368-47b05b152a1e-maas-api-tls\") pod \"26fd9d67-dec9-4ade-b368-47b05b152a1e\" (UID: \"26fd9d67-dec9-4ade-b368-47b05b152a1e\") "
Jul 22 02:20:34.944160 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:34.944140    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26fd9d67-dec9-4ade-b368-47b05b152a1e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "26fd9d67-dec9-4ade-b368-47b05b152a1e" (UID: "26fd9d67-dec9-4ade-b368-47b05b152a1e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:20:34.945992 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:34.945967    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26fd9d67-dec9-4ade-b368-47b05b152a1e-kube-api-access-9957f" (OuterVolumeSpecName: "kube-api-access-9957f") pod "26fd9d67-dec9-4ade-b368-47b05b152a1e" (UID: "26fd9d67-dec9-4ade-b368-47b05b152a1e"). InnerVolumeSpecName "kube-api-access-9957f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:20:34.946076 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:34.946041    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26fd9d67-dec9-4ade-b368-47b05b152a1e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "26fd9d67-dec9-4ade-b368-47b05b152a1e" (UID: "26fd9d67-dec9-4ade-b368-47b05b152a1e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:20:35.044541 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:35.044515    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26fd9d67-dec9-4ade-b368-47b05b152a1e-openshift-service-ca\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:20:35.044541 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:35.044538    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9957f\" (UniqueName: \"kubernetes.io/projected/26fd9d67-dec9-4ade-b368-47b05b152a1e-kube-api-access-9957f\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:20:35.044541 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:35.044548    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26fd9d67-dec9-4ade-b368-47b05b152a1e-maas-api-tls\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:20:35.304420 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:35.304327    2563 generic.go:356] "Generic (PLEG): container finished" podID="26fd9d67-dec9-4ade-b368-47b05b152a1e" containerID="30578ea8ab428f5a52cdbadaf2eb015bae3fb18e581cf4881fc6c91549107295" exitCode=0
Jul 22 02:20:35.304557 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:35.304370    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4f16202b-5d85df9b5-854l4"
Jul 22 02:20:35.304557 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:35.304427    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4f16202b-5d85df9b5-854l4" event={"ID":"26fd9d67-dec9-4ade-b368-47b05b152a1e","Type":"ContainerDied","Data":"30578ea8ab428f5a52cdbadaf2eb015bae3fb18e581cf4881fc6c91549107295"}
Jul 22 02:20:35.304557 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:35.304495    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4f16202b-5d85df9b5-854l4" event={"ID":"26fd9d67-dec9-4ade-b368-47b05b152a1e","Type":"ContainerDied","Data":"df15c5f75c3682250fff13c1ee3443e8a68fe8d4eba04b93410de82fc40dcabf"}
Jul 22 02:20:35.304557 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:35.304527    2563 scope.go:117] "RemoveContainer" containerID="30578ea8ab428f5a52cdbadaf2eb015bae3fb18e581cf4881fc6c91549107295"
Jul 22 02:20:35.314158 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:35.314139    2563 scope.go:117] "RemoveContainer" containerID="30578ea8ab428f5a52cdbadaf2eb015bae3fb18e581cf4881fc6c91549107295"
Jul 22 02:20:35.314498 ip-10-0-135-230 kubenswrapper[2563]: E0722 02:20:35.314479    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30578ea8ab428f5a52cdbadaf2eb015bae3fb18e581cf4881fc6c91549107295\": container with ID starting with 30578ea8ab428f5a52cdbadaf2eb015bae3fb18e581cf4881fc6c91549107295 not found: ID does not exist" containerID="30578ea8ab428f5a52cdbadaf2eb015bae3fb18e581cf4881fc6c91549107295"
Jul 22 02:20:35.314567 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:35.314511    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30578ea8ab428f5a52cdbadaf2eb015bae3fb18e581cf4881fc6c91549107295"} err="failed to get container status \"30578ea8ab428f5a52cdbadaf2eb015bae3fb18e581cf4881fc6c91549107295\": rpc error: code = NotFound desc = could not find container \"30578ea8ab428f5a52cdbadaf2eb015bae3fb18e581cf4881fc6c91549107295\": container with ID starting with 30578ea8ab428f5a52cdbadaf2eb015bae3fb18e581cf4881fc6c91549107295 not found: ID does not exist"
Jul 22 02:20:35.326685 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:35.326662    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4f16202b-5d85df9b5-854l4"]
Jul 22 02:20:35.330913 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:35.330891    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4f16202b-5d85df9b5-854l4"]
Jul 22 02:20:35.664043 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:35.663975    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26fd9d67-dec9-4ade-b368-47b05b152a1e" path="/var/lib/kubelet/pods/26fd9d67-dec9-4ade-b368-47b05b152a1e/volumes"
Jul 22 02:20:39.209860 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:39.209824    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4f16202b-hh8r8"]
Jul 22 02:20:39.212514 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:39.212490    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4f16202b-hh8r8"]
Jul 22 02:20:39.668589 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:39.668499    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="656077ce-1466-49d1-985e-04302113079a" path="/var/lib/kubelet/pods/656077ce-1466-49d1-985e-04302113079a/volumes"
Jul 22 02:20:46.623024 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:20:46.622985    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-4f16202b/e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv"]
Jul 22 02:21:01.643172 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:21:01.643134    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-4f16202b/e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv" podUID="85441672-38c0-4c4f-8ebf-c2d1caac406a" containerName="main" containerID="cri-o://a967d4c2beafc25367089228cf2bb9f919164f88f68deabb4ec3b0065a5dfd89" gracePeriod=45
Jul 22 02:21:01.896065 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:21:01.896006    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4f16202b/e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv"
Jul 22 02:21:01.957617 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:21:01.957585    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85441672-38c0-4c4f-8ebf-c2d1caac406a-dshm\") pod \"85441672-38c0-4c4f-8ebf-c2d1caac406a\" (UID: \"85441672-38c0-4c4f-8ebf-c2d1caac406a\") "
Jul 22 02:21:01.957798 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:21:01.957633    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85441672-38c0-4c4f-8ebf-c2d1caac406a-home\") pod \"85441672-38c0-4c4f-8ebf-c2d1caac406a\" (UID: \"85441672-38c0-4c4f-8ebf-c2d1caac406a\") "
Jul 22 02:21:01.957798 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:21:01.957668    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dhzk\" (UniqueName: \"kubernetes.io/projected/85441672-38c0-4c4f-8ebf-c2d1caac406a-kube-api-access-4dhzk\") pod \"85441672-38c0-4c4f-8ebf-c2d1caac406a\" (UID: \"85441672-38c0-4c4f-8ebf-c2d1caac406a\") "
Jul 22 02:21:01.957798 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:21:01.957743    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85441672-38c0-4c4f-8ebf-c2d1caac406a-tls-certs\") pod \"85441672-38c0-4c4f-8ebf-c2d1caac406a\" (UID: \"85441672-38c0-4c4f-8ebf-c2d1caac406a\") "
Jul 22 02:21:01.957798 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:21:01.957785    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85441672-38c0-4c4f-8ebf-c2d1caac406a-tmp-dir\") pod \"85441672-38c0-4c4f-8ebf-c2d1caac406a\" (UID: \"85441672-38c0-4c4f-8ebf-c2d1caac406a\") "
Jul 22 02:21:01.958017 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:21:01.957931    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85441672-38c0-4c4f-8ebf-c2d1caac406a-home" (OuterVolumeSpecName: "home") pod "85441672-38c0-4c4f-8ebf-c2d1caac406a" (UID: "85441672-38c0-4c4f-8ebf-c2d1caac406a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:21:01.958017 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:21:01.957936    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85441672-38c0-4c4f-8ebf-c2d1caac406a-model-cache\") pod \"85441672-38c0-4c4f-8ebf-c2d1caac406a\" (UID: \"85441672-38c0-4c4f-8ebf-c2d1caac406a\") "
Jul 22 02:21:01.958124 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:21:01.958044    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85441672-38c0-4c4f-8ebf-c2d1caac406a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "85441672-38c0-4c4f-8ebf-c2d1caac406a" (UID: "85441672-38c0-4c4f-8ebf-c2d1caac406a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:21:01.958175 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:21:01.958138    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85441672-38c0-4c4f-8ebf-c2d1caac406a-model-cache" (OuterVolumeSpecName: "model-cache") pod "85441672-38c0-4c4f-8ebf-c2d1caac406a" (UID: "85441672-38c0-4c4f-8ebf-c2d1caac406a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:21:01.958295 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:21:01.958271    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85441672-38c0-4c4f-8ebf-c2d1caac406a-tmp-dir\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:21:01.958295 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:21:01.958293    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85441672-38c0-4c4f-8ebf-c2d1caac406a-model-cache\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:21:01.958510 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:21:01.958304    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85441672-38c0-4c4f-8ebf-c2d1caac406a-home\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:21:01.960023 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:21:01.959998    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85441672-38c0-4c4f-8ebf-c2d1caac406a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "85441672-38c0-4c4f-8ebf-c2d1caac406a" (UID: "85441672-38c0-4c4f-8ebf-c2d1caac406a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:21:01.960133 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:21:01.960096    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85441672-38c0-4c4f-8ebf-c2d1caac406a-kube-api-access-4dhzk" (OuterVolumeSpecName: "kube-api-access-4dhzk") pod "85441672-38c0-4c4f-8ebf-c2d1caac406a" (UID: "85441672-38c0-4c4f-8ebf-c2d1caac406a"). InnerVolumeSpecName "kube-api-access-4dhzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:21:01.960133 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:21:01.960095    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85441672-38c0-4c4f-8ebf-c2d1caac406a-dshm" (OuterVolumeSpecName: "dshm") pod "85441672-38c0-4c4f-8ebf-c2d1caac406a" (UID: "85441672-38c0-4c4f-8ebf-c2d1caac406a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:21:02.058955 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:21:02.058926    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85441672-38c0-4c4f-8ebf-c2d1caac406a-tls-certs\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:21:02.058955 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:21:02.058950    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85441672-38c0-4c4f-8ebf-c2d1caac406a-dshm\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:21:02.058955 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:21:02.058959    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dhzk\" (UniqueName: \"kubernetes.io/projected/85441672-38c0-4c4f-8ebf-c2d1caac406a-kube-api-access-4dhzk\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:21:02.453326 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:21:02.453290    2563 generic.go:356] "Generic (PLEG): container finished" podID="85441672-38c0-4c4f-8ebf-c2d1caac406a" containerID="a967d4c2beafc25367089228cf2bb9f919164f88f68deabb4ec3b0065a5dfd89" exitCode=0
Jul 22 02:21:02.453488 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:21:02.453349    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4f16202b/e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv"
Jul 22 02:21:02.453488 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:21:02.453398    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4f16202b/e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv" event={"ID":"85441672-38c0-4c4f-8ebf-c2d1caac406a","Type":"ContainerDied","Data":"a967d4c2beafc25367089228cf2bb9f919164f88f68deabb4ec3b0065a5dfd89"}
Jul 22 02:21:02.453488 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:21:02.453436    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4f16202b/e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv" event={"ID":"85441672-38c0-4c4f-8ebf-c2d1caac406a","Type":"ContainerDied","Data":"b5ed27572f1945aafe63d3c2949e0fce1127235391dcec24a2806039ec2497a7"}
Jul 22 02:21:02.453488 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:21:02.453453    2563 scope.go:117] "RemoveContainer" containerID="a967d4c2beafc25367089228cf2bb9f919164f88f68deabb4ec3b0065a5dfd89"
Jul 22 02:21:02.463265 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:21:02.463250    2563 scope.go:117] "RemoveContainer" containerID="a967d4c2beafc25367089228cf2bb9f919164f88f68deabb4ec3b0065a5dfd89"
Jul 22 02:21:02.463546 ip-10-0-135-230 kubenswrapper[2563]: E0722 02:21:02.463530    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a967d4c2beafc25367089228cf2bb9f919164f88f68deabb4ec3b0065a5dfd89\": container with ID starting with a967d4c2beafc25367089228cf2bb9f919164f88f68deabb4ec3b0065a5dfd89 not found: ID does not exist" containerID="a967d4c2beafc25367089228cf2bb9f919164f88f68deabb4ec3b0065a5dfd89"
Jul 22 02:21:02.463608 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:21:02.463553    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a967d4c2beafc25367089228cf2bb9f919164f88f68deabb4ec3b0065a5dfd89"} err="failed to get container status \"a967d4c2beafc25367089228cf2bb9f919164f88f68deabb4ec3b0065a5dfd89\": rpc error: code = NotFound desc = could not find container \"a967d4c2beafc25367089228cf2bb9f919164f88f68deabb4ec3b0065a5dfd89\": container with ID starting with a967d4c2beafc25367089228cf2bb9f919164f88f68deabb4ec3b0065a5dfd89 not found: ID does not exist"
Jul 22 02:21:02.474669 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:21:02.474647    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-4f16202b/e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv"]
Jul 22 02:21:02.480516 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:21:02.480493    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-4f16202b/e2e-lifecycle-model-4f16202b-kserve-54c9bcb8c5-8m4kv"]
Jul 22 02:21:03.662901 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:21:03.662869    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85441672-38c0-4c4f-8ebf-c2d1caac406a" path="/var/lib/kubelet/pods/85441672-38c0-4c4f-8ebf-c2d1caac406a/volumes"
Jul 22 02:21:27.920552 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:21:27.920520    2563 scope.go:117] "RemoveContainer" containerID="25f77f234e28ce445cb62d41b287dd4983af527ccf3d09dd0161eb4df8a7c2f3"
Jul 22 02:21:27.928512 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:21:27.928491    2563 scope.go:117] "RemoveContainer" containerID="0a06e7e9f25e1cce50f92dcf40426007f50d102bbc60adc3a02d254b7a290f80"
Jul 22 02:22:27.960588 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:27.960553    2563 scope.go:117] "RemoveContainer" containerID="76aff8eadb996ba334d99cff9f90439541bb10f98930f18527075dc36c6e49dc"
Jul 22 02:22:28.909436 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:28.909403    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-196560-599b4d7967-gdrw8"]
Jul 22 02:22:28.909782 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:28.909766    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26fd9d67-dec9-4ade-b368-47b05b152a1e" containerName="maas-api"
Jul 22 02:22:28.909782 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:28.909780    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="26fd9d67-dec9-4ade-b368-47b05b152a1e" containerName="maas-api"
Jul 22 02:22:28.909959 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:28.909796    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85441672-38c0-4c4f-8ebf-c2d1caac406a" containerName="main"
Jul 22 02:22:28.909959 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:28.909802    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="85441672-38c0-4c4f-8ebf-c2d1caac406a" containerName="main"
Jul 22 02:22:28.909959 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:28.909812    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="656077ce-1466-49d1-985e-04302113079a" containerName="revoke-keys"
Jul 22 02:22:28.909959 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:28.909817    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="656077ce-1466-49d1-985e-04302113079a" containerName="revoke-keys"
Jul 22 02:22:28.909959 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:28.909872    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="656077ce-1466-49d1-985e-04302113079a" containerName="revoke-keys"
Jul 22 02:22:28.909959 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:28.909881    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="85441672-38c0-4c4f-8ebf-c2d1caac406a" containerName="main"
Jul 22 02:22:28.909959 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:28.909888    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="26fd9d67-dec9-4ade-b368-47b05b152a1e" containerName="maas-api"
Jul 22 02:22:28.914305 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:28.914281    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-196560-599b4d7967-gdrw8"
Jul 22 02:22:28.916754 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:28.916723    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-196560-serving-cert\""
Jul 22 02:22:28.930156 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:28.930132    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-196560-599b4d7967-gdrw8"]
Jul 22 02:22:28.946583 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:28.946551    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08818997-3e72-464f-a8d0-d32504bed40b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-196560-599b4d7967-gdrw8\" (UID: \"08818997-3e72-464f-a8d0-d32504bed40b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-196560-599b4d7967-gdrw8"
Jul 22 02:22:28.946892 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:28.946871    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08818997-3e72-464f-a8d0-d32504bed40b-maas-api-tls\") pod \"maas-api-e2e-inf-a-196560-599b4d7967-gdrw8\" (UID: \"08818997-3e72-464f-a8d0-d32504bed40b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-196560-599b4d7967-gdrw8"
Jul 22 02:22:28.946958 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:28.946908    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2qkt\" (UniqueName: \"kubernetes.io/projected/08818997-3e72-464f-a8d0-d32504bed40b-kube-api-access-l2qkt\") pod \"maas-api-e2e-inf-a-196560-599b4d7967-gdrw8\" (UID: \"08818997-3e72-464f-a8d0-d32504bed40b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-196560-599b4d7967-gdrw8"
Jul 22 02:22:29.047830 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:29.047811    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08818997-3e72-464f-a8d0-d32504bed40b-maas-api-tls\") pod \"maas-api-e2e-inf-a-196560-599b4d7967-gdrw8\" (UID: \"08818997-3e72-464f-a8d0-d32504bed40b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-196560-599b4d7967-gdrw8"
Jul 22 02:22:29.048108 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:29.047840    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2qkt\" (UniqueName: \"kubernetes.io/projected/08818997-3e72-464f-a8d0-d32504bed40b-kube-api-access-l2qkt\") pod \"maas-api-e2e-inf-a-196560-599b4d7967-gdrw8\" (UID: \"08818997-3e72-464f-a8d0-d32504bed40b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-196560-599b4d7967-gdrw8"
Jul 22 02:22:29.048108 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:29.047893    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08818997-3e72-464f-a8d0-d32504bed40b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-196560-599b4d7967-gdrw8\" (UID: \"08818997-3e72-464f-a8d0-d32504bed40b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-196560-599b4d7967-gdrw8"
Jul 22 02:22:29.048108 ip-10-0-135-230 kubenswrapper[2563]: E0722 02:22:29.047958    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-196560-serving-cert: secret "maas-api-e2e-inf-a-196560-serving-cert" not found
Jul 22 02:22:29.048108 ip-10-0-135-230 kubenswrapper[2563]: E0722 02:22:29.048034    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08818997-3e72-464f-a8d0-d32504bed40b-maas-api-tls podName:08818997-3e72-464f-a8d0-d32504bed40b nodeName:}" failed. No retries permitted until 2026-07-22 02:22:29.548016291 +0000 UTC m=+2102.391845402 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/08818997-3e72-464f-a8d0-d32504bed40b-maas-api-tls") pod "maas-api-e2e-inf-a-196560-599b4d7967-gdrw8" (UID: "08818997-3e72-464f-a8d0-d32504bed40b") : secret "maas-api-e2e-inf-a-196560-serving-cert" not found
Jul 22 02:22:29.048453 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:29.048438    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08818997-3e72-464f-a8d0-d32504bed40b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-196560-599b4d7967-gdrw8\" (UID: \"08818997-3e72-464f-a8d0-d32504bed40b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-196560-599b4d7967-gdrw8"
Jul 22 02:22:29.056491 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:29.056473    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2qkt\" (UniqueName: \"kubernetes.io/projected/08818997-3e72-464f-a8d0-d32504bed40b-kube-api-access-l2qkt\") pod \"maas-api-e2e-inf-a-196560-599b4d7967-gdrw8\" (UID: \"08818997-3e72-464f-a8d0-d32504bed40b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-196560-599b4d7967-gdrw8"
Jul 22 02:22:29.552575 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:29.552540    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08818997-3e72-464f-a8d0-d32504bed40b-maas-api-tls\") pod \"maas-api-e2e-inf-a-196560-599b4d7967-gdrw8\" (UID: \"08818997-3e72-464f-a8d0-d32504bed40b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-196560-599b4d7967-gdrw8"
Jul 22 02:22:29.552740 ip-10-0-135-230 kubenswrapper[2563]: E0722 02:22:29.552636    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-196560-serving-cert: secret "maas-api-e2e-inf-a-196560-serving-cert" not found
Jul 22 02:22:29.552740 ip-10-0-135-230 kubenswrapper[2563]: E0722 02:22:29.552696    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08818997-3e72-464f-a8d0-d32504bed40b-maas-api-tls podName:08818997-3e72-464f-a8d0-d32504bed40b nodeName:}" failed. No retries permitted until 2026-07-22 02:22:30.552682253 +0000 UTC m=+2103.396511363 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/08818997-3e72-464f-a8d0-d32504bed40b-maas-api-tls") pod "maas-api-e2e-inf-a-196560-599b4d7967-gdrw8" (UID: "08818997-3e72-464f-a8d0-d32504bed40b") : secret "maas-api-e2e-inf-a-196560-serving-cert" not found
Jul 22 02:22:30.561295 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:30.561260    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08818997-3e72-464f-a8d0-d32504bed40b-maas-api-tls\") pod \"maas-api-e2e-inf-a-196560-599b4d7967-gdrw8\" (UID: \"08818997-3e72-464f-a8d0-d32504bed40b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-196560-599b4d7967-gdrw8"
Jul 22 02:22:30.563667 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:30.563643    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08818997-3e72-464f-a8d0-d32504bed40b-maas-api-tls\") pod \"maas-api-e2e-inf-a-196560-599b4d7967-gdrw8\" (UID: \"08818997-3e72-464f-a8d0-d32504bed40b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-196560-599b4d7967-gdrw8"
Jul 22 02:22:30.725240 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:30.725218    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-196560-599b4d7967-gdrw8"
Jul 22 02:22:30.856991 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:30.856968    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-196560-599b4d7967-gdrw8"]
Jul 22 02:22:30.858845 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:22:30.858807    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08818997_3e72_464f_a8d0_d32504bed40b.slice/crio-7fdcd24199224b01f011b7b0ddd4d631d92a5dd6422b3cc3063125b93e6e7c61 WatchSource:0}: Error finding container 7fdcd24199224b01f011b7b0ddd4d631d92a5dd6422b3cc3063125b93e6e7c61: Status 404 returned error can't find the container with id 7fdcd24199224b01f011b7b0ddd4d631d92a5dd6422b3cc3063125b93e6e7c61
Jul 22 02:22:30.935307 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:30.935266    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-196560-599b4d7967-gdrw8" event={"ID":"08818997-3e72-464f-a8d0-d32504bed40b","Type":"ContainerStarted","Data":"7fdcd24199224b01f011b7b0ddd4d631d92a5dd6422b3cc3063125b93e6e7c61"}
Jul 22 02:22:31.941992 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:31.941959    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-196560-599b4d7967-gdrw8" event={"ID":"08818997-3e72-464f-a8d0-d32504bed40b","Type":"ContainerStarted","Data":"d36e5127a6b7630e735d806db16c53413eb0d1efaa3feaa55713bf6d5fcb6ebf"}
Jul 22 02:22:31.942360 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:31.942099    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-196560-599b4d7967-gdrw8"
Jul 22 02:22:31.959955 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:31.959893    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-196560-599b4d7967-gdrw8" podStartSLOduration=3.5522608829999998 podStartE2EDuration="3.95987163s" totalImagesPullingTime="407.610747ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:22:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:22:31.956679733 +0000 UTC m=+2104.800508868" watchObservedRunningTime="2026-07-22 02:22:31.95987163 +0000 UTC m=+2104.803700763"
Jul 22 02:22:36.927613 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:36.927581    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6708c8-d6767ddb5-889pc"]
Jul 22 02:22:36.937201 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:36.937170    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6708c8-d6767ddb5-889pc"
Jul 22 02:22:36.939570 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:36.939545    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-6708c8-serving-cert\""
Jul 22 02:22:36.940838 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:36.940806    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6708c8-d6767ddb5-889pc"]
Jul 22 02:22:37.007549 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:37.007527    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd329baf-2bb2-4edb-a19f-f259982cbb1b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6708c8-d6767ddb5-889pc\" (UID: \"cd329baf-2bb2-4edb-a19f-f259982cbb1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6708c8-d6767ddb5-889pc"
Jul 22 02:22:37.007653 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:37.007597    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwrpj\" (UniqueName: \"kubernetes.io/projected/cd329baf-2bb2-4edb-a19f-f259982cbb1b-kube-api-access-qwrpj\") pod \"maas-api-e2e-inf-b-6708c8-d6767ddb5-889pc\" (UID: \"cd329baf-2bb2-4edb-a19f-f259982cbb1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6708c8-d6767ddb5-889pc"
Jul 22 02:22:37.007653 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:37.007626    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd329baf-2bb2-4edb-a19f-f259982cbb1b-maas-api-tls\") pod \"maas-api-e2e-inf-b-6708c8-d6767ddb5-889pc\" (UID: \"cd329baf-2bb2-4edb-a19f-f259982cbb1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6708c8-d6767ddb5-889pc"
Jul 22 02:22:37.108614 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:37.108592    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwrpj\" (UniqueName: \"kubernetes.io/projected/cd329baf-2bb2-4edb-a19f-f259982cbb1b-kube-api-access-qwrpj\") pod \"maas-api-e2e-inf-b-6708c8-d6767ddb5-889pc\" (UID: \"cd329baf-2bb2-4edb-a19f-f259982cbb1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6708c8-d6767ddb5-889pc"
Jul 22 02:22:37.108727 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:37.108623    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd329baf-2bb2-4edb-a19f-f259982cbb1b-maas-api-tls\") pod \"maas-api-e2e-inf-b-6708c8-d6767ddb5-889pc\" (UID: \"cd329baf-2bb2-4edb-a19f-f259982cbb1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6708c8-d6767ddb5-889pc"
Jul 22 02:22:37.108727 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:37.108652    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd329baf-2bb2-4edb-a19f-f259982cbb1b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6708c8-d6767ddb5-889pc\" (UID: \"cd329baf-2bb2-4edb-a19f-f259982cbb1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6708c8-d6767ddb5-889pc"
Jul 22 02:22:37.108836 ip-10-0-135-230 kubenswrapper[2563]: E0722 02:22:37.108745    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-6708c8-serving-cert: secret "maas-api-e2e-inf-b-6708c8-serving-cert" not found
Jul 22 02:22:37.108836 ip-10-0-135-230 kubenswrapper[2563]: E0722 02:22:37.108799    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd329baf-2bb2-4edb-a19f-f259982cbb1b-maas-api-tls podName:cd329baf-2bb2-4edb-a19f-f259982cbb1b nodeName:}" failed. No retries permitted until 2026-07-22 02:22:37.608784918 +0000 UTC m=+2110.452614029 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cd329baf-2bb2-4edb-a19f-f259982cbb1b-maas-api-tls") pod "maas-api-e2e-inf-b-6708c8-d6767ddb5-889pc" (UID: "cd329baf-2bb2-4edb-a19f-f259982cbb1b") : secret "maas-api-e2e-inf-b-6708c8-serving-cert" not found
Jul 22 02:22:37.109242 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:37.109224    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd329baf-2bb2-4edb-a19f-f259982cbb1b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6708c8-d6767ddb5-889pc\" (UID: \"cd329baf-2bb2-4edb-a19f-f259982cbb1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6708c8-d6767ddb5-889pc"
Jul 22 02:22:37.120611 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:37.120588    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwrpj\" (UniqueName: \"kubernetes.io/projected/cd329baf-2bb2-4edb-a19f-f259982cbb1b-kube-api-access-qwrpj\") pod \"maas-api-e2e-inf-b-6708c8-d6767ddb5-889pc\" (UID: \"cd329baf-2bb2-4edb-a19f-f259982cbb1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6708c8-d6767ddb5-889pc"
Jul 22 02:22:37.612661 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:37.612633    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd329baf-2bb2-4edb-a19f-f259982cbb1b-maas-api-tls\") pod \"maas-api-e2e-inf-b-6708c8-d6767ddb5-889pc\" (UID: \"cd329baf-2bb2-4edb-a19f-f259982cbb1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6708c8-d6767ddb5-889pc"
Jul 22 02:22:37.614981 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:37.614958    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd329baf-2bb2-4edb-a19f-f259982cbb1b-maas-api-tls\") pod \"maas-api-e2e-inf-b-6708c8-d6767ddb5-889pc\" (UID: \"cd329baf-2bb2-4edb-a19f-f259982cbb1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6708c8-d6767ddb5-889pc"
Jul 22 02:22:37.851348 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:37.851322    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6708c8-d6767ddb5-889pc"
Jul 22 02:22:37.952880 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:37.952854    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-196560-599b4d7967-gdrw8"
Jul 22 02:22:37.988484 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:37.988455    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6708c8-d6767ddb5-889pc"]
Jul 22 02:22:37.989636 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:22:37.989609    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd329baf_2bb2_4edb_a19f_f259982cbb1b.slice/crio-b418b747b15b78fc6f67afea4225503a59bdec01c21fe4fee781ff226bec04fa WatchSource:0}: Error finding container b418b747b15b78fc6f67afea4225503a59bdec01c21fe4fee781ff226bec04fa: Status 404 returned error can't find the container with id b418b747b15b78fc6f67afea4225503a59bdec01c21fe4fee781ff226bec04fa
Jul 22 02:22:38.981549 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:38.981516    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6708c8-d6767ddb5-889pc" event={"ID":"cd329baf-2bb2-4edb-a19f-f259982cbb1b","Type":"ContainerStarted","Data":"f29a68fd5a4b097dbc0f79c6f0ce9ee431bfcb2bc08b0f1dea7a4f097ae5139e"}
Jul 22 02:22:38.981891 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:38.981566    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6708c8-d6767ddb5-889pc" event={"ID":"cd329baf-2bb2-4edb-a19f-f259982cbb1b","Type":"ContainerStarted","Data":"b418b747b15b78fc6f67afea4225503a59bdec01c21fe4fee781ff226bec04fa"}
Jul 22 02:22:38.981891 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:38.981604    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6708c8-d6767ddb5-889pc"
Jul 22 02:22:38.999424 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:38.999353    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6708c8-d6767ddb5-889pc" podStartSLOduration=2.646390001 podStartE2EDuration="2.999335378s" totalImagesPullingTime="352.945377ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:22:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:22:38.997226084 +0000 UTC m=+2111.841055218" watchObservedRunningTime="2026-07-22 02:22:38.999335378 +0000 UTC m=+2111.843164512"
Jul 22 02:22:44.992074 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:44.992046    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6708c8-d6767ddb5-889pc"
Jul 22 02:22:53.467150 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:53.467117    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-196560/test-model-196560-kserve-f6f959f79-rkg47"]
Jul 22 02:22:53.470847 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:53.470830    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-196560/test-model-196560-kserve-f6f959f79-rkg47"
Jul 22 02:22:53.473297 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:53.473273    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-196560\"/\"openshift-service-ca.crt\""
Jul 22 02:22:53.474207 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:53.474188    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-196560\"/\"test-model-196560-kserve-self-signed-certs\""
Jul 22 02:22:53.474336 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:53.474189    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-196560\"/\"default-dockercfg-nz75x\""
Jul 22 02:22:53.474336 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:53.474231    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-196560\"/\"kube-root-ca.crt\""
Jul 22 02:22:53.480812 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:53.480791    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-196560/test-model-196560-kserve-f6f959f79-rkg47"]
Jul 22 02:22:53.636236 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:53.636206    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72ae7e9d-4fca-49cd-80bd-629a3d4618e3-tmp-dir\") pod \"test-model-196560-kserve-f6f959f79-rkg47\" (UID: \"72ae7e9d-4fca-49cd-80bd-629a3d4618e3\") " pod="ai-tenant-e2e-inf-a-196560/test-model-196560-kserve-f6f959f79-rkg47"
Jul 22 02:22:53.636399 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:53.636260    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh57w\" (UniqueName: \"kubernetes.io/projected/72ae7e9d-4fca-49cd-80bd-629a3d4618e3-kube-api-access-gh57w\") pod \"test-model-196560-kserve-f6f959f79-rkg47\" (UID: \"72ae7e9d-4fca-49cd-80bd-629a3d4618e3\") " pod="ai-tenant-e2e-inf-a-196560/test-model-196560-kserve-f6f959f79-rkg47"
Jul 22 02:22:53.636399 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:53.636312    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72ae7e9d-4fca-49cd-80bd-629a3d4618e3-model-cache\") pod \"test-model-196560-kserve-f6f959f79-rkg47\" (UID: \"72ae7e9d-4fca-49cd-80bd-629a3d4618e3\") " pod="ai-tenant-e2e-inf-a-196560/test-model-196560-kserve-f6f959f79-rkg47"
Jul 22 02:22:53.636399 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:53.636352    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72ae7e9d-4fca-49cd-80bd-629a3d4618e3-home\") pod \"test-model-196560-kserve-f6f959f79-rkg47\" (UID: \"72ae7e9d-4fca-49cd-80bd-629a3d4618e3\") " pod="ai-tenant-e2e-inf-a-196560/test-model-196560-kserve-f6f959f79-rkg47"
Jul 22 02:22:53.636537 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:53.636402    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72ae7e9d-4fca-49cd-80bd-629a3d4618e3-tls-certs\") pod \"test-model-196560-kserve-f6f959f79-rkg47\" (UID: \"72ae7e9d-4fca-49cd-80bd-629a3d4618e3\") " pod="ai-tenant-e2e-inf-a-196560/test-model-196560-kserve-f6f959f79-rkg47"
Jul 22 02:22:53.636537 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:53.636428    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72ae7e9d-4fca-49cd-80bd-629a3d4618e3-dshm\") pod \"test-model-196560-kserve-f6f959f79-rkg47\" (UID: \"72ae7e9d-4fca-49cd-80bd-629a3d4618e3\") " pod="ai-tenant-e2e-inf-a-196560/test-model-196560-kserve-f6f959f79-rkg47"
Jul 22 02:22:53.737672 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:53.737607    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh57w\" (UniqueName: \"kubernetes.io/projected/72ae7e9d-4fca-49cd-80bd-629a3d4618e3-kube-api-access-gh57w\") pod \"test-model-196560-kserve-f6f959f79-rkg47\" (UID: \"72ae7e9d-4fca-49cd-80bd-629a3d4618e3\") " pod="ai-tenant-e2e-inf-a-196560/test-model-196560-kserve-f6f959f79-rkg47"
Jul 22 02:22:53.737672 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:53.737642    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72ae7e9d-4fca-49cd-80bd-629a3d4618e3-model-cache\") pod \"test-model-196560-kserve-f6f959f79-rkg47\" (UID: \"72ae7e9d-4fca-49cd-80bd-629a3d4618e3\") " pod="ai-tenant-e2e-inf-a-196560/test-model-196560-kserve-f6f959f79-rkg47"
Jul 22 02:22:53.737672 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:53.737668    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72ae7e9d-4fca-49cd-80bd-629a3d4618e3-home\") pod \"test-model-196560-kserve-f6f959f79-rkg47\" (UID: \"72ae7e9d-4fca-49cd-80bd-629a3d4618e3\") " pod="ai-tenant-e2e-inf-a-196560/test-model-196560-kserve-f6f959f79-rkg47"
Jul 22 02:22:53.737911 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:53.737700    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72ae7e9d-4fca-49cd-80bd-629a3d4618e3-tls-certs\") pod \"test-model-196560-kserve-f6f959f79-rkg47\" (UID: \"72ae7e9d-4fca-49cd-80bd-629a3d4618e3\") " pod="ai-tenant-e2e-inf-a-196560/test-model-196560-kserve-f6f959f79-rkg47"
Jul 22 02:22:53.737911 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:53.737811    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72ae7e9d-4fca-49cd-80bd-629a3d4618e3-dshm\") pod \"test-model-196560-kserve-f6f959f79-rkg47\" (UID: \"72ae7e9d-4fca-49cd-80bd-629a3d4618e3\") " pod="ai-tenant-e2e-inf-a-196560/test-model-196560-kserve-f6f959f79-rkg47"
Jul 22 02:22:53.737911 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:53.737873    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72ae7e9d-4fca-49cd-80bd-629a3d4618e3-tmp-dir\") pod \"test-model-196560-kserve-f6f959f79-rkg47\" (UID: \"72ae7e9d-4fca-49cd-80bd-629a3d4618e3\") " pod="ai-tenant-e2e-inf-a-196560/test-model-196560-kserve-f6f959f79-rkg47"
Jul 22 02:22:53.738071 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:53.738029    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72ae7e9d-4fca-49cd-80bd-629a3d4618e3-model-cache\") pod \"test-model-196560-kserve-f6f959f79-rkg47\" (UID: \"72ae7e9d-4fca-49cd-80bd-629a3d4618e3\") " pod="ai-tenant-e2e-inf-a-196560/test-model-196560-kserve-f6f959f79-rkg47"
Jul 22 02:22:53.738127 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:53.738082    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72ae7e9d-4fca-49cd-80bd-629a3d4618e3-home\") pod \"test-model-196560-kserve-f6f959f79-rkg47\" (UID: \"72ae7e9d-4fca-49cd-80bd-629a3d4618e3\") " pod="ai-tenant-e2e-inf-a-196560/test-model-196560-kserve-f6f959f79-rkg47"
Jul 22 02:22:53.738297 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:53.738276    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72ae7e9d-4fca-49cd-80bd-629a3d4618e3-tmp-dir\") pod \"test-model-196560-kserve-f6f959f79-rkg47\" (UID: \"72ae7e9d-4fca-49cd-80bd-629a3d4618e3\") " pod="ai-tenant-e2e-inf-a-196560/test-model-196560-kserve-f6f959f79-rkg47"
Jul 22 02:22:53.739863 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:53.739842    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72ae7e9d-4fca-49cd-80bd-629a3d4618e3-dshm\") pod \"test-model-196560-kserve-f6f959f79-rkg47\" (UID: \"72ae7e9d-4fca-49cd-80bd-629a3d4618e3\") " pod="ai-tenant-e2e-inf-a-196560/test-model-196560-kserve-f6f959f79-rkg47"
Jul 22 02:22:53.740224 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:53.740205    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72ae7e9d-4fca-49cd-80bd-629a3d4618e3-tls-certs\") pod \"test-model-196560-kserve-f6f959f79-rkg47\" (UID: \"72ae7e9d-4fca-49cd-80bd-629a3d4618e3\") " pod="ai-tenant-e2e-inf-a-196560/test-model-196560-kserve-f6f959f79-rkg47"
Jul 22 02:22:53.745626 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:53.745608    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh57w\" (UniqueName: \"kubernetes.io/projected/72ae7e9d-4fca-49cd-80bd-629a3d4618e3-kube-api-access-gh57w\") pod \"test-model-196560-kserve-f6f959f79-rkg47\" (UID: \"72ae7e9d-4fca-49cd-80bd-629a3d4618e3\") " pod="ai-tenant-e2e-inf-a-196560/test-model-196560-kserve-f6f959f79-rkg47"
Jul 22 02:22:53.781132 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:53.781115    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-196560/test-model-196560-kserve-f6f959f79-rkg47"
Jul 22 02:22:53.909898 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:53.909875    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-196560/test-model-196560-kserve-f6f959f79-rkg47"]
Jul 22 02:22:53.911863 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:22:53.911833    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72ae7e9d_4fca_49cd_80bd_629a3d4618e3.slice/crio-88ad50fbf5bd18bd9e426e5cc12e7fada406d0e74a1a69fa9a60ee1c68a73d7a WatchSource:0}: Error finding container 88ad50fbf5bd18bd9e426e5cc12e7fada406d0e74a1a69fa9a60ee1c68a73d7a: Status 404 returned error can't find the container with id 88ad50fbf5bd18bd9e426e5cc12e7fada406d0e74a1a69fa9a60ee1c68a73d7a
Jul 22 02:22:54.057112 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:54.057032    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-196560/test-model-196560-kserve-f6f959f79-rkg47" event={"ID":"72ae7e9d-4fca-49cd-80bd-629a3d4618e3","Type":"ContainerStarted","Data":"21fe75ed72cbd29586918e8895db9057a2fabf306f2e719999ccafa41283ffd7"}
Jul 22 02:22:54.057112 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:54.057081    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-196560/test-model-196560-kserve-f6f959f79-rkg47" event={"ID":"72ae7e9d-4fca-49cd-80bd-629a3d4618e3","Type":"ContainerStarted","Data":"88ad50fbf5bd18bd9e426e5cc12e7fada406d0e74a1a69fa9a60ee1c68a73d7a"}
Jul 22 02:22:54.076032 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:22:54.075992    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-196560/test-model-196560-kserve-f6f959f79-rkg47" podStartSLOduration=1.075977404 podStartE2EDuration="1.075977404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:22:54.07371417 +0000 UTC m=+2126.917543303" watchObservedRunningTime="2026-07-22 02:22:54.075977404 +0000 UTC m=+2126.919806536"
Jul 22 02:23:03.781949 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:23:03.781914    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-196560/test-model-196560-kserve-f6f959f79-rkg47"
Jul 22 02:23:03.781949 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:23:03.781957    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-196560/test-model-196560-kserve-f6f959f79-rkg47"
Jul 22 02:23:03.795458 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:23:03.795432    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-196560/test-model-196560-kserve-f6f959f79-rkg47"
Jul 22 02:23:04.120603 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:23:04.120524    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-196560/test-model-196560-kserve-f6f959f79-rkg47"
Jul 22 02:23:27.981823 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:23:27.981746    2563 scope.go:117] "RemoveContainer" containerID="e5bec0542a0462926dfe4714a63bf904705a87da629024b97d15c6a26020d743"
Jul 22 02:24:01.228060 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:01.224133    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-196560/test-model-196560-kserve-f6f959f79-rkg47"]
Jul 22 02:24:02.321192 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:02.321152    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-196560-7rqgv"]
Jul 22 02:24:02.324845 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:02.324827    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-196560-7rqgv"
Jul 22 02:24:02.327330 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:02.327305    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wrtw7\""
Jul 22 02:24:02.333118 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:02.333094    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-196560-7rqgv"]
Jul 22 02:24:02.371919 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:02.371884    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3e3d239-40bf-492d-9693-f5e011267c3b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-196560-7rqgv\" (UID: \"c3e3d239-40bf-492d-9693-f5e011267c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-196560-7rqgv"
Jul 22 02:24:02.472712 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:02.472623    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3e3d239-40bf-492d-9693-f5e011267c3b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-196560-7rqgv\" (UID: \"c3e3d239-40bf-492d-9693-f5e011267c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-196560-7rqgv"
Jul 22 02:24:02.473180 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:02.473159    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3e3d239-40bf-492d-9693-f5e011267c3b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-196560-7rqgv\" (UID: \"c3e3d239-40bf-492d-9693-f5e011267c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-196560-7rqgv"
Jul 22 02:24:02.636837 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:02.636765    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-196560-7rqgv"
Jul 22 02:24:02.771642 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:02.771600    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-196560-7rqgv"]
Jul 22 02:24:02.775612 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:24:02.775583    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e3d239_40bf_492d_9693_f5e011267c3b.slice/crio-852fbf6a4fe95665f2bdffc864a74cbcd8254a671e391456828518967a2ebbb2 WatchSource:0}: Error finding container 852fbf6a4fe95665f2bdffc864a74cbcd8254a671e391456828518967a2ebbb2: Status 404 returned error can't find the container with id 852fbf6a4fe95665f2bdffc864a74cbcd8254a671e391456828518967a2ebbb2
Jul 22 02:24:03.413046 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:03.412970    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-196560-7rqgv" event={"ID":"c3e3d239-40bf-492d-9693-f5e011267c3b","Type":"ContainerStarted","Data":"58dc6ea711fc49c72316f31b671c4e101ec03ff3d5b385b2726ad7616b9b1b9d"}
Jul 22 02:24:03.413046 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:03.413020    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-196560-7rqgv" event={"ID":"c3e3d239-40bf-492d-9693-f5e011267c3b","Type":"ContainerStarted","Data":"852fbf6a4fe95665f2bdffc864a74cbcd8254a671e391456828518967a2ebbb2"}
Jul 22 02:24:03.428695 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:03.428650    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-196560-7rqgv" podStartSLOduration=1.428637707 podStartE2EDuration="1.428637707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:24:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:24:03.42685756 +0000 UTC m=+2196.270686687" watchObservedRunningTime="2026-07-22 02:24:03.428637707 +0000 UTC m=+2196.272466839"
Jul 22 02:24:04.419565 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:04.419531    2563 generic.go:356] "Generic (PLEG): container finished" podID="c3e3d239-40bf-492d-9693-f5e011267c3b" containerID="58dc6ea711fc49c72316f31b671c4e101ec03ff3d5b385b2726ad7616b9b1b9d" exitCode=0
Jul 22 02:24:04.419952 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:04.419621    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-196560-7rqgv" event={"ID":"c3e3d239-40bf-492d-9693-f5e011267c3b","Type":"ContainerDied","Data":"58dc6ea711fc49c72316f31b671c4e101ec03ff3d5b385b2726ad7616b9b1b9d"}
Jul 22 02:24:05.558707 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:05.558682    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-196560-7rqgv"
Jul 22 02:24:05.700144 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:05.700077    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3e3d239-40bf-492d-9693-f5e011267c3b-maas-api-service-ca\") pod \"c3e3d239-40bf-492d-9693-f5e011267c3b\" (UID: \"c3e3d239-40bf-492d-9693-f5e011267c3b\") "
Jul 22 02:24:05.700483 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:05.700455    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3e3d239-40bf-492d-9693-f5e011267c3b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c3e3d239-40bf-492d-9693-f5e011267c3b" (UID: "c3e3d239-40bf-492d-9693-f5e011267c3b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:24:05.800953 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:05.800930    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3e3d239-40bf-492d-9693-f5e011267c3b-maas-api-service-ca\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:24:06.430683 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:06.430657    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-196560-7rqgv"
Jul 22 02:24:06.430830 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:06.430643    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-196560-7rqgv" event={"ID":"c3e3d239-40bf-492d-9693-f5e011267c3b","Type":"ContainerDied","Data":"852fbf6a4fe95665f2bdffc864a74cbcd8254a671e391456828518967a2ebbb2"}
Jul 22 02:24:06.430830 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:06.430780    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="852fbf6a4fe95665f2bdffc864a74cbcd8254a671e391456828518967a2ebbb2"
Jul 22 02:24:12.898997 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:12.898963    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-196560-599b4d7967-gdrw8"]
Jul 22 02:24:12.899498 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:12.899192    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-196560-599b4d7967-gdrw8" podUID="08818997-3e72-464f-a8d0-d32504bed40b" containerName="maas-api" containerID="cri-o://d36e5127a6b7630e735d806db16c53413eb0d1efaa3feaa55713bf6d5fcb6ebf" gracePeriod=30
Jul 22 02:24:12.948046 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:12.948015    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-196560-599b4d7967-gdrw8" podUID="08818997-3e72-464f-a8d0-d32504bed40b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.65:8443/health\": dial tcp 10.134.0.65:8443: connect: connection refused"
Jul 22 02:24:13.141883 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:13.141861    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-196560-599b4d7967-gdrw8"
Jul 22 02:24:13.267464 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:13.267435    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08818997-3e72-464f-a8d0-d32504bed40b-maas-api-tls\") pod \"08818997-3e72-464f-a8d0-d32504bed40b\" (UID: \"08818997-3e72-464f-a8d0-d32504bed40b\") "
Jul 22 02:24:13.267629 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:13.267483    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2qkt\" (UniqueName: \"kubernetes.io/projected/08818997-3e72-464f-a8d0-d32504bed40b-kube-api-access-l2qkt\") pod \"08818997-3e72-464f-a8d0-d32504bed40b\" (UID: \"08818997-3e72-464f-a8d0-d32504bed40b\") "
Jul 22 02:24:13.267629 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:13.267578    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08818997-3e72-464f-a8d0-d32504bed40b-openshift-service-ca\") pod \"08818997-3e72-464f-a8d0-d32504bed40b\" (UID: \"08818997-3e72-464f-a8d0-d32504bed40b\") "
Jul 22 02:24:13.267946 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:13.267920    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08818997-3e72-464f-a8d0-d32504bed40b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "08818997-3e72-464f-a8d0-d32504bed40b" (UID: "08818997-3e72-464f-a8d0-d32504bed40b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:24:13.269469 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:13.269443    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08818997-3e72-464f-a8d0-d32504bed40b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "08818997-3e72-464f-a8d0-d32504bed40b" (UID: "08818997-3e72-464f-a8d0-d32504bed40b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:24:13.269595 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:13.269573    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08818997-3e72-464f-a8d0-d32504bed40b-kube-api-access-l2qkt" (OuterVolumeSpecName: "kube-api-access-l2qkt") pod "08818997-3e72-464f-a8d0-d32504bed40b" (UID: "08818997-3e72-464f-a8d0-d32504bed40b"). InnerVolumeSpecName "kube-api-access-l2qkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:24:13.368685 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:13.368660    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08818997-3e72-464f-a8d0-d32504bed40b-openshift-service-ca\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:24:13.368685 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:13.368683    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08818997-3e72-464f-a8d0-d32504bed40b-maas-api-tls\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:24:13.368825 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:13.368692    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2qkt\" (UniqueName: \"kubernetes.io/projected/08818997-3e72-464f-a8d0-d32504bed40b-kube-api-access-l2qkt\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:24:13.465163 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:13.465137    2563 generic.go:356] "Generic (PLEG): container finished" podID="08818997-3e72-464f-a8d0-d32504bed40b" containerID="d36e5127a6b7630e735d806db16c53413eb0d1efaa3feaa55713bf6d5fcb6ebf" exitCode=0
Jul 22 02:24:13.465258 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:13.465196    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-196560-599b4d7967-gdrw8"
Jul 22 02:24:13.465258 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:13.465227    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-196560-599b4d7967-gdrw8" event={"ID":"08818997-3e72-464f-a8d0-d32504bed40b","Type":"ContainerDied","Data":"d36e5127a6b7630e735d806db16c53413eb0d1efaa3feaa55713bf6d5fcb6ebf"}
Jul 22 02:24:13.465330 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:13.465309    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-196560-599b4d7967-gdrw8" event={"ID":"08818997-3e72-464f-a8d0-d32504bed40b","Type":"ContainerDied","Data":"7fdcd24199224b01f011b7b0ddd4d631d92a5dd6422b3cc3063125b93e6e7c61"}
Jul 22 02:24:13.465368 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:13.465338    2563 scope.go:117] "RemoveContainer" containerID="d36e5127a6b7630e735d806db16c53413eb0d1efaa3feaa55713bf6d5fcb6ebf"
Jul 22 02:24:13.474769 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:13.474751    2563 scope.go:117] "RemoveContainer" containerID="d36e5127a6b7630e735d806db16c53413eb0d1efaa3feaa55713bf6d5fcb6ebf"
Jul 22 02:24:13.475024 ip-10-0-135-230 kubenswrapper[2563]: E0722 02:24:13.475006    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d36e5127a6b7630e735d806db16c53413eb0d1efaa3feaa55713bf6d5fcb6ebf\": container with ID starting with d36e5127a6b7630e735d806db16c53413eb0d1efaa3feaa55713bf6d5fcb6ebf not found: ID does not exist" containerID="d36e5127a6b7630e735d806db16c53413eb0d1efaa3feaa55713bf6d5fcb6ebf"
Jul 22 02:24:13.475087 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:13.475035    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d36e5127a6b7630e735d806db16c53413eb0d1efaa3feaa55713bf6d5fcb6ebf"} err="failed to get container status \"d36e5127a6b7630e735d806db16c53413eb0d1efaa3feaa55713bf6d5fcb6ebf\": rpc error: code = NotFound desc = could not find container \"d36e5127a6b7630e735d806db16c53413eb0d1efaa3feaa55713bf6d5fcb6ebf\": container with ID starting with d36e5127a6b7630e735d806db16c53413eb0d1efaa3feaa55713bf6d5fcb6ebf not found: ID does not exist"
Jul 22 02:24:13.487783 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:13.487762    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-196560-599b4d7967-gdrw8"]
Jul 22 02:24:13.489718 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:13.489698    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-196560-599b4d7967-gdrw8"]
Jul 22 02:24:13.664065 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:13.664001    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08818997-3e72-464f-a8d0-d32504bed40b" path="/var/lib/kubelet/pods/08818997-3e72-464f-a8d0-d32504bed40b/volumes"
Jul 22 02:24:16.248932 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:16.248887    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-196560/test-model-196560-kserve-f6f959f79-rkg47" podUID="72ae7e9d-4fca-49cd-80bd-629a3d4618e3" containerName="main" containerID="cri-o://21fe75ed72cbd29586918e8895db9057a2fabf306f2e719999ccafa41283ffd7" gracePeriod=45
Jul 22 02:24:16.484645 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:16.484619    2563 generic.go:356] "Generic (PLEG): container finished" podID="72ae7e9d-4fca-49cd-80bd-629a3d4618e3" containerID="21fe75ed72cbd29586918e8895db9057a2fabf306f2e719999ccafa41283ffd7" exitCode=0
Jul 22 02:24:16.484786 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:16.484658    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-196560/test-model-196560-kserve-f6f959f79-rkg47" event={"ID":"72ae7e9d-4fca-49cd-80bd-629a3d4618e3","Type":"ContainerDied","Data":"21fe75ed72cbd29586918e8895db9057a2fabf306f2e719999ccafa41283ffd7"}
Jul 22 02:24:16.484786 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:16.484696    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-196560/test-model-196560-kserve-f6f959f79-rkg47" event={"ID":"72ae7e9d-4fca-49cd-80bd-629a3d4618e3","Type":"ContainerDied","Data":"88ad50fbf5bd18bd9e426e5cc12e7fada406d0e74a1a69fa9a60ee1c68a73d7a"}
Jul 22 02:24:16.484786 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:16.484708    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88ad50fbf5bd18bd9e426e5cc12e7fada406d0e74a1a69fa9a60ee1c68a73d7a"
Jul 22 02:24:16.489228 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:16.489209    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-196560/test-model-196560-kserve-f6f959f79-rkg47"
Jul 22 02:24:16.594762 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:16.594694    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72ae7e9d-4fca-49cd-80bd-629a3d4618e3-home\") pod \"72ae7e9d-4fca-49cd-80bd-629a3d4618e3\" (UID: \"72ae7e9d-4fca-49cd-80bd-629a3d4618e3\") "
Jul 22 02:24:16.594762 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:16.594725    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72ae7e9d-4fca-49cd-80bd-629a3d4618e3-model-cache\") pod \"72ae7e9d-4fca-49cd-80bd-629a3d4618e3\" (UID: \"72ae7e9d-4fca-49cd-80bd-629a3d4618e3\") "
Jul 22 02:24:16.594956 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:16.594773    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gh57w\" (UniqueName: \"kubernetes.io/projected/72ae7e9d-4fca-49cd-80bd-629a3d4618e3-kube-api-access-gh57w\") pod \"72ae7e9d-4fca-49cd-80bd-629a3d4618e3\" (UID: \"72ae7e9d-4fca-49cd-80bd-629a3d4618e3\") "
Jul 22 02:24:16.594956 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:16.594798    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72ae7e9d-4fca-49cd-80bd-629a3d4618e3-tmp-dir\") pod \"72ae7e9d-4fca-49cd-80bd-629a3d4618e3\" (UID: \"72ae7e9d-4fca-49cd-80bd-629a3d4618e3\") "
Jul 22 02:24:16.594956 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:16.594917    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72ae7e9d-4fca-49cd-80bd-629a3d4618e3-dshm\") pod \"72ae7e9d-4fca-49cd-80bd-629a3d4618e3\" (UID: \"72ae7e9d-4fca-49cd-80bd-629a3d4618e3\") "
Jul 22 02:24:16.595127 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:16.594975    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72ae7e9d-4fca-49cd-80bd-629a3d4618e3-tls-certs\") pod \"72ae7e9d-4fca-49cd-80bd-629a3d4618e3\" (UID: \"72ae7e9d-4fca-49cd-80bd-629a3d4618e3\") "
Jul 22 02:24:16.595127 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:16.594992    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72ae7e9d-4fca-49cd-80bd-629a3d4618e3-model-cache" (OuterVolumeSpecName: "model-cache") pod "72ae7e9d-4fca-49cd-80bd-629a3d4618e3" (UID: "72ae7e9d-4fca-49cd-80bd-629a3d4618e3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:24:16.595127 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:16.595008    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72ae7e9d-4fca-49cd-80bd-629a3d4618e3-home" (OuterVolumeSpecName: "home") pod "72ae7e9d-4fca-49cd-80bd-629a3d4618e3" (UID: "72ae7e9d-4fca-49cd-80bd-629a3d4618e3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:24:16.595127 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:16.595016    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72ae7e9d-4fca-49cd-80bd-629a3d4618e3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "72ae7e9d-4fca-49cd-80bd-629a3d4618e3" (UID: "72ae7e9d-4fca-49cd-80bd-629a3d4618e3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:24:16.595320 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:16.595237    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72ae7e9d-4fca-49cd-80bd-629a3d4618e3-home\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:24:16.595320 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:16.595253    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72ae7e9d-4fca-49cd-80bd-629a3d4618e3-model-cache\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:24:16.595320 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:16.595267    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72ae7e9d-4fca-49cd-80bd-629a3d4618e3-tmp-dir\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:24:16.596991 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:16.596958    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72ae7e9d-4fca-49cd-80bd-629a3d4618e3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "72ae7e9d-4fca-49cd-80bd-629a3d4618e3" (UID: "72ae7e9d-4fca-49cd-80bd-629a3d4618e3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:24:16.597095 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:16.597028    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72ae7e9d-4fca-49cd-80bd-629a3d4618e3-kube-api-access-gh57w" (OuterVolumeSpecName: "kube-api-access-gh57w") pod "72ae7e9d-4fca-49cd-80bd-629a3d4618e3" (UID: "72ae7e9d-4fca-49cd-80bd-629a3d4618e3"). InnerVolumeSpecName "kube-api-access-gh57w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:24:16.597095 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:16.597060    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72ae7e9d-4fca-49cd-80bd-629a3d4618e3-dshm" (OuterVolumeSpecName: "dshm") pod "72ae7e9d-4fca-49cd-80bd-629a3d4618e3" (UID: "72ae7e9d-4fca-49cd-80bd-629a3d4618e3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:24:16.696065 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:16.696044    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh57w\" (UniqueName: \"kubernetes.io/projected/72ae7e9d-4fca-49cd-80bd-629a3d4618e3-kube-api-access-gh57w\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:24:16.696065 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:16.696064    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72ae7e9d-4fca-49cd-80bd-629a3d4618e3-dshm\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:24:16.696188 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:16.696074    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72ae7e9d-4fca-49cd-80bd-629a3d4618e3-tls-certs\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:24:17.446464 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:17.446426    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-196560-7rqgv"]
Jul 22 02:24:17.449630 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:17.449607    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-196560-7rqgv"]
Jul 22 02:24:17.490168 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:17.490140    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-196560/test-model-196560-kserve-f6f959f79-rkg47"
Jul 22 02:24:17.517505 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:17.517470    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-196560/test-model-196560-kserve-f6f959f79-rkg47"]
Jul 22 02:24:17.522150 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:17.522127    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-196560/test-model-196560-kserve-f6f959f79-rkg47"]
Jul 22 02:24:17.663766 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:17.663735    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72ae7e9d-4fca-49cd-80bd-629a3d4618e3" path="/var/lib/kubelet/pods/72ae7e9d-4fca-49cd-80bd-629a3d4618e3/volumes"
Jul 22 02:24:17.664074 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:17.664061    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3e3d239-40bf-492d-9693-f5e011267c3b" path="/var/lib/kubelet/pods/c3e3d239-40bf-492d-9693-f5e011267c3b/volumes"
Jul 22 02:24:24.801475 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:24.801438    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6708c8-bnb5v"]
Jul 22 02:24:24.801846 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:24.801784    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3e3d239-40bf-492d-9693-f5e011267c3b" containerName="revoke-keys"
Jul 22 02:24:24.801846 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:24.801796    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3e3d239-40bf-492d-9693-f5e011267c3b" containerName="revoke-keys"
Jul 22 02:24:24.801846 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:24.801812    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08818997-3e72-464f-a8d0-d32504bed40b" containerName="maas-api"
Jul 22 02:24:24.801846 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:24.801818    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="08818997-3e72-464f-a8d0-d32504bed40b" containerName="maas-api"
Jul 22 02:24:24.801846 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:24.801829    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72ae7e9d-4fca-49cd-80bd-629a3d4618e3" containerName="main"
Jul 22 02:24:24.801846 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:24.801834    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="72ae7e9d-4fca-49cd-80bd-629a3d4618e3" containerName="main"
Jul 22 02:24:24.802073 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:24.801884    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="72ae7e9d-4fca-49cd-80bd-629a3d4618e3" containerName="main"
Jul 22 02:24:24.802073 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:24.801894    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="08818997-3e72-464f-a8d0-d32504bed40b" containerName="maas-api"
Jul 22 02:24:24.802073 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:24.801899    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3e3d239-40bf-492d-9693-f5e011267c3b" containerName="revoke-keys"
Jul 22 02:24:24.806043 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:24.806025    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6708c8-bnb5v"
Jul 22 02:24:24.808686 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:24.808664    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wrtw7\""
Jul 22 02:24:24.815755 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:24.815725    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6708c8-bnb5v"]
Jul 22 02:24:24.963643 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:24.963600    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afddf0a4-ad33-44b6-ac14-9a45877dfef3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6708c8-bnb5v\" (UID: \"afddf0a4-ad33-44b6-ac14-9a45877dfef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6708c8-bnb5v"
Jul 22 02:24:25.064535 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:25.064442    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afddf0a4-ad33-44b6-ac14-9a45877dfef3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6708c8-bnb5v\" (UID: \"afddf0a4-ad33-44b6-ac14-9a45877dfef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6708c8-bnb5v"
Jul 22 02:24:25.065074 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:25.065055    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afddf0a4-ad33-44b6-ac14-9a45877dfef3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6708c8-bnb5v\" (UID: \"afddf0a4-ad33-44b6-ac14-9a45877dfef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6708c8-bnb5v"
Jul 22 02:24:25.118233 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:25.118192    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6708c8-bnb5v"
Jul 22 02:24:25.252960 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:25.252934    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6708c8-bnb5v"]
Jul 22 02:24:25.254393 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:24:25.254353    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafddf0a4_ad33_44b6_ac14_9a45877dfef3.slice/crio-02d2daf603bf1a868031ab32f4b2d4d3e9953d810b5b079fb0942ff6995a6ca8 WatchSource:0}: Error finding container 02d2daf603bf1a868031ab32f4b2d4d3e9953d810b5b079fb0942ff6995a6ca8: Status 404 returned error can't find the container with id 02d2daf603bf1a868031ab32f4b2d4d3e9953d810b5b079fb0942ff6995a6ca8
Jul 22 02:24:25.533112 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:25.533075    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6708c8-bnb5v" event={"ID":"afddf0a4-ad33-44b6-ac14-9a45877dfef3","Type":"ContainerStarted","Data":"02d2daf603bf1a868031ab32f4b2d4d3e9953d810b5b079fb0942ff6995a6ca8"}
Jul 22 02:24:26.541496 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:26.541464    2563 generic.go:356] "Generic (PLEG): container finished" podID="afddf0a4-ad33-44b6-ac14-9a45877dfef3" containerID="f822c571292915cc13097ffd1d323d0ba18558696ee80870991801f54e97efba" exitCode=0
Jul 22 02:24:26.541868 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:26.541554    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6708c8-bnb5v" event={"ID":"afddf0a4-ad33-44b6-ac14-9a45877dfef3","Type":"ContainerDied","Data":"f822c571292915cc13097ffd1d323d0ba18558696ee80870991801f54e97efba"}
Jul 22 02:24:27.680290 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:27.680268    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6708c8-bnb5v"
Jul 22 02:24:27.789391 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:27.789349    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afddf0a4-ad33-44b6-ac14-9a45877dfef3-maas-api-service-ca\") pod \"afddf0a4-ad33-44b6-ac14-9a45877dfef3\" (UID: \"afddf0a4-ad33-44b6-ac14-9a45877dfef3\") "
Jul 22 02:24:27.789773 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:27.789744    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afddf0a4-ad33-44b6-ac14-9a45877dfef3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "afddf0a4-ad33-44b6-ac14-9a45877dfef3" (UID: "afddf0a4-ad33-44b6-ac14-9a45877dfef3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:24:27.890958 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:27.890891    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afddf0a4-ad33-44b6-ac14-9a45877dfef3-maas-api-service-ca\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:24:28.553961 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:28.553926    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6708c8-bnb5v" event={"ID":"afddf0a4-ad33-44b6-ac14-9a45877dfef3","Type":"ContainerDied","Data":"02d2daf603bf1a868031ab32f4b2d4d3e9953d810b5b079fb0942ff6995a6ca8"}
Jul 22 02:24:28.553961 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:28.553961    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6708c8-bnb5v"
Jul 22 02:24:28.554162 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:28.553999    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02d2daf603bf1a868031ab32f4b2d4d3e9953d810b5b079fb0942ff6995a6ca8"
Jul 22 02:24:35.444692 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:35.444657    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6708c8-d6767ddb5-889pc"]
Jul 22 02:24:35.445215 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:35.444921    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6708c8-d6767ddb5-889pc" podUID="cd329baf-2bb2-4edb-a19f-f259982cbb1b" containerName="maas-api" containerID="cri-o://f29a68fd5a4b097dbc0f79c6f0ce9ee431bfcb2bc08b0f1dea7a4f097ae5139e" gracePeriod=30
Jul 22 02:24:35.591789 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:35.591758    2563 generic.go:356] "Generic (PLEG): container finished" podID="cd329baf-2bb2-4edb-a19f-f259982cbb1b" containerID="f29a68fd5a4b097dbc0f79c6f0ce9ee431bfcb2bc08b0f1dea7a4f097ae5139e" exitCode=0
Jul 22 02:24:35.591929 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:35.591855    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6708c8-d6767ddb5-889pc" event={"ID":"cd329baf-2bb2-4edb-a19f-f259982cbb1b","Type":"ContainerDied","Data":"f29a68fd5a4b097dbc0f79c6f0ce9ee431bfcb2bc08b0f1dea7a4f097ae5139e"}
Jul 22 02:24:35.693474 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:35.693449    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6708c8-d6767ddb5-889pc"
Jul 22 02:24:35.746779 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:35.746714    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd329baf-2bb2-4edb-a19f-f259982cbb1b-openshift-service-ca\") pod \"cd329baf-2bb2-4edb-a19f-f259982cbb1b\" (UID: \"cd329baf-2bb2-4edb-a19f-f259982cbb1b\") "
Jul 22 02:24:35.746779 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:35.746749    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwrpj\" (UniqueName: \"kubernetes.io/projected/cd329baf-2bb2-4edb-a19f-f259982cbb1b-kube-api-access-qwrpj\") pod \"cd329baf-2bb2-4edb-a19f-f259982cbb1b\" (UID: \"cd329baf-2bb2-4edb-a19f-f259982cbb1b\") "
Jul 22 02:24:35.746779 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:35.746778    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd329baf-2bb2-4edb-a19f-f259982cbb1b-maas-api-tls\") pod \"cd329baf-2bb2-4edb-a19f-f259982cbb1b\" (UID: \"cd329baf-2bb2-4edb-a19f-f259982cbb1b\") "
Jul 22 02:24:35.747082 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:35.747056    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd329baf-2bb2-4edb-a19f-f259982cbb1b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cd329baf-2bb2-4edb-a19f-f259982cbb1b" (UID: "cd329baf-2bb2-4edb-a19f-f259982cbb1b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:24:35.748687 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:35.748668    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd329baf-2bb2-4edb-a19f-f259982cbb1b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cd329baf-2bb2-4edb-a19f-f259982cbb1b" (UID: "cd329baf-2bb2-4edb-a19f-f259982cbb1b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:24:35.748797 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:35.748780    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd329baf-2bb2-4edb-a19f-f259982cbb1b-kube-api-access-qwrpj" (OuterVolumeSpecName: "kube-api-access-qwrpj") pod "cd329baf-2bb2-4edb-a19f-f259982cbb1b" (UID: "cd329baf-2bb2-4edb-a19f-f259982cbb1b"). InnerVolumeSpecName "kube-api-access-qwrpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:24:35.848255 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:35.848234    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd329baf-2bb2-4edb-a19f-f259982cbb1b-openshift-service-ca\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:24:35.848255 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:35.848253    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwrpj\" (UniqueName: \"kubernetes.io/projected/cd329baf-2bb2-4edb-a19f-f259982cbb1b-kube-api-access-qwrpj\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:24:35.848403 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:35.848263    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd329baf-2bb2-4edb-a19f-f259982cbb1b-maas-api-tls\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:24:36.598173 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:36.598139    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6708c8-d6767ddb5-889pc" event={"ID":"cd329baf-2bb2-4edb-a19f-f259982cbb1b","Type":"ContainerDied","Data":"b418b747b15b78fc6f67afea4225503a59bdec01c21fe4fee781ff226bec04fa"}
Jul 22 02:24:36.598173 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:36.598165    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6708c8-d6767ddb5-889pc"
Jul 22 02:24:36.598664 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:36.598206    2563 scope.go:117] "RemoveContainer" containerID="f29a68fd5a4b097dbc0f79c6f0ce9ee431bfcb2bc08b0f1dea7a4f097ae5139e"
Jul 22 02:24:36.625099 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:36.625026    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6708c8-d6767ddb5-889pc"]
Jul 22 02:24:36.630518 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:36.630496    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6708c8-d6767ddb5-889pc"]
Jul 22 02:24:37.663436 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:37.663407    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd329baf-2bb2-4edb-a19f-f259982cbb1b" path="/var/lib/kubelet/pods/cd329baf-2bb2-4edb-a19f-f259982cbb1b/volumes"
Jul 22 02:24:39.891796 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:39.891752    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6708c8-bnb5v"]
Jul 22 02:24:39.894530 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:39.894497    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6708c8-bnb5v"]
Jul 22 02:24:41.663068 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:41.663025    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afddf0a4-ad33-44b6-ac14-9a45877dfef3" path="/var/lib/kubelet/pods/afddf0a4-ad33-44b6-ac14-9a45877dfef3/volumes"
Jul 22 02:24:50.502412 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:50.502329    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-828ede-5d6f774db9-7m4gt"]
Jul 22 02:24:50.502776 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:50.502733    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afddf0a4-ad33-44b6-ac14-9a45877dfef3" containerName="revoke-keys"
Jul 22 02:24:50.502776 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:50.502746    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="afddf0a4-ad33-44b6-ac14-9a45877dfef3" containerName="revoke-keys"
Jul 22 02:24:50.502776 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:50.502765    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd329baf-2bb2-4edb-a19f-f259982cbb1b" containerName="maas-api"
Jul 22 02:24:50.502776 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:50.502772    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd329baf-2bb2-4edb-a19f-f259982cbb1b" containerName="maas-api"
Jul 22 02:24:50.502923 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:50.502837    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd329baf-2bb2-4edb-a19f-f259982cbb1b" containerName="maas-api"
Jul 22 02:24:50.502923 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:50.502848    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="afddf0a4-ad33-44b6-ac14-9a45877dfef3" containerName="revoke-keys"
Jul 22 02:24:50.507067 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:50.507050    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-828ede-5d6f774db9-7m4gt"
Jul 22 02:24:50.509415 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:50.509390    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-828ede-serving-cert\""
Jul 22 02:24:50.518823 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:50.518792    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-828ede-5d6f774db9-7m4gt"]
Jul 22 02:24:50.563739 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:50.563715    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b99b658-4ce2-42c4-baba-fbf9f8a8025d-openshift-service-ca\") pod \"maas-api-e2e-api-a-828ede-5d6f774db9-7m4gt\" (UID: \"1b99b658-4ce2-42c4-baba-fbf9f8a8025d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-828ede-5d6f774db9-7m4gt"
Jul 22 02:24:50.563842 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:50.563779    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l99b7\" (UniqueName: \"kubernetes.io/projected/1b99b658-4ce2-42c4-baba-fbf9f8a8025d-kube-api-access-l99b7\") pod \"maas-api-e2e-api-a-828ede-5d6f774db9-7m4gt\" (UID: \"1b99b658-4ce2-42c4-baba-fbf9f8a8025d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-828ede-5d6f774db9-7m4gt"
Jul 22 02:24:50.563842 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:50.563808    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b99b658-4ce2-42c4-baba-fbf9f8a8025d-maas-api-tls\") pod \"maas-api-e2e-api-a-828ede-5d6f774db9-7m4gt\" (UID: \"1b99b658-4ce2-42c4-baba-fbf9f8a8025d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-828ede-5d6f774db9-7m4gt"
Jul 22 02:24:50.664519 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:50.664494    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b99b658-4ce2-42c4-baba-fbf9f8a8025d-openshift-service-ca\") pod \"maas-api-e2e-api-a-828ede-5d6f774db9-7m4gt\" (UID: \"1b99b658-4ce2-42c4-baba-fbf9f8a8025d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-828ede-5d6f774db9-7m4gt"
Jul 22 02:24:50.664648 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:50.664555    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l99b7\" (UniqueName: \"kubernetes.io/projected/1b99b658-4ce2-42c4-baba-fbf9f8a8025d-kube-api-access-l99b7\") pod \"maas-api-e2e-api-a-828ede-5d6f774db9-7m4gt\" (UID: \"1b99b658-4ce2-42c4-baba-fbf9f8a8025d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-828ede-5d6f774db9-7m4gt"
Jul 22 02:24:50.664648 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:50.664583    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b99b658-4ce2-42c4-baba-fbf9f8a8025d-maas-api-tls\") pod \"maas-api-e2e-api-a-828ede-5d6f774db9-7m4gt\" (UID: \"1b99b658-4ce2-42c4-baba-fbf9f8a8025d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-828ede-5d6f774db9-7m4gt"
Jul 22 02:24:50.665136 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:50.665116    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b99b658-4ce2-42c4-baba-fbf9f8a8025d-openshift-service-ca\") pod \"maas-api-e2e-api-a-828ede-5d6f774db9-7m4gt\" (UID: \"1b99b658-4ce2-42c4-baba-fbf9f8a8025d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-828ede-5d6f774db9-7m4gt"
Jul 22 02:24:50.666873 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:50.666854    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b99b658-4ce2-42c4-baba-fbf9f8a8025d-maas-api-tls\") pod \"maas-api-e2e-api-a-828ede-5d6f774db9-7m4gt\" (UID: \"1b99b658-4ce2-42c4-baba-fbf9f8a8025d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-828ede-5d6f774db9-7m4gt"
Jul 22 02:24:50.673736 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:50.673714    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l99b7\" (UniqueName: \"kubernetes.io/projected/1b99b658-4ce2-42c4-baba-fbf9f8a8025d-kube-api-access-l99b7\") pod \"maas-api-e2e-api-a-828ede-5d6f774db9-7m4gt\" (UID: \"1b99b658-4ce2-42c4-baba-fbf9f8a8025d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-828ede-5d6f774db9-7m4gt"
Jul 22 02:24:50.818353 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:50.818294    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-828ede-5d6f774db9-7m4gt"
Jul 22 02:24:50.952864 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:50.952840    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-828ede-5d6f774db9-7m4gt"]
Jul 22 02:24:50.954660 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:24:50.954635    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b99b658_4ce2_42c4_baba_fbf9f8a8025d.slice/crio-1d13229accca9b48eefae9ea1ef32adbd686e5263859cadf546b68905f157ada WatchSource:0}: Error finding container 1d13229accca9b48eefae9ea1ef32adbd686e5263859cadf546b68905f157ada: Status 404 returned error can't find the container with id 1d13229accca9b48eefae9ea1ef32adbd686e5263859cadf546b68905f157ada
Jul 22 02:24:50.956276 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:50.956258    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 22 02:24:51.679947 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:51.679915    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-828ede-5d6f774db9-7m4gt" event={"ID":"1b99b658-4ce2-42c4-baba-fbf9f8a8025d","Type":"ContainerStarted","Data":"f4dde2a6fab4fc1385405805403d37e747976abe573aaa46ce030a889fcf71d0"}
Jul 22 02:24:51.680299 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:51.679957    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-828ede-5d6f774db9-7m4gt" event={"ID":"1b99b658-4ce2-42c4-baba-fbf9f8a8025d","Type":"ContainerStarted","Data":"1d13229accca9b48eefae9ea1ef32adbd686e5263859cadf546b68905f157ada"}
Jul 22 02:24:51.680299 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:51.680033    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-828ede-5d6f774db9-7m4gt"
Jul 22 02:24:51.702048 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:51.702001    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-828ede-5d6f774db9-7m4gt" podStartSLOduration=1.204185791 podStartE2EDuration="1.701986367s" totalImagesPullingTime="497.800576ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:24:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:24:51.70034127 +0000 UTC m=+2244.544170401" watchObservedRunningTime="2026-07-22 02:24:51.701986367 +0000 UTC m=+2244.545815521"
Jul 22 02:24:57.690476 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:57.690451    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-828ede-5d6f774db9-7m4gt"
Jul 22 02:24:58.823473 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:58.823442    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-da3c46-85d75d8d99-6v899"]
Jul 22 02:24:58.827024 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:58.827008    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da3c46-85d75d8d99-6v899"
Jul 22 02:24:58.829721 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:58.829698    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-da3c46-serving-cert\""
Jul 22 02:24:58.841996 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:58.841973    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-da3c46-85d75d8d99-6v899"]
Jul 22 02:24:58.931820 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:58.931783    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh4s4\" (UniqueName: \"kubernetes.io/projected/cb1c4611-8f9a-42b5-ad22-d0562be962de-kube-api-access-wh4s4\") pod \"maas-api-e2e-api-b-da3c46-85d75d8d99-6v899\" (UID: \"cb1c4611-8f9a-42b5-ad22-d0562be962de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da3c46-85d75d8d99-6v899"
Jul 22 02:24:58.931820 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:58.931822    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb1c4611-8f9a-42b5-ad22-d0562be962de-maas-api-tls\") pod \"maas-api-e2e-api-b-da3c46-85d75d8d99-6v899\" (UID: \"cb1c4611-8f9a-42b5-ad22-d0562be962de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da3c46-85d75d8d99-6v899"
Jul 22 02:24:58.932028 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:58.931881    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb1c4611-8f9a-42b5-ad22-d0562be962de-openshift-service-ca\") pod \"maas-api-e2e-api-b-da3c46-85d75d8d99-6v899\" (UID: \"cb1c4611-8f9a-42b5-ad22-d0562be962de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da3c46-85d75d8d99-6v899"
Jul 22 02:24:59.032709 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:59.032668    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb1c4611-8f9a-42b5-ad22-d0562be962de-openshift-service-ca\") pod \"maas-api-e2e-api-b-da3c46-85d75d8d99-6v899\" (UID: \"cb1c4611-8f9a-42b5-ad22-d0562be962de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da3c46-85d75d8d99-6v899"
Jul 22 02:24:59.032896 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:59.032790    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh4s4\" (UniqueName: \"kubernetes.io/projected/cb1c4611-8f9a-42b5-ad22-d0562be962de-kube-api-access-wh4s4\") pod \"maas-api-e2e-api-b-da3c46-85d75d8d99-6v899\" (UID: \"cb1c4611-8f9a-42b5-ad22-d0562be962de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da3c46-85d75d8d99-6v899"
Jul 22 02:24:59.032896 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:59.032823    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb1c4611-8f9a-42b5-ad22-d0562be962de-maas-api-tls\") pod \"maas-api-e2e-api-b-da3c46-85d75d8d99-6v899\" (UID: \"cb1c4611-8f9a-42b5-ad22-d0562be962de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da3c46-85d75d8d99-6v899"
Jul 22 02:24:59.033016 ip-10-0-135-230 kubenswrapper[2563]: E0722 02:24:59.032964    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-da3c46-serving-cert: secret "maas-api-e2e-api-b-da3c46-serving-cert" not found
Jul 22 02:24:59.033076 ip-10-0-135-230 kubenswrapper[2563]: E0722 02:24:59.033034    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb1c4611-8f9a-42b5-ad22-d0562be962de-maas-api-tls podName:cb1c4611-8f9a-42b5-ad22-d0562be962de nodeName:}" failed. No retries permitted until 2026-07-22 02:24:59.533012899 +0000 UTC m=+2252.376842022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cb1c4611-8f9a-42b5-ad22-d0562be962de-maas-api-tls") pod "maas-api-e2e-api-b-da3c46-85d75d8d99-6v899" (UID: "cb1c4611-8f9a-42b5-ad22-d0562be962de") : secret "maas-api-e2e-api-b-da3c46-serving-cert" not found
Jul 22 02:24:59.033404 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:59.033364    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb1c4611-8f9a-42b5-ad22-d0562be962de-openshift-service-ca\") pod \"maas-api-e2e-api-b-da3c46-85d75d8d99-6v899\" (UID: \"cb1c4611-8f9a-42b5-ad22-d0562be962de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da3c46-85d75d8d99-6v899"
Jul 22 02:24:59.041203 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:59.041182    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh4s4\" (UniqueName: \"kubernetes.io/projected/cb1c4611-8f9a-42b5-ad22-d0562be962de-kube-api-access-wh4s4\") pod \"maas-api-e2e-api-b-da3c46-85d75d8d99-6v899\" (UID: \"cb1c4611-8f9a-42b5-ad22-d0562be962de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da3c46-85d75d8d99-6v899"
Jul 22 02:24:59.537126 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:59.537094    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb1c4611-8f9a-42b5-ad22-d0562be962de-maas-api-tls\") pod \"maas-api-e2e-api-b-da3c46-85d75d8d99-6v899\" (UID: \"cb1c4611-8f9a-42b5-ad22-d0562be962de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da3c46-85d75d8d99-6v899"
Jul 22 02:24:59.539643 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:59.539612    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb1c4611-8f9a-42b5-ad22-d0562be962de-maas-api-tls\") pod \"maas-api-e2e-api-b-da3c46-85d75d8d99-6v899\" (UID: \"cb1c4611-8f9a-42b5-ad22-d0562be962de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da3c46-85d75d8d99-6v899"
Jul 22 02:24:59.737206 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:59.737171    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da3c46-85d75d8d99-6v899"
Jul 22 02:24:59.871918 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:24:59.871895    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-da3c46-85d75d8d99-6v899"]
Jul 22 02:24:59.873453 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:24:59.873429    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb1c4611_8f9a_42b5_ad22_d0562be962de.slice/crio-a004afcf81bff2673e2f5a24caa1daa3647ff7a1e1b82701f719fb572b586c28 WatchSource:0}: Error finding container a004afcf81bff2673e2f5a24caa1daa3647ff7a1e1b82701f719fb572b586c28: Status 404 returned error can't find the container with id a004afcf81bff2673e2f5a24caa1daa3647ff7a1e1b82701f719fb572b586c28
Jul 22 02:25:00.742628 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:00.742596    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da3c46-85d75d8d99-6v899" event={"ID":"cb1c4611-8f9a-42b5-ad22-d0562be962de","Type":"ContainerStarted","Data":"8aca5d2c0c516ce02e4e3058fc95d948cc0ea1c7ef9116a16b28782feeeebd60"}
Jul 22 02:25:00.742808 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:00.742649    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da3c46-85d75d8d99-6v899" event={"ID":"cb1c4611-8f9a-42b5-ad22-d0562be962de","Type":"ContainerStarted","Data":"a004afcf81bff2673e2f5a24caa1daa3647ff7a1e1b82701f719fb572b586c28"}
Jul 22 02:25:00.742808 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:00.742679    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da3c46-85d75d8d99-6v899"
Jul 22 02:25:00.759897 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:00.759854    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da3c46-85d75d8d99-6v899" podStartSLOduration=2.345074161 podStartE2EDuration="2.759841495s" totalImagesPullingTime="414.767334ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:24:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:25:00.75814488 +0000 UTC m=+2253.601974011" watchObservedRunningTime="2026-07-22 02:25:00.759841495 +0000 UTC m=+2253.603670629"
Jul 22 02:25:06.753593 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:06.753560    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da3c46-85d75d8d99-6v899"
Jul 22 02:25:11.455659 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:11.455622    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-828ede-2ddkp"]
Jul 22 02:25:11.460244 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:11.460216    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-828ede-2ddkp"
Jul 22 02:25:11.462659 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:11.462618    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wrtw7\""
Jul 22 02:25:11.464410 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:11.464372    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-828ede-2ddkp"]
Jul 22 02:25:11.533844 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:11.533819    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a16b9b7e-fe37-4049-81e7-9cbc928cbcb2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-828ede-2ddkp\" (UID: \"a16b9b7e-fe37-4049-81e7-9cbc928cbcb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-828ede-2ddkp"
Jul 22 02:25:11.634620 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:11.634599    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a16b9b7e-fe37-4049-81e7-9cbc928cbcb2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-828ede-2ddkp\" (UID: \"a16b9b7e-fe37-4049-81e7-9cbc928cbcb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-828ede-2ddkp"
Jul 22 02:25:11.635126 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:11.635109    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a16b9b7e-fe37-4049-81e7-9cbc928cbcb2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-828ede-2ddkp\" (UID: \"a16b9b7e-fe37-4049-81e7-9cbc928cbcb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-828ede-2ddkp"
Jul 22 02:25:11.771867 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:11.771811    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-828ede-2ddkp"
Jul 22 02:25:11.900593 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:11.900545    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-828ede-2ddkp"]
Jul 22 02:25:11.902478 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:25:11.902453    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda16b9b7e_fe37_4049_81e7_9cbc928cbcb2.slice/crio-5b76d3489c92a3943d60eeacf2a34e7f43d59cfd9c4b276ddde63860b0b7eecb WatchSource:0}: Error finding container 5b76d3489c92a3943d60eeacf2a34e7f43d59cfd9c4b276ddde63860b0b7eecb: Status 404 returned error can't find the container with id 5b76d3489c92a3943d60eeacf2a34e7f43d59cfd9c4b276ddde63860b0b7eecb
Jul 22 02:25:12.805120 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:12.805087    2563 generic.go:356] "Generic (PLEG): container finished" podID="a16b9b7e-fe37-4049-81e7-9cbc928cbcb2" containerID="acfc0372620f1fb7ebc95bec95208dfe1a9cc8cd3c1644af30b9d482e333ecf3" exitCode=0
Jul 22 02:25:12.805430 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:12.805170    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-828ede-2ddkp" event={"ID":"a16b9b7e-fe37-4049-81e7-9cbc928cbcb2","Type":"ContainerDied","Data":"acfc0372620f1fb7ebc95bec95208dfe1a9cc8cd3c1644af30b9d482e333ecf3"}
Jul 22 02:25:12.805430 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:12.805228    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-828ede-2ddkp" event={"ID":"a16b9b7e-fe37-4049-81e7-9cbc928cbcb2","Type":"ContainerStarted","Data":"5b76d3489c92a3943d60eeacf2a34e7f43d59cfd9c4b276ddde63860b0b7eecb"}
Jul 22 02:25:13.936224 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:13.936202    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-828ede-2ddkp"
Jul 22 02:25:14.054453 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:14.054433    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a16b9b7e-fe37-4049-81e7-9cbc928cbcb2-maas-api-service-ca\") pod \"a16b9b7e-fe37-4049-81e7-9cbc928cbcb2\" (UID: \"a16b9b7e-fe37-4049-81e7-9cbc928cbcb2\") "
Jul 22 02:25:14.054736 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:14.054717    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a16b9b7e-fe37-4049-81e7-9cbc928cbcb2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a16b9b7e-fe37-4049-81e7-9cbc928cbcb2" (UID: "a16b9b7e-fe37-4049-81e7-9cbc928cbcb2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:25:14.155322 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:14.155270    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a16b9b7e-fe37-4049-81e7-9cbc928cbcb2-maas-api-service-ca\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:25:14.816466 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:14.816433    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-828ede-2ddkp" event={"ID":"a16b9b7e-fe37-4049-81e7-9cbc928cbcb2","Type":"ContainerDied","Data":"5b76d3489c92a3943d60eeacf2a34e7f43d59cfd9c4b276ddde63860b0b7eecb"}
Jul 22 02:25:14.816603 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:14.816478    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b76d3489c92a3943d60eeacf2a34e7f43d59cfd9c4b276ddde63860b0b7eecb"
Jul 22 02:25:14.816603 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:14.816445    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-828ede-2ddkp"
Jul 22 02:25:27.224266 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:27.224230    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-828ede-5d6f774db9-7m4gt"]
Jul 22 02:25:27.224731 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:27.224544    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-828ede-5d6f774db9-7m4gt" podUID="1b99b658-4ce2-42c4-baba-fbf9f8a8025d" containerName="maas-api" containerID="cri-o://f4dde2a6fab4fc1385405805403d37e747976abe573aaa46ce030a889fcf71d0" gracePeriod=30
Jul 22 02:25:27.489534 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:27.489507    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-828ede-5d6f774db9-7m4gt"
Jul 22 02:25:27.654030 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:27.654006    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b99b658-4ce2-42c4-baba-fbf9f8a8025d-maas-api-tls\") pod \"1b99b658-4ce2-42c4-baba-fbf9f8a8025d\" (UID: \"1b99b658-4ce2-42c4-baba-fbf9f8a8025d\") "
Jul 22 02:25:27.654199 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:27.654040    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l99b7\" (UniqueName: \"kubernetes.io/projected/1b99b658-4ce2-42c4-baba-fbf9f8a8025d-kube-api-access-l99b7\") pod \"1b99b658-4ce2-42c4-baba-fbf9f8a8025d\" (UID: \"1b99b658-4ce2-42c4-baba-fbf9f8a8025d\") "
Jul 22 02:25:27.654199 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:27.654098    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b99b658-4ce2-42c4-baba-fbf9f8a8025d-openshift-service-ca\") pod \"1b99b658-4ce2-42c4-baba-fbf9f8a8025d\" (UID: \"1b99b658-4ce2-42c4-baba-fbf9f8a8025d\") "
Jul 22 02:25:27.654515 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:27.654491    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b99b658-4ce2-42c4-baba-fbf9f8a8025d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1b99b658-4ce2-42c4-baba-fbf9f8a8025d" (UID: "1b99b658-4ce2-42c4-baba-fbf9f8a8025d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:25:27.655896 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:27.655874    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b99b658-4ce2-42c4-baba-fbf9f8a8025d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1b99b658-4ce2-42c4-baba-fbf9f8a8025d" (UID: "1b99b658-4ce2-42c4-baba-fbf9f8a8025d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:25:27.655998 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:27.655983    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b99b658-4ce2-42c4-baba-fbf9f8a8025d-kube-api-access-l99b7" (OuterVolumeSpecName: "kube-api-access-l99b7") pod "1b99b658-4ce2-42c4-baba-fbf9f8a8025d" (UID: "1b99b658-4ce2-42c4-baba-fbf9f8a8025d"). InnerVolumeSpecName "kube-api-access-l99b7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:25:27.755327 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:27.755278    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b99b658-4ce2-42c4-baba-fbf9f8a8025d-openshift-service-ca\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:25:27.755327 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:27.755299    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b99b658-4ce2-42c4-baba-fbf9f8a8025d-maas-api-tls\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:25:27.755327 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:27.755308    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l99b7\" (UniqueName: \"kubernetes.io/projected/1b99b658-4ce2-42c4-baba-fbf9f8a8025d-kube-api-access-l99b7\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:25:27.879561 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:27.879524    2563 generic.go:356] "Generic (PLEG): container finished" podID="1b99b658-4ce2-42c4-baba-fbf9f8a8025d" containerID="f4dde2a6fab4fc1385405805403d37e747976abe573aaa46ce030a889fcf71d0" exitCode=0
Jul 22 02:25:27.879666 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:27.879573    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-828ede-5d6f774db9-7m4gt"
Jul 22 02:25:27.879666 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:27.879581    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-828ede-5d6f774db9-7m4gt" event={"ID":"1b99b658-4ce2-42c4-baba-fbf9f8a8025d","Type":"ContainerDied","Data":"f4dde2a6fab4fc1385405805403d37e747976abe573aaa46ce030a889fcf71d0"}
Jul 22 02:25:27.879666 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:27.879635    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-828ede-5d6f774db9-7m4gt" event={"ID":"1b99b658-4ce2-42c4-baba-fbf9f8a8025d","Type":"ContainerDied","Data":"1d13229accca9b48eefae9ea1ef32adbd686e5263859cadf546b68905f157ada"}
Jul 22 02:25:27.879666 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:27.879657    2563 scope.go:117] "RemoveContainer" containerID="f4dde2a6fab4fc1385405805403d37e747976abe573aaa46ce030a889fcf71d0"
Jul 22 02:25:27.888438 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:27.888365    2563 scope.go:117] "RemoveContainer" containerID="f4dde2a6fab4fc1385405805403d37e747976abe573aaa46ce030a889fcf71d0"
Jul 22 02:25:27.888652 ip-10-0-135-230 kubenswrapper[2563]: E0722 02:25:27.888634    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4dde2a6fab4fc1385405805403d37e747976abe573aaa46ce030a889fcf71d0\": container with ID starting with f4dde2a6fab4fc1385405805403d37e747976abe573aaa46ce030a889fcf71d0 not found: ID does not exist" containerID="f4dde2a6fab4fc1385405805403d37e747976abe573aaa46ce030a889fcf71d0"
Jul 22 02:25:27.888710 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:27.888664    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4dde2a6fab4fc1385405805403d37e747976abe573aaa46ce030a889fcf71d0"} err="failed to get container status \"f4dde2a6fab4fc1385405805403d37e747976abe573aaa46ce030a889fcf71d0\": rpc error: code = NotFound desc = could not find container \"f4dde2a6fab4fc1385405805403d37e747976abe573aaa46ce030a889fcf71d0\": container with ID starting with f4dde2a6fab4fc1385405805403d37e747976abe573aaa46ce030a889fcf71d0 not found: ID does not exist"
Jul 22 02:25:27.899209 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:27.899184    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-828ede-5d6f774db9-7m4gt"]
Jul 22 02:25:27.906353 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:27.906332    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-828ede-5d6f774db9-7m4gt"]
Jul 22 02:25:28.018545 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:28.018486    2563 scope.go:117] "RemoveContainer" containerID="a56994ca1ba0bfc0b7fc7a6ad5e2f4a0a8a3092790e874810d5033d6146237aa"
Jul 22 02:25:29.663250 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:29.663217    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b99b658-4ce2-42c4-baba-fbf9f8a8025d" path="/var/lib/kubelet/pods/1b99b658-4ce2-42c4-baba-fbf9f8a8025d/volumes"
Jul 22 02:25:31.549513 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:31.549481    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-828ede-2ddkp"]
Jul 22 02:25:31.552764 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:31.552741    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-828ede-2ddkp"]
Jul 22 02:25:31.664722 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:31.664690    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a16b9b7e-fe37-4049-81e7-9cbc928cbcb2" path="/var/lib/kubelet/pods/a16b9b7e-fe37-4049-81e7-9cbc928cbcb2/volumes"
Jul 22 02:25:38.491938 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:38.491910    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da3c46-c4vkh"]
Jul 22 02:25:38.492444 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:38.492229    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b99b658-4ce2-42c4-baba-fbf9f8a8025d" containerName="maas-api"
Jul 22 02:25:38.492444 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:38.492255    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b99b658-4ce2-42c4-baba-fbf9f8a8025d" containerName="maas-api"
Jul 22 02:25:38.492444 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:38.492276    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a16b9b7e-fe37-4049-81e7-9cbc928cbcb2" containerName="revoke-keys"
Jul 22 02:25:38.492444 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:38.492281    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a16b9b7e-fe37-4049-81e7-9cbc928cbcb2" containerName="revoke-keys"
Jul 22 02:25:38.492444 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:38.492341    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a16b9b7e-fe37-4049-81e7-9cbc928cbcb2" containerName="revoke-keys"
Jul 22 02:25:38.492444 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:38.492348    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b99b658-4ce2-42c4-baba-fbf9f8a8025d" containerName="maas-api"
Jul 22 02:25:38.495152 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:38.495130    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da3c46-c4vkh"
Jul 22 02:25:38.497511 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:38.497489    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wrtw7\""
Jul 22 02:25:38.506998 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:38.506977    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da3c46-c4vkh"]
Jul 22 02:25:38.646531 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:38.646504    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c3c076b-9a2e-4b7d-aa44-c0e17b1ddd1d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-da3c46-c4vkh\" (UID: \"8c3c076b-9a2e-4b7d-aa44-c0e17b1ddd1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da3c46-c4vkh"
Jul 22 02:25:38.747719 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:38.747659    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c3c076b-9a2e-4b7d-aa44-c0e17b1ddd1d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-da3c46-c4vkh\" (UID: \"8c3c076b-9a2e-4b7d-aa44-c0e17b1ddd1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da3c46-c4vkh"
Jul 22 02:25:38.748263 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:38.748241    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c3c076b-9a2e-4b7d-aa44-c0e17b1ddd1d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-da3c46-c4vkh\" (UID: \"8c3c076b-9a2e-4b7d-aa44-c0e17b1ddd1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da3c46-c4vkh"
Jul 22 02:25:38.804793 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:38.804771    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da3c46-c4vkh"
Jul 22 02:25:38.959634 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:38.959572    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da3c46-c4vkh"]
Jul 22 02:25:38.961930 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:25:38.961902    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c3c076b_9a2e_4b7d_aa44_c0e17b1ddd1d.slice/crio-02c4517624c4e5911b1fbfdde8d9281f493b578f022fe6f8f4df5b6144675f25 WatchSource:0}: Error finding container 02c4517624c4e5911b1fbfdde8d9281f493b578f022fe6f8f4df5b6144675f25: Status 404 returned error can't find the container with id 02c4517624c4e5911b1fbfdde8d9281f493b578f022fe6f8f4df5b6144675f25
Jul 22 02:25:39.947780 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:39.947714    2563 generic.go:356] "Generic (PLEG): container finished" podID="8c3c076b-9a2e-4b7d-aa44-c0e17b1ddd1d" containerID="8cd040bae1aec9569743ca52107358094dd08b04890b51f4eb417da9358f7e95" exitCode=0
Jul 22 02:25:39.948095 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:39.947796    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da3c46-c4vkh" event={"ID":"8c3c076b-9a2e-4b7d-aa44-c0e17b1ddd1d","Type":"ContainerDied","Data":"8cd040bae1aec9569743ca52107358094dd08b04890b51f4eb417da9358f7e95"}
Jul 22 02:25:39.948095 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:39.947851    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da3c46-c4vkh" event={"ID":"8c3c076b-9a2e-4b7d-aa44-c0e17b1ddd1d","Type":"ContainerStarted","Data":"02c4517624c4e5911b1fbfdde8d9281f493b578f022fe6f8f4df5b6144675f25"}
Jul 22 02:25:41.076116 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:41.076093    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da3c46-c4vkh"
Jul 22 02:25:41.164894 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:41.164872    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c3c076b-9a2e-4b7d-aa44-c0e17b1ddd1d-maas-api-service-ca\") pod \"8c3c076b-9a2e-4b7d-aa44-c0e17b1ddd1d\" (UID: \"8c3c076b-9a2e-4b7d-aa44-c0e17b1ddd1d\") "
Jul 22 02:25:41.165171 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:41.165150    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c3c076b-9a2e-4b7d-aa44-c0e17b1ddd1d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8c3c076b-9a2e-4b7d-aa44-c0e17b1ddd1d" (UID: "8c3c076b-9a2e-4b7d-aa44-c0e17b1ddd1d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:25:41.266420 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:41.266352    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c3c076b-9a2e-4b7d-aa44-c0e17b1ddd1d-maas-api-service-ca\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:25:41.958315 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:41.958288    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da3c46-c4vkh"
Jul 22 02:25:41.958454 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:41.958310    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da3c46-c4vkh" event={"ID":"8c3c076b-9a2e-4b7d-aa44-c0e17b1ddd1d","Type":"ContainerDied","Data":"02c4517624c4e5911b1fbfdde8d9281f493b578f022fe6f8f4df5b6144675f25"}
Jul 22 02:25:41.958454 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:41.958350    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02c4517624c4e5911b1fbfdde8d9281f493b578f022fe6f8f4df5b6144675f25"
Jul 22 02:25:53.995974 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:53.995939    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-da3c46-85d75d8d99-6v899"]
Jul 22 02:25:53.996511 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:53.996195    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da3c46-85d75d8d99-6v899" podUID="cb1c4611-8f9a-42b5-ad22-d0562be962de" containerName="maas-api" containerID="cri-o://8aca5d2c0c516ce02e4e3058fc95d948cc0ea1c7ef9116a16b28782feeeebd60" gracePeriod=30
Jul 22 02:25:54.239414 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:54.239392    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da3c46-85d75d8d99-6v899"
Jul 22 02:25:54.262067 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:54.262003    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb1c4611-8f9a-42b5-ad22-d0562be962de-maas-api-tls\") pod \"cb1c4611-8f9a-42b5-ad22-d0562be962de\" (UID: \"cb1c4611-8f9a-42b5-ad22-d0562be962de\") "
Jul 22 02:25:54.262067 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:54.262063    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb1c4611-8f9a-42b5-ad22-d0562be962de-openshift-service-ca\") pod \"cb1c4611-8f9a-42b5-ad22-d0562be962de\" (UID: \"cb1c4611-8f9a-42b5-ad22-d0562be962de\") "
Jul 22 02:25:54.262236 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:54.262100    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wh4s4\" (UniqueName: \"kubernetes.io/projected/cb1c4611-8f9a-42b5-ad22-d0562be962de-kube-api-access-wh4s4\") pod \"cb1c4611-8f9a-42b5-ad22-d0562be962de\" (UID: \"cb1c4611-8f9a-42b5-ad22-d0562be962de\") "
Jul 22 02:25:54.262506 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:54.262479    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb1c4611-8f9a-42b5-ad22-d0562be962de-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cb1c4611-8f9a-42b5-ad22-d0562be962de" (UID: "cb1c4611-8f9a-42b5-ad22-d0562be962de"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:25:54.264072 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:54.264046    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb1c4611-8f9a-42b5-ad22-d0562be962de-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cb1c4611-8f9a-42b5-ad22-d0562be962de" (UID: "cb1c4611-8f9a-42b5-ad22-d0562be962de"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:25:54.264175 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:54.264140    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb1c4611-8f9a-42b5-ad22-d0562be962de-kube-api-access-wh4s4" (OuterVolumeSpecName: "kube-api-access-wh4s4") pod "cb1c4611-8f9a-42b5-ad22-d0562be962de" (UID: "cb1c4611-8f9a-42b5-ad22-d0562be962de"). InnerVolumeSpecName "kube-api-access-wh4s4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:25:54.363021 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:54.363000    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb1c4611-8f9a-42b5-ad22-d0562be962de-maas-api-tls\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:25:54.363021 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:54.363021    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb1c4611-8f9a-42b5-ad22-d0562be962de-openshift-service-ca\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:25:54.363140 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:54.363030    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh4s4\" (UniqueName: \"kubernetes.io/projected/cb1c4611-8f9a-42b5-ad22-d0562be962de-kube-api-access-wh4s4\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:25:55.032113 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:55.032081    2563 generic.go:356] "Generic (PLEG): container finished" podID="cb1c4611-8f9a-42b5-ad22-d0562be962de" containerID="8aca5d2c0c516ce02e4e3058fc95d948cc0ea1c7ef9116a16b28782feeeebd60" exitCode=0
Jul 22 02:25:55.032547 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:55.032128    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da3c46-85d75d8d99-6v899"
Jul 22 02:25:55.032547 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:55.032166    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da3c46-85d75d8d99-6v899" event={"ID":"cb1c4611-8f9a-42b5-ad22-d0562be962de","Type":"ContainerDied","Data":"8aca5d2c0c516ce02e4e3058fc95d948cc0ea1c7ef9116a16b28782feeeebd60"}
Jul 22 02:25:55.032547 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:55.032214    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-da3c46-85d75d8d99-6v899" event={"ID":"cb1c4611-8f9a-42b5-ad22-d0562be962de","Type":"ContainerDied","Data":"a004afcf81bff2673e2f5a24caa1daa3647ff7a1e1b82701f719fb572b586c28"}
Jul 22 02:25:55.032547 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:55.032234    2563 scope.go:117] "RemoveContainer" containerID="8aca5d2c0c516ce02e4e3058fc95d948cc0ea1c7ef9116a16b28782feeeebd60"
Jul 22 02:25:55.043621 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:55.043598    2563 scope.go:117] "RemoveContainer" containerID="8aca5d2c0c516ce02e4e3058fc95d948cc0ea1c7ef9116a16b28782feeeebd60"
Jul 22 02:25:55.043900 ip-10-0-135-230 kubenswrapper[2563]: E0722 02:25:55.043877    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aca5d2c0c516ce02e4e3058fc95d948cc0ea1c7ef9116a16b28782feeeebd60\": container with ID starting with 8aca5d2c0c516ce02e4e3058fc95d948cc0ea1c7ef9116a16b28782feeeebd60 not found: ID does not exist" containerID="8aca5d2c0c516ce02e4e3058fc95d948cc0ea1c7ef9116a16b28782feeeebd60"
Jul 22 02:25:55.043943 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:55.043910    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aca5d2c0c516ce02e4e3058fc95d948cc0ea1c7ef9116a16b28782feeeebd60"} err="failed to get container status \"8aca5d2c0c516ce02e4e3058fc95d948cc0ea1c7ef9116a16b28782feeeebd60\": rpc error: code = NotFound desc = could not find container \"8aca5d2c0c516ce02e4e3058fc95d948cc0ea1c7ef9116a16b28782feeeebd60\": container with ID starting with 8aca5d2c0c516ce02e4e3058fc95d948cc0ea1c7ef9116a16b28782feeeebd60 not found: ID does not exist"
Jul 22 02:25:55.056437 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:55.056372    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-da3c46-85d75d8d99-6v899"]
Jul 22 02:25:55.063796 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:55.063771    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-da3c46-85d75d8d99-6v899"]
Jul 22 02:25:55.664525 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:55.664494    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb1c4611-8f9a-42b5-ad22-d0562be962de" path="/var/lib/kubelet/pods/cb1c4611-8f9a-42b5-ad22-d0562be962de/volumes"
Jul 22 02:25:58.594369 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:58.594334    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da3c46-c4vkh"]
Jul 22 02:25:58.601855 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:58.601829    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da3c46-c4vkh"]
Jul 22 02:25:59.663276 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:25:59.663237    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c3c076b-9a2e-4b7d-aa44-c0e17b1ddd1d" path="/var/lib/kubelet/pods/8c3c076b-9a2e-4b7d-aa44-c0e17b1ddd1d/volumes"
Jul 22 02:26:10.105786 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:10.105756    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-9d0056-6f9c55556b-8l464"]
Jul 22 02:26:10.106341 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:10.106270    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c3c076b-9a2e-4b7d-aa44-c0e17b1ddd1d" containerName="revoke-keys"
Jul 22 02:26:10.106341 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:10.106291    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c3c076b-9a2e-4b7d-aa44-c0e17b1ddd1d" containerName="revoke-keys"
Jul 22 02:26:10.106341 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:10.106331    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb1c4611-8f9a-42b5-ad22-d0562be962de" containerName="maas-api"
Jul 22 02:26:10.106341 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:10.106339    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb1c4611-8f9a-42b5-ad22-d0562be962de" containerName="maas-api"
Jul 22 02:26:10.106586 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:10.106443    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb1c4611-8f9a-42b5-ad22-d0562be962de" containerName="maas-api"
Jul 22 02:26:10.106586 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:10.106459    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c3c076b-9a2e-4b7d-aa44-c0e17b1ddd1d" containerName="revoke-keys"
Jul 22 02:26:10.109369 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:10.109349    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9d0056-6f9c55556b-8l464"
Jul 22 02:26:10.112448 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:10.112427    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-9d0056-serving-cert\""
Jul 22 02:26:10.121210 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:10.121188    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-9d0056-6f9c55556b-8l464"]
Jul 22 02:26:10.187502 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:10.187472    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/816d1cf3-7171-455e-9e87-405792553a60-maas-api-tls\") pod \"maas-api-e2e-shared-a-9d0056-6f9c55556b-8l464\" (UID: \"816d1cf3-7171-455e-9e87-405792553a60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9d0056-6f9c55556b-8l464"
Jul 22 02:26:10.187610 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:10.187512    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rz6b\" (UniqueName: \"kubernetes.io/projected/816d1cf3-7171-455e-9e87-405792553a60-kube-api-access-2rz6b\") pod \"maas-api-e2e-shared-a-9d0056-6f9c55556b-8l464\" (UID: \"816d1cf3-7171-455e-9e87-405792553a60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9d0056-6f9c55556b-8l464"
Jul 22 02:26:10.187610 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:10.187575    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/816d1cf3-7171-455e-9e87-405792553a60-openshift-service-ca\") pod \"maas-api-e2e-shared-a-9d0056-6f9c55556b-8l464\" (UID: \"816d1cf3-7171-455e-9e87-405792553a60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9d0056-6f9c55556b-8l464"
Jul 22 02:26:10.288209 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:10.288181    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/816d1cf3-7171-455e-9e87-405792553a60-maas-api-tls\") pod \"maas-api-e2e-shared-a-9d0056-6f9c55556b-8l464\" (UID: \"816d1cf3-7171-455e-9e87-405792553a60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9d0056-6f9c55556b-8l464"
Jul 22 02:26:10.288328 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:10.288216    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rz6b\" (UniqueName: \"kubernetes.io/projected/816d1cf3-7171-455e-9e87-405792553a60-kube-api-access-2rz6b\") pod \"maas-api-e2e-shared-a-9d0056-6f9c55556b-8l464\" (UID: \"816d1cf3-7171-455e-9e87-405792553a60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9d0056-6f9c55556b-8l464"
Jul 22 02:26:10.288328 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:10.288243    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/816d1cf3-7171-455e-9e87-405792553a60-openshift-service-ca\") pod \"maas-api-e2e-shared-a-9d0056-6f9c55556b-8l464\" (UID: \"816d1cf3-7171-455e-9e87-405792553a60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9d0056-6f9c55556b-8l464"
Jul 22 02:26:10.288485 ip-10-0-135-230 kubenswrapper[2563]: E0722 02:26:10.288325    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-9d0056-serving-cert: secret "maas-api-e2e-shared-a-9d0056-serving-cert" not found
Jul 22 02:26:10.288485 ip-10-0-135-230 kubenswrapper[2563]: E0722 02:26:10.288404    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/816d1cf3-7171-455e-9e87-405792553a60-maas-api-tls podName:816d1cf3-7171-455e-9e87-405792553a60 nodeName:}" failed. No retries permitted until 2026-07-22 02:26:10.788367164 +0000 UTC m=+2323.632196273 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/816d1cf3-7171-455e-9e87-405792553a60-maas-api-tls") pod "maas-api-e2e-shared-a-9d0056-6f9c55556b-8l464" (UID: "816d1cf3-7171-455e-9e87-405792553a60") : secret "maas-api-e2e-shared-a-9d0056-serving-cert" not found
Jul 22 02:26:10.288899 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:10.288879    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/816d1cf3-7171-455e-9e87-405792553a60-openshift-service-ca\") pod \"maas-api-e2e-shared-a-9d0056-6f9c55556b-8l464\" (UID: \"816d1cf3-7171-455e-9e87-405792553a60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9d0056-6f9c55556b-8l464"
Jul 22 02:26:10.298546 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:10.298525    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rz6b\" (UniqueName: \"kubernetes.io/projected/816d1cf3-7171-455e-9e87-405792553a60-kube-api-access-2rz6b\") pod \"maas-api-e2e-shared-a-9d0056-6f9c55556b-8l464\" (UID: \"816d1cf3-7171-455e-9e87-405792553a60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9d0056-6f9c55556b-8l464"
Jul 22 02:26:10.792741 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:10.792703    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/816d1cf3-7171-455e-9e87-405792553a60-maas-api-tls\") pod \"maas-api-e2e-shared-a-9d0056-6f9c55556b-8l464\" (UID: \"816d1cf3-7171-455e-9e87-405792553a60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9d0056-6f9c55556b-8l464"
Jul 22 02:26:10.795794 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:10.795757    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/816d1cf3-7171-455e-9e87-405792553a60-maas-api-tls\") pod \"maas-api-e2e-shared-a-9d0056-6f9c55556b-8l464\" (UID: \"816d1cf3-7171-455e-9e87-405792553a60\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9d0056-6f9c55556b-8l464"
Jul 22 02:26:11.021251 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:11.021218    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9d0056-6f9c55556b-8l464"
Jul 22 02:26:11.150930 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:11.150904    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-9d0056-6f9c55556b-8l464"]
Jul 22 02:26:11.152331 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:26:11.152297    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod816d1cf3_7171_455e_9e87_405792553a60.slice/crio-e79f84a274ec2d9f4000f5aa78bf57fb1fa66ba0d4057146b183de60c14fc9d1 WatchSource:0}: Error finding container e79f84a274ec2d9f4000f5aa78bf57fb1fa66ba0d4057146b183de60c14fc9d1: Status 404 returned error can't find the container with id e79f84a274ec2d9f4000f5aa78bf57fb1fa66ba0d4057146b183de60c14fc9d1
Jul 22 02:26:12.123119 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:12.123087    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9d0056-6f9c55556b-8l464" event={"ID":"816d1cf3-7171-455e-9e87-405792553a60","Type":"ContainerStarted","Data":"820ca8244401e0e2db58d56d1c0e099e83edba187a396749b323fd56c81e0860"}
Jul 22 02:26:12.123293 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:12.123134    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9d0056-6f9c55556b-8l464" event={"ID":"816d1cf3-7171-455e-9e87-405792553a60","Type":"ContainerStarted","Data":"e79f84a274ec2d9f4000f5aa78bf57fb1fa66ba0d4057146b183de60c14fc9d1"}
Jul 22 02:26:12.123293 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:12.123231    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9d0056-6f9c55556b-8l464"
Jul 22 02:26:12.145621 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:12.145576    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9d0056-6f9c55556b-8l464" podStartSLOduration=1.744422583 podStartE2EDuration="2.145563192s" totalImagesPullingTime="401.140609ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:26:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:26:12.142050099 +0000 UTC m=+2324.985879229" watchObservedRunningTime="2026-07-22 02:26:12.145563192 +0000 UTC m=+2324.989392322"
Jul 22 02:26:18.135649 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:18.135621    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9d0056-6f9c55556b-8l464"
Jul 22 02:26:19.430125 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:19.430031    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a73722-546b4bf4bf-jmtkb"]
Jul 22 02:26:19.433573 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:19.433555    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a73722-546b4bf4bf-jmtkb"
Jul 22 02:26:19.436933 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:19.436917    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-a73722-serving-cert\""
Jul 22 02:26:19.462856 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:19.462831    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a73722-546b4bf4bf-jmtkb"]
Jul 22 02:26:19.559585 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:19.559523    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbc1c1da-fc82-44e3-8362-dd0872803a46-maas-api-tls\") pod \"maas-api-e2e-shared-b-a73722-546b4bf4bf-jmtkb\" (UID: \"cbc1c1da-fc82-44e3-8362-dd0872803a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a73722-546b4bf4bf-jmtkb"
Jul 22 02:26:19.559699 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:19.559587    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfn97\" (UniqueName: \"kubernetes.io/projected/cbc1c1da-fc82-44e3-8362-dd0872803a46-kube-api-access-hfn97\") pod \"maas-api-e2e-shared-b-a73722-546b4bf4bf-jmtkb\" (UID: \"cbc1c1da-fc82-44e3-8362-dd0872803a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a73722-546b4bf4bf-jmtkb"
Jul 22 02:26:19.559765 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:19.559705    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc1c1da-fc82-44e3-8362-dd0872803a46-openshift-service-ca\") pod \"maas-api-e2e-shared-b-a73722-546b4bf4bf-jmtkb\" (UID: \"cbc1c1da-fc82-44e3-8362-dd0872803a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a73722-546b4bf4bf-jmtkb"
Jul 22 02:26:19.660204 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:19.660181    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc1c1da-fc82-44e3-8362-dd0872803a46-openshift-service-ca\") pod \"maas-api-e2e-shared-b-a73722-546b4bf4bf-jmtkb\" (UID: \"cbc1c1da-fc82-44e3-8362-dd0872803a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a73722-546b4bf4bf-jmtkb"
Jul 22 02:26:19.660315 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:19.660218    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbc1c1da-fc82-44e3-8362-dd0872803a46-maas-api-tls\") pod \"maas-api-e2e-shared-b-a73722-546b4bf4bf-jmtkb\" (UID: \"cbc1c1da-fc82-44e3-8362-dd0872803a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a73722-546b4bf4bf-jmtkb"
Jul 22 02:26:19.660315 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:19.660236    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfn97\" (UniqueName: \"kubernetes.io/projected/cbc1c1da-fc82-44e3-8362-dd0872803a46-kube-api-access-hfn97\") pod \"maas-api-e2e-shared-b-a73722-546b4bf4bf-jmtkb\" (UID: \"cbc1c1da-fc82-44e3-8362-dd0872803a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a73722-546b4bf4bf-jmtkb"
Jul 22 02:26:19.660456 ip-10-0-135-230 kubenswrapper[2563]: E0722 02:26:19.660345    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-a73722-serving-cert: secret "maas-api-e2e-shared-b-a73722-serving-cert" not found
Jul 22 02:26:19.660514 ip-10-0-135-230 kubenswrapper[2563]: E0722 02:26:19.660456    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbc1c1da-fc82-44e3-8362-dd0872803a46-maas-api-tls podName:cbc1c1da-fc82-44e3-8362-dd0872803a46 nodeName:}" failed. No retries permitted until 2026-07-22 02:26:20.160434321 +0000 UTC m=+2333.004263435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cbc1c1da-fc82-44e3-8362-dd0872803a46-maas-api-tls") pod "maas-api-e2e-shared-b-a73722-546b4bf4bf-jmtkb" (UID: "cbc1c1da-fc82-44e3-8362-dd0872803a46") : secret "maas-api-e2e-shared-b-a73722-serving-cert" not found
Jul 22 02:26:19.660882 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:19.660864    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc1c1da-fc82-44e3-8362-dd0872803a46-openshift-service-ca\") pod \"maas-api-e2e-shared-b-a73722-546b4bf4bf-jmtkb\" (UID: \"cbc1c1da-fc82-44e3-8362-dd0872803a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a73722-546b4bf4bf-jmtkb"
Jul 22 02:26:19.678854 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:19.678827    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfn97\" (UniqueName: \"kubernetes.io/projected/cbc1c1da-fc82-44e3-8362-dd0872803a46-kube-api-access-hfn97\") pod \"maas-api-e2e-shared-b-a73722-546b4bf4bf-jmtkb\" (UID: \"cbc1c1da-fc82-44e3-8362-dd0872803a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a73722-546b4bf4bf-jmtkb"
Jul 22 02:26:20.164353 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:20.164325    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbc1c1da-fc82-44e3-8362-dd0872803a46-maas-api-tls\") pod \"maas-api-e2e-shared-b-a73722-546b4bf4bf-jmtkb\" (UID: \"cbc1c1da-fc82-44e3-8362-dd0872803a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a73722-546b4bf4bf-jmtkb"
Jul 22 02:26:20.166650 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:20.166630    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbc1c1da-fc82-44e3-8362-dd0872803a46-maas-api-tls\") pod \"maas-api-e2e-shared-b-a73722-546b4bf4bf-jmtkb\" (UID: \"cbc1c1da-fc82-44e3-8362-dd0872803a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a73722-546b4bf4bf-jmtkb"
Jul 22 02:26:20.342508 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:20.342484    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a73722-546b4bf4bf-jmtkb"
Jul 22 02:26:20.493136 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:20.493108    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a73722-546b4bf4bf-jmtkb"]
Jul 22 02:26:20.497085 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:26:20.497058    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbc1c1da_fc82_44e3_8362_dd0872803a46.slice/crio-b7fec58869add0b4e0abb2935f48ebdb39f533ce5c511c2f3c606333c9517707 WatchSource:0}: Error finding container b7fec58869add0b4e0abb2935f48ebdb39f533ce5c511c2f3c606333c9517707: Status 404 returned error can't find the container with id b7fec58869add0b4e0abb2935f48ebdb39f533ce5c511c2f3c606333c9517707
Jul 22 02:26:21.177329 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:21.177292    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a73722-546b4bf4bf-jmtkb" event={"ID":"cbc1c1da-fc82-44e3-8362-dd0872803a46","Type":"ContainerStarted","Data":"3d5232c4b51b1f135585965fb97086c891bcc0ded9f779bdde7540a8351672de"}
Jul 22 02:26:21.177525 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:21.177363    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a73722-546b4bf4bf-jmtkb" event={"ID":"cbc1c1da-fc82-44e3-8362-dd0872803a46","Type":"ContainerStarted","Data":"b7fec58869add0b4e0abb2935f48ebdb39f533ce5c511c2f3c606333c9517707"}
Jul 22 02:26:21.177525 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:21.177419    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a73722-546b4bf4bf-jmtkb"
Jul 22 02:26:21.201805 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:21.201758    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a73722-546b4bf4bf-jmtkb" podStartSLOduration=1.91333211 podStartE2EDuration="2.201745866s" totalImagesPullingTime="288.413756ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:26:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:26:21.199674201 +0000 UTC m=+2334.043503332" watchObservedRunningTime="2026-07-22 02:26:21.201745866 +0000 UTC m=+2334.045575035"
Jul 22 02:26:27.188016 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:27.187990    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a73722-546b4bf4bf-jmtkb"
Jul 22 02:26:28.049573 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:28.049543    2563 scope.go:117] "RemoveContainer" containerID="f93b5cc1dae4ed3035763c6fbb07130b16f6be3c5a4d6311ac89ea2a73ac3d39"
Jul 22 02:26:33.624720 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:33.624688    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-9d0056/auth-test-model-9d0056-kserve-55484f6d59-9htdf"]
Jul 22 02:26:33.628012 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:33.627996    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9d0056/auth-test-model-9d0056-kserve-55484f6d59-9htdf"
Jul 22 02:26:33.649358 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:33.648649    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9d0056\"/\"auth-test-model-9d0056-kserve-self-signed-certs\""
Jul 22 02:26:33.649358 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:33.648869    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-9d0056\"/\"openshift-service-ca.crt\""
Jul 22 02:26:33.649358 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:33.649069    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-9d0056\"/\"kube-root-ca.crt\""
Jul 22 02:26:33.649358 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:33.649233    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9d0056\"/\"default-dockercfg-mwkhg\""
Jul 22 02:26:33.675159 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:33.675133    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9d0056/auth-test-model-9d0056-kserve-55484f6d59-9htdf"]
Jul 22 02:26:33.773839 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:33.773812    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0196dc09-0ee8-4010-8b9b-ed4a645d7b66-dshm\") pod \"auth-test-model-9d0056-kserve-55484f6d59-9htdf\" (UID: \"0196dc09-0ee8-4010-8b9b-ed4a645d7b66\") " pod="ai-tenant-e2e-shared-a-9d0056/auth-test-model-9d0056-kserve-55484f6d59-9htdf"
Jul 22 02:26:33.773975 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:33.773852    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0196dc09-0ee8-4010-8b9b-ed4a645d7b66-model-cache\") pod \"auth-test-model-9d0056-kserve-55484f6d59-9htdf\" (UID: \"0196dc09-0ee8-4010-8b9b-ed4a645d7b66\") " pod="ai-tenant-e2e-shared-a-9d0056/auth-test-model-9d0056-kserve-55484f6d59-9htdf"
Jul 22 02:26:33.773975 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:33.773873    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m42hn\" (UniqueName: \"kubernetes.io/projected/0196dc09-0ee8-4010-8b9b-ed4a645d7b66-kube-api-access-m42hn\") pod \"auth-test-model-9d0056-kserve-55484f6d59-9htdf\" (UID: \"0196dc09-0ee8-4010-8b9b-ed4a645d7b66\") " pod="ai-tenant-e2e-shared-a-9d0056/auth-test-model-9d0056-kserve-55484f6d59-9htdf"
Jul 22 02:26:33.773975 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:33.773932    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0196dc09-0ee8-4010-8b9b-ed4a645d7b66-home\") pod \"auth-test-model-9d0056-kserve-55484f6d59-9htdf\" (UID: \"0196dc09-0ee8-4010-8b9b-ed4a645d7b66\") " pod="ai-tenant-e2e-shared-a-9d0056/auth-test-model-9d0056-kserve-55484f6d59-9htdf"
Jul 22 02:26:33.773975 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:33.773970    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0196dc09-0ee8-4010-8b9b-ed4a645d7b66-tmp-dir\") pod \"auth-test-model-9d0056-kserve-55484f6d59-9htdf\" (UID: \"0196dc09-0ee8-4010-8b9b-ed4a645d7b66\") " pod="ai-tenant-e2e-shared-a-9d0056/auth-test-model-9d0056-kserve-55484f6d59-9htdf"
Jul 22 02:26:33.774127 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:33.773993    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0196dc09-0ee8-4010-8b9b-ed4a645d7b66-tls-certs\") pod \"auth-test-model-9d0056-kserve-55484f6d59-9htdf\" (UID: \"0196dc09-0ee8-4010-8b9b-ed4a645d7b66\") " pod="ai-tenant-e2e-shared-a-9d0056/auth-test-model-9d0056-kserve-55484f6d59-9htdf"
Jul 22 02:26:33.874499 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:33.874472    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0196dc09-0ee8-4010-8b9b-ed4a645d7b66-dshm\") pod \"auth-test-model-9d0056-kserve-55484f6d59-9htdf\" (UID: \"0196dc09-0ee8-4010-8b9b-ed4a645d7b66\") " pod="ai-tenant-e2e-shared-a-9d0056/auth-test-model-9d0056-kserve-55484f6d59-9htdf"
Jul 22 02:26:33.874624 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:33.874517    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0196dc09-0ee8-4010-8b9b-ed4a645d7b66-model-cache\") pod \"auth-test-model-9d0056-kserve-55484f6d59-9htdf\" (UID: \"0196dc09-0ee8-4010-8b9b-ed4a645d7b66\") " pod="ai-tenant-e2e-shared-a-9d0056/auth-test-model-9d0056-kserve-55484f6d59-9htdf"
Jul 22 02:26:33.874624 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:33.874535    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m42hn\" (UniqueName: \"kubernetes.io/projected/0196dc09-0ee8-4010-8b9b-ed4a645d7b66-kube-api-access-m42hn\") pod \"auth-test-model-9d0056-kserve-55484f6d59-9htdf\" (UID: \"0196dc09-0ee8-4010-8b9b-ed4a645d7b66\") " pod="ai-tenant-e2e-shared-a-9d0056/auth-test-model-9d0056-kserve-55484f6d59-9htdf"
Jul 22 02:26:33.874624 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:33.874558    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0196dc09-0ee8-4010-8b9b-ed4a645d7b66-home\") pod \"auth-test-model-9d0056-kserve-55484f6d59-9htdf\" (UID: \"0196dc09-0ee8-4010-8b9b-ed4a645d7b66\") " pod="ai-tenant-e2e-shared-a-9d0056/auth-test-model-9d0056-kserve-55484f6d59-9htdf"
Jul 22 02:26:33.874624 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:33.874598    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0196dc09-0ee8-4010-8b9b-ed4a645d7b66-tmp-dir\") pod \"auth-test-model-9d0056-kserve-55484f6d59-9htdf\" (UID: \"0196dc09-0ee8-4010-8b9b-ed4a645d7b66\") " pod="ai-tenant-e2e-shared-a-9d0056/auth-test-model-9d0056-kserve-55484f6d59-9htdf"
Jul 22 02:26:33.874839 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:33.874630    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0196dc09-0ee8-4010-8b9b-ed4a645d7b66-tls-certs\") pod \"auth-test-model-9d0056-kserve-55484f6d59-9htdf\" (UID: \"0196dc09-0ee8-4010-8b9b-ed4a645d7b66\") " pod="ai-tenant-e2e-shared-a-9d0056/auth-test-model-9d0056-kserve-55484f6d59-9htdf"
Jul 22 02:26:33.874909 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:33.874879    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0196dc09-0ee8-4010-8b9b-ed4a645d7b66-model-cache\") pod \"auth-test-model-9d0056-kserve-55484f6d59-9htdf\" (UID: \"0196dc09-0ee8-4010-8b9b-ed4a645d7b66\") " pod="ai-tenant-e2e-shared-a-9d0056/auth-test-model-9d0056-kserve-55484f6d59-9htdf"
Jul 22 02:26:33.875035 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:33.875007    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0196dc09-0ee8-4010-8b9b-ed4a645d7b66-tmp-dir\") pod \"auth-test-model-9d0056-kserve-55484f6d59-9htdf\" (UID: \"0196dc09-0ee8-4010-8b9b-ed4a645d7b66\") " pod="ai-tenant-e2e-shared-a-9d0056/auth-test-model-9d0056-kserve-55484f6d59-9htdf"
Jul 22 02:26:33.875147 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:33.875039    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0196dc09-0ee8-4010-8b9b-ed4a645d7b66-home\") pod \"auth-test-model-9d0056-kserve-55484f6d59-9htdf\" (UID: \"0196dc09-0ee8-4010-8b9b-ed4a645d7b66\") " pod="ai-tenant-e2e-shared-a-9d0056/auth-test-model-9d0056-kserve-55484f6d59-9htdf"
Jul 22 02:26:33.876746 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:33.876722    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0196dc09-0ee8-4010-8b9b-ed4a645d7b66-dshm\") pod \"auth-test-model-9d0056-kserve-55484f6d59-9htdf\" (UID: \"0196dc09-0ee8-4010-8b9b-ed4a645d7b66\") " pod="ai-tenant-e2e-shared-a-9d0056/auth-test-model-9d0056-kserve-55484f6d59-9htdf"
Jul 22 02:26:33.877210 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:33.877191    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0196dc09-0ee8-4010-8b9b-ed4a645d7b66-tls-certs\") pod \"auth-test-model-9d0056-kserve-55484f6d59-9htdf\" (UID: \"0196dc09-0ee8-4010-8b9b-ed4a645d7b66\") " pod="ai-tenant-e2e-shared-a-9d0056/auth-test-model-9d0056-kserve-55484f6d59-9htdf"
Jul 22 02:26:33.895063 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:33.895035    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m42hn\" (UniqueName: \"kubernetes.io/projected/0196dc09-0ee8-4010-8b9b-ed4a645d7b66-kube-api-access-m42hn\") pod \"auth-test-model-9d0056-kserve-55484f6d59-9htdf\" (UID: \"0196dc09-0ee8-4010-8b9b-ed4a645d7b66\") " pod="ai-tenant-e2e-shared-a-9d0056/auth-test-model-9d0056-kserve-55484f6d59-9htdf"
Jul 22 02:26:33.939495 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:33.939474    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9d0056/auth-test-model-9d0056-kserve-55484f6d59-9htdf"
Jul 22 02:26:34.073838 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:34.073797    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9d0056/auth-test-model-9d0056-kserve-55484f6d59-9htdf"]
Jul 22 02:26:34.078692 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:26:34.078660    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0196dc09_0ee8_4010_8b9b_ed4a645d7b66.slice/crio-57a3465460de06f344ad94af68ac6680b88d5d5b660d177b2b51482a7f9368ed WatchSource:0}: Error finding container 57a3465460de06f344ad94af68ac6680b88d5d5b660d177b2b51482a7f9368ed: Status 404 returned error can't find the container with id 57a3465460de06f344ad94af68ac6680b88d5d5b660d177b2b51482a7f9368ed
Jul 22 02:26:34.242811 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:34.242776    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9d0056/auth-test-model-9d0056-kserve-55484f6d59-9htdf" event={"ID":"0196dc09-0ee8-4010-8b9b-ed4a645d7b66","Type":"ContainerStarted","Data":"4667a01b1e02a55057121a9fe5c7c5132c87743c66261459fb38f30cb656d669"}
Jul 22 02:26:34.242939 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:34.242823    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9d0056/auth-test-model-9d0056-kserve-55484f6d59-9htdf" event={"ID":"0196dc09-0ee8-4010-8b9b-ed4a645d7b66","Type":"ContainerStarted","Data":"57a3465460de06f344ad94af68ac6680b88d5d5b660d177b2b51482a7f9368ed"}
Jul 22 02:26:34.272297 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:34.272251    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-9d0056/auth-test-model-9d0056-kserve-55484f6d59-9htdf" podStartSLOduration=1.272236291 podStartE2EDuration="1.272236291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:26:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:26:34.270889381 +0000 UTC m=+2347.114718507" watchObservedRunningTime="2026-07-22 02:26:34.272236291 +0000 UTC m=+2347.116065421"
Jul 22 02:26:43.939793 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:43.939765    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-9d0056/auth-test-model-9d0056-kserve-55484f6d59-9htdf"
Jul 22 02:26:43.939793 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:43.939810    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-9d0056/auth-test-model-9d0056-kserve-55484f6d59-9htdf"
Jul 22 02:26:43.951564 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:43.951542    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-9d0056/auth-test-model-9d0056-kserve-55484f6d59-9htdf"
Jul 22 02:26:44.303165 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:26:44.303089    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-9d0056/auth-test-model-9d0056-kserve-55484f6d59-9htdf"
Jul 22 02:27:07.567875 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:27:07.567847    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-a73722/auth-test-model-a73722-kserve-669549f979-rj7m7"]
Jul 22 02:27:07.571092 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:27:07.571074    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a73722/auth-test-model-a73722-kserve-669549f979-rj7m7"
Jul 22 02:27:07.575111 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:27:07.575087    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a73722\"/\"default-dockercfg-wlc4l\""
Jul 22 02:27:07.575223 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:27:07.575114    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a73722\"/\"auth-test-model-a73722-kserve-self-signed-certs\""
Jul 22 02:27:07.575343 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:27:07.575329    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a73722\"/\"openshift-service-ca.crt\""
Jul 22 02:27:07.575452 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:27:07.575438    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a73722\"/\"kube-root-ca.crt\""
Jul 22 02:27:07.582078 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:27:07.582060    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a73722/auth-test-model-a73722-kserve-669549f979-rj7m7"]
Jul 22 02:27:07.630798 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:27:07.630775    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnldw\" (UniqueName: \"kubernetes.io/projected/0a68fd46-2cba-4c64-a28d-10bf41f47391-kube-api-access-qnldw\") pod \"auth-test-model-a73722-kserve-669549f979-rj7m7\" (UID: \"0a68fd46-2cba-4c64-a28d-10bf41f47391\") " pod="ai-tenant-e2e-shared-b-a73722/auth-test-model-a73722-kserve-669549f979-rj7m7"
Jul 22 02:27:07.630923 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:27:07.630811    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a68fd46-2cba-4c64-a28d-10bf41f47391-tmp-dir\") pod \"auth-test-model-a73722-kserve-669549f979-rj7m7\" (UID: \"0a68fd46-2cba-4c64-a28d-10bf41f47391\") " pod="ai-tenant-e2e-shared-b-a73722/auth-test-model-a73722-kserve-669549f979-rj7m7"
Jul 22 02:27:07.630923 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:27:07.630860    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a68fd46-2cba-4c64-a28d-10bf41f47391-home\") pod \"auth-test-model-a73722-kserve-669549f979-rj7m7\" (UID: \"0a68fd46-2cba-4c64-a28d-10bf41f47391\") " pod="ai-tenant-e2e-shared-b-a73722/auth-test-model-a73722-kserve-669549f979-rj7m7"
Jul 22 02:27:07.630923 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:27:07.630908    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a68fd46-2cba-4c64-a28d-10bf41f47391-model-cache\") pod \"auth-test-model-a73722-kserve-669549f979-rj7m7\" (UID: \"0a68fd46-2cba-4c64-a28d-10bf41f47391\") " pod="ai-tenant-e2e-shared-b-a73722/auth-test-model-a73722-kserve-669549f979-rj7m7"
Jul 22 02:27:07.631055 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:27:07.630969    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a68fd46-2cba-4c64-a28d-10bf41f47391-tls-certs\") pod \"auth-test-model-a73722-kserve-669549f979-rj7m7\" (UID: \"0a68fd46-2cba-4c64-a28d-10bf41f47391\") " pod="ai-tenant-e2e-shared-b-a73722/auth-test-model-a73722-kserve-669549f979-rj7m7"
Jul 22 02:27:07.631055 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:27:07.631006    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a68fd46-2cba-4c64-a28d-10bf41f47391-dshm\") pod \"auth-test-model-a73722-kserve-669549f979-rj7m7\" (UID: \"0a68fd46-2cba-4c64-a28d-10bf41f47391\") " pod="ai-tenant-e2e-shared-b-a73722/auth-test-model-a73722-kserve-669549f979-rj7m7"
Jul 22 02:27:07.732308 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:27:07.732277    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a68fd46-2cba-4c64-a28d-10bf41f47391-home\") pod \"auth-test-model-a73722-kserve-669549f979-rj7m7\" (UID: \"0a68fd46-2cba-4c64-a28d-10bf41f47391\") " pod="ai-tenant-e2e-shared-b-a73722/auth-test-model-a73722-kserve-669549f979-rj7m7"
Jul 22 02:27:07.732308 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:27:07.732308    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a68fd46-2cba-4c64-a28d-10bf41f47391-model-cache\") pod \"auth-test-model-a73722-kserve-669549f979-rj7m7\" (UID: \"0a68fd46-2cba-4c64-a28d-10bf41f47391\") " pod="ai-tenant-e2e-shared-b-a73722/auth-test-model-a73722-kserve-669549f979-rj7m7"
Jul 22 02:27:07.732518 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:27:07.732333    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a68fd46-2cba-4c64-a28d-10bf41f47391-tls-certs\") pod \"auth-test-model-a73722-kserve-669549f979-rj7m7\" (UID: \"0a68fd46-2cba-4c64-a28d-10bf41f47391\") " pod="ai-tenant-e2e-shared-b-a73722/auth-test-model-a73722-kserve-669549f979-rj7m7"
Jul 22 02:27:07.732518 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:27:07.732353    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a68fd46-2cba-4c64-a28d-10bf41f47391-dshm\") pod \"auth-test-model-a73722-kserve-669549f979-rj7m7\" (UID: \"0a68fd46-2cba-4c64-a28d-10bf41f47391\") " pod="ai-tenant-e2e-shared-b-a73722/auth-test-model-a73722-kserve-669549f979-rj7m7"
Jul 22 02:27:07.732631 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:27:07.732546    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnldw\" (UniqueName: \"kubernetes.io/projected/0a68fd46-2cba-4c64-a28d-10bf41f47391-kube-api-access-qnldw\") pod \"auth-test-model-a73722-kserve-669549f979-rj7m7\" (UID: \"0a68fd46-2cba-4c64-a28d-10bf41f47391\") " pod="ai-tenant-e2e-shared-b-a73722/auth-test-model-a73722-kserve-669549f979-rj7m7"
Jul 22 02:27:07.732631 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:27:07.732608    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a68fd46-2cba-4c64-a28d-10bf41f47391-tmp-dir\") pod \"auth-test-model-a73722-kserve-669549f979-rj7m7\" (UID: \"0a68fd46-2cba-4c64-a28d-10bf41f47391\") " pod="ai-tenant-e2e-shared-b-a73722/auth-test-model-a73722-kserve-669549f979-rj7m7"
Jul 22 02:27:07.732737 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:27:07.732714    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a68fd46-2cba-4c64-a28d-10bf41f47391-home\") pod \"auth-test-model-a73722-kserve-669549f979-rj7m7\" (UID: \"0a68fd46-2cba-4c64-a28d-10bf41f47391\") " pod="ai-tenant-e2e-shared-b-a73722/auth-test-model-a73722-kserve-669549f979-rj7m7"
Jul 22 02:27:07.733025 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:27:07.733002    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a68fd46-2cba-4c64-a28d-10bf41f47391-model-cache\") pod \"auth-test-model-a73722-kserve-669549f979-rj7m7\" (UID: \"0a68fd46-2cba-4c64-a28d-10bf41f47391\") " pod="ai-tenant-e2e-shared-b-a73722/auth-test-model-a73722-kserve-669549f979-rj7m7"
Jul 22 02:27:07.733106 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:27:07.733053    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a68fd46-2cba-4c64-a28d-10bf41f47391-tmp-dir\") pod \"auth-test-model-a73722-kserve-669549f979-rj7m7\" (UID: \"0a68fd46-2cba-4c64-a28d-10bf41f47391\") " pod="ai-tenant-e2e-shared-b-a73722/auth-test-model-a73722-kserve-669549f979-rj7m7"
Jul 22 02:27:07.734568 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:27:07.734550    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a68fd46-2cba-4c64-a28d-10bf41f47391-dshm\") pod \"auth-test-model-a73722-kserve-669549f979-rj7m7\" (UID: \"0a68fd46-2cba-4c64-a28d-10bf41f47391\") " pod="ai-tenant-e2e-shared-b-a73722/auth-test-model-a73722-kserve-669549f979-rj7m7"
Jul 22 02:27:07.734824 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:27:07.734807    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a68fd46-2cba-4c64-a28d-10bf41f47391-tls-certs\") pod \"auth-test-model-a73722-kserve-669549f979-rj7m7\" (UID: \"0a68fd46-2cba-4c64-a28d-10bf41f47391\") " pod="ai-tenant-e2e-shared-b-a73722/auth-test-model-a73722-kserve-669549f979-rj7m7"
Jul 22 02:27:07.741358 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:27:07.741339    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnldw\" (UniqueName: \"kubernetes.io/projected/0a68fd46-2cba-4c64-a28d-10bf41f47391-kube-api-access-qnldw\") pod \"auth-test-model-a73722-kserve-669549f979-rj7m7\" (UID: \"0a68fd46-2cba-4c64-a28d-10bf41f47391\") " pod="ai-tenant-e2e-shared-b-a73722/auth-test-model-a73722-kserve-669549f979-rj7m7"
Jul 22 02:27:07.880562 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:27:07.880507    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a73722/auth-test-model-a73722-kserve-669549f979-rj7m7"
Jul 22 02:27:08.032148 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:27:08.032124    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a73722/auth-test-model-a73722-kserve-669549f979-rj7m7"]
Jul 22 02:27:08.034112 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:27:08.034079    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a68fd46_2cba_4c64_a28d_10bf41f47391.slice/crio-dcc9ee8477731d3f17192b7defe0496d53e8ac9508bc4a59937bcfa358e3f7e9 WatchSource:0}: Error finding container dcc9ee8477731d3f17192b7defe0496d53e8ac9508bc4a59937bcfa358e3f7e9: Status 404 returned error can't find the container with id dcc9ee8477731d3f17192b7defe0496d53e8ac9508bc4a59937bcfa358e3f7e9
Jul 22 02:27:08.410229 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:27:08.410187    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a73722/auth-test-model-a73722-kserve-669549f979-rj7m7" event={"ID":"0a68fd46-2cba-4c64-a28d-10bf41f47391","Type":"ContainerStarted","Data":"8d0f14adfad4ee4e4ac8fc965b4262abde03c2094a50556554a23980e4203595"}
Jul 22 02:27:08.410417 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:27:08.410237    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a73722/auth-test-model-a73722-kserve-669549f979-rj7m7" event={"ID":"0a68fd46-2cba-4c64-a28d-10bf41f47391","Type":"ContainerStarted","Data":"dcc9ee8477731d3f17192b7defe0496d53e8ac9508bc4a59937bcfa358e3f7e9"}
Jul 22 02:27:08.434807 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:27:08.434764    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-a73722/auth-test-model-a73722-kserve-669549f979-rj7m7" podStartSLOduration=1.434749678 podStartE2EDuration="1.434749678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:27:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:27:08.432937093 +0000 UTC m=+2381.276766224" watchObservedRunningTime="2026-07-22 02:27:08.434749678 +0000 UTC m=+2381.278578809"
Jul 22 02:27:17.881523 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:27:17.881489    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-a73722/auth-test-model-a73722-kserve-669549f979-rj7m7"
Jul 22 02:27:17.882094 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:27:17.881537    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-a73722/auth-test-model-a73722-kserve-669549f979-rj7m7"
Jul 22 02:27:17.893447 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:27:17.893427    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-a73722/auth-test-model-a73722-kserve-669549f979-rj7m7"
Jul 22 02:27:18.475296 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:27:18.475269    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-a73722/auth-test-model-a73722-kserve-669549f979-rj7m7"
Jul 22 02:29:07.647554 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:07.647518    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9d0056/auth-test-model-9d0056-kserve-55484f6d59-9htdf"]
Jul 22 02:29:08.666249 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:08.666223    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a73722/auth-test-model-a73722-kserve-669549f979-rj7m7"]
Jul 22 02:29:22.667677 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:22.667586    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-9d0056/auth-test-model-9d0056-kserve-55484f6d59-9htdf" podUID="0196dc09-0ee8-4010-8b9b-ed4a645d7b66" containerName="main" containerID="cri-o://4667a01b1e02a55057121a9fe5c7c5132c87743c66261459fb38f30cb656d669" gracePeriod=45
Jul 22 02:29:22.909968 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:22.909944    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9d0056/auth-test-model-9d0056-kserve-55484f6d59-9htdf"
Jul 22 02:29:23.085985 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:23.085959    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0196dc09-0ee8-4010-8b9b-ed4a645d7b66-dshm\") pod \"0196dc09-0ee8-4010-8b9b-ed4a645d7b66\" (UID: \"0196dc09-0ee8-4010-8b9b-ed4a645d7b66\") "
Jul 22 02:29:23.086120 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:23.085990    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0196dc09-0ee8-4010-8b9b-ed4a645d7b66-tmp-dir\") pod \"0196dc09-0ee8-4010-8b9b-ed4a645d7b66\" (UID: \"0196dc09-0ee8-4010-8b9b-ed4a645d7b66\") "
Jul 22 02:29:23.086120 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:23.086022    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0196dc09-0ee8-4010-8b9b-ed4a645d7b66-model-cache\") pod \"0196dc09-0ee8-4010-8b9b-ed4a645d7b66\" (UID: \"0196dc09-0ee8-4010-8b9b-ed4a645d7b66\") "
Jul 22 02:29:23.086120 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:23.086046    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m42hn\" (UniqueName: \"kubernetes.io/projected/0196dc09-0ee8-4010-8b9b-ed4a645d7b66-kube-api-access-m42hn\") pod \"0196dc09-0ee8-4010-8b9b-ed4a645d7b66\" (UID: \"0196dc09-0ee8-4010-8b9b-ed4a645d7b66\") "
Jul 22 02:29:23.086120 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:23.086102    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0196dc09-0ee8-4010-8b9b-ed4a645d7b66-home\") pod \"0196dc09-0ee8-4010-8b9b-ed4a645d7b66\" (UID: \"0196dc09-0ee8-4010-8b9b-ed4a645d7b66\") "
Jul 22 02:29:23.086327 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:23.086124    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0196dc09-0ee8-4010-8b9b-ed4a645d7b66-tls-certs\") pod \"0196dc09-0ee8-4010-8b9b-ed4a645d7b66\" (UID: \"0196dc09-0ee8-4010-8b9b-ed4a645d7b66\") "
Jul 22 02:29:23.086327 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:23.086205    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0196dc09-0ee8-4010-8b9b-ed4a645d7b66-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0196dc09-0ee8-4010-8b9b-ed4a645d7b66" (UID: "0196dc09-0ee8-4010-8b9b-ed4a645d7b66"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:29:23.086327 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:23.086304    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0196dc09-0ee8-4010-8b9b-ed4a645d7b66-model-cache" (OuterVolumeSpecName: "model-cache") pod "0196dc09-0ee8-4010-8b9b-ed4a645d7b66" (UID: "0196dc09-0ee8-4010-8b9b-ed4a645d7b66"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:29:23.086477 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:23.086356    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0196dc09-0ee8-4010-8b9b-ed4a645d7b66-home" (OuterVolumeSpecName: "home") pod "0196dc09-0ee8-4010-8b9b-ed4a645d7b66" (UID: "0196dc09-0ee8-4010-8b9b-ed4a645d7b66"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:29:23.086477 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:23.086447    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0196dc09-0ee8-4010-8b9b-ed4a645d7b66-model-cache\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:29:23.086477 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:23.086467    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0196dc09-0ee8-4010-8b9b-ed4a645d7b66-tmp-dir\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:29:23.088120 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:23.088100    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0196dc09-0ee8-4010-8b9b-ed4a645d7b66-dshm" (OuterVolumeSpecName: "dshm") pod "0196dc09-0ee8-4010-8b9b-ed4a645d7b66" (UID: "0196dc09-0ee8-4010-8b9b-ed4a645d7b66"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:29:23.088191 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:23.088118    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0196dc09-0ee8-4010-8b9b-ed4a645d7b66-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0196dc09-0ee8-4010-8b9b-ed4a645d7b66" (UID: "0196dc09-0ee8-4010-8b9b-ed4a645d7b66"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:29:23.088191 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:23.088174    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0196dc09-0ee8-4010-8b9b-ed4a645d7b66-kube-api-access-m42hn" (OuterVolumeSpecName: "kube-api-access-m42hn") pod "0196dc09-0ee8-4010-8b9b-ed4a645d7b66" (UID: "0196dc09-0ee8-4010-8b9b-ed4a645d7b66"). InnerVolumeSpecName "kube-api-access-m42hn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:29:23.119198 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:23.119168    2563 generic.go:356] "Generic (PLEG): container finished" podID="0196dc09-0ee8-4010-8b9b-ed4a645d7b66" containerID="4667a01b1e02a55057121a9fe5c7c5132c87743c66261459fb38f30cb656d669" exitCode=0
Jul 22 02:29:23.119300 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:23.119208    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9d0056/auth-test-model-9d0056-kserve-55484f6d59-9htdf" event={"ID":"0196dc09-0ee8-4010-8b9b-ed4a645d7b66","Type":"ContainerDied","Data":"4667a01b1e02a55057121a9fe5c7c5132c87743c66261459fb38f30cb656d669"}
Jul 22 02:29:23.119300 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:23.119211    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9d0056/auth-test-model-9d0056-kserve-55484f6d59-9htdf"
Jul 22 02:29:23.119300 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:23.119236    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9d0056/auth-test-model-9d0056-kserve-55484f6d59-9htdf" event={"ID":"0196dc09-0ee8-4010-8b9b-ed4a645d7b66","Type":"ContainerDied","Data":"57a3465460de06f344ad94af68ac6680b88d5d5b660d177b2b51482a7f9368ed"}
Jul 22 02:29:23.119300 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:23.119252    2563 scope.go:117] "RemoveContainer" containerID="4667a01b1e02a55057121a9fe5c7c5132c87743c66261459fb38f30cb656d669"
Jul 22 02:29:23.128146 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:23.128130    2563 scope.go:117] "RemoveContainer" containerID="4667a01b1e02a55057121a9fe5c7c5132c87743c66261459fb38f30cb656d669"
Jul 22 02:29:23.128366 ip-10-0-135-230 kubenswrapper[2563]: E0722 02:29:23.128351    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4667a01b1e02a55057121a9fe5c7c5132c87743c66261459fb38f30cb656d669\": container with ID starting with 4667a01b1e02a55057121a9fe5c7c5132c87743c66261459fb38f30cb656d669 not found: ID does not exist" containerID="4667a01b1e02a55057121a9fe5c7c5132c87743c66261459fb38f30cb656d669"
Jul 22 02:29:23.128432 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:23.128396    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4667a01b1e02a55057121a9fe5c7c5132c87743c66261459fb38f30cb656d669"} err="failed to get container status \"4667a01b1e02a55057121a9fe5c7c5132c87743c66261459fb38f30cb656d669\": rpc error: code = NotFound desc = could not find container \"4667a01b1e02a55057121a9fe5c7c5132c87743c66261459fb38f30cb656d669\": container with ID starting with 4667a01b1e02a55057121a9fe5c7c5132c87743c66261459fb38f30cb656d669 not found: ID does not exist"
Jul 22 02:29:23.154428 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:23.154406    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9d0056/auth-test-model-9d0056-kserve-55484f6d59-9htdf"]
Jul 22 02:29:23.159019 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:23.158996    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-9d0056/auth-test-model-9d0056-kserve-55484f6d59-9htdf"]
Jul 22 02:29:23.187481 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:23.187462    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0196dc09-0ee8-4010-8b9b-ed4a645d7b66-dshm\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:29:23.187481 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:23.187480    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m42hn\" (UniqueName: \"kubernetes.io/projected/0196dc09-0ee8-4010-8b9b-ed4a645d7b66-kube-api-access-m42hn\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:29:23.187598 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:23.187489    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0196dc09-0ee8-4010-8b9b-ed4a645d7b66-home\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:29:23.187598 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:23.187499    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0196dc09-0ee8-4010-8b9b-ed4a645d7b66-tls-certs\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:29:23.663736 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:23.663710    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0196dc09-0ee8-4010-8b9b-ed4a645d7b66" path="/var/lib/kubelet/pods/0196dc09-0ee8-4010-8b9b-ed4a645d7b66/volumes"
Jul 22 02:29:23.685813 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:23.685784    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-a73722/auth-test-model-a73722-kserve-669549f979-rj7m7" podUID="0a68fd46-2cba-4c64-a28d-10bf41f47391" containerName="main" containerID="cri-o://8d0f14adfad4ee4e4ac8fc965b4262abde03c2094a50556554a23980e4203595" gracePeriod=45
Jul 22 02:29:23.937223 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:23.937200    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a73722/auth-test-model-a73722-kserve-669549f979-rj7m7"
Jul 22 02:29:24.095003 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:24.094974    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a68fd46-2cba-4c64-a28d-10bf41f47391-dshm\") pod \"0a68fd46-2cba-4c64-a28d-10bf41f47391\" (UID: \"0a68fd46-2cba-4c64-a28d-10bf41f47391\") "
Jul 22 02:29:24.095145 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:24.095029    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a68fd46-2cba-4c64-a28d-10bf41f47391-tls-certs\") pod \"0a68fd46-2cba-4c64-a28d-10bf41f47391\" (UID: \"0a68fd46-2cba-4c64-a28d-10bf41f47391\") "
Jul 22 02:29:24.095145 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:24.095083    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a68fd46-2cba-4c64-a28d-10bf41f47391-home\") pod \"0a68fd46-2cba-4c64-a28d-10bf41f47391\" (UID: \"0a68fd46-2cba-4c64-a28d-10bf41f47391\") "
Jul 22 02:29:24.095145 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:24.095108    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a68fd46-2cba-4c64-a28d-10bf41f47391-tmp-dir\") pod \"0a68fd46-2cba-4c64-a28d-10bf41f47391\" (UID: \"0a68fd46-2cba-4c64-a28d-10bf41f47391\") "
Jul 22 02:29:24.095303 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:24.095159    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnldw\" (UniqueName: \"kubernetes.io/projected/0a68fd46-2cba-4c64-a28d-10bf41f47391-kube-api-access-qnldw\") pod \"0a68fd46-2cba-4c64-a28d-10bf41f47391\" (UID: \"0a68fd46-2cba-4c64-a28d-10bf41f47391\") "
Jul 22 02:29:24.095303 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:24.095182    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a68fd46-2cba-4c64-a28d-10bf41f47391-model-cache\") pod \"0a68fd46-2cba-4c64-a28d-10bf41f47391\" (UID: \"0a68fd46-2cba-4c64-a28d-10bf41f47391\") "
Jul 22 02:29:24.095420 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:24.095340    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a68fd46-2cba-4c64-a28d-10bf41f47391-home" (OuterVolumeSpecName: "home") pod "0a68fd46-2cba-4c64-a28d-10bf41f47391" (UID: "0a68fd46-2cba-4c64-a28d-10bf41f47391"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:29:24.095485 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:24.095462    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a68fd46-2cba-4c64-a28d-10bf41f47391-home\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:29:24.095535 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:24.095495    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a68fd46-2cba-4c64-a28d-10bf41f47391-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0a68fd46-2cba-4c64-a28d-10bf41f47391" (UID: "0a68fd46-2cba-4c64-a28d-10bf41f47391"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:29:24.095711 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:24.095693    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a68fd46-2cba-4c64-a28d-10bf41f47391-model-cache" (OuterVolumeSpecName: "model-cache") pod "0a68fd46-2cba-4c64-a28d-10bf41f47391" (UID: "0a68fd46-2cba-4c64-a28d-10bf41f47391"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:29:24.097118 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:24.097090    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a68fd46-2cba-4c64-a28d-10bf41f47391-dshm" (OuterVolumeSpecName: "dshm") pod "0a68fd46-2cba-4c64-a28d-10bf41f47391" (UID: "0a68fd46-2cba-4c64-a28d-10bf41f47391"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:29:24.097230 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:24.097214    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a68fd46-2cba-4c64-a28d-10bf41f47391-kube-api-access-qnldw" (OuterVolumeSpecName: "kube-api-access-qnldw") pod "0a68fd46-2cba-4c64-a28d-10bf41f47391" (UID: "0a68fd46-2cba-4c64-a28d-10bf41f47391"). InnerVolumeSpecName "kube-api-access-qnldw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:29:24.097285 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:24.097269    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a68fd46-2cba-4c64-a28d-10bf41f47391-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0a68fd46-2cba-4c64-a28d-10bf41f47391" (UID: "0a68fd46-2cba-4c64-a28d-10bf41f47391"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:29:24.125006 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:24.124978    2563 generic.go:356] "Generic (PLEG): container finished" podID="0a68fd46-2cba-4c64-a28d-10bf41f47391" containerID="8d0f14adfad4ee4e4ac8fc965b4262abde03c2094a50556554a23980e4203595" exitCode=0
Jul 22 02:29:24.125101 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:24.125018    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a73722/auth-test-model-a73722-kserve-669549f979-rj7m7"
Jul 22 02:29:24.125101 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:24.125054    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a73722/auth-test-model-a73722-kserve-669549f979-rj7m7" event={"ID":"0a68fd46-2cba-4c64-a28d-10bf41f47391","Type":"ContainerDied","Data":"8d0f14adfad4ee4e4ac8fc965b4262abde03c2094a50556554a23980e4203595"}
Jul 22 02:29:24.125101 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:24.125093    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a73722/auth-test-model-a73722-kserve-669549f979-rj7m7" event={"ID":"0a68fd46-2cba-4c64-a28d-10bf41f47391","Type":"ContainerDied","Data":"dcc9ee8477731d3f17192b7defe0496d53e8ac9508bc4a59937bcfa358e3f7e9"}
Jul 22 02:29:24.125232 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:24.125109    2563 scope.go:117] "RemoveContainer" containerID="8d0f14adfad4ee4e4ac8fc965b4262abde03c2094a50556554a23980e4203595"
Jul 22 02:29:24.133423 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:24.133408    2563 scope.go:117] "RemoveContainer" containerID="8d0f14adfad4ee4e4ac8fc965b4262abde03c2094a50556554a23980e4203595"
Jul 22 02:29:24.133663 ip-10-0-135-230 kubenswrapper[2563]: E0722 02:29:24.133644    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d0f14adfad4ee4e4ac8fc965b4262abde03c2094a50556554a23980e4203595\": container with ID starting with 8d0f14adfad4ee4e4ac8fc965b4262abde03c2094a50556554a23980e4203595 not found: ID does not exist" containerID="8d0f14adfad4ee4e4ac8fc965b4262abde03c2094a50556554a23980e4203595"
Jul 22 02:29:24.133722 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:24.133668    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d0f14adfad4ee4e4ac8fc965b4262abde03c2094a50556554a23980e4203595"} err="failed to get container status \"8d0f14adfad4ee4e4ac8fc965b4262abde03c2094a50556554a23980e4203595\": rpc error: code = NotFound desc = could not find container \"8d0f14adfad4ee4e4ac8fc965b4262abde03c2094a50556554a23980e4203595\": container with ID starting with 8d0f14adfad4ee4e4ac8fc965b4262abde03c2094a50556554a23980e4203595 not found: ID does not exist"
Jul 22 02:29:24.145846 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:24.145825    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a73722/auth-test-model-a73722-kserve-669549f979-rj7m7"]
Jul 22 02:29:24.151705 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:24.151688    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-a73722/auth-test-model-a73722-kserve-669549f979-rj7m7"]
Jul 22 02:29:24.195955 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:24.195932    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a68fd46-2cba-4c64-a28d-10bf41f47391-dshm\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:29:24.196063 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:24.195964    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a68fd46-2cba-4c64-a28d-10bf41f47391-tls-certs\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:29:24.196063 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:24.196009    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a68fd46-2cba-4c64-a28d-10bf41f47391-tmp-dir\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:29:24.196063 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:24.196020    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnldw\" (UniqueName: \"kubernetes.io/projected/0a68fd46-2cba-4c64-a28d-10bf41f47391-kube-api-access-qnldw\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:29:24.196063 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:24.196029    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a68fd46-2cba-4c64-a28d-10bf41f47391-model-cache\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:29:25.417134 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:25.417097    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-9d0056/sub-test-model-9d0056-kserve-79b776984c-d2js5"]
Jul 22 02:29:25.417515 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:25.417476    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0196dc09-0ee8-4010-8b9b-ed4a645d7b66" containerName="main"
Jul 22 02:29:25.417515 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:25.417487    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0196dc09-0ee8-4010-8b9b-ed4a645d7b66" containerName="main"
Jul 22 02:29:25.417515 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:25.417495    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a68fd46-2cba-4c64-a28d-10bf41f47391" containerName="main"
Jul 22 02:29:25.417515 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:25.417500    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a68fd46-2cba-4c64-a28d-10bf41f47391" containerName="main"
Jul 22 02:29:25.417643 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:25.417572    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0196dc09-0ee8-4010-8b9b-ed4a645d7b66" containerName="main"
Jul 22 02:29:25.417643 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:25.417579    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a68fd46-2cba-4c64-a28d-10bf41f47391" containerName="main"
Jul 22 02:29:25.421872 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:25.421844    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9d0056/sub-test-model-9d0056-kserve-79b776984c-d2js5"
Jul 22 02:29:25.428264 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:25.428243    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9d0056\"/\"default-dockercfg-mwkhg\""
Jul 22 02:29:25.428566 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:25.428548    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-9d0056\"/\"openshift-service-ca.crt\""
Jul 22 02:29:25.428998 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:25.428979    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-9d0056\"/\"kube-root-ca.crt\""
Jul 22 02:29:25.430759 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:25.430744    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9d0056\"/\"sub-test-model-9d0056-kserve-self-signed-certs\""
Jul 22 02:29:25.437543 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:25.437524    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9d0056/sub-test-model-9d0056-kserve-79b776984c-d2js5"]
Jul 22 02:29:25.607750 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:25.607710    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2add91d7-07c5-4d33-8bd2-6388aacc7867-dshm\") pod \"sub-test-model-9d0056-kserve-79b776984c-d2js5\" (UID: \"2add91d7-07c5-4d33-8bd2-6388aacc7867\") " pod="ai-tenant-e2e-shared-a-9d0056/sub-test-model-9d0056-kserve-79b776984c-d2js5"
Jul 22 02:29:25.607922 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:25.607772    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnmsq\" (UniqueName: \"kubernetes.io/projected/2add91d7-07c5-4d33-8bd2-6388aacc7867-kube-api-access-xnmsq\") pod \"sub-test-model-9d0056-kserve-79b776984c-d2js5\" (UID: \"2add91d7-07c5-4d33-8bd2-6388aacc7867\") " pod="ai-tenant-e2e-shared-a-9d0056/sub-test-model-9d0056-kserve-79b776984c-d2js5"
Jul 22 02:29:25.607922 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:25.607834    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2add91d7-07c5-4d33-8bd2-6388aacc7867-home\") pod \"sub-test-model-9d0056-kserve-79b776984c-d2js5\" (UID: \"2add91d7-07c5-4d33-8bd2-6388aacc7867\") " pod="ai-tenant-e2e-shared-a-9d0056/sub-test-model-9d0056-kserve-79b776984c-d2js5"
Jul 22 02:29:25.607922 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:25.607876    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2add91d7-07c5-4d33-8bd2-6388aacc7867-model-cache\") pod \"sub-test-model-9d0056-kserve-79b776984c-d2js5\" (UID: \"2add91d7-07c5-4d33-8bd2-6388aacc7867\") " pod="ai-tenant-e2e-shared-a-9d0056/sub-test-model-9d0056-kserve-79b776984c-d2js5"
Jul 22 02:29:25.607922 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:25.607915    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2add91d7-07c5-4d33-8bd2-6388aacc7867-tmp-dir\") pod \"sub-test-model-9d0056-kserve-79b776984c-d2js5\" (UID: \"2add91d7-07c5-4d33-8bd2-6388aacc7867\") " pod="ai-tenant-e2e-shared-a-9d0056/sub-test-model-9d0056-kserve-79b776984c-d2js5"
Jul 22 02:29:25.608068 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:25.607946    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2add91d7-07c5-4d33-8bd2-6388aacc7867-tls-certs\") pod \"sub-test-model-9d0056-kserve-79b776984c-d2js5\" (UID: \"2add91d7-07c5-4d33-8bd2-6388aacc7867\") " pod="ai-tenant-e2e-shared-a-9d0056/sub-test-model-9d0056-kserve-79b776984c-d2js5"
Jul 22 02:29:25.664794 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:25.664764    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a68fd46-2cba-4c64-a28d-10bf41f47391" path="/var/lib/kubelet/pods/0a68fd46-2cba-4c64-a28d-10bf41f47391/volumes"
Jul 22 02:29:25.709358 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:25.709332    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2add91d7-07c5-4d33-8bd2-6388aacc7867-model-cache\") pod \"sub-test-model-9d0056-kserve-79b776984c-d2js5\" (UID: \"2add91d7-07c5-4d33-8bd2-6388aacc7867\") " pod="ai-tenant-e2e-shared-a-9d0056/sub-test-model-9d0056-kserve-79b776984c-d2js5"
Jul 22 02:29:25.709530 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:25.709396    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2add91d7-07c5-4d33-8bd2-6388aacc7867-tmp-dir\") pod \"sub-test-model-9d0056-kserve-79b776984c-d2js5\" (UID: \"2add91d7-07c5-4d33-8bd2-6388aacc7867\") " pod="ai-tenant-e2e-shared-a-9d0056/sub-test-model-9d0056-kserve-79b776984c-d2js5"
Jul 22 02:29:25.709530 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:25.709430    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2add91d7-07c5-4d33-8bd2-6388aacc7867-tls-certs\") pod \"sub-test-model-9d0056-kserve-79b776984c-d2js5\" (UID: \"2add91d7-07c5-4d33-8bd2-6388aacc7867\") " pod="ai-tenant-e2e-shared-a-9d0056/sub-test-model-9d0056-kserve-79b776984c-d2js5"
Jul 22 02:29:25.709530 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:25.709457    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2add91d7-07c5-4d33-8bd2-6388aacc7867-dshm\") pod \"sub-test-model-9d0056-kserve-79b776984c-d2js5\" (UID: \"2add91d7-07c5-4d33-8bd2-6388aacc7867\") " pod="ai-tenant-e2e-shared-a-9d0056/sub-test-model-9d0056-kserve-79b776984c-d2js5"
Jul 22 02:29:25.709530 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:25.709495    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnmsq\" (UniqueName: \"kubernetes.io/projected/2add91d7-07c5-4d33-8bd2-6388aacc7867-kube-api-access-xnmsq\") pod \"sub-test-model-9d0056-kserve-79b776984c-d2js5\" (UID: \"2add91d7-07c5-4d33-8bd2-6388aacc7867\") " pod="ai-tenant-e2e-shared-a-9d0056/sub-test-model-9d0056-kserve-79b776984c-d2js5"
Jul 22 02:29:25.709530 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:25.709514    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2add91d7-07c5-4d33-8bd2-6388aacc7867-home\") pod \"sub-test-model-9d0056-kserve-79b776984c-d2js5\" (UID: \"2add91d7-07c5-4d33-8bd2-6388aacc7867\") " pod="ai-tenant-e2e-shared-a-9d0056/sub-test-model-9d0056-kserve-79b776984c-d2js5"
Jul 22 02:29:25.709790 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:25.709767    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2add91d7-07c5-4d33-8bd2-6388aacc7867-model-cache\") pod \"sub-test-model-9d0056-kserve-79b776984c-d2js5\" (UID: \"2add91d7-07c5-4d33-8bd2-6388aacc7867\") " pod="ai-tenant-e2e-shared-a-9d0056/sub-test-model-9d0056-kserve-79b776984c-d2js5"
Jul 22 02:29:25.709850 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:25.709813    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2add91d7-07c5-4d33-8bd2-6388aacc7867-tmp-dir\") pod \"sub-test-model-9d0056-kserve-79b776984c-d2js5\" (UID: \"2add91d7-07c5-4d33-8bd2-6388aacc7867\") " pod="ai-tenant-e2e-shared-a-9d0056/sub-test-model-9d0056-kserve-79b776984c-d2js5"
Jul 22 02:29:25.709910 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:25.709862    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2add91d7-07c5-4d33-8bd2-6388aacc7867-home\") pod \"sub-test-model-9d0056-kserve-79b776984c-d2js5\" (UID: \"2add91d7-07c5-4d33-8bd2-6388aacc7867\") " pod="ai-tenant-e2e-shared-a-9d0056/sub-test-model-9d0056-kserve-79b776984c-d2js5"
Jul 22 02:29:25.711700 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:25.711680    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2add91d7-07c5-4d33-8bd2-6388aacc7867-dshm\") pod \"sub-test-model-9d0056-kserve-79b776984c-d2js5\" (UID: \"2add91d7-07c5-4d33-8bd2-6388aacc7867\") " pod="ai-tenant-e2e-shared-a-9d0056/sub-test-model-9d0056-kserve-79b776984c-d2js5"
Jul 22 02:29:25.711927 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:25.711907    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2add91d7-07c5-4d33-8bd2-6388aacc7867-tls-certs\") pod \"sub-test-model-9d0056-kserve-79b776984c-d2js5\" (UID: \"2add91d7-07c5-4d33-8bd2-6388aacc7867\") " pod="ai-tenant-e2e-shared-a-9d0056/sub-test-model-9d0056-kserve-79b776984c-d2js5"
Jul 22 02:29:25.720681 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:25.720662    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnmsq\" (UniqueName: \"kubernetes.io/projected/2add91d7-07c5-4d33-8bd2-6388aacc7867-kube-api-access-xnmsq\") pod \"sub-test-model-9d0056-kserve-79b776984c-d2js5\" (UID: \"2add91d7-07c5-4d33-8bd2-6388aacc7867\") " pod="ai-tenant-e2e-shared-a-9d0056/sub-test-model-9d0056-kserve-79b776984c-d2js5"
Jul 22 02:29:25.730527 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:25.730506    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9d0056/sub-test-model-9d0056-kserve-79b776984c-d2js5"
Jul 22 02:29:25.866369 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:29:25.866339    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2add91d7_07c5_4d33_8bd2_6388aacc7867.slice/crio-0871c941c9a98dbec9e3d6a52008b297e3ec0818eeb1ddd415b1f8b62f694710 WatchSource:0}: Error finding container 0871c941c9a98dbec9e3d6a52008b297e3ec0818eeb1ddd415b1f8b62f694710: Status 404 returned error can't find the container with id 0871c941c9a98dbec9e3d6a52008b297e3ec0818eeb1ddd415b1f8b62f694710
Jul 22 02:29:25.871168 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:25.871147    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9d0056/sub-test-model-9d0056-kserve-79b776984c-d2js5"]
Jul 22 02:29:26.139609 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:26.139530    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9d0056/sub-test-model-9d0056-kserve-79b776984c-d2js5" event={"ID":"2add91d7-07c5-4d33-8bd2-6388aacc7867","Type":"ContainerStarted","Data":"37b9502b11b48c2893fa024dfccb2e40cf09114e017459771012740f51fc1fb9"}
Jul 22 02:29:26.139609 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:26.139569    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9d0056/sub-test-model-9d0056-kserve-79b776984c-d2js5" event={"ID":"2add91d7-07c5-4d33-8bd2-6388aacc7867","Type":"ContainerStarted","Data":"0871c941c9a98dbec9e3d6a52008b297e3ec0818eeb1ddd415b1f8b62f694710"}
Jul 22 02:29:26.164254 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:26.164206    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-9d0056/sub-test-model-9d0056-kserve-79b776984c-d2js5" podStartSLOduration=1.164192482 podStartE2EDuration="1.164192482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:29:26.16281682 +0000 UTC m=+2519.006645952" watchObservedRunningTime="2026-07-22 02:29:26.164192482 +0000 UTC m=+2519.008021613"
Jul 22 02:29:28.089707 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:28.089670    2563 scope.go:117] "RemoveContainer" containerID="21fe75ed72cbd29586918e8895db9057a2fabf306f2e719999ccafa41283ffd7"
Jul 22 02:29:35.731022 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:35.730987    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-9d0056/sub-test-model-9d0056-kserve-79b776984c-d2js5"
Jul 22 02:29:35.731505 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:35.731041    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-9d0056/sub-test-model-9d0056-kserve-79b776984c-d2js5"
Jul 22 02:29:35.743030 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:35.743004    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-9d0056/sub-test-model-9d0056-kserve-79b776984c-d2js5"
Jul 22 02:29:36.203492 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:36.203465    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-9d0056/sub-test-model-9d0056-kserve-79b776984c-d2js5"
Jul 22 02:29:47.259423 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:47.259395    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-a73722/sub-test-model-a73722-kserve-6598dd5c47-d8jbb"]
Jul 22 02:29:47.262846 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:47.262829    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a73722/sub-test-model-a73722-kserve-6598dd5c47-d8jbb"
Jul 22 02:29:47.265514 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:47.265494    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a73722\"/\"default-dockercfg-wlc4l\""
Jul 22 02:29:47.266645 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:47.266611    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a73722\"/\"openshift-service-ca.crt\""
Jul 22 02:29:47.266645 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:47.266620    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a73722\"/\"kube-root-ca.crt\""
Jul 22 02:29:47.266824 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:47.266809    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a73722\"/\"sub-test-model-a73722-kserve-self-signed-certs\""
Jul 22 02:29:47.274535 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:47.274511    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a73722/sub-test-model-a73722-kserve-6598dd5c47-d8jbb"]
Jul 22 02:29:47.375405 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:47.375353    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0927033-4a6e-4eb6-9821-22bad184a967-model-cache\") pod \"sub-test-model-a73722-kserve-6598dd5c47-d8jbb\" (UID: \"f0927033-4a6e-4eb6-9821-22bad184a967\") " pod="ai-tenant-e2e-shared-b-a73722/sub-test-model-a73722-kserve-6598dd5c47-d8jbb"
Jul 22 02:29:47.375577 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:47.375411    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0927033-4a6e-4eb6-9821-22bad184a967-tmp-dir\") pod \"sub-test-model-a73722-kserve-6598dd5c47-d8jbb\" (UID: \"f0927033-4a6e-4eb6-9821-22bad184a967\") " pod="ai-tenant-e2e-shared-b-a73722/sub-test-model-a73722-kserve-6598dd5c47-d8jbb"
Jul 22 02:29:47.375577 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:47.375476    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0927033-4a6e-4eb6-9821-22bad184a967-home\") pod \"sub-test-model-a73722-kserve-6598dd5c47-d8jbb\" (UID: \"f0927033-4a6e-4eb6-9821-22bad184a967\") " pod="ai-tenant-e2e-shared-b-a73722/sub-test-model-a73722-kserve-6598dd5c47-d8jbb"
Jul 22 02:29:47.375577 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:47.375540    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jg64\" (UniqueName: \"kubernetes.io/projected/f0927033-4a6e-4eb6-9821-22bad184a967-kube-api-access-7jg64\") pod \"sub-test-model-a73722-kserve-6598dd5c47-d8jbb\" (UID: \"f0927033-4a6e-4eb6-9821-22bad184a967\") " pod="ai-tenant-e2e-shared-b-a73722/sub-test-model-a73722-kserve-6598dd5c47-d8jbb"
Jul 22 02:29:47.375577 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:47.375577    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0927033-4a6e-4eb6-9821-22bad184a967-tls-certs\") pod \"sub-test-model-a73722-kserve-6598dd5c47-d8jbb\" (UID: \"f0927033-4a6e-4eb6-9821-22bad184a967\") " pod="ai-tenant-e2e-shared-b-a73722/sub-test-model-a73722-kserve-6598dd5c47-d8jbb"
Jul 22 02:29:47.375743 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:47.375660    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0927033-4a6e-4eb6-9821-22bad184a967-dshm\") pod \"sub-test-model-a73722-kserve-6598dd5c47-d8jbb\" (UID: \"f0927033-4a6e-4eb6-9821-22bad184a967\") " pod="ai-tenant-e2e-shared-b-a73722/sub-test-model-a73722-kserve-6598dd5c47-d8jbb"
Jul 22 02:29:47.477058 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:47.477025    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0927033-4a6e-4eb6-9821-22bad184a967-tls-certs\") pod \"sub-test-model-a73722-kserve-6598dd5c47-d8jbb\" (UID: \"f0927033-4a6e-4eb6-9821-22bad184a967\") " pod="ai-tenant-e2e-shared-b-a73722/sub-test-model-a73722-kserve-6598dd5c47-d8jbb"
Jul 22 02:29:47.477208 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:47.477077    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0927033-4a6e-4eb6-9821-22bad184a967-dshm\") pod \"sub-test-model-a73722-kserve-6598dd5c47-d8jbb\" (UID: \"f0927033-4a6e-4eb6-9821-22bad184a967\") " pod="ai-tenant-e2e-shared-b-a73722/sub-test-model-a73722-kserve-6598dd5c47-d8jbb"
Jul 22 02:29:47.477208 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:47.477106    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0927033-4a6e-4eb6-9821-22bad184a967-model-cache\") pod \"sub-test-model-a73722-kserve-6598dd5c47-d8jbb\" (UID: \"f0927033-4a6e-4eb6-9821-22bad184a967\") " pod="ai-tenant-e2e-shared-b-a73722/sub-test-model-a73722-kserve-6598dd5c47-d8jbb"
Jul 22 02:29:47.477208 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:47.477181    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0927033-4a6e-4eb6-9821-22bad184a967-tmp-dir\") pod \"sub-test-model-a73722-kserve-6598dd5c47-d8jbb\" (UID: \"f0927033-4a6e-4eb6-9821-22bad184a967\") " pod="ai-tenant-e2e-shared-b-a73722/sub-test-model-a73722-kserve-6598dd5c47-d8jbb"
Jul 22 02:29:47.477317 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:47.477240    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0927033-4a6e-4eb6-9821-22bad184a967-home\") pod \"sub-test-model-a73722-kserve-6598dd5c47-d8jbb\" (UID: \"f0927033-4a6e-4eb6-9821-22bad184a967\") " pod="ai-tenant-e2e-shared-b-a73722/sub-test-model-a73722-kserve-6598dd5c47-d8jbb"
Jul 22 02:29:47.477317 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:47.477289    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jg64\" (UniqueName: \"kubernetes.io/projected/f0927033-4a6e-4eb6-9821-22bad184a967-kube-api-access-7jg64\") pod \"sub-test-model-a73722-kserve-6598dd5c47-d8jbb\" (UID: \"f0927033-4a6e-4eb6-9821-22bad184a967\") " pod="ai-tenant-e2e-shared-b-a73722/sub-test-model-a73722-kserve-6598dd5c47-d8jbb"
Jul 22 02:29:47.477518 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:47.477500    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0927033-4a6e-4eb6-9821-22bad184a967-tmp-dir\") pod \"sub-test-model-a73722-kserve-6598dd5c47-d8jbb\" (UID: \"f0927033-4a6e-4eb6-9821-22bad184a967\") " pod="ai-tenant-e2e-shared-b-a73722/sub-test-model-a73722-kserve-6598dd5c47-d8jbb"
Jul 22 02:29:47.477588 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:47.477559    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0927033-4a6e-4eb6-9821-22bad184a967-model-cache\") pod \"sub-test-model-a73722-kserve-6598dd5c47-d8jbb\" (UID: \"f0927033-4a6e-4eb6-9821-22bad184a967\") " pod="ai-tenant-e2e-shared-b-a73722/sub-test-model-a73722-kserve-6598dd5c47-d8jbb"
Jul 22 02:29:47.477647 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:47.477631    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0927033-4a6e-4eb6-9821-22bad184a967-home\") pod \"sub-test-model-a73722-kserve-6598dd5c47-d8jbb\" (UID: \"f0927033-4a6e-4eb6-9821-22bad184a967\") " pod="ai-tenant-e2e-shared-b-a73722/sub-test-model-a73722-kserve-6598dd5c47-d8jbb"
Jul 22 02:29:47.479356 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:47.479333    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0927033-4a6e-4eb6-9821-22bad184a967-dshm\") pod \"sub-test-model-a73722-kserve-6598dd5c47-d8jbb\" (UID: \"f0927033-4a6e-4eb6-9821-22bad184a967\") " pod="ai-tenant-e2e-shared-b-a73722/sub-test-model-a73722-kserve-6598dd5c47-d8jbb"
Jul 22 02:29:47.479577 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:47.479562    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0927033-4a6e-4eb6-9821-22bad184a967-tls-certs\") pod \"sub-test-model-a73722-kserve-6598dd5c47-d8jbb\" (UID: \"f0927033-4a6e-4eb6-9821-22bad184a967\") " pod="ai-tenant-e2e-shared-b-a73722/sub-test-model-a73722-kserve-6598dd5c47-d8jbb"
Jul 22 02:29:47.485196 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:47.485175    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jg64\" (UniqueName: \"kubernetes.io/projected/f0927033-4a6e-4eb6-9821-22bad184a967-kube-api-access-7jg64\") pod \"sub-test-model-a73722-kserve-6598dd5c47-d8jbb\" (UID: \"f0927033-4a6e-4eb6-9821-22bad184a967\") " pod="ai-tenant-e2e-shared-b-a73722/sub-test-model-a73722-kserve-6598dd5c47-d8jbb"
Jul 22 02:29:47.572070 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:47.572000    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a73722/sub-test-model-a73722-kserve-6598dd5c47-d8jbb"
Jul 22 02:29:47.702156 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:47.702133    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a73722/sub-test-model-a73722-kserve-6598dd5c47-d8jbb"]
Jul 22 02:29:47.703547 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:29:47.703522    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0927033_4a6e_4eb6_9821_22bad184a967.slice/crio-7f0e0cdfeee6e73b584026324cd91a28c8d0cfa06a8577bac899afa395a3296a WatchSource:0}: Error finding container 7f0e0cdfeee6e73b584026324cd91a28c8d0cfa06a8577bac899afa395a3296a: Status 404 returned error can't find the container with id 7f0e0cdfeee6e73b584026324cd91a28c8d0cfa06a8577bac899afa395a3296a
Jul 22 02:29:48.252882 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:48.252847    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a73722/sub-test-model-a73722-kserve-6598dd5c47-d8jbb" event={"ID":"f0927033-4a6e-4eb6-9821-22bad184a967","Type":"ContainerStarted","Data":"acd949a63ce93ce5cd44f1d1f65b451c1233a802e8fdc22f175ef0aee196b37c"}
Jul 22 02:29:48.253042 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:48.252891    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a73722/sub-test-model-a73722-kserve-6598dd5c47-d8jbb" event={"ID":"f0927033-4a6e-4eb6-9821-22bad184a967","Type":"ContainerStarted","Data":"7f0e0cdfeee6e73b584026324cd91a28c8d0cfa06a8577bac899afa395a3296a"}
Jul 22 02:29:48.271344 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:48.271295    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-a73722/sub-test-model-a73722-kserve-6598dd5c47-d8jbb" podStartSLOduration=1.2712809950000001 podStartE2EDuration="1.271280995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:29:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:29:48.268770052 +0000 UTC m=+2541.112599181" watchObservedRunningTime="2026-07-22 02:29:48.271280995 +0000 UTC m=+2541.115110124"
Jul 22 02:29:57.572688 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:57.572658    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-a73722/sub-test-model-a73722-kserve-6598dd5c47-d8jbb"
Jul 22 02:29:57.572688 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:57.572697    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-a73722/sub-test-model-a73722-kserve-6598dd5c47-d8jbb"
Jul 22 02:29:57.584768 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:57.584745    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-a73722/sub-test-model-a73722-kserve-6598dd5c47-d8jbb"
Jul 22 02:29:58.320258 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:29:58.320227    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-a73722/sub-test-model-a73722-kserve-6598dd5c47-d8jbb"
Jul 22 02:30:00.145786 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:00.145758    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744790-z6p59"]
Jul 22 02:30:00.149349 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:00.149328    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-9d0056-29744790-twvmv"]
Jul 22 02:30:00.149485 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:00.149468    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744790-z6p59"
Jul 22 02:30:00.152334 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:00.152308    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wrtw7\""
Jul 22 02:30:00.152500 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:00.152453    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a73722-29744790-8zmjt"]
Jul 22 02:30:00.152572 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:00.152546    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-9d0056-29744790-twvmv"
Jul 22 02:30:00.155542 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:00.155522    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a73722-29744790-8zmjt"
Jul 22 02:30:00.160315 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:00.160293    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744790-z6p59"]
Jul 22 02:30:00.161768 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:00.161750    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744790-z6p59"
Jul 22 02:30:00.162203 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:00.162167    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-9d0056-29744790-twvmv"]
Jul 22 02:30:00.164933 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:00.164842    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a73722-29744790-8zmjt"]
Jul 22 02:30:00.170622 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:00.170604    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-9d0056-29744790-twvmv"
Jul 22 02:30:00.204773 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:00.204752    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a73722-29744790-8zmjt"
Jul 22 02:30:00.316696 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:00.316452    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744790-z6p59"]
Jul 22 02:30:00.319365 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:30:00.319339    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe20609b_0f9c_4e66_8a91_81e8b8d8bace.slice/crio-6eaa1610624972e21508a3bc4ffdc68476c2d11e8256b2da0882009d28b1eeb0 WatchSource:0}: Error finding container 6eaa1610624972e21508a3bc4ffdc68476c2d11e8256b2da0882009d28b1eeb0: Status 404 returned error can't find the container with id 6eaa1610624972e21508a3bc4ffdc68476c2d11e8256b2da0882009d28b1eeb0
Jul 22 02:30:00.354907 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:00.354884    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-9d0056-29744790-twvmv"]
Jul 22 02:30:00.356733 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:30:00.356706    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb5b1928_d7b8_4c7c_9567_2235fdff4837.slice/crio-e22e5fdc4e721c51d03ba6e4334d8d1dfed8e741766d9bebccc0c6e6a9753068 WatchSource:0}: Error finding container e22e5fdc4e721c51d03ba6e4334d8d1dfed8e741766d9bebccc0c6e6a9753068: Status 404 returned error can't find the container with id e22e5fdc4e721c51d03ba6e4334d8d1dfed8e741766d9bebccc0c6e6a9753068
Jul 22 02:30:00.381925 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:00.381903    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a73722-29744790-8zmjt"]
Jul 22 02:30:00.393698 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:30:00.393664    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce656997_5a4c_4ec5_8224_204ab040e681.slice/crio-b8338e6bb34b02b31ff3380042953bb5670faca799bcbcff1d3d5843d1640bee WatchSource:0}: Error finding container b8338e6bb34b02b31ff3380042953bb5670faca799bcbcff1d3d5843d1640bee: Status 404 returned error can't find the container with id b8338e6bb34b02b31ff3380042953bb5670faca799bcbcff1d3d5843d1640bee
Jul 22 02:30:01.328982 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:01.328956    2563 generic.go:356] "Generic (PLEG): container finished" podID="be20609b-0f9c-4e66-8a91-81e8b8d8bace" containerID="56497c245c494402527806e12d648a81f7ee79a55853d6fc9f37ce5bb5ff561f" exitCode=0
Jul 22 02:30:01.329347 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:01.329041    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744790-z6p59" event={"ID":"be20609b-0f9c-4e66-8a91-81e8b8d8bace","Type":"ContainerDied","Data":"56497c245c494402527806e12d648a81f7ee79a55853d6fc9f37ce5bb5ff561f"}
Jul 22 02:30:01.329347 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:01.329104    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744790-z6p59" event={"ID":"be20609b-0f9c-4e66-8a91-81e8b8d8bace","Type":"ContainerStarted","Data":"6eaa1610624972e21508a3bc4ffdc68476c2d11e8256b2da0882009d28b1eeb0"}
Jul 22 02:30:01.330563 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:01.330542    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-9d0056-29744790-twvmv" event={"ID":"bb5b1928-d7b8-4c7c-9567-2235fdff4837","Type":"ContainerStarted","Data":"332313c4055a06ccdffb798867a70f00fed9b32c969701d81ac3e536bca0ec07"}
Jul 22 02:30:01.330680 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:01.330589    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-9d0056-29744790-twvmv" event={"ID":"bb5b1928-d7b8-4c7c-9567-2235fdff4837","Type":"ContainerStarted","Data":"e22e5fdc4e721c51d03ba6e4334d8d1dfed8e741766d9bebccc0c6e6a9753068"}
Jul 22 02:30:01.332003 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:01.331981    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a73722-29744790-8zmjt" event={"ID":"ce656997-5a4c-4ec5-8224-204ab040e681","Type":"ContainerStarted","Data":"963c9708a7ec4a7af1ca9fd509b3fe33a94137d57507d9762e1f8f62620f064f"}
Jul 22 02:30:01.332092 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:01.332027    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a73722-29744790-8zmjt" event={"ID":"ce656997-5a4c-4ec5-8224-204ab040e681","Type":"ContainerStarted","Data":"b8338e6bb34b02b31ff3380042953bb5670faca799bcbcff1d3d5843d1640bee"}
Jul 22 02:30:01.382649 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:01.382610    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a73722-29744790-8zmjt" podStartSLOduration=1.3825989060000001 podStartE2EDuration="1.382598906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:30:01.381577447 +0000 UTC m=+2554.225406577" watchObservedRunningTime="2026-07-22 02:30:01.382598906 +0000 UTC m=+2554.226428048"
Jul 22 02:30:02.338720 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:02.338681    2563 generic.go:356] "Generic (PLEG): container finished" podID="ce656997-5a4c-4ec5-8224-204ab040e681" containerID="963c9708a7ec4a7af1ca9fd509b3fe33a94137d57507d9762e1f8f62620f064f" exitCode=0
Jul 22 02:30:02.339147 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:02.338766    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a73722-29744790-8zmjt" event={"ID":"ce656997-5a4c-4ec5-8224-204ab040e681","Type":"ContainerDied","Data":"963c9708a7ec4a7af1ca9fd509b3fe33a94137d57507d9762e1f8f62620f064f"}
Jul 22 02:30:02.340337 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:02.340311    2563 generic.go:356] "Generic (PLEG): container finished" podID="bb5b1928-d7b8-4c7c-9567-2235fdff4837" containerID="332313c4055a06ccdffb798867a70f00fed9b32c969701d81ac3e536bca0ec07" exitCode=0
Jul 22 02:30:02.340478 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:02.340349    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-9d0056-29744790-twvmv" event={"ID":"bb5b1928-d7b8-4c7c-9567-2235fdff4837","Type":"ContainerDied","Data":"332313c4055a06ccdffb798867a70f00fed9b32c969701d81ac3e536bca0ec07"}
Jul 22 02:30:02.354191 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:02.354151    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-9d0056-29744790-twvmv" podStartSLOduration=2.35413855 podStartE2EDuration="2.35413855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:30:01.415848719 +0000 UTC m=+2554.259677850" watchObservedRunningTime="2026-07-22 02:30:02.35413855 +0000 UTC m=+2555.197967681"
Jul 22 02:30:02.475440 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:02.475417    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744790-z6p59"
Jul 22 02:30:03.346669 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:03.346641    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744790-z6p59"
Jul 22 02:30:03.347140 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:03.346685    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744790-z6p59" event={"ID":"be20609b-0f9c-4e66-8a91-81e8b8d8bace","Type":"ContainerDied","Data":"6eaa1610624972e21508a3bc4ffdc68476c2d11e8256b2da0882009d28b1eeb0"}
Jul 22 02:30:03.347140 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:03.346731    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6eaa1610624972e21508a3bc4ffdc68476c2d11e8256b2da0882009d28b1eeb0"
Jul 22 02:30:03.493678 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:03.493658    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a73722-29744790-8zmjt"
Jul 22 02:30:03.497641 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:03.497618    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-9d0056-29744790-twvmv"
Jul 22 02:30:04.353582 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:04.353553    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-9d0056-29744790-twvmv"
Jul 22 02:30:04.354002 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:04.353584    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-9d0056-29744790-twvmv" event={"ID":"bb5b1928-d7b8-4c7c-9567-2235fdff4837","Type":"ContainerDied","Data":"e22e5fdc4e721c51d03ba6e4334d8d1dfed8e741766d9bebccc0c6e6a9753068"}
Jul 22 02:30:04.354002 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:04.353638    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e22e5fdc4e721c51d03ba6e4334d8d1dfed8e741766d9bebccc0c6e6a9753068"
Jul 22 02:30:04.355056 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:04.355029    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a73722-29744790-8zmjt" event={"ID":"ce656997-5a4c-4ec5-8224-204ab040e681","Type":"ContainerDied","Data":"b8338e6bb34b02b31ff3380042953bb5670faca799bcbcff1d3d5843d1640bee"}
Jul 22 02:30:04.355182 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:04.355058    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a73722-29744790-8zmjt"
Jul 22 02:30:04.355182 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:04.355072    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8338e6bb34b02b31ff3380042953bb5670faca799bcbcff1d3d5843d1640bee"
Jul 22 02:30:28.123550 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:28.123521    2563 scope.go:117] "RemoveContainer" containerID="f822c571292915cc13097ffd1d323d0ba18558696ee80870991801f54e97efba"
Jul 22 02:30:28.131731 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:28.131708    2563 scope.go:117] "RemoveContainer" containerID="58dc6ea711fc49c72316f31b671c4e101ec03ff3d5b385b2726ad7616b9b1b9d"
Jul 22 02:30:28.372397 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:28.366096    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9d0056/sub-test-model-9d0056-kserve-79b776984c-d2js5"]
Jul 22 02:30:29.161532 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:29.161501    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a73722/sub-test-model-a73722-kserve-6598dd5c47-d8jbb"]
Jul 22 02:30:43.399196 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:43.399130    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-9d0056/sub-test-model-9d0056-kserve-79b776984c-d2js5" podUID="2add91d7-07c5-4d33-8bd2-6388aacc7867" containerName="main" containerID="cri-o://37b9502b11b48c2893fa024dfccb2e40cf09114e017459771012740f51fc1fb9" gracePeriod=45
Jul 22 02:30:43.565733 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:43.565526    2563 generic.go:356] "Generic (PLEG): container finished" podID="2add91d7-07c5-4d33-8bd2-6388aacc7867" containerID="37b9502b11b48c2893fa024dfccb2e40cf09114e017459771012740f51fc1fb9" exitCode=0
Jul 22 02:30:43.565733 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:43.565572    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9d0056/sub-test-model-9d0056-kserve-79b776984c-d2js5" event={"ID":"2add91d7-07c5-4d33-8bd2-6388aacc7867","Type":"ContainerDied","Data":"37b9502b11b48c2893fa024dfccb2e40cf09114e017459771012740f51fc1fb9"}
Jul 22 02:30:43.647872 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:43.647852    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9d0056/sub-test-model-9d0056-kserve-79b776984c-d2js5"
Jul 22 02:30:43.722030 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:43.722003    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2add91d7-07c5-4d33-8bd2-6388aacc7867-tls-certs\") pod \"2add91d7-07c5-4d33-8bd2-6388aacc7867\" (UID: \"2add91d7-07c5-4d33-8bd2-6388aacc7867\") "
Jul 22 02:30:43.722153 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:43.722041    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2add91d7-07c5-4d33-8bd2-6388aacc7867-tmp-dir\") pod \"2add91d7-07c5-4d33-8bd2-6388aacc7867\" (UID: \"2add91d7-07c5-4d33-8bd2-6388aacc7867\") "
Jul 22 02:30:43.722153 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:43.722065    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2add91d7-07c5-4d33-8bd2-6388aacc7867-home\") pod \"2add91d7-07c5-4d33-8bd2-6388aacc7867\" (UID: \"2add91d7-07c5-4d33-8bd2-6388aacc7867\") "
Jul 22 02:30:43.722153 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:43.722096    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2add91d7-07c5-4d33-8bd2-6388aacc7867-dshm\") pod \"2add91d7-07c5-4d33-8bd2-6388aacc7867\" (UID: \"2add91d7-07c5-4d33-8bd2-6388aacc7867\") "
Jul 22 02:30:43.722307 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:43.722148    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnmsq\" (UniqueName: \"kubernetes.io/projected/2add91d7-07c5-4d33-8bd2-6388aacc7867-kube-api-access-xnmsq\") pod \"2add91d7-07c5-4d33-8bd2-6388aacc7867\" (UID: \"2add91d7-07c5-4d33-8bd2-6388aacc7867\") "
Jul 22 02:30:43.722307 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:43.722245    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2add91d7-07c5-4d33-8bd2-6388aacc7867-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2add91d7-07c5-4d33-8bd2-6388aacc7867" (UID: "2add91d7-07c5-4d33-8bd2-6388aacc7867"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:30:43.722307 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:43.722279    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2add91d7-07c5-4d33-8bd2-6388aacc7867-model-cache\") pod \"2add91d7-07c5-4d33-8bd2-6388aacc7867\" (UID: \"2add91d7-07c5-4d33-8bd2-6388aacc7867\") "
Jul 22 02:30:43.722489 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:43.722354    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2add91d7-07c5-4d33-8bd2-6388aacc7867-home" (OuterVolumeSpecName: "home") pod "2add91d7-07c5-4d33-8bd2-6388aacc7867" (UID: "2add91d7-07c5-4d33-8bd2-6388aacc7867"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:30:43.722540 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:43.722499    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2add91d7-07c5-4d33-8bd2-6388aacc7867-model-cache" (OuterVolumeSpecName: "model-cache") pod "2add91d7-07c5-4d33-8bd2-6388aacc7867" (UID: "2add91d7-07c5-4d33-8bd2-6388aacc7867"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:30:43.722603 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:43.722584    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2add91d7-07c5-4d33-8bd2-6388aacc7867-tmp-dir\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:30:43.722657 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:43.722608    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2add91d7-07c5-4d33-8bd2-6388aacc7867-home\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:30:43.722657 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:43.722621    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2add91d7-07c5-4d33-8bd2-6388aacc7867-model-cache\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:30:43.724001 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:43.723980    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2add91d7-07c5-4d33-8bd2-6388aacc7867-dshm" (OuterVolumeSpecName: "dshm") pod "2add91d7-07c5-4d33-8bd2-6388aacc7867" (UID: "2add91d7-07c5-4d33-8bd2-6388aacc7867"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:30:43.724117 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:43.724060    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2add91d7-07c5-4d33-8bd2-6388aacc7867-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2add91d7-07c5-4d33-8bd2-6388aacc7867" (UID: "2add91d7-07c5-4d33-8bd2-6388aacc7867"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:30:43.724117 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:43.724077    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2add91d7-07c5-4d33-8bd2-6388aacc7867-kube-api-access-xnmsq" (OuterVolumeSpecName: "kube-api-access-xnmsq") pod "2add91d7-07c5-4d33-8bd2-6388aacc7867" (UID: "2add91d7-07c5-4d33-8bd2-6388aacc7867"). InnerVolumeSpecName "kube-api-access-xnmsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:30:43.823461 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:43.823444    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnmsq\" (UniqueName: \"kubernetes.io/projected/2add91d7-07c5-4d33-8bd2-6388aacc7867-kube-api-access-xnmsq\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:30:43.823557 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:43.823463    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2add91d7-07c5-4d33-8bd2-6388aacc7867-tls-certs\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:30:43.823557 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:43.823474    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2add91d7-07c5-4d33-8bd2-6388aacc7867-dshm\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:30:44.180444 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:44.180414    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-a73722/sub-test-model-a73722-kserve-6598dd5c47-d8jbb" podUID="f0927033-4a6e-4eb6-9821-22bad184a967" containerName="main" containerID="cri-o://acd949a63ce93ce5cd44f1d1f65b451c1233a802e8fdc22f175ef0aee196b37c" gracePeriod=45
Jul 22 02:30:44.425678 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:44.425656    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a73722/sub-test-model-a73722-kserve-6598dd5c47-d8jbb"
Jul 22 02:30:44.528697 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:44.528642    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0927033-4a6e-4eb6-9821-22bad184a967-home\") pod \"f0927033-4a6e-4eb6-9821-22bad184a967\" (UID: \"f0927033-4a6e-4eb6-9821-22bad184a967\") "
Jul 22 02:30:44.528697 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:44.528690    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0927033-4a6e-4eb6-9821-22bad184a967-tmp-dir\") pod \"f0927033-4a6e-4eb6-9821-22bad184a967\" (UID: \"f0927033-4a6e-4eb6-9821-22bad184a967\") "
Jul 22 02:30:44.528881 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:44.528712    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jg64\" (UniqueName: \"kubernetes.io/projected/f0927033-4a6e-4eb6-9821-22bad184a967-kube-api-access-7jg64\") pod \"f0927033-4a6e-4eb6-9821-22bad184a967\" (UID: \"f0927033-4a6e-4eb6-9821-22bad184a967\") "
Jul 22 02:30:44.528881 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:44.528740    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0927033-4a6e-4eb6-9821-22bad184a967-tls-certs\") pod \"f0927033-4a6e-4eb6-9821-22bad184a967\" (UID: \"f0927033-4a6e-4eb6-9821-22bad184a967\") "
Jul 22 02:30:44.528881 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:44.528769    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0927033-4a6e-4eb6-9821-22bad184a967-dshm\") pod \"f0927033-4a6e-4eb6-9821-22bad184a967\" (UID: \"f0927033-4a6e-4eb6-9821-22bad184a967\") "
Jul 22 02:30:44.528881 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:44.528788    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0927033-4a6e-4eb6-9821-22bad184a967-model-cache\") pod \"f0927033-4a6e-4eb6-9821-22bad184a967\" (UID: \"f0927033-4a6e-4eb6-9821-22bad184a967\") "
Jul 22 02:30:44.528881 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:44.528871    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0927033-4a6e-4eb6-9821-22bad184a967-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f0927033-4a6e-4eb6-9821-22bad184a967" (UID: "f0927033-4a6e-4eb6-9821-22bad184a967"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:30:44.529101 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:44.528887    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0927033-4a6e-4eb6-9821-22bad184a967-home" (OuterVolumeSpecName: "home") pod "f0927033-4a6e-4eb6-9821-22bad184a967" (UID: "f0927033-4a6e-4eb6-9821-22bad184a967"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:30:44.529139 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:44.529105    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0927033-4a6e-4eb6-9821-22bad184a967-model-cache" (OuterVolumeSpecName: "model-cache") pod "f0927033-4a6e-4eb6-9821-22bad184a967" (UID: "f0927033-4a6e-4eb6-9821-22bad184a967"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:30:44.529349 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:44.529328    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0927033-4a6e-4eb6-9821-22bad184a967-home\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:30:44.529454 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:44.529352    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0927033-4a6e-4eb6-9821-22bad184a967-tmp-dir\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:30:44.529454 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:44.529367    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0927033-4a6e-4eb6-9821-22bad184a967-model-cache\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:30:44.530928 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:44.530899    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0927033-4a6e-4eb6-9821-22bad184a967-kube-api-access-7jg64" (OuterVolumeSpecName: "kube-api-access-7jg64") pod "f0927033-4a6e-4eb6-9821-22bad184a967" (UID: "f0927033-4a6e-4eb6-9821-22bad184a967"). InnerVolumeSpecName "kube-api-access-7jg64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:30:44.531039 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:44.530934    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0927033-4a6e-4eb6-9821-22bad184a967-dshm" (OuterVolumeSpecName: "dshm") pod "f0927033-4a6e-4eb6-9821-22bad184a967" (UID: "f0927033-4a6e-4eb6-9821-22bad184a967"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:30:44.531039 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:44.531001    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0927033-4a6e-4eb6-9821-22bad184a967-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f0927033-4a6e-4eb6-9821-22bad184a967" (UID: "f0927033-4a6e-4eb6-9821-22bad184a967"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:30:44.571837 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:44.571808    2563 generic.go:356] "Generic (PLEG): container finished" podID="f0927033-4a6e-4eb6-9821-22bad184a967" containerID="acd949a63ce93ce5cd44f1d1f65b451c1233a802e8fdc22f175ef0aee196b37c" exitCode=0
Jul 22 02:30:44.571940 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:44.571853    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a73722/sub-test-model-a73722-kserve-6598dd5c47-d8jbb"
Jul 22 02:30:44.571940 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:44.571898    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a73722/sub-test-model-a73722-kserve-6598dd5c47-d8jbb" event={"ID":"f0927033-4a6e-4eb6-9821-22bad184a967","Type":"ContainerDied","Data":"acd949a63ce93ce5cd44f1d1f65b451c1233a802e8fdc22f175ef0aee196b37c"}
Jul 22 02:30:44.572048 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:44.571939    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a73722/sub-test-model-a73722-kserve-6598dd5c47-d8jbb" event={"ID":"f0927033-4a6e-4eb6-9821-22bad184a967","Type":"ContainerDied","Data":"7f0e0cdfeee6e73b584026324cd91a28c8d0cfa06a8577bac899afa395a3296a"}
Jul 22 02:30:44.572048 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:44.571962    2563 scope.go:117] "RemoveContainer" containerID="acd949a63ce93ce5cd44f1d1f65b451c1233a802e8fdc22f175ef0aee196b37c"
Jul 22 02:30:44.573293 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:44.573269    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9d0056/sub-test-model-9d0056-kserve-79b776984c-d2js5" event={"ID":"2add91d7-07c5-4d33-8bd2-6388aacc7867","Type":"ContainerDied","Data":"0871c941c9a98dbec9e3d6a52008b297e3ec0818eeb1ddd415b1f8b62f694710"}
Jul 22 02:30:44.573368 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:44.573273    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9d0056/sub-test-model-9d0056-kserve-79b776984c-d2js5"
Jul 22 02:30:44.585015 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:44.584981    2563 scope.go:117] "RemoveContainer" containerID="acd949a63ce93ce5cd44f1d1f65b451c1233a802e8fdc22f175ef0aee196b37c"
Jul 22 02:30:44.585700 ip-10-0-135-230 kubenswrapper[2563]: E0722 02:30:44.585671    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acd949a63ce93ce5cd44f1d1f65b451c1233a802e8fdc22f175ef0aee196b37c\": container with ID starting with acd949a63ce93ce5cd44f1d1f65b451c1233a802e8fdc22f175ef0aee196b37c not found: ID does not exist" containerID="acd949a63ce93ce5cd44f1d1f65b451c1233a802e8fdc22f175ef0aee196b37c"
Jul 22 02:30:44.585789 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:44.585710    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acd949a63ce93ce5cd44f1d1f65b451c1233a802e8fdc22f175ef0aee196b37c"} err="failed to get container status \"acd949a63ce93ce5cd44f1d1f65b451c1233a802e8fdc22f175ef0aee196b37c\": rpc error: code = NotFound desc = could not find container \"acd949a63ce93ce5cd44f1d1f65b451c1233a802e8fdc22f175ef0aee196b37c\": container with ID starting with acd949a63ce93ce5cd44f1d1f65b451c1233a802e8fdc22f175ef0aee196b37c not found: ID does not exist"
Jul 22 02:30:44.585789 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:44.585734    2563 scope.go:117] "RemoveContainer" containerID="37b9502b11b48c2893fa024dfccb2e40cf09114e017459771012740f51fc1fb9"
Jul 22 02:30:44.602153 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:44.602130    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9d0056/sub-test-model-9d0056-kserve-79b776984c-d2js5"]
Jul 22 02:30:44.604966 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:44.604947    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-9d0056/sub-test-model-9d0056-kserve-79b776984c-d2js5"]
Jul 22 02:30:44.616181 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:44.616160    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a73722/sub-test-model-a73722-kserve-6598dd5c47-d8jbb"]
Jul 22 02:30:44.624779 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:44.624760    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-a73722/sub-test-model-a73722-kserve-6598dd5c47-d8jbb"]
Jul 22 02:30:44.630068 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:44.630048    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jg64\" (UniqueName: \"kubernetes.io/projected/f0927033-4a6e-4eb6-9821-22bad184a967-kube-api-access-7jg64\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:30:44.630147 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:44.630071    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0927033-4a6e-4eb6-9821-22bad184a967-tls-certs\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:30:44.630147 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:44.630085    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0927033-4a6e-4eb6-9821-22bad184a967-dshm\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:30:45.663792 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:45.663764    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2add91d7-07c5-4d33-8bd2-6388aacc7867" path="/var/lib/kubelet/pods/2add91d7-07c5-4d33-8bd2-6388aacc7867/volumes"
Jul 22 02:30:45.664133 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:45.664113    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0927033-4a6e-4eb6-9821-22bad184a967" path="/var/lib/kubelet/pods/f0927033-4a6e-4eb6-9821-22bad184a967/volumes"
Jul 22 02:30:55.263682 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:55.263608    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-9d0056/rate-test-model-9d0056-kserve-7b8767cc55-hdvzg"]
Jul 22 02:30:55.264119 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:55.263988    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce656997-5a4c-4ec5-8224-204ab040e681" containerName="cleanup"
Jul 22 02:30:55.264119 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:55.264001    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce656997-5a4c-4ec5-8224-204ab040e681" containerName="cleanup"
Jul 22 02:30:55.264119 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:55.264020    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2add91d7-07c5-4d33-8bd2-6388aacc7867" containerName="main"
Jul 22 02:30:55.264119 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:55.264025    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2add91d7-07c5-4d33-8bd2-6388aacc7867" containerName="main"
Jul 22 02:30:55.264119 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:55.264032    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0927033-4a6e-4eb6-9821-22bad184a967" containerName="main"
Jul 22 02:30:55.264119 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:55.264037    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0927033-4a6e-4eb6-9821-22bad184a967" containerName="main"
Jul 22 02:30:55.264119 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:55.264043    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb5b1928-d7b8-4c7c-9567-2235fdff4837" containerName="cleanup"
Jul 22 02:30:55.264119 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:55.264048    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb5b1928-d7b8-4c7c-9567-2235fdff4837" containerName="cleanup"
Jul 22 02:30:55.264119 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:55.264061    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be20609b-0f9c-4e66-8a91-81e8b8d8bace" containerName="cleanup"
Jul 22 02:30:55.264119 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:55.264065    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="be20609b-0f9c-4e66-8a91-81e8b8d8bace" containerName="cleanup"
Jul 22 02:30:55.264119 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:55.264118    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb5b1928-d7b8-4c7c-9567-2235fdff4837" containerName="cleanup"
Jul 22 02:30:55.264493 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:55.264126    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2add91d7-07c5-4d33-8bd2-6388aacc7867" containerName="main"
Jul 22 02:30:55.264493 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:55.264131    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce656997-5a4c-4ec5-8224-204ab040e681" containerName="cleanup"
Jul 22 02:30:55.264493 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:55.264137    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="be20609b-0f9c-4e66-8a91-81e8b8d8bace" containerName="cleanup"
Jul 22 02:30:55.264493 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:55.264143    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0927033-4a6e-4eb6-9821-22bad184a967" containerName="main"
Jul 22 02:30:55.268658 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:55.268643    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9d0056/rate-test-model-9d0056-kserve-7b8767cc55-hdvzg"
Jul 22 02:30:55.272278 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:55.272258    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9d0056\"/\"default-dockercfg-mwkhg\""
Jul 22 02:30:55.272278 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:55.272270    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9d0056\"/\"rate-test-model-9d0056-kserve-self-signed-certs\""
Jul 22 02:30:55.272463 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:55.272316    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-9d0056\"/\"openshift-service-ca.crt\""
Jul 22 02:30:55.272463 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:55.272361    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-9d0056\"/\"kube-root-ca.crt\""
Jul 22 02:30:55.279988 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:55.279968    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9d0056/rate-test-model-9d0056-kserve-7b8767cc55-hdvzg"]
Jul 22 02:30:55.308676 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:55.308647    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6b762dc-f195-4fd4-bc65-6534a8f85e49-tls-certs\") pod \"rate-test-model-9d0056-kserve-7b8767cc55-hdvzg\" (UID: \"a6b762dc-f195-4fd4-bc65-6534a8f85e49\") " pod="ai-tenant-e2e-shared-a-9d0056/rate-test-model-9d0056-kserve-7b8767cc55-hdvzg"
Jul 22 02:30:55.308772 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:55.308690    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6b762dc-f195-4fd4-bc65-6534a8f85e49-model-cache\") pod \"rate-test-model-9d0056-kserve-7b8767cc55-hdvzg\" (UID: \"a6b762dc-f195-4fd4-bc65-6534a8f85e49\") " pod="ai-tenant-e2e-shared-a-9d0056/rate-test-model-9d0056-kserve-7b8767cc55-hdvzg"
Jul 22 02:30:55.308772 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:55.308716    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6b762dc-f195-4fd4-bc65-6534a8f85e49-tmp-dir\") pod \"rate-test-model-9d0056-kserve-7b8767cc55-hdvzg\" (UID: \"a6b762dc-f195-4fd4-bc65-6534a8f85e49\") " pod="ai-tenant-e2e-shared-a-9d0056/rate-test-model-9d0056-kserve-7b8767cc55-hdvzg"
Jul 22 02:30:55.308855 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:55.308813    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfbqw\" (UniqueName: \"kubernetes.io/projected/a6b762dc-f195-4fd4-bc65-6534a8f85e49-kube-api-access-kfbqw\") pod \"rate-test-model-9d0056-kserve-7b8767cc55-hdvzg\" (UID: \"a6b762dc-f195-4fd4-bc65-6534a8f85e49\") " pod="ai-tenant-e2e-shared-a-9d0056/rate-test-model-9d0056-kserve-7b8767cc55-hdvzg"
Jul 22 02:30:55.308899 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:55.308857    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6b762dc-f195-4fd4-bc65-6534a8f85e49-dshm\") pod \"rate-test-model-9d0056-kserve-7b8767cc55-hdvzg\" (UID: \"a6b762dc-f195-4fd4-bc65-6534a8f85e49\") " pod="ai-tenant-e2e-shared-a-9d0056/rate-test-model-9d0056-kserve-7b8767cc55-hdvzg"
Jul 22 02:30:55.308940 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:55.308912    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6b762dc-f195-4fd4-bc65-6534a8f85e49-home\") pod \"rate-test-model-9d0056-kserve-7b8767cc55-hdvzg\" (UID: \"a6b762dc-f195-4fd4-bc65-6534a8f85e49\") " pod="ai-tenant-e2e-shared-a-9d0056/rate-test-model-9d0056-kserve-7b8767cc55-hdvzg"
Jul 22 02:30:55.409715 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:55.409689    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfbqw\" (UniqueName: \"kubernetes.io/projected/a6b762dc-f195-4fd4-bc65-6534a8f85e49-kube-api-access-kfbqw\") pod \"rate-test-model-9d0056-kserve-7b8767cc55-hdvzg\" (UID: \"a6b762dc-f195-4fd4-bc65-6534a8f85e49\") " pod="ai-tenant-e2e-shared-a-9d0056/rate-test-model-9d0056-kserve-7b8767cc55-hdvzg"
Jul 22 02:30:55.409829 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:55.409721    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6b762dc-f195-4fd4-bc65-6534a8f85e49-dshm\") pod \"rate-test-model-9d0056-kserve-7b8767cc55-hdvzg\" (UID: \"a6b762dc-f195-4fd4-bc65-6534a8f85e49\") " pod="ai-tenant-e2e-shared-a-9d0056/rate-test-model-9d0056-kserve-7b8767cc55-hdvzg"
Jul 22 02:30:55.409829 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:55.409749    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6b762dc-f195-4fd4-bc65-6534a8f85e49-home\") pod \"rate-test-model-9d0056-kserve-7b8767cc55-hdvzg\" (UID: \"a6b762dc-f195-4fd4-bc65-6534a8f85e49\") " pod="ai-tenant-e2e-shared-a-9d0056/rate-test-model-9d0056-kserve-7b8767cc55-hdvzg"
Jul 22 02:30:55.409829 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:55.409802    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6b762dc-f195-4fd4-bc65-6534a8f85e49-tls-certs\") pod \"rate-test-model-9d0056-kserve-7b8767cc55-hdvzg\" (UID: \"a6b762dc-f195-4fd4-bc65-6534a8f85e49\") " pod="ai-tenant-e2e-shared-a-9d0056/rate-test-model-9d0056-kserve-7b8767cc55-hdvzg"
Jul 22 02:30:55.409987 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:55.409836    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6b762dc-f195-4fd4-bc65-6534a8f85e49-model-cache\") pod \"rate-test-model-9d0056-kserve-7b8767cc55-hdvzg\" (UID: \"a6b762dc-f195-4fd4-bc65-6534a8f85e49\") " pod="ai-tenant-e2e-shared-a-9d0056/rate-test-model-9d0056-kserve-7b8767cc55-hdvzg"
Jul 22 02:30:55.409987 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:55.409859    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6b762dc-f195-4fd4-bc65-6534a8f85e49-tmp-dir\") pod \"rate-test-model-9d0056-kserve-7b8767cc55-hdvzg\" (UID: \"a6b762dc-f195-4fd4-bc65-6534a8f85e49\") " pod="ai-tenant-e2e-shared-a-9d0056/rate-test-model-9d0056-kserve-7b8767cc55-hdvzg"
Jul 22 02:30:55.410208 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:55.410186    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6b762dc-f195-4fd4-bc65-6534a8f85e49-home\") pod \"rate-test-model-9d0056-kserve-7b8767cc55-hdvzg\" (UID: \"a6b762dc-f195-4fd4-bc65-6534a8f85e49\") " pod="ai-tenant-e2e-shared-a-9d0056/rate-test-model-9d0056-kserve-7b8767cc55-hdvzg"
Jul 22 02:30:55.410323 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:55.410230    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6b762dc-f195-4fd4-bc65-6534a8f85e49-tmp-dir\") pod \"rate-test-model-9d0056-kserve-7b8767cc55-hdvzg\" (UID: \"a6b762dc-f195-4fd4-bc65-6534a8f85e49\") " pod="ai-tenant-e2e-shared-a-9d0056/rate-test-model-9d0056-kserve-7b8767cc55-hdvzg"
Jul 22 02:30:55.410323 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:55.410256    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6b762dc-f195-4fd4-bc65-6534a8f85e49-model-cache\") pod \"rate-test-model-9d0056-kserve-7b8767cc55-hdvzg\" (UID: \"a6b762dc-f195-4fd4-bc65-6534a8f85e49\") " pod="ai-tenant-e2e-shared-a-9d0056/rate-test-model-9d0056-kserve-7b8767cc55-hdvzg"
Jul 22 02:30:55.411847 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:55.411819    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6b762dc-f195-4fd4-bc65-6534a8f85e49-dshm\") pod \"rate-test-model-9d0056-kserve-7b8767cc55-hdvzg\" (UID: \"a6b762dc-f195-4fd4-bc65-6534a8f85e49\") " pod="ai-tenant-e2e-shared-a-9d0056/rate-test-model-9d0056-kserve-7b8767cc55-hdvzg"
Jul 22 02:30:55.412154 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:55.412137    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6b762dc-f195-4fd4-bc65-6534a8f85e49-tls-certs\") pod \"rate-test-model-9d0056-kserve-7b8767cc55-hdvzg\" (UID: \"a6b762dc-f195-4fd4-bc65-6534a8f85e49\") " pod="ai-tenant-e2e-shared-a-9d0056/rate-test-model-9d0056-kserve-7b8767cc55-hdvzg"
Jul 22 02:30:55.417817 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:55.417798    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfbqw\" (UniqueName: \"kubernetes.io/projected/a6b762dc-f195-4fd4-bc65-6534a8f85e49-kube-api-access-kfbqw\") pod \"rate-test-model-9d0056-kserve-7b8767cc55-hdvzg\" (UID: \"a6b762dc-f195-4fd4-bc65-6534a8f85e49\") " pod="ai-tenant-e2e-shared-a-9d0056/rate-test-model-9d0056-kserve-7b8767cc55-hdvzg"
Jul 22 02:30:55.578175 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:55.578117    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9d0056/rate-test-model-9d0056-kserve-7b8767cc55-hdvzg"
Jul 22 02:30:55.720945 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:55.720922    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9d0056/rate-test-model-9d0056-kserve-7b8767cc55-hdvzg"]
Jul 22 02:30:55.723240 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:30:55.723210    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6b762dc_f195_4fd4_bc65_6534a8f85e49.slice/crio-7c558b21c057c7112705c187a003880b2670cd0142be009efd7ddf5fc1289271 WatchSource:0}: Error finding container 7c558b21c057c7112705c187a003880b2670cd0142be009efd7ddf5fc1289271: Status 404 returned error can't find the container with id 7c558b21c057c7112705c187a003880b2670cd0142be009efd7ddf5fc1289271
Jul 22 02:30:56.648315 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:56.648281    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9d0056/rate-test-model-9d0056-kserve-7b8767cc55-hdvzg" event={"ID":"a6b762dc-f195-4fd4-bc65-6534a8f85e49","Type":"ContainerStarted","Data":"d71ce2ba831a449984b9fa2ca2f50d1844a915dc9669c9bcf1f854ce141647cd"}
Jul 22 02:30:56.648746 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:56.648322    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9d0056/rate-test-model-9d0056-kserve-7b8767cc55-hdvzg" event={"ID":"a6b762dc-f195-4fd4-bc65-6534a8f85e49","Type":"ContainerStarted","Data":"7c558b21c057c7112705c187a003880b2670cd0142be009efd7ddf5fc1289271"}
Jul 22 02:30:56.671171 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:30:56.671131    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-9d0056/rate-test-model-9d0056-kserve-7b8767cc55-hdvzg" podStartSLOduration=1.6711167310000001 podStartE2EDuration="1.671116731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:30:56.670974813 +0000 UTC m=+2609.514803943" watchObservedRunningTime="2026-07-22 02:30:56.671116731 +0000 UTC m=+2609.514945920"
Jul 22 02:31:05.578835 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:05.578799    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-9d0056/rate-test-model-9d0056-kserve-7b8767cc55-hdvzg"
Jul 22 02:31:05.579330 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:05.578849    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-9d0056/rate-test-model-9d0056-kserve-7b8767cc55-hdvzg"
Jul 22 02:31:05.590775 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:05.590753    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-9d0056/rate-test-model-9d0056-kserve-7b8767cc55-hdvzg"
Jul 22 02:31:05.709537 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:05.709509    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-9d0056/rate-test-model-9d0056-kserve-7b8767cc55-hdvzg"
Jul 22 02:31:11.155827 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:11.155795    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-a73722/rate-test-model-a73722-kserve-5cc99c85bc-pdmvf"]
Jul 22 02:31:11.159145 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:11.159131    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a73722/rate-test-model-a73722-kserve-5cc99c85bc-pdmvf"
Jul 22 02:31:11.161863 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:11.161841    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a73722\"/\"openshift-service-ca.crt\""
Jul 22 02:31:11.162088 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:11.161975    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a73722\"/\"kube-root-ca.crt\""
Jul 22 02:31:11.162634 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:11.162615    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a73722\"/\"default-dockercfg-wlc4l\""
Jul 22 02:31:11.162846 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:11.162831    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a73722\"/\"rate-test-model-a73722-kserve-self-signed-certs\""
Jul 22 02:31:11.171335 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:11.171314    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a73722/rate-test-model-a73722-kserve-5cc99c85bc-pdmvf"]
Jul 22 02:31:11.224867 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:11.224835    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/545a838e-a0e7-49fe-aab7-2a9146f33073-home\") pod \"rate-test-model-a73722-kserve-5cc99c85bc-pdmvf\" (UID: \"545a838e-a0e7-49fe-aab7-2a9146f33073\") " pod="ai-tenant-e2e-shared-b-a73722/rate-test-model-a73722-kserve-5cc99c85bc-pdmvf"
Jul 22 02:31:11.225015 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:11.224883    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/545a838e-a0e7-49fe-aab7-2a9146f33073-dshm\") pod \"rate-test-model-a73722-kserve-5cc99c85bc-pdmvf\" (UID: \"545a838e-a0e7-49fe-aab7-2a9146f33073\") " pod="ai-tenant-e2e-shared-b-a73722/rate-test-model-a73722-kserve-5cc99c85bc-pdmvf"
Jul 22 02:31:11.225015 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:11.224914    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/545a838e-a0e7-49fe-aab7-2a9146f33073-model-cache\") pod \"rate-test-model-a73722-kserve-5cc99c85bc-pdmvf\" (UID: \"545a838e-a0e7-49fe-aab7-2a9146f33073\") " pod="ai-tenant-e2e-shared-b-a73722/rate-test-model-a73722-kserve-5cc99c85bc-pdmvf"
Jul 22 02:31:11.225015 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:11.224974    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/545a838e-a0e7-49fe-aab7-2a9146f33073-tls-certs\") pod \"rate-test-model-a73722-kserve-5cc99c85bc-pdmvf\" (UID: \"545a838e-a0e7-49fe-aab7-2a9146f33073\") " pod="ai-tenant-e2e-shared-b-a73722/rate-test-model-a73722-kserve-5cc99c85bc-pdmvf"
Jul 22 02:31:11.225180 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:11.225026    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpjpn\" (UniqueName: \"kubernetes.io/projected/545a838e-a0e7-49fe-aab7-2a9146f33073-kube-api-access-tpjpn\") pod \"rate-test-model-a73722-kserve-5cc99c85bc-pdmvf\" (UID: \"545a838e-a0e7-49fe-aab7-2a9146f33073\") " pod="ai-tenant-e2e-shared-b-a73722/rate-test-model-a73722-kserve-5cc99c85bc-pdmvf"
Jul 22 02:31:11.225180 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:11.225106    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/545a838e-a0e7-49fe-aab7-2a9146f33073-tmp-dir\") pod \"rate-test-model-a73722-kserve-5cc99c85bc-pdmvf\" (UID: \"545a838e-a0e7-49fe-aab7-2a9146f33073\") " pod="ai-tenant-e2e-shared-b-a73722/rate-test-model-a73722-kserve-5cc99c85bc-pdmvf"
Jul 22 02:31:11.326361 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:11.326319    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/545a838e-a0e7-49fe-aab7-2a9146f33073-dshm\") pod \"rate-test-model-a73722-kserve-5cc99c85bc-pdmvf\" (UID: \"545a838e-a0e7-49fe-aab7-2a9146f33073\") " pod="ai-tenant-e2e-shared-b-a73722/rate-test-model-a73722-kserve-5cc99c85bc-pdmvf"
Jul 22 02:31:11.326548 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:11.326395    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/545a838e-a0e7-49fe-aab7-2a9146f33073-model-cache\") pod \"rate-test-model-a73722-kserve-5cc99c85bc-pdmvf\" (UID: \"545a838e-a0e7-49fe-aab7-2a9146f33073\") " pod="ai-tenant-e2e-shared-b-a73722/rate-test-model-a73722-kserve-5cc99c85bc-pdmvf"
Jul 22 02:31:11.326548 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:11.326431    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/545a838e-a0e7-49fe-aab7-2a9146f33073-tls-certs\") pod \"rate-test-model-a73722-kserve-5cc99c85bc-pdmvf\" (UID: \"545a838e-a0e7-49fe-aab7-2a9146f33073\") " pod="ai-tenant-e2e-shared-b-a73722/rate-test-model-a73722-kserve-5cc99c85bc-pdmvf"
Jul 22 02:31:11.326548 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:11.326473    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpjpn\" (UniqueName: \"kubernetes.io/projected/545a838e-a0e7-49fe-aab7-2a9146f33073-kube-api-access-tpjpn\") pod \"rate-test-model-a73722-kserve-5cc99c85bc-pdmvf\" (UID: \"545a838e-a0e7-49fe-aab7-2a9146f33073\") " pod="ai-tenant-e2e-shared-b-a73722/rate-test-model-a73722-kserve-5cc99c85bc-pdmvf"
Jul 22 02:31:11.326548 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:11.326528    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/545a838e-a0e7-49fe-aab7-2a9146f33073-tmp-dir\") pod \"rate-test-model-a73722-kserve-5cc99c85bc-pdmvf\" (UID: \"545a838e-a0e7-49fe-aab7-2a9146f33073\") " pod="ai-tenant-e2e-shared-b-a73722/rate-test-model-a73722-kserve-5cc99c85bc-pdmvf"
Jul 22 02:31:11.326771 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:11.326580    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/545a838e-a0e7-49fe-aab7-2a9146f33073-home\") pod \"rate-test-model-a73722-kserve-5cc99c85bc-pdmvf\" (UID: \"545a838e-a0e7-49fe-aab7-2a9146f33073\") " pod="ai-tenant-e2e-shared-b-a73722/rate-test-model-a73722-kserve-5cc99c85bc-pdmvf"
Jul 22 02:31:11.326830 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:11.326803    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/545a838e-a0e7-49fe-aab7-2a9146f33073-model-cache\") pod \"rate-test-model-a73722-kserve-5cc99c85bc-pdmvf\" (UID: \"545a838e-a0e7-49fe-aab7-2a9146f33073\") " pod="ai-tenant-e2e-shared-b-a73722/rate-test-model-a73722-kserve-5cc99c85bc-pdmvf"
Jul 22 02:31:11.326886 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:11.326869    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/545a838e-a0e7-49fe-aab7-2a9146f33073-home\") pod \"rate-test-model-a73722-kserve-5cc99c85bc-pdmvf\" (UID: \"545a838e-a0e7-49fe-aab7-2a9146f33073\") " pod="ai-tenant-e2e-shared-b-a73722/rate-test-model-a73722-kserve-5cc99c85bc-pdmvf"
Jul 22 02:31:11.327089 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:11.327054    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/545a838e-a0e7-49fe-aab7-2a9146f33073-tmp-dir\") pod \"rate-test-model-a73722-kserve-5cc99c85bc-pdmvf\" (UID: \"545a838e-a0e7-49fe-aab7-2a9146f33073\") " pod="ai-tenant-e2e-shared-b-a73722/rate-test-model-a73722-kserve-5cc99c85bc-pdmvf"
Jul 22 02:31:11.329153 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:11.329128    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/545a838e-a0e7-49fe-aab7-2a9146f33073-dshm\") pod \"rate-test-model-a73722-kserve-5cc99c85bc-pdmvf\" (UID: \"545a838e-a0e7-49fe-aab7-2a9146f33073\") " pod="ai-tenant-e2e-shared-b-a73722/rate-test-model-a73722-kserve-5cc99c85bc-pdmvf"
Jul 22 02:31:11.334111 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:11.334090    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/545a838e-a0e7-49fe-aab7-2a9146f33073-tls-certs\") pod \"rate-test-model-a73722-kserve-5cc99c85bc-pdmvf\" (UID: \"545a838e-a0e7-49fe-aab7-2a9146f33073\") " pod="ai-tenant-e2e-shared-b-a73722/rate-test-model-a73722-kserve-5cc99c85bc-pdmvf"
Jul 22 02:31:11.335496 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:11.335477    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpjpn\" (UniqueName: \"kubernetes.io/projected/545a838e-a0e7-49fe-aab7-2a9146f33073-kube-api-access-tpjpn\") pod \"rate-test-model-a73722-kserve-5cc99c85bc-pdmvf\" (UID: \"545a838e-a0e7-49fe-aab7-2a9146f33073\") " pod="ai-tenant-e2e-shared-b-a73722/rate-test-model-a73722-kserve-5cc99c85bc-pdmvf"
Jul 22 02:31:11.469536 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:11.469503    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a73722/rate-test-model-a73722-kserve-5cc99c85bc-pdmvf"
Jul 22 02:31:11.612980 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:11.612952    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a73722/rate-test-model-a73722-kserve-5cc99c85bc-pdmvf"]
Jul 22 02:31:11.614525 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:31:11.614491    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod545a838e_a0e7_49fe_aab7_2a9146f33073.slice/crio-7bc99222b5020fc8d3f9556c61edb0be19ec06a5ba5e21494375391b1cce026f WatchSource:0}: Error finding container 7bc99222b5020fc8d3f9556c61edb0be19ec06a5ba5e21494375391b1cce026f: Status 404 returned error can't find the container with id 7bc99222b5020fc8d3f9556c61edb0be19ec06a5ba5e21494375391b1cce026f
Jul 22 02:31:11.728691 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:11.728612    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a73722/rate-test-model-a73722-kserve-5cc99c85bc-pdmvf" event={"ID":"545a838e-a0e7-49fe-aab7-2a9146f33073","Type":"ContainerStarted","Data":"08cd5e8289f59ed9781aa229ef98b0ca503c3a9aa6896a0e9d2276ca2c105210"}
Jul 22 02:31:11.728691 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:11.728662    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a73722/rate-test-model-a73722-kserve-5cc99c85bc-pdmvf" event={"ID":"545a838e-a0e7-49fe-aab7-2a9146f33073","Type":"ContainerStarted","Data":"7bc99222b5020fc8d3f9556c61edb0be19ec06a5ba5e21494375391b1cce026f"}
Jul 22 02:31:11.747973 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:11.747926    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-a73722/rate-test-model-a73722-kserve-5cc99c85bc-pdmvf" podStartSLOduration=0.747910324 podStartE2EDuration="747.910324ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:31:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:31:11.746658104 +0000 UTC m=+2624.590487236" watchObservedRunningTime="2026-07-22 02:31:11.747910324 +0000 UTC m=+2624.591739454"
Jul 22 02:31:21.469923 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:21.469883    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-a73722/rate-test-model-a73722-kserve-5cc99c85bc-pdmvf"
Jul 22 02:31:21.470456 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:21.469954    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-a73722/rate-test-model-a73722-kserve-5cc99c85bc-pdmvf"
Jul 22 02:31:21.482291 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:21.482271    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-a73722/rate-test-model-a73722-kserve-5cc99c85bc-pdmvf"
Jul 22 02:31:21.793823 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:21.793731    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-a73722/rate-test-model-a73722-kserve-5cc99c85bc-pdmvf"
Jul 22 02:31:28.153237 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:28.153204    2563 scope.go:117] "RemoveContainer" containerID="acfc0372620f1fb7ebc95bec95208dfe1a9cc8cd3c1644af30b9d482e333ecf3"
Jul 22 02:31:57.188396 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:57.188353    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9d0056/rate-test-model-9d0056-kserve-7b8767cc55-hdvzg"]
Jul 22 02:31:58.136538 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:58.136508    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a73722/rate-test-model-a73722-kserve-5cc99c85bc-pdmvf"]
Jul 22 02:31:59.190771 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:59.190736    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7285-140585246288944"]
Jul 22 02:31:59.194310 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:59.194290    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7285-140585246288944"
Jul 22 02:31:59.198574 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:59.198556    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wfg6c\""
Jul 22 02:31:59.206566 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:59.206545    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7285-140585246288944"]
Jul 22 02:31:59.246390 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:59.246357    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blnkq\" (UniqueName: \"kubernetes.io/projected/42c364b1-71f7-493d-b206-3f2f31b2d04b-kube-api-access-blnkq\") pod \"test-curl-7285-140585246288944\" (UID: \"42c364b1-71f7-493d-b206-3f2f31b2d04b\") " pod="opendatahub/test-curl-7285-140585246288944"
Jul 22 02:31:59.347004 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:59.346966    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blnkq\" (UniqueName: \"kubernetes.io/projected/42c364b1-71f7-493d-b206-3f2f31b2d04b-kube-api-access-blnkq\") pod \"test-curl-7285-140585246288944\" (UID: \"42c364b1-71f7-493d-b206-3f2f31b2d04b\") " pod="opendatahub/test-curl-7285-140585246288944"
Jul 22 02:31:59.355735 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:59.355703    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blnkq\" (UniqueName: \"kubernetes.io/projected/42c364b1-71f7-493d-b206-3f2f31b2d04b-kube-api-access-blnkq\") pod \"test-curl-7285-140585246288944\" (UID: \"42c364b1-71f7-493d-b206-3f2f31b2d04b\") " pod="opendatahub/test-curl-7285-140585246288944"
Jul 22 02:31:59.504176 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:59.504092    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7285-140585246288944"
Jul 22 02:31:59.640746 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:59.640326    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7285-140585246288944"]
Jul 22 02:31:59.646079 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:59.646053    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 22 02:31:59.978665 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:31:59.978618    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7285-140585246288944" event={"ID":"42c364b1-71f7-493d-b206-3f2f31b2d04b","Type":"ContainerStarted","Data":"332e2d787a8831621cb340cc4655ae221dfe517313a938a86d299e004fd0c04b"}
Jul 22 02:32:00.985566 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:00.985534    2563 generic.go:356] "Generic (PLEG): container finished" podID="42c364b1-71f7-493d-b206-3f2f31b2d04b" containerID="ee3b27682089b0713d9b32dfc7d13ee964a2afd7e74c2a9d430a59459fdb3734" exitCode=0
Jul 22 02:32:00.986123 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:00.985614    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7285-140585246288944" event={"ID":"42c364b1-71f7-493d-b206-3f2f31b2d04b","Type":"ContainerDied","Data":"ee3b27682089b0713d9b32dfc7d13ee964a2afd7e74c2a9d430a59459fdb3734"}
Jul 22 02:32:02.122656 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:02.122633    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7285-140585246288944"
Jul 22 02:32:02.161312 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:02.161286    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7285-140585246288944_42c364b1-71f7-493d-b206-3f2f31b2d04b/test-curl-7285-140585246288944/0.log"
Jul 22 02:32:02.171523 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:02.171504    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blnkq\" (UniqueName: \"kubernetes.io/projected/42c364b1-71f7-493d-b206-3f2f31b2d04b-kube-api-access-blnkq\") pod \"42c364b1-71f7-493d-b206-3f2f31b2d04b\" (UID: \"42c364b1-71f7-493d-b206-3f2f31b2d04b\") "
Jul 22 02:32:02.173542 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:02.173517    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42c364b1-71f7-493d-b206-3f2f31b2d04b-kube-api-access-blnkq" (OuterVolumeSpecName: "kube-api-access-blnkq") pod "42c364b1-71f7-493d-b206-3f2f31b2d04b" (UID: "42c364b1-71f7-493d-b206-3f2f31b2d04b"). InnerVolumeSpecName "kube-api-access-blnkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:32:02.191054 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:02.191018    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7285-140585246288944"]
Jul 22 02:32:02.194535 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:02.194511    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7285-140585246288944"]
Jul 22 02:32:02.272966 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:02.272897    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blnkq\" (UniqueName: \"kubernetes.io/projected/42c364b1-71f7-493d-b206-3f2f31b2d04b-kube-api-access-blnkq\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:32:02.338036 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:02.338006    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7285-140585246288816"]
Jul 22 02:32:02.338410 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:02.338396    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42c364b1-71f7-493d-b206-3f2f31b2d04b" containerName="test-curl-7285-140585246288944"
Jul 22 02:32:02.338476 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:02.338412    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="42c364b1-71f7-493d-b206-3f2f31b2d04b" containerName="test-curl-7285-140585246288944"
Jul 22 02:32:02.338515 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:02.338486    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="42c364b1-71f7-493d-b206-3f2f31b2d04b" containerName="test-curl-7285-140585246288944"
Jul 22 02:32:02.341545 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:02.341531    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7285-140585246288816"
Jul 22 02:32:02.349676 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:02.349651    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7285-140585246288816"]
Jul 22 02:32:02.474789 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:02.474756    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v7v4\" (UniqueName: \"kubernetes.io/projected/6d76ec13-1e16-4df9-8ff7-96411c0237d1-kube-api-access-9v7v4\") pod \"test-curl-7285-140585246288816\" (UID: \"6d76ec13-1e16-4df9-8ff7-96411c0237d1\") " pod="opendatahub/test-curl-7285-140585246288816"
Jul 22 02:32:02.575786 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:02.575697    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v7v4\" (UniqueName: \"kubernetes.io/projected/6d76ec13-1e16-4df9-8ff7-96411c0237d1-kube-api-access-9v7v4\") pod \"test-curl-7285-140585246288816\" (UID: \"6d76ec13-1e16-4df9-8ff7-96411c0237d1\") " pod="opendatahub/test-curl-7285-140585246288816"
Jul 22 02:32:02.584789 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:02.584764    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v7v4\" (UniqueName: \"kubernetes.io/projected/6d76ec13-1e16-4df9-8ff7-96411c0237d1-kube-api-access-9v7v4\") pod \"test-curl-7285-140585246288816\" (UID: \"6d76ec13-1e16-4df9-8ff7-96411c0237d1\") " pod="opendatahub/test-curl-7285-140585246288816"
Jul 22 02:32:02.651802 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:02.651767    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7285-140585246288816"
Jul 22 02:32:02.792838 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:02.792811    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7285-140585246288816"]
Jul 22 02:32:02.794170 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:32:02.794148    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d76ec13_1e16_4df9_8ff7_96411c0237d1.slice/crio-73da5aca41b23bcad08e5a2330f06b67144c5fadcb196b35da56a930a6eaf397 WatchSource:0}: Error finding container 73da5aca41b23bcad08e5a2330f06b67144c5fadcb196b35da56a930a6eaf397: Status 404 returned error can't find the container with id 73da5aca41b23bcad08e5a2330f06b67144c5fadcb196b35da56a930a6eaf397
Jul 22 02:32:02.998903 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:02.998872    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7285-140585246288944"
Jul 22 02:32:02.998903 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:02.998901    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="332e2d787a8831621cb340cc4655ae221dfe517313a938a86d299e004fd0c04b"
Jul 22 02:32:03.000141 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:03.000114    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7285-140585246288816" event={"ID":"6d76ec13-1e16-4df9-8ff7-96411c0237d1","Type":"ContainerStarted","Data":"73da5aca41b23bcad08e5a2330f06b67144c5fadcb196b35da56a930a6eaf397"}
Jul 22 02:32:03.015604 ip-10-0-135-230 kubenswrapper[2563]: E0722 02:32:03.015577    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7285-140585246288944\" is forbidden: User \"system:node:ip-10-0-135-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-230.ec2.internal' and this object" podUID="42c364b1-71f7-493d-b206-3f2f31b2d04b" pod="opendatahub/test-curl-7285-140585246288944"
Jul 22 02:32:03.663247 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:03.663214    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42c364b1-71f7-493d-b206-3f2f31b2d04b" path="/var/lib/kubelet/pods/42c364b1-71f7-493d-b206-3f2f31b2d04b/volumes"
Jul 22 02:32:04.008862 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:04.008832    2563 generic.go:356] "Generic (PLEG): container finished" podID="6d76ec13-1e16-4df9-8ff7-96411c0237d1" containerID="d09c6bb56ad170ee6182d424d746c7164b5e77dfba965ece5b65211b80b1e253" exitCode=0
Jul 22 02:32:04.009046 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:04.008880    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7285-140585246288816" event={"ID":"6d76ec13-1e16-4df9-8ff7-96411c0237d1","Type":"ContainerDied","Data":"d09c6bb56ad170ee6182d424d746c7164b5e77dfba965ece5b65211b80b1e253"}
Jul 22 02:32:05.146807 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:05.146783    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7285-140585246288816"
Jul 22 02:32:05.167958 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:05.167928    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7285-140585246288816_6d76ec13-1e16-4df9-8ff7-96411c0237d1/test-curl-7285-140585246288816/0.log"
Jul 22 02:32:05.197414 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:05.197361    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9v7v4\" (UniqueName: \"kubernetes.io/projected/6d76ec13-1e16-4df9-8ff7-96411c0237d1-kube-api-access-9v7v4\") pod \"6d76ec13-1e16-4df9-8ff7-96411c0237d1\" (UID: \"6d76ec13-1e16-4df9-8ff7-96411c0237d1\") "
Jul 22 02:32:05.197914 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:05.197880    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7285-140585246288816"]
Jul 22 02:32:05.199513 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:05.199486    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d76ec13-1e16-4df9-8ff7-96411c0237d1-kube-api-access-9v7v4" (OuterVolumeSpecName: "kube-api-access-9v7v4") pod "6d76ec13-1e16-4df9-8ff7-96411c0237d1" (UID: "6d76ec13-1e16-4df9-8ff7-96411c0237d1"). InnerVolumeSpecName "kube-api-access-9v7v4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:32:05.200171 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:05.200151    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7285-140585246288816"]
Jul 22 02:32:05.298709 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:05.298618    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v7v4\" (UniqueName: \"kubernetes.io/projected/6d76ec13-1e16-4df9-8ff7-96411c0237d1-kube-api-access-9v7v4\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:32:05.378887 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:05.378856    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7285-140585246295072"]
Jul 22 02:32:05.379241 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:05.379228    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d76ec13-1e16-4df9-8ff7-96411c0237d1" containerName="test-curl-7285-140585246288816"
Jul 22 02:32:05.379283 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:05.379242    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d76ec13-1e16-4df9-8ff7-96411c0237d1" containerName="test-curl-7285-140585246288816"
Jul 22 02:32:05.379315 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:05.379309    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d76ec13-1e16-4df9-8ff7-96411c0237d1" containerName="test-curl-7285-140585246288816"
Jul 22 02:32:05.382242 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:05.382227    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7285-140585246295072"
Jul 22 02:32:05.390362 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:05.390337    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7285-140585246295072"]
Jul 22 02:32:05.500874 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:05.500825    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6qds\" (UniqueName: \"kubernetes.io/projected/53feeba4-b8eb-49fe-9ebd-a67f0809f4a9-kube-api-access-p6qds\") pod \"test-curl-7285-140585246295072\" (UID: \"53feeba4-b8eb-49fe-9ebd-a67f0809f4a9\") " pod="opendatahub/test-curl-7285-140585246295072"
Jul 22 02:32:05.601950 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:05.601868    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6qds\" (UniqueName: \"kubernetes.io/projected/53feeba4-b8eb-49fe-9ebd-a67f0809f4a9-kube-api-access-p6qds\") pod \"test-curl-7285-140585246295072\" (UID: \"53feeba4-b8eb-49fe-9ebd-a67f0809f4a9\") " pod="opendatahub/test-curl-7285-140585246295072"
Jul 22 02:32:05.610289 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:05.610255    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6qds\" (UniqueName: \"kubernetes.io/projected/53feeba4-b8eb-49fe-9ebd-a67f0809f4a9-kube-api-access-p6qds\") pod \"test-curl-7285-140585246295072\" (UID: \"53feeba4-b8eb-49fe-9ebd-a67f0809f4a9\") " pod="opendatahub/test-curl-7285-140585246295072"
Jul 22 02:32:05.664588 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:05.664560    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d76ec13-1e16-4df9-8ff7-96411c0237d1" path="/var/lib/kubelet/pods/6d76ec13-1e16-4df9-8ff7-96411c0237d1/volumes"
Jul 22 02:32:05.691971 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:05.691942    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7285-140585246295072"
Jul 22 02:32:05.826777 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:05.826747    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7285-140585246295072"]
Jul 22 02:32:05.828557 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:32:05.828530    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53feeba4_b8eb_49fe_9ebd_a67f0809f4a9.slice/crio-001810667e965051ad445c6a625c01f3440d8fbbe289312540f0da821abc1d6c WatchSource:0}: Error finding container 001810667e965051ad445c6a625c01f3440d8fbbe289312540f0da821abc1d6c: Status 404 returned error can't find the container with id 001810667e965051ad445c6a625c01f3440d8fbbe289312540f0da821abc1d6c
Jul 22 02:32:06.021032 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:06.020998    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7285-140585246295072" event={"ID":"53feeba4-b8eb-49fe-9ebd-a67f0809f4a9","Type":"ContainerStarted","Data":"001810667e965051ad445c6a625c01f3440d8fbbe289312540f0da821abc1d6c"}
Jul 22 02:32:06.022067 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:06.022044    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7285-140585246288816"
Jul 22 02:32:06.022201 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:06.022094    2563 scope.go:117] "RemoveContainer" containerID="d09c6bb56ad170ee6182d424d746c7164b5e77dfba965ece5b65211b80b1e253"
Jul 22 02:32:07.028648 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:07.028610    2563 generic.go:356] "Generic (PLEG): container finished" podID="53feeba4-b8eb-49fe-9ebd-a67f0809f4a9" containerID="3362f5d418de79318f72a9521e10e13029a21d2dbf352594fd4c6f54e2bc6f77" exitCode=0
Jul 22 02:32:07.029058 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:07.028693    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7285-140585246295072" event={"ID":"53feeba4-b8eb-49fe-9ebd-a67f0809f4a9","Type":"ContainerDied","Data":"3362f5d418de79318f72a9521e10e13029a21d2dbf352594fd4c6f54e2bc6f77"}
Jul 22 02:32:08.171894 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:08.171869    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7285-140585246295072"
Jul 22 02:32:08.196591 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:08.196567    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7285-140585246295072_53feeba4-b8eb-49fe-9ebd-a67f0809f4a9/test-curl-7285-140585246295072/0.log"
Jul 22 02:32:08.226034 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:08.225996    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7285-140585246295072"]
Jul 22 02:32:08.226262 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:08.226238    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6qds\" (UniqueName: \"kubernetes.io/projected/53feeba4-b8eb-49fe-9ebd-a67f0809f4a9-kube-api-access-p6qds\") pod \"53feeba4-b8eb-49fe-9ebd-a67f0809f4a9\" (UID: \"53feeba4-b8eb-49fe-9ebd-a67f0809f4a9\") "
Jul 22 02:32:08.228735 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:08.228707    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53feeba4-b8eb-49fe-9ebd-a67f0809f4a9-kube-api-access-p6qds" (OuterVolumeSpecName: "kube-api-access-p6qds") pod "53feeba4-b8eb-49fe-9ebd-a67f0809f4a9" (UID: "53feeba4-b8eb-49fe-9ebd-a67f0809f4a9"). InnerVolumeSpecName "kube-api-access-p6qds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:32:08.230049 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:08.230029    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7285-140585246295072"]
Jul 22 02:32:08.327441 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:08.327349    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6qds\" (UniqueName: \"kubernetes.io/projected/53feeba4-b8eb-49fe-9ebd-a67f0809f4a9-kube-api-access-p6qds\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:32:08.360864 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:08.360834    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7285-140585246295072"]
Jul 22 02:32:08.361441 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:08.361372    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53feeba4-b8eb-49fe-9ebd-a67f0809f4a9" containerName="test-curl-7285-140585246295072"
Jul 22 02:32:08.361441 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:08.361438    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="53feeba4-b8eb-49fe-9ebd-a67f0809f4a9" containerName="test-curl-7285-140585246295072"
Jul 22 02:32:08.361611 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:08.361535    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="53feeba4-b8eb-49fe-9ebd-a67f0809f4a9" containerName="test-curl-7285-140585246295072"
Jul 22 02:32:08.364687 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:08.364671    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7285-140585246295072"
Jul 22 02:32:08.374405 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:08.369670    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7285-140585246295072"]
Jul 22 02:32:08.428566 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:08.428537    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5b84\" (UniqueName: \"kubernetes.io/projected/044ee63e-8825-4eb5-9b63-a2aab363182d-kube-api-access-w5b84\") pod \"test-curl-7285-140585246295072\" (UID: \"044ee63e-8825-4eb5-9b63-a2aab363182d\") " pod="opendatahub/test-curl-7285-140585246295072"
Jul 22 02:32:08.529109 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:08.529069    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5b84\" (UniqueName: \"kubernetes.io/projected/044ee63e-8825-4eb5-9b63-a2aab363182d-kube-api-access-w5b84\") pod \"test-curl-7285-140585246295072\" (UID: \"044ee63e-8825-4eb5-9b63-a2aab363182d\") " pod="opendatahub/test-curl-7285-140585246295072"
Jul 22 02:32:08.538964 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:08.538930    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5b84\" (UniqueName: \"kubernetes.io/projected/044ee63e-8825-4eb5-9b63-a2aab363182d-kube-api-access-w5b84\") pod \"test-curl-7285-140585246295072\" (UID: \"044ee63e-8825-4eb5-9b63-a2aab363182d\") " pod="opendatahub/test-curl-7285-140585246295072"
Jul 22 02:32:08.680293 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:08.680258    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7285-140585246295072"
Jul 22 02:32:08.816241 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:08.816215    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7285-140585246295072"]
Jul 22 02:32:08.817726 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:32:08.817698    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod044ee63e_8825_4eb5_9b63_a2aab363182d.slice/crio-ae5f9f819691205115bdde11e2cc65d20f22604bd3699409957152655fa665a9 WatchSource:0}: Error finding container ae5f9f819691205115bdde11e2cc65d20f22604bd3699409957152655fa665a9: Status 404 returned error can't find the container with id ae5f9f819691205115bdde11e2cc65d20f22604bd3699409957152655fa665a9
Jul 22 02:32:09.040827 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:09.040749    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="001810667e965051ad445c6a625c01f3440d8fbbe289312540f0da821abc1d6c"
Jul 22 02:32:09.040827 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:09.040744    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7285-140585246295072"
Jul 22 02:32:09.041853 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:09.041817    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7285-140585246295072" event={"ID":"044ee63e-8825-4eb5-9b63-a2aab363182d","Type":"ContainerStarted","Data":"ae5f9f819691205115bdde11e2cc65d20f22604bd3699409957152655fa665a9"}
Jul 22 02:32:09.056845 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:09.056812    2563 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7285-140585246295072" oldPodUID="53feeba4-b8eb-49fe-9ebd-a67f0809f4a9" podUID="044ee63e-8825-4eb5-9b63-a2aab363182d"
Jul 22 02:32:09.663327 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:09.663292    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53feeba4-b8eb-49fe-9ebd-a67f0809f4a9" path="/var/lib/kubelet/pods/53feeba4-b8eb-49fe-9ebd-a67f0809f4a9/volumes"
Jul 22 02:32:10.048746 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:10.048714    2563 generic.go:356] "Generic (PLEG): container finished" podID="044ee63e-8825-4eb5-9b63-a2aab363182d" containerID="e555ba3f884a0ad99e73e24f27bf0ea5fb952c0b61684744d4eb7bdf48c9765d" exitCode=0
Jul 22 02:32:10.048920 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:10.048795    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7285-140585246295072" event={"ID":"044ee63e-8825-4eb5-9b63-a2aab363182d","Type":"ContainerDied","Data":"e555ba3f884a0ad99e73e24f27bf0ea5fb952c0b61684744d4eb7bdf48c9765d"}
Jul 22 02:32:11.196479 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:11.196450    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7285-140585246295072"
Jul 22 02:32:11.218761 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:11.218738    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7285-140585246295072_044ee63e-8825-4eb5-9b63-a2aab363182d/test-curl-7285-140585246295072/0.log"
Jul 22 02:32:11.246115 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:11.246086    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7285-140585246295072"]
Jul 22 02:32:11.250985 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:11.250957    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7285-140585246295072"]
Jul 22 02:32:11.253077 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:11.253058    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5b84\" (UniqueName: \"kubernetes.io/projected/044ee63e-8825-4eb5-9b63-a2aab363182d-kube-api-access-w5b84\") pod \"044ee63e-8825-4eb5-9b63-a2aab363182d\" (UID: \"044ee63e-8825-4eb5-9b63-a2aab363182d\") "
Jul 22 02:32:11.255079 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:11.255057    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/044ee63e-8825-4eb5-9b63-a2aab363182d-kube-api-access-w5b84" (OuterVolumeSpecName: "kube-api-access-w5b84") pod "044ee63e-8825-4eb5-9b63-a2aab363182d" (UID: "044ee63e-8825-4eb5-9b63-a2aab363182d"). InnerVolumeSpecName "kube-api-access-w5b84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:32:11.353787 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:11.353703    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5b84\" (UniqueName: \"kubernetes.io/projected/044ee63e-8825-4eb5-9b63-a2aab363182d-kube-api-access-w5b84\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:32:11.663299 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:11.663220    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="044ee63e-8825-4eb5-9b63-a2aab363182d" path="/var/lib/kubelet/pods/044ee63e-8825-4eb5-9b63-a2aab363182d/volumes"
Jul 22 02:32:12.060499 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:12.060472    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7285-140585246295072"
Jul 22 02:32:12.060671 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:12.060474    2563 scope.go:117] "RemoveContainer" containerID="e555ba3f884a0ad99e73e24f27bf0ea5fb952c0b61684744d4eb7bdf48c9765d"
Jul 22 02:32:12.209193 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:12.209155    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-9d0056/rate-test-model-9d0056-kserve-7b8767cc55-hdvzg" podUID="a6b762dc-f195-4fd4-bc65-6534a8f85e49" containerName="main" containerID="cri-o://d71ce2ba831a449984b9fa2ca2f50d1844a915dc9669c9bcf1f854ce141647cd" gracePeriod=45
Jul 22 02:32:12.483406 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:12.483368    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9d0056/rate-test-model-9d0056-kserve-7b8767cc55-hdvzg"
Jul 22 02:32:12.564514 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:12.564482    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6b762dc-f195-4fd4-bc65-6534a8f85e49-home\") pod \"a6b762dc-f195-4fd4-bc65-6534a8f85e49\" (UID: \"a6b762dc-f195-4fd4-bc65-6534a8f85e49\") "
Jul 22 02:32:12.564677 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:12.564527    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6b762dc-f195-4fd4-bc65-6534a8f85e49-tmp-dir\") pod \"a6b762dc-f195-4fd4-bc65-6534a8f85e49\" (UID: \"a6b762dc-f195-4fd4-bc65-6534a8f85e49\") "
Jul 22 02:32:12.564677 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:12.564555    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfbqw\" (UniqueName: \"kubernetes.io/projected/a6b762dc-f195-4fd4-bc65-6534a8f85e49-kube-api-access-kfbqw\") pod \"a6b762dc-f195-4fd4-bc65-6534a8f85e49\" (UID: \"a6b762dc-f195-4fd4-bc65-6534a8f85e49\") "
Jul 22 02:32:12.564677 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:12.564591    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6b762dc-f195-4fd4-bc65-6534a8f85e49-tls-certs\") pod \"a6b762dc-f195-4fd4-bc65-6534a8f85e49\" (UID: \"a6b762dc-f195-4fd4-bc65-6534a8f85e49\") "
Jul 22 02:32:12.564677 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:12.564622    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6b762dc-f195-4fd4-bc65-6534a8f85e49-model-cache\") pod \"a6b762dc-f195-4fd4-bc65-6534a8f85e49\" (UID: \"a6b762dc-f195-4fd4-bc65-6534a8f85e49\") "
Jul 22 02:32:12.564834 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:12.564688    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6b762dc-f195-4fd4-bc65-6534a8f85e49-dshm\") pod \"a6b762dc-f195-4fd4-bc65-6534a8f85e49\" (UID: \"a6b762dc-f195-4fd4-bc65-6534a8f85e49\") "
Jul 22 02:32:12.564834 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:12.564760    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6b762dc-f195-4fd4-bc65-6534a8f85e49-home" (OuterVolumeSpecName: "home") pod "a6b762dc-f195-4fd4-bc65-6534a8f85e49" (UID: "a6b762dc-f195-4fd4-bc65-6534a8f85e49"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:32:12.564948 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:12.564926    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6b762dc-f195-4fd4-bc65-6534a8f85e49-model-cache" (OuterVolumeSpecName: "model-cache") pod "a6b762dc-f195-4fd4-bc65-6534a8f85e49" (UID: "a6b762dc-f195-4fd4-bc65-6534a8f85e49"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:32:12.565003 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:12.564934    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6b762dc-f195-4fd4-bc65-6534a8f85e49-home\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:32:12.565003 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:12.564974    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6b762dc-f195-4fd4-bc65-6534a8f85e49-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a6b762dc-f195-4fd4-bc65-6534a8f85e49" (UID: "a6b762dc-f195-4fd4-bc65-6534a8f85e49"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:32:12.566761 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:12.566735    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6b762dc-f195-4fd4-bc65-6534a8f85e49-dshm" (OuterVolumeSpecName: "dshm") pod "a6b762dc-f195-4fd4-bc65-6534a8f85e49" (UID: "a6b762dc-f195-4fd4-bc65-6534a8f85e49"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:32:12.566761 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:12.566743    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6b762dc-f195-4fd4-bc65-6534a8f85e49-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a6b762dc-f195-4fd4-bc65-6534a8f85e49" (UID: "a6b762dc-f195-4fd4-bc65-6534a8f85e49"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:32:12.566944 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:12.566800    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6b762dc-f195-4fd4-bc65-6534a8f85e49-kube-api-access-kfbqw" (OuterVolumeSpecName: "kube-api-access-kfbqw") pod "a6b762dc-f195-4fd4-bc65-6534a8f85e49" (UID: "a6b762dc-f195-4fd4-bc65-6534a8f85e49"). InnerVolumeSpecName "kube-api-access-kfbqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:32:12.665735 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:12.665645    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6b762dc-f195-4fd4-bc65-6534a8f85e49-tls-certs\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:32:12.665735 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:12.665674    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6b762dc-f195-4fd4-bc65-6534a8f85e49-model-cache\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:32:12.665735 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:12.665688    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6b762dc-f195-4fd4-bc65-6534a8f85e49-dshm\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:32:12.665735 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:12.665718    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6b762dc-f195-4fd4-bc65-6534a8f85e49-tmp-dir\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:32:12.665735 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:12.665731    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfbqw\" (UniqueName: \"kubernetes.io/projected/a6b762dc-f195-4fd4-bc65-6534a8f85e49-kube-api-access-kfbqw\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:32:13.068246 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:13.068211    2563 generic.go:356] "Generic (PLEG): container finished" podID="a6b762dc-f195-4fd4-bc65-6534a8f85e49" containerID="d71ce2ba831a449984b9fa2ca2f50d1844a915dc9669c9bcf1f854ce141647cd" exitCode=0
Jul 22 02:32:13.068452 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:13.068252    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9d0056/rate-test-model-9d0056-kserve-7b8767cc55-hdvzg"
Jul 22 02:32:13.068452 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:13.068276    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9d0056/rate-test-model-9d0056-kserve-7b8767cc55-hdvzg" event={"ID":"a6b762dc-f195-4fd4-bc65-6534a8f85e49","Type":"ContainerDied","Data":"d71ce2ba831a449984b9fa2ca2f50d1844a915dc9669c9bcf1f854ce141647cd"}
Jul 22 02:32:13.068452 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:13.068311    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9d0056/rate-test-model-9d0056-kserve-7b8767cc55-hdvzg" event={"ID":"a6b762dc-f195-4fd4-bc65-6534a8f85e49","Type":"ContainerDied","Data":"7c558b21c057c7112705c187a003880b2670cd0142be009efd7ddf5fc1289271"}
Jul 22 02:32:13.068452 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:13.068331    2563 scope.go:117] "RemoveContainer" containerID="d71ce2ba831a449984b9fa2ca2f50d1844a915dc9669c9bcf1f854ce141647cd"
Jul 22 02:32:13.077628 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:13.077609    2563 scope.go:117] "RemoveContainer" containerID="d71ce2ba831a449984b9fa2ca2f50d1844a915dc9669c9bcf1f854ce141647cd"
Jul 22 02:32:13.077865 ip-10-0-135-230 kubenswrapper[2563]: E0722 02:32:13.077847    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d71ce2ba831a449984b9fa2ca2f50d1844a915dc9669c9bcf1f854ce141647cd\": container with ID starting with d71ce2ba831a449984b9fa2ca2f50d1844a915dc9669c9bcf1f854ce141647cd not found: ID does not exist" containerID="d71ce2ba831a449984b9fa2ca2f50d1844a915dc9669c9bcf1f854ce141647cd"
Jul 22 02:32:13.077928 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:13.077872    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d71ce2ba831a449984b9fa2ca2f50d1844a915dc9669c9bcf1f854ce141647cd"} err="failed to get container status \"d71ce2ba831a449984b9fa2ca2f50d1844a915dc9669c9bcf1f854ce141647cd\": rpc error: code = NotFound desc = could not find container \"d71ce2ba831a449984b9fa2ca2f50d1844a915dc9669c9bcf1f854ce141647cd\": container with ID starting with d71ce2ba831a449984b9fa2ca2f50d1844a915dc9669c9bcf1f854ce141647cd not found: ID does not exist"
Jul 22 02:32:13.090455 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:13.090433    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9d0056/rate-test-model-9d0056-kserve-7b8767cc55-hdvzg"]
Jul 22 02:32:13.095158 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:13.095137    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-9d0056/rate-test-model-9d0056-kserve-7b8767cc55-hdvzg"]
Jul 22 02:32:13.156430 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:13.156392    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-a73722/rate-test-model-a73722-kserve-5cc99c85bc-pdmvf" podUID="545a838e-a0e7-49fe-aab7-2a9146f33073" containerName="main" containerID="cri-o://08cd5e8289f59ed9781aa229ef98b0ca503c3a9aa6896a0e9d2276ca2c105210" gracePeriod=45
Jul 22 02:32:13.404949 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:13.404919    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a73722/rate-test-model-a73722-kserve-5cc99c85bc-pdmvf"
Jul 22 02:32:13.471533 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:13.471500    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/545a838e-a0e7-49fe-aab7-2a9146f33073-model-cache\") pod \"545a838e-a0e7-49fe-aab7-2a9146f33073\" (UID: \"545a838e-a0e7-49fe-aab7-2a9146f33073\") "
Jul 22 02:32:13.471533 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:13.471538    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/545a838e-a0e7-49fe-aab7-2a9146f33073-tmp-dir\") pod \"545a838e-a0e7-49fe-aab7-2a9146f33073\" (UID: \"545a838e-a0e7-49fe-aab7-2a9146f33073\") "
Jul 22 02:32:13.471773 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:13.471567    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpjpn\" (UniqueName: \"kubernetes.io/projected/545a838e-a0e7-49fe-aab7-2a9146f33073-kube-api-access-tpjpn\") pod \"545a838e-a0e7-49fe-aab7-2a9146f33073\" (UID: \"545a838e-a0e7-49fe-aab7-2a9146f33073\") "
Jul 22 02:32:13.471773 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:13.471650    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/545a838e-a0e7-49fe-aab7-2a9146f33073-home\") pod \"545a838e-a0e7-49fe-aab7-2a9146f33073\" (UID: \"545a838e-a0e7-49fe-aab7-2a9146f33073\") "
Jul 22 02:32:13.471773 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:13.471675    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/545a838e-a0e7-49fe-aab7-2a9146f33073-tls-certs\") pod \"545a838e-a0e7-49fe-aab7-2a9146f33073\" (UID: \"545a838e-a0e7-49fe-aab7-2a9146f33073\") "
Jul 22 02:32:13.471773 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:13.471734    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/545a838e-a0e7-49fe-aab7-2a9146f33073-dshm\") pod \"545a838e-a0e7-49fe-aab7-2a9146f33073\" (UID: \"545a838e-a0e7-49fe-aab7-2a9146f33073\") "
Jul 22 02:32:13.472024 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:13.471818    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/545a838e-a0e7-49fe-aab7-2a9146f33073-model-cache" (OuterVolumeSpecName: "model-cache") pod "545a838e-a0e7-49fe-aab7-2a9146f33073" (UID: "545a838e-a0e7-49fe-aab7-2a9146f33073"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:32:13.472024 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:13.471855    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/545a838e-a0e7-49fe-aab7-2a9146f33073-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "545a838e-a0e7-49fe-aab7-2a9146f33073" (UID: "545a838e-a0e7-49fe-aab7-2a9146f33073"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:32:13.472024 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:13.471892    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/545a838e-a0e7-49fe-aab7-2a9146f33073-home" (OuterVolumeSpecName: "home") pod "545a838e-a0e7-49fe-aab7-2a9146f33073" (UID: "545a838e-a0e7-49fe-aab7-2a9146f33073"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:32:13.472150 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:13.472027    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/545a838e-a0e7-49fe-aab7-2a9146f33073-model-cache\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:32:13.472150 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:13.472046    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/545a838e-a0e7-49fe-aab7-2a9146f33073-tmp-dir\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:32:13.472150 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:13.472059    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/545a838e-a0e7-49fe-aab7-2a9146f33073-home\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:32:13.473740 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:13.473711    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/545a838e-a0e7-49fe-aab7-2a9146f33073-dshm" (OuterVolumeSpecName: "dshm") pod "545a838e-a0e7-49fe-aab7-2a9146f33073" (UID: "545a838e-a0e7-49fe-aab7-2a9146f33073"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:32:13.473846 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:13.473742    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/545a838e-a0e7-49fe-aab7-2a9146f33073-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "545a838e-a0e7-49fe-aab7-2a9146f33073" (UID: "545a838e-a0e7-49fe-aab7-2a9146f33073"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:32:13.473846 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:13.473783    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/545a838e-a0e7-49fe-aab7-2a9146f33073-kube-api-access-tpjpn" (OuterVolumeSpecName: "kube-api-access-tpjpn") pod "545a838e-a0e7-49fe-aab7-2a9146f33073" (UID: "545a838e-a0e7-49fe-aab7-2a9146f33073"). InnerVolumeSpecName "kube-api-access-tpjpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:32:13.572796 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:13.572702    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/545a838e-a0e7-49fe-aab7-2a9146f33073-dshm\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:32:13.572796 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:13.572734    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpjpn\" (UniqueName: \"kubernetes.io/projected/545a838e-a0e7-49fe-aab7-2a9146f33073-kube-api-access-tpjpn\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:32:13.572796 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:13.572745    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/545a838e-a0e7-49fe-aab7-2a9146f33073-tls-certs\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:32:13.663435 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:13.663404    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6b762dc-f195-4fd4-bc65-6534a8f85e49" path="/var/lib/kubelet/pods/a6b762dc-f195-4fd4-bc65-6534a8f85e49/volumes"
Jul 22 02:32:14.075106 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:14.075073    2563 generic.go:356] "Generic (PLEG): container finished" podID="545a838e-a0e7-49fe-aab7-2a9146f33073" containerID="08cd5e8289f59ed9781aa229ef98b0ca503c3a9aa6896a0e9d2276ca2c105210" exitCode=0
Jul 22 02:32:14.075284 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:14.075116    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a73722/rate-test-model-a73722-kserve-5cc99c85bc-pdmvf"
Jul 22 02:32:14.075284 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:14.075161    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a73722/rate-test-model-a73722-kserve-5cc99c85bc-pdmvf" event={"ID":"545a838e-a0e7-49fe-aab7-2a9146f33073","Type":"ContainerDied","Data":"08cd5e8289f59ed9781aa229ef98b0ca503c3a9aa6896a0e9d2276ca2c105210"}
Jul 22 02:32:14.075284 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:14.075209    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a73722/rate-test-model-a73722-kserve-5cc99c85bc-pdmvf" event={"ID":"545a838e-a0e7-49fe-aab7-2a9146f33073","Type":"ContainerDied","Data":"7bc99222b5020fc8d3f9556c61edb0be19ec06a5ba5e21494375391b1cce026f"}
Jul 22 02:32:14.075284 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:14.075233    2563 scope.go:117] "RemoveContainer" containerID="08cd5e8289f59ed9781aa229ef98b0ca503c3a9aa6896a0e9d2276ca2c105210"
Jul 22 02:32:14.083884 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:14.083869    2563 scope.go:117] "RemoveContainer" containerID="08cd5e8289f59ed9781aa229ef98b0ca503c3a9aa6896a0e9d2276ca2c105210"
Jul 22 02:32:14.087044 ip-10-0-135-230 kubenswrapper[2563]: E0722 02:32:14.084327    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08cd5e8289f59ed9781aa229ef98b0ca503c3a9aa6896a0e9d2276ca2c105210\": container with ID starting with 08cd5e8289f59ed9781aa229ef98b0ca503c3a9aa6896a0e9d2276ca2c105210 not found: ID does not exist" containerID="08cd5e8289f59ed9781aa229ef98b0ca503c3a9aa6896a0e9d2276ca2c105210"
Jul 22 02:32:14.087044 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:14.084361    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08cd5e8289f59ed9781aa229ef98b0ca503c3a9aa6896a0e9d2276ca2c105210"} err="failed to get container status \"08cd5e8289f59ed9781aa229ef98b0ca503c3a9aa6896a0e9d2276ca2c105210\": rpc error: code = NotFound desc = could not find container \"08cd5e8289f59ed9781aa229ef98b0ca503c3a9aa6896a0e9d2276ca2c105210\": container with ID starting with 08cd5e8289f59ed9781aa229ef98b0ca503c3a9aa6896a0e9d2276ca2c105210 not found: ID does not exist"
Jul 22 02:32:14.093673 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:14.093652    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a73722/rate-test-model-a73722-kserve-5cc99c85bc-pdmvf"]
Jul 22 02:32:14.096863 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:14.096843    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-a73722/rate-test-model-a73722-kserve-5cc99c85bc-pdmvf"]
Jul 22 02:32:14.962197 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:14.962160    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7285-140585246296080"]
Jul 22 02:32:14.962582 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:14.962531    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="044ee63e-8825-4eb5-9b63-a2aab363182d" containerName="test-curl-7285-140585246295072"
Jul 22 02:32:14.962582 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:14.962542    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="044ee63e-8825-4eb5-9b63-a2aab363182d" containerName="test-curl-7285-140585246295072"
Jul 22 02:32:14.962582 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:14.962565    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="545a838e-a0e7-49fe-aab7-2a9146f33073" containerName="main"
Jul 22 02:32:14.962582 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:14.962570    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="545a838e-a0e7-49fe-aab7-2a9146f33073" containerName="main"
Jul 22 02:32:14.962582 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:14.962580    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6b762dc-f195-4fd4-bc65-6534a8f85e49" containerName="main"
Jul 22 02:32:14.962582 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:14.962585    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6b762dc-f195-4fd4-bc65-6534a8f85e49" containerName="main"
Jul 22 02:32:14.962767 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:14.962635    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="044ee63e-8825-4eb5-9b63-a2aab363182d" containerName="test-curl-7285-140585246295072"
Jul 22 02:32:14.962767 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:14.962644    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6b762dc-f195-4fd4-bc65-6534a8f85e49" containerName="main"
Jul 22 02:32:14.962767 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:14.962650    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="545a838e-a0e7-49fe-aab7-2a9146f33073" containerName="main"
Jul 22 02:32:14.966875 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:14.966860    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7285-140585246296080"
Jul 22 02:32:14.969493 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:14.969472    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wfg6c\""
Jul 22 02:32:14.972129 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:14.972101    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7285-140585246296080"]
Jul 22 02:32:15.082714 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:15.082686    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gb8p\" (UniqueName: \"kubernetes.io/projected/e3e79109-cada-4dc3-a50c-c3c7c3e9b24e-kube-api-access-7gb8p\") pod \"test-curl-7285-140585246296080\" (UID: \"e3e79109-cada-4dc3-a50c-c3c7c3e9b24e\") " pod="opendatahub/test-curl-7285-140585246296080"
Jul 22 02:32:15.183703 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:15.183644    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gb8p\" (UniqueName: \"kubernetes.io/projected/e3e79109-cada-4dc3-a50c-c3c7c3e9b24e-kube-api-access-7gb8p\") pod \"test-curl-7285-140585246296080\" (UID: \"e3e79109-cada-4dc3-a50c-c3c7c3e9b24e\") " pod="opendatahub/test-curl-7285-140585246296080"
Jul 22 02:32:15.193353 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:15.193327    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gb8p\" (UniqueName: \"kubernetes.io/projected/e3e79109-cada-4dc3-a50c-c3c7c3e9b24e-kube-api-access-7gb8p\") pod \"test-curl-7285-140585246296080\" (UID: \"e3e79109-cada-4dc3-a50c-c3c7c3e9b24e\") " pod="opendatahub/test-curl-7285-140585246296080"
Jul 22 02:32:15.277091 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:15.277006    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7285-140585246296080"
Jul 22 02:32:15.406707 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:15.406683    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7285-140585246296080"]
Jul 22 02:32:15.408119 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:32:15.408097    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3e79109_cada_4dc3_a50c_c3c7c3e9b24e.slice/crio-15351bae231b17f9bf8a0f52b94c4668c83bdda41f70073015b991ac39e4aa33 WatchSource:0}: Error finding container 15351bae231b17f9bf8a0f52b94c4668c83bdda41f70073015b991ac39e4aa33: Status 404 returned error can't find the container with id 15351bae231b17f9bf8a0f52b94c4668c83bdda41f70073015b991ac39e4aa33
Jul 22 02:32:15.663337 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:15.663254    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="545a838e-a0e7-49fe-aab7-2a9146f33073" path="/var/lib/kubelet/pods/545a838e-a0e7-49fe-aab7-2a9146f33073/volumes"
Jul 22 02:32:16.089784 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:16.089750    2563 generic.go:356] "Generic (PLEG): container finished" podID="e3e79109-cada-4dc3-a50c-c3c7c3e9b24e" containerID="13da1c45cf0e1151b9416e573f1505376378071259a752c54d4340ac596e0602" exitCode=0
Jul 22 02:32:16.090156 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:16.089826    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7285-140585246296080" event={"ID":"e3e79109-cada-4dc3-a50c-c3c7c3e9b24e","Type":"ContainerDied","Data":"13da1c45cf0e1151b9416e573f1505376378071259a752c54d4340ac596e0602"}
Jul 22 02:32:16.090156 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:16.089864    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7285-140585246296080" event={"ID":"e3e79109-cada-4dc3-a50c-c3c7c3e9b24e","Type":"ContainerStarted","Data":"15351bae231b17f9bf8a0f52b94c4668c83bdda41f70073015b991ac39e4aa33"}
Jul 22 02:32:17.224168 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:17.224147    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7285-140585246296080"
Jul 22 02:32:17.247321 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:17.247302    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7285-140585246296080_e3e79109-cada-4dc3-a50c-c3c7c3e9b24e/test-curl-7285-140585246296080/0.log"
Jul 22 02:32:17.278873 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:17.278847    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7285-140585246296080"]
Jul 22 02:32:17.282016 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:17.281993    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7285-140585246296080"]
Jul 22 02:32:17.301204 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:17.301182    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gb8p\" (UniqueName: \"kubernetes.io/projected/e3e79109-cada-4dc3-a50c-c3c7c3e9b24e-kube-api-access-7gb8p\") pod \"e3e79109-cada-4dc3-a50c-c3c7c3e9b24e\" (UID: \"e3e79109-cada-4dc3-a50c-c3c7c3e9b24e\") "
Jul 22 02:32:17.303161 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:17.303141    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3e79109-cada-4dc3-a50c-c3c7c3e9b24e-kube-api-access-7gb8p" (OuterVolumeSpecName: "kube-api-access-7gb8p") pod "e3e79109-cada-4dc3-a50c-c3c7c3e9b24e" (UID: "e3e79109-cada-4dc3-a50c-c3c7c3e9b24e"). InnerVolumeSpecName "kube-api-access-7gb8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:32:17.402487 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:17.402425    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gb8p\" (UniqueName: \"kubernetes.io/projected/e3e79109-cada-4dc3-a50c-c3c7c3e9b24e-kube-api-access-7gb8p\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:32:17.479396 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:17.479359    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9d0056-5wqtx"]
Jul 22 02:32:17.479740 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:17.479728    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3e79109-cada-4dc3-a50c-c3c7c3e9b24e" containerName="test-curl-7285-140585246296080"
Jul 22 02:32:17.479790 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:17.479743    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3e79109-cada-4dc3-a50c-c3c7c3e9b24e" containerName="test-curl-7285-140585246296080"
Jul 22 02:32:17.479826 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:17.479821    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3e79109-cada-4dc3-a50c-c3c7c3e9b24e" containerName="test-curl-7285-140585246296080"
Jul 22 02:32:17.482785 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:17.482771    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9d0056-5wqtx"
Jul 22 02:32:17.485809 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:17.485791    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wrtw7\""
Jul 22 02:32:17.491683 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:17.491662    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9d0056-5wqtx"]
Jul 22 02:32:17.604302 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:17.604266    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4f25650-0d01-4ae4-9796-eebb28cfa7f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-9d0056-5wqtx\" (UID: \"c4f25650-0d01-4ae4-9796-eebb28cfa7f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9d0056-5wqtx"
Jul 22 02:32:17.663169 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:17.663112    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3e79109-cada-4dc3-a50c-c3c7c3e9b24e" path="/var/lib/kubelet/pods/e3e79109-cada-4dc3-a50c-c3c7c3e9b24e/volumes"
Jul 22 02:32:17.704754 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:17.704731    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4f25650-0d01-4ae4-9796-eebb28cfa7f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-9d0056-5wqtx\" (UID: \"c4f25650-0d01-4ae4-9796-eebb28cfa7f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9d0056-5wqtx"
Jul 22 02:32:17.705309 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:17.705291    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4f25650-0d01-4ae4-9796-eebb28cfa7f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-9d0056-5wqtx\" (UID: \"c4f25650-0d01-4ae4-9796-eebb28cfa7f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9d0056-5wqtx"
Jul 22 02:32:17.794254 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:17.794221    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9d0056-5wqtx"
Jul 22 02:32:17.922806 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:17.922782    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9d0056-5wqtx"]
Jul 22 02:32:17.924725 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:32:17.924701    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4f25650_0d01_4ae4_9796_eebb28cfa7f1.slice/crio-2c6b04dd70b509ad3c41414ecddb2208d0deec3d7eee7d8ce8267e7ad6917185 WatchSource:0}: Error finding container 2c6b04dd70b509ad3c41414ecddb2208d0deec3d7eee7d8ce8267e7ad6917185: Status 404 returned error can't find the container with id 2c6b04dd70b509ad3c41414ecddb2208d0deec3d7eee7d8ce8267e7ad6917185
Jul 22 02:32:18.101588 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:18.101553    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9d0056-5wqtx" event={"ID":"c4f25650-0d01-4ae4-9796-eebb28cfa7f1","Type":"ContainerStarted","Data":"2c6b04dd70b509ad3c41414ecddb2208d0deec3d7eee7d8ce8267e7ad6917185"}
Jul 22 02:32:18.102742 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:18.102724    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7285-140585246296080"
Jul 22 02:32:18.102829 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:18.102772    2563 scope.go:117] "RemoveContainer" containerID="13da1c45cf0e1151b9416e573f1505376378071259a752c54d4340ac596e0602"
Jul 22 02:32:19.109952 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:19.109920    2563 generic.go:356] "Generic (PLEG): container finished" podID="c4f25650-0d01-4ae4-9796-eebb28cfa7f1" containerID="a3bd189fe5df82c5f70a2487e96e8e3712302a96b66ed3ef02605708c6fe7e4e" exitCode=0
Jul 22 02:32:19.110347 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:19.110004    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9d0056-5wqtx" event={"ID":"c4f25650-0d01-4ae4-9796-eebb28cfa7f1","Type":"ContainerDied","Data":"a3bd189fe5df82c5f70a2487e96e8e3712302a96b66ed3ef02605708c6fe7e4e"}
Jul 22 02:32:20.239997 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:20.239975    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9d0056-5wqtx"
Jul 22 02:32:20.325596 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:20.325573    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4f25650-0d01-4ae4-9796-eebb28cfa7f1-maas-api-service-ca\") pod \"c4f25650-0d01-4ae4-9796-eebb28cfa7f1\" (UID: \"c4f25650-0d01-4ae4-9796-eebb28cfa7f1\") "
Jul 22 02:32:20.325918 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:20.325895    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4f25650-0d01-4ae4-9796-eebb28cfa7f1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c4f25650-0d01-4ae4-9796-eebb28cfa7f1" (UID: "c4f25650-0d01-4ae4-9796-eebb28cfa7f1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:32:20.426334 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:20.426313    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4f25650-0d01-4ae4-9796-eebb28cfa7f1-maas-api-service-ca\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:32:21.122864 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:21.122840    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9d0056-5wqtx"
Jul 22 02:32:21.123009 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:21.122870    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9d0056-5wqtx" event={"ID":"c4f25650-0d01-4ae4-9796-eebb28cfa7f1","Type":"ContainerDied","Data":"2c6b04dd70b509ad3c41414ecddb2208d0deec3d7eee7d8ce8267e7ad6917185"}
Jul 22 02:32:21.123009 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:21.122918    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c6b04dd70b509ad3c41414ecddb2208d0deec3d7eee7d8ce8267e7ad6917185"
Jul 22 02:32:27.993181 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:27.993146    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-9d0056-6f9c55556b-8l464"]
Jul 22 02:32:27.993941 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:27.993907    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9d0056-6f9c55556b-8l464" podUID="816d1cf3-7171-455e-9e87-405792553a60" containerName="maas-api" containerID="cri-o://820ca8244401e0e2db58d56d1c0e099e83edba187a396749b323fd56c81e0860" gracePeriod=30
Jul 22 02:32:28.034840 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:28.034815    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-9d0056-29744790-twvmv"]
Jul 22 02:32:28.037174 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:28.037153    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-9d0056-29744790-twvmv"]
Jul 22 02:32:28.131582 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:28.131542    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9d0056-6f9c55556b-8l464" podUID="816d1cf3-7171-455e-9e87-405792553a60" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.74:8443/health\": dial tcp 10.134.0.74:8443: connect: connection refused"
Jul 22 02:32:28.162921 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:28.162889    2563 generic.go:356] "Generic (PLEG): container finished" podID="816d1cf3-7171-455e-9e87-405792553a60" containerID="820ca8244401e0e2db58d56d1c0e099e83edba187a396749b323fd56c81e0860" exitCode=0
Jul 22 02:32:28.163058 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:28.162970    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9d0056-6f9c55556b-8l464" event={"ID":"816d1cf3-7171-455e-9e87-405792553a60","Type":"ContainerDied","Data":"820ca8244401e0e2db58d56d1c0e099e83edba187a396749b323fd56c81e0860"}
Jul 22 02:32:28.181747 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:28.181603    2563 scope.go:117] "RemoveContainer" containerID="8cd040bae1aec9569743ca52107358094dd08b04890b51f4eb417da9358f7e95"
Jul 22 02:32:28.245920 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:28.245868    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9d0056-6f9c55556b-8l464"
Jul 22 02:32:28.388326 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:28.388300    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rz6b\" (UniqueName: \"kubernetes.io/projected/816d1cf3-7171-455e-9e87-405792553a60-kube-api-access-2rz6b\") pod \"816d1cf3-7171-455e-9e87-405792553a60\" (UID: \"816d1cf3-7171-455e-9e87-405792553a60\") "
Jul 22 02:32:28.388650 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:28.388356    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/816d1cf3-7171-455e-9e87-405792553a60-maas-api-tls\") pod \"816d1cf3-7171-455e-9e87-405792553a60\" (UID: \"816d1cf3-7171-455e-9e87-405792553a60\") "
Jul 22 02:32:28.388650 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:28.388400    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/816d1cf3-7171-455e-9e87-405792553a60-openshift-service-ca\") pod \"816d1cf3-7171-455e-9e87-405792553a60\" (UID: \"816d1cf3-7171-455e-9e87-405792553a60\") "
Jul 22 02:32:28.388780 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:28.388753    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/816d1cf3-7171-455e-9e87-405792553a60-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "816d1cf3-7171-455e-9e87-405792553a60" (UID: "816d1cf3-7171-455e-9e87-405792553a60"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:32:28.390287 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:28.390267    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/816d1cf3-7171-455e-9e87-405792553a60-kube-api-access-2rz6b" (OuterVolumeSpecName: "kube-api-access-2rz6b") pod "816d1cf3-7171-455e-9e87-405792553a60" (UID: "816d1cf3-7171-455e-9e87-405792553a60"). InnerVolumeSpecName "kube-api-access-2rz6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:32:28.390406 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:28.390364    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/816d1cf3-7171-455e-9e87-405792553a60-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "816d1cf3-7171-455e-9e87-405792553a60" (UID: "816d1cf3-7171-455e-9e87-405792553a60"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:32:28.488921 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:28.488894    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rz6b\" (UniqueName: \"kubernetes.io/projected/816d1cf3-7171-455e-9e87-405792553a60-kube-api-access-2rz6b\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:32:28.488921 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:28.488917    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/816d1cf3-7171-455e-9e87-405792553a60-maas-api-tls\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:32:28.488921 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:28.488925    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/816d1cf3-7171-455e-9e87-405792553a60-openshift-service-ca\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:32:29.169507 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:29.169472    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9d0056-6f9c55556b-8l464" event={"ID":"816d1cf3-7171-455e-9e87-405792553a60","Type":"ContainerDied","Data":"e79f84a274ec2d9f4000f5aa78bf57fb1fa66ba0d4057146b183de60c14fc9d1"}
Jul 22 02:32:29.169507 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:29.169504    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9d0056-6f9c55556b-8l464"
Jul 22 02:32:29.169966 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:29.169534    2563 scope.go:117] "RemoveContainer" containerID="820ca8244401e0e2db58d56d1c0e099e83edba187a396749b323fd56c81e0860"
Jul 22 02:32:29.196171 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:29.196139    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-9d0056-6f9c55556b-8l464"]
Jul 22 02:32:29.203642 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:29.203610    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-9d0056-6f9c55556b-8l464"]
Jul 22 02:32:29.664169 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:29.664137    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="816d1cf3-7171-455e-9e87-405792553a60" path="/var/lib/kubelet/pods/816d1cf3-7171-455e-9e87-405792553a60/volumes"
Jul 22 02:32:29.664446 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:29.664432    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb5b1928-d7b8-4c7c-9567-2235fdff4837" path="/var/lib/kubelet/pods/bb5b1928-d7b8-4c7c-9567-2235fdff4837/volumes"
Jul 22 02:32:32.620869 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:32.620834    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9d0056-5wqtx"]
Jul 22 02:32:32.627095 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:32.627065    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9d0056-5wqtx"]
Jul 22 02:32:33.662825 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:33.662794    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4f25650-0d01-4ae4-9796-eebb28cfa7f1" path="/var/lib/kubelet/pods/c4f25650-0d01-4ae4-9796-eebb28cfa7f1/volumes"
Jul 22 02:32:40.862313 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:40.862275    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a73722-8mmvv"]
Jul 22 02:32:40.862801 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:40.862780    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4f25650-0d01-4ae4-9796-eebb28cfa7f1" containerName="revoke-keys"
Jul 22 02:32:40.862872 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:40.862805    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4f25650-0d01-4ae4-9796-eebb28cfa7f1" containerName="revoke-keys"
Jul 22 02:32:40.862872 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:40.862836    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="816d1cf3-7171-455e-9e87-405792553a60" containerName="maas-api"
Jul 22 02:32:40.862872 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:40.862845    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="816d1cf3-7171-455e-9e87-405792553a60" containerName="maas-api"
Jul 22 02:32:40.863029 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:40.862937    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4f25650-0d01-4ae4-9796-eebb28cfa7f1" containerName="revoke-keys"
Jul 22 02:32:40.863029 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:40.862950    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="816d1cf3-7171-455e-9e87-405792553a60" containerName="maas-api"
Jul 22 02:32:40.866097 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:40.866078    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a73722-8mmvv"
Jul 22 02:32:40.868881 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:40.868863    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wrtw7\""
Jul 22 02:32:40.878773 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:40.878750    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a73722-8mmvv"]
Jul 22 02:32:40.994666 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:40.994635    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f821c00-2af0-47a0-aedc-dc1ceaf0d948-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-a73722-8mmvv\" (UID: \"2f821c00-2af0-47a0-aedc-dc1ceaf0d948\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a73722-8mmvv"
Jul 22 02:32:41.095544 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:41.095505    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f821c00-2af0-47a0-aedc-dc1ceaf0d948-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-a73722-8mmvv\" (UID: \"2f821c00-2af0-47a0-aedc-dc1ceaf0d948\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a73722-8mmvv"
Jul 22 02:32:41.096142 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:41.096119    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f821c00-2af0-47a0-aedc-dc1ceaf0d948-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-a73722-8mmvv\" (UID: \"2f821c00-2af0-47a0-aedc-dc1ceaf0d948\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a73722-8mmvv"
Jul 22 02:32:41.175958 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:41.175929    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a73722-8mmvv"
Jul 22 02:32:41.305967 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:41.305943    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a73722-8mmvv"]
Jul 22 02:32:41.307681 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:32:41.307654    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f821c00_2af0_47a0_aedc_dc1ceaf0d948.slice/crio-0f6e5cf5490dc3677a55e898bb9f508cbbcb38e5fd5ff514b6fe46cb935e870a WatchSource:0}: Error finding container 0f6e5cf5490dc3677a55e898bb9f508cbbcb38e5fd5ff514b6fe46cb935e870a: Status 404 returned error can't find the container with id 0f6e5cf5490dc3677a55e898bb9f508cbbcb38e5fd5ff514b6fe46cb935e870a
Jul 22 02:32:42.240391 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:42.240343    2563 generic.go:356] "Generic (PLEG): container finished" podID="2f821c00-2af0-47a0-aedc-dc1ceaf0d948" containerID="20af2898942199d5a0f811409825b47eb4cb5014924651270ec7632635b2d3e7" exitCode=0
Jul 22 02:32:42.240786 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:42.240415    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a73722-8mmvv" event={"ID":"2f821c00-2af0-47a0-aedc-dc1ceaf0d948","Type":"ContainerDied","Data":"20af2898942199d5a0f811409825b47eb4cb5014924651270ec7632635b2d3e7"}
Jul 22 02:32:42.240786 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:42.240489    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a73722-8mmvv" event={"ID":"2f821c00-2af0-47a0-aedc-dc1ceaf0d948","Type":"ContainerStarted","Data":"0f6e5cf5490dc3677a55e898bb9f508cbbcb38e5fd5ff514b6fe46cb935e870a"}
Jul 22 02:32:43.376656 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:43.376628    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a73722-8mmvv"
Jul 22 02:32:43.518283 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:43.518206    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f821c00-2af0-47a0-aedc-dc1ceaf0d948-maas-api-service-ca\") pod \"2f821c00-2af0-47a0-aedc-dc1ceaf0d948\" (UID: \"2f821c00-2af0-47a0-aedc-dc1ceaf0d948\") "
Jul 22 02:32:43.518566 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:43.518544    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f821c00-2af0-47a0-aedc-dc1ceaf0d948-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2f821c00-2af0-47a0-aedc-dc1ceaf0d948" (UID: "2f821c00-2af0-47a0-aedc-dc1ceaf0d948"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:32:43.619240 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:43.619207    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f821c00-2af0-47a0-aedc-dc1ceaf0d948-maas-api-service-ca\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:32:44.252964 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:44.252931    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a73722-8mmvv" event={"ID":"2f821c00-2af0-47a0-aedc-dc1ceaf0d948","Type":"ContainerDied","Data":"0f6e5cf5490dc3677a55e898bb9f508cbbcb38e5fd5ff514b6fe46cb935e870a"}
Jul 22 02:32:44.252964 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:44.252951    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a73722-8mmvv"
Jul 22 02:32:44.253169 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:44.252992    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f6e5cf5490dc3677a55e898bb9f508cbbcb38e5fd5ff514b6fe46cb935e870a"
Jul 22 02:32:51.462415 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:51.462366    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a73722-546b4bf4bf-jmtkb"]
Jul 22 02:32:51.462891 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:51.462648    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a73722-546b4bf4bf-jmtkb" podUID="cbc1c1da-fc82-44e3-8362-dd0872803a46" containerName="maas-api" containerID="cri-o://3d5232c4b51b1f135585965fb97086c891bcc0ded9f779bdde7540a8351672de" gracePeriod=30
Jul 22 02:32:51.505452 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:51.505423    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a73722-29744790-8zmjt"]
Jul 22 02:32:51.507018 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:51.506997    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a73722-29744790-8zmjt"]
Jul 22 02:32:51.664160 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:51.664131    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce656997-5a4c-4ec5-8224-204ab040e681" path="/var/lib/kubelet/pods/ce656997-5a4c-4ec5-8224-204ab040e681/volumes"
Jul 22 02:32:51.712532 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:51.712481    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a73722-546b4bf4bf-jmtkb"
Jul 22 02:32:51.775035 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:51.775006    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc1c1da-fc82-44e3-8362-dd0872803a46-openshift-service-ca\") pod \"cbc1c1da-fc82-44e3-8362-dd0872803a46\" (UID: \"cbc1c1da-fc82-44e3-8362-dd0872803a46\") "
Jul 22 02:32:51.775155 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:51.775060    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbc1c1da-fc82-44e3-8362-dd0872803a46-maas-api-tls\") pod \"cbc1c1da-fc82-44e3-8362-dd0872803a46\" (UID: \"cbc1c1da-fc82-44e3-8362-dd0872803a46\") "
Jul 22 02:32:51.775155 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:51.775082    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfn97\" (UniqueName: \"kubernetes.io/projected/cbc1c1da-fc82-44e3-8362-dd0872803a46-kube-api-access-hfn97\") pod \"cbc1c1da-fc82-44e3-8362-dd0872803a46\" (UID: \"cbc1c1da-fc82-44e3-8362-dd0872803a46\") "
Jul 22 02:32:51.775353 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:51.775329    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbc1c1da-fc82-44e3-8362-dd0872803a46-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cbc1c1da-fc82-44e3-8362-dd0872803a46" (UID: "cbc1c1da-fc82-44e3-8362-dd0872803a46"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:32:51.776961 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:51.776940    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbc1c1da-fc82-44e3-8362-dd0872803a46-kube-api-access-hfn97" (OuterVolumeSpecName: "kube-api-access-hfn97") pod "cbc1c1da-fc82-44e3-8362-dd0872803a46" (UID: "cbc1c1da-fc82-44e3-8362-dd0872803a46"). InnerVolumeSpecName "kube-api-access-hfn97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:32:51.777024 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:51.776969    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbc1c1da-fc82-44e3-8362-dd0872803a46-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cbc1c1da-fc82-44e3-8362-dd0872803a46" (UID: "cbc1c1da-fc82-44e3-8362-dd0872803a46"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:32:51.875570 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:51.875549    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc1c1da-fc82-44e3-8362-dd0872803a46-openshift-service-ca\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:32:51.875570 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:51.875568    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbc1c1da-fc82-44e3-8362-dd0872803a46-maas-api-tls\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:32:51.875691 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:51.875577    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfn97\" (UniqueName: \"kubernetes.io/projected/cbc1c1da-fc82-44e3-8362-dd0872803a46-kube-api-access-hfn97\") on node \"ip-10-0-135-230.ec2.internal\" DevicePath \"\""
Jul 22 02:32:52.295602 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:52.295566    2563 generic.go:356] "Generic (PLEG): container finished" podID="cbc1c1da-fc82-44e3-8362-dd0872803a46" containerID="3d5232c4b51b1f135585965fb97086c891bcc0ded9f779bdde7540a8351672de" exitCode=0
Jul 22 02:32:52.295746 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:52.295609    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a73722-546b4bf4bf-jmtkb"
Jul 22 02:32:52.295746 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:52.295647    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a73722-546b4bf4bf-jmtkb" event={"ID":"cbc1c1da-fc82-44e3-8362-dd0872803a46","Type":"ContainerDied","Data":"3d5232c4b51b1f135585965fb97086c891bcc0ded9f779bdde7540a8351672de"}
Jul 22 02:32:52.295746 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:52.295694    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a73722-546b4bf4bf-jmtkb" event={"ID":"cbc1c1da-fc82-44e3-8362-dd0872803a46","Type":"ContainerDied","Data":"b7fec58869add0b4e0abb2935f48ebdb39f533ce5c511c2f3c606333c9517707"}
Jul 22 02:32:52.295746 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:52.295716    2563 scope.go:117] "RemoveContainer" containerID="3d5232c4b51b1f135585965fb97086c891bcc0ded9f779bdde7540a8351672de"
Jul 22 02:32:52.305277 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:52.305261    2563 scope.go:117] "RemoveContainer" containerID="3d5232c4b51b1f135585965fb97086c891bcc0ded9f779bdde7540a8351672de"
Jul 22 02:32:52.305537 ip-10-0-135-230 kubenswrapper[2563]: E0722 02:32:52.305518    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d5232c4b51b1f135585965fb97086c891bcc0ded9f779bdde7540a8351672de\": container with ID starting with 3d5232c4b51b1f135585965fb97086c891bcc0ded9f779bdde7540a8351672de not found: ID does not exist" containerID="3d5232c4b51b1f135585965fb97086c891bcc0ded9f779bdde7540a8351672de"
Jul 22 02:32:52.305582 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:52.305547    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d5232c4b51b1f135585965fb97086c891bcc0ded9f779bdde7540a8351672de"} err="failed to get container status \"3d5232c4b51b1f135585965fb97086c891bcc0ded9f779bdde7540a8351672de\": rpc error: code = NotFound desc = could not find container \"3d5232c4b51b1f135585965fb97086c891bcc0ded9f779bdde7540a8351672de\": container with ID starting with 3d5232c4b51b1f135585965fb97086c891bcc0ded9f779bdde7540a8351672de not found: ID does not exist"
Jul 22 02:32:52.318892 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:52.318871    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a73722-546b4bf4bf-jmtkb"]
Jul 22 02:32:52.325489 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:52.325461    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a73722-546b4bf4bf-jmtkb"]
Jul 22 02:32:53.663177 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:53.663143    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbc1c1da-fc82-44e3-8362-dd0872803a46" path="/var/lib/kubelet/pods/cbc1c1da-fc82-44e3-8362-dd0872803a46/volumes"
Jul 22 02:32:55.969624 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:55.969590    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a73722-8mmvv"]
Jul 22 02:32:55.973301 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:55.973277    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a73722-8mmvv"]
Jul 22 02:32:57.662588 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:32:57.662554    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f821c00-2af0-47a0-aedc-dc1ceaf0d948" path="/var/lib/kubelet/pods/2f821c00-2af0-47a0-aedc-dc1ceaf0d948/volumes"
Jul 22 02:33:03.962604 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:03.962574    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-f6578bcf6-s5zfn_f522f711-4278-4185-b9ce-95a84ae51f40/authorino/0.log"
Jul 22 02:33:10.236663 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:10.236629    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-62wkf_958bb2e8-a856-4e85-9482-36370ce716c6/manager/0.log"
Jul 22 02:33:10.368052 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:10.368024    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-9qxz4_92e08f68-a6f8-482b-ba21-4dab98b2a400/manager/0.log"
Jul 22 02:33:10.786405 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:10.786367    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-lgqv9_250b2d23-5084-4267-acf0-857731a2fe8c/manager/0.log"
Jul 22 02:33:11.238922 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:11.238897    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76db4bc599-kvtx4_f8dd657c-fe0f-4329-87bd-5381cbb98728/manager/0.log"
Jul 22 02:33:11.249042 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:11.249023    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76db4bc599-kvtx4_f8dd657c-fe0f-4329-87bd-5381cbb98728/copy-manifests/0.log"
Jul 22 02:33:11.644834 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:11.644772    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-8459478d64-8xtx5_33bc4a81-9a52-468d-b37d-dfbfd3edb289/maas-api/0.log"
Jul 22 02:33:11.923921 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:11.923901    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29744775-pt2wf_2db07ffd-f96e-4d29-a4d1-846ee8796eed/cleanup/0.log"
Jul 22 02:33:12.054223 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:12.054199    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29744790-z6p59_be20609b-0f9c-4e66-8a91-81e8b8d8bace/cleanup/0.log"
Jul 22 02:33:12.184562 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:12.184513    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-d96gh_61a18758-f622-4cd7-a6ce-1d918b0d120b/postgres/0.log"
Jul 22 02:33:13.070339 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:13.070316    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gzzv6_5f575f0b-aaa2-413d-87f2-ce9d69096fb8/pull/0.log"
Jul 22 02:33:13.090869 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:13.090853    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gzzv6_5f575f0b-aaa2-413d-87f2-ce9d69096fb8/extract/0.log"
Jul 22 02:33:13.106873 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:13.106855    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gzzv6_5f575f0b-aaa2-413d-87f2-ce9d69096fb8/util/0.log"
Jul 22 02:33:13.236591 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:13.236575    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bmkm4_24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9/util/0.log"
Jul 22 02:33:13.247726 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:13.247707    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bmkm4_24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9/pull/0.log"
Jul 22 02:33:13.261588 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:13.261563    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bmkm4_24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9/extract/0.log"
Jul 22 02:33:13.404031 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:13.403974    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vjn4s_43423d8a-faea-4db0-84ae-8af34f0eb834/util/0.log"
Jul 22 02:33:13.415885 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:13.415864    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vjn4s_43423d8a-faea-4db0-84ae-8af34f0eb834/pull/0.log"
Jul 22 02:33:13.430717 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:13.430703    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vjn4s_43423d8a-faea-4db0-84ae-8af34f0eb834/extract/0.log"
Jul 22 02:33:13.570672 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:13.570651    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfjcd_62e08628-fa98-48a4-99cc-3dc660dbc84e/util/0.log"
Jul 22 02:33:13.580268 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:13.580249    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfjcd_62e08628-fa98-48a4-99cc-3dc660dbc84e/pull/0.log"
Jul 22 02:33:13.593876 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:13.593856    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfjcd_62e08628-fa98-48a4-99cc-3dc660dbc84e/extract/0.log"
Jul 22 02:33:13.746076 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:13.746046    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-f6578bcf6-s5zfn_f522f711-4278-4185-b9ce-95a84ae51f40/authorino/0.log"
Jul 22 02:33:13.890512 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:13.890484    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-pvwj2_7497ee29-5208-4496-97c4-e4b083d32712/manager/0.log"
Jul 22 02:33:14.023059 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:14.022990    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-kzbvr_2c568a93-ab9d-453e-82dc-a943b46669ff/manager/0.log"
Jul 22 02:33:14.291069 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:14.291005    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-bhvjd_869bc209-0252-4361-a934-c57c0284885e/registry-server/0.log"
Jul 22 02:33:14.435964 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:14.435944    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-dcm6p_e61078f7-e62c-4c53-9186-80e33d10ab4f/manager/0.log"
Jul 22 02:33:14.713476 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:14.713445    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-2ghjg_be5832ee-cedb-4e5e-8262-1e63d96e3c22/manager/0.log"
Jul 22 02:33:15.506736 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:15.506713    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-75f64fb6dd-xrvcm_179aab03-1b60-45e8-a9a0-1dce658c6971/kube-auth-proxy/0.log"
Jul 22 02:33:15.895590 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:15.895525    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-tcq4d_e71fb6d5-df07-41bc-96fe-95df778f814d/payload-processing/0.log"
Jul 22 02:33:16.396443 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:16.396414    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-l2wpc_8fe97e5a-cb67-49a0-9f85-318daeb2cca3/main/0.log"
Jul 22 02:33:23.167661 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:23.167625    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-49wdk/must-gather-5t2sf"]
Jul 22 02:33:23.168091 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:23.167945    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbc1c1da-fc82-44e3-8362-dd0872803a46" containerName="maas-api"
Jul 22 02:33:23.168091 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:23.167955    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbc1c1da-fc82-44e3-8362-dd0872803a46" containerName="maas-api"
Jul 22 02:33:23.168091 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:23.167963    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f821c00-2af0-47a0-aedc-dc1ceaf0d948" containerName="revoke-keys"
Jul 22 02:33:23.168091 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:23.167969    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f821c00-2af0-47a0-aedc-dc1ceaf0d948" containerName="revoke-keys"
Jul 22 02:33:23.168091 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:23.168024    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f821c00-2af0-47a0-aedc-dc1ceaf0d948" containerName="revoke-keys"
Jul 22 02:33:23.168091 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:23.168035    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbc1c1da-fc82-44e3-8362-dd0872803a46" containerName="maas-api"
Jul 22 02:33:23.171615 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:23.171595    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-49wdk/must-gather-5t2sf"
Jul 22 02:33:23.174715 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:23.174692    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-49wdk\"/\"openshift-service-ca.crt\""
Jul 22 02:33:23.174819 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:23.174714    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-49wdk\"/\"kube-root-ca.crt\""
Jul 22 02:33:23.175549 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:23.175531    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-49wdk\"/\"default-dockercfg-fbnjc\""
Jul 22 02:33:23.190453 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:23.190435    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-49wdk/must-gather-5t2sf"]
Jul 22 02:33:23.218102 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:23.218078    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cbd8b491-3e11-4e4d-83ea-c35efa28e3af-must-gather-output\") pod \"must-gather-5t2sf\" (UID: \"cbd8b491-3e11-4e4d-83ea-c35efa28e3af\") " pod="openshift-must-gather-49wdk/must-gather-5t2sf"
Jul 22 02:33:23.218203 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:23.218112    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swvcj\" (UniqueName: \"kubernetes.io/projected/cbd8b491-3e11-4e4d-83ea-c35efa28e3af-kube-api-access-swvcj\") pod \"must-gather-5t2sf\" (UID: \"cbd8b491-3e11-4e4d-83ea-c35efa28e3af\") " pod="openshift-must-gather-49wdk/must-gather-5t2sf"
Jul 22 02:33:23.319471 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:23.319439    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cbd8b491-3e11-4e4d-83ea-c35efa28e3af-must-gather-output\") pod \"must-gather-5t2sf\" (UID: \"cbd8b491-3e11-4e4d-83ea-c35efa28e3af\") " pod="openshift-must-gather-49wdk/must-gather-5t2sf"
Jul 22 02:33:23.319625 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:23.319482    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swvcj\" (UniqueName: \"kubernetes.io/projected/cbd8b491-3e11-4e4d-83ea-c35efa28e3af-kube-api-access-swvcj\") pod \"must-gather-5t2sf\" (UID: \"cbd8b491-3e11-4e4d-83ea-c35efa28e3af\") " pod="openshift-must-gather-49wdk/must-gather-5t2sf"
Jul 22 02:33:23.319779 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:23.319760    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cbd8b491-3e11-4e4d-83ea-c35efa28e3af-must-gather-output\") pod \"must-gather-5t2sf\" (UID: \"cbd8b491-3e11-4e4d-83ea-c35efa28e3af\") " pod="openshift-must-gather-49wdk/must-gather-5t2sf"
Jul 22 02:33:23.330114 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:23.330081    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swvcj\" (UniqueName: \"kubernetes.io/projected/cbd8b491-3e11-4e4d-83ea-c35efa28e3af-kube-api-access-swvcj\") pod \"must-gather-5t2sf\" (UID: \"cbd8b491-3e11-4e4d-83ea-c35efa28e3af\") " pod="openshift-must-gather-49wdk/must-gather-5t2sf"
Jul 22 02:33:23.480934 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:23.480905    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-49wdk/must-gather-5t2sf"
Jul 22 02:33:23.614017 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:23.613959    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-49wdk/must-gather-5t2sf"]
Jul 22 02:33:23.616041 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:33:23.616010    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcbd8b491_3e11_4e4d_83ea_c35efa28e3af.slice/crio-24bff739907e4ba1e024c9bd25930ebd653ae3096e3003483e58a717490ee5a3 WatchSource:0}: Error finding container 24bff739907e4ba1e024c9bd25930ebd653ae3096e3003483e58a717490ee5a3: Status 404 returned error can't find the container with id 24bff739907e4ba1e024c9bd25930ebd653ae3096e3003483e58a717490ee5a3
Jul 22 02:33:24.460436 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:24.460322    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-49wdk/must-gather-5t2sf" event={"ID":"cbd8b491-3e11-4e4d-83ea-c35efa28e3af","Type":"ContainerStarted","Data":"4a071ab4bb68a699671dab7d832040f2943887fd189396eae6bd013c15a3688b"}
Jul 22 02:33:24.460436 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:24.460396    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-49wdk/must-gather-5t2sf" event={"ID":"cbd8b491-3e11-4e4d-83ea-c35efa28e3af","Type":"ContainerStarted","Data":"24bff739907e4ba1e024c9bd25930ebd653ae3096e3003483e58a717490ee5a3"}
Jul 22 02:33:25.470455 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:25.470417    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-49wdk/must-gather-5t2sf" event={"ID":"cbd8b491-3e11-4e4d-83ea-c35efa28e3af","Type":"ContainerStarted","Data":"bfaa1cde5eb523a327b22b0507e07bb800c58f33c1e299f71f3735c0540a8d17"}
Jul 22 02:33:25.494344 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:25.494287    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-49wdk/must-gather-5t2sf" podStartSLOduration=1.7612743339999999 podStartE2EDuration="2.494265722s" totalImagesPullingTime="732.991388ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:33:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:33:25.491134562 +0000 UTC m=+2758.334963693" watchObservedRunningTime="2026-07-22 02:33:25.494265722 +0000 UTC m=+2758.338094856"
Jul 22 02:33:26.081823 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:26.081792    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mjkvg_2f67b074-15ce-4e39-afdc-5c30c8addc97/global-pull-secret-syncer/0.log"
Jul 22 02:33:26.182239 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:26.182212    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rd82t_04c89d69-a674-49e4-9630-9302cec0fad4/konnectivity-agent/0.log"
Jul 22 02:33:26.283833 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:26.283807    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-230.ec2.internal_a67e292d73c90fc17a80248cb369ca7d/haproxy/0.log"
Jul 22 02:33:30.042157 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:30.042123    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gzzv6_5f575f0b-aaa2-413d-87f2-ce9d69096fb8/extract/0.log"
Jul 22 02:33:30.082186 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:30.082154    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gzzv6_5f575f0b-aaa2-413d-87f2-ce9d69096fb8/util/0.log"
Jul 22 02:33:30.121512 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:30.121482    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759gzzv6_5f575f0b-aaa2-413d-87f2-ce9d69096fb8/pull/0.log"
Jul 22 02:33:30.169898 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:30.169868    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bmkm4_24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9/extract/0.log"
Jul 22 02:33:30.221305 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:30.221271    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bmkm4_24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9/util/0.log"
Jul 22 02:33:30.255645 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:30.255481    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bmkm4_24fb3a78-b34a-466a-bb1f-ec9bbf1d5bb9/pull/0.log"
Jul 22 02:33:30.303042 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:30.302961    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vjn4s_43423d8a-faea-4db0-84ae-8af34f0eb834/extract/0.log"
Jul 22 02:33:30.331881 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:30.331853    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vjn4s_43423d8a-faea-4db0-84ae-8af34f0eb834/util/0.log"
Jul 22 02:33:30.364637 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:30.364614    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vjn4s_43423d8a-faea-4db0-84ae-8af34f0eb834/pull/0.log"
Jul 22 02:33:30.401486 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:30.401446    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfjcd_62e08628-fa98-48a4-99cc-3dc660dbc84e/extract/0.log"
Jul 22 02:33:30.441649 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:30.441557    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfjcd_62e08628-fa98-48a4-99cc-3dc660dbc84e/util/0.log"
Jul 22 02:33:30.477729 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:30.477686    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfjcd_62e08628-fa98-48a4-99cc-3dc660dbc84e/pull/0.log"
Jul 22 02:33:30.924935 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:30.924847    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-f6578bcf6-s5zfn_f522f711-4278-4185-b9ce-95a84ae51f40/authorino/0.log"
Jul 22 02:33:30.962504 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:30.962469    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-pvwj2_7497ee29-5208-4496-97c4-e4b083d32712/manager/0.log"
Jul 22 02:33:31.008446 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:31.008415    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-kzbvr_2c568a93-ab9d-453e-82dc-a943b46669ff/manager/0.log"
Jul 22 02:33:31.117296 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:31.117262    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-bhvjd_869bc209-0252-4361-a934-c57c0284885e/registry-server/0.log"
Jul 22 02:33:31.294056 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:31.293770    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-dcm6p_e61078f7-e62c-4c53-9186-80e33d10ab4f/manager/0.log"
Jul 22 02:33:31.430248 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:31.430096    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-2ghjg_be5832ee-cedb-4e5e-8262-1e63d96e3c22/manager/0.log"
Jul 22 02:33:33.726217 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:33.726186    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8lmr8_14610eac-7fab-4d74-ac5d-10ea5d09716b/node-exporter/0.log"
Jul 22 02:33:33.795949 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:33.795921    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8lmr8_14610eac-7fab-4d74-ac5d-10ea5d09716b/kube-rbac-proxy/0.log"
Jul 22 02:33:33.829905 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:33.829873    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8lmr8_14610eac-7fab-4d74-ac5d-10ea5d09716b/init-textfile/0.log"
Jul 22 02:33:35.705195 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:35.705151    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-49wdk/perf-node-gather-daemonset-zv6fv"]
Jul 22 02:33:35.713144 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:35.713117    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-49wdk/perf-node-gather-daemonset-zv6fv"
Jul 22 02:33:35.718771 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:35.718740    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-49wdk/perf-node-gather-daemonset-zv6fv"]
Jul 22 02:33:35.857193 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:35.857163    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cfc3b73f-cb8a-4fda-bd66-774085f8ad2b-sys\") pod \"perf-node-gather-daemonset-zv6fv\" (UID: \"cfc3b73f-cb8a-4fda-bd66-774085f8ad2b\") " pod="openshift-must-gather-49wdk/perf-node-gather-daemonset-zv6fv"
Jul 22 02:33:35.857193 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:35.857194    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m8lq\" (UniqueName: \"kubernetes.io/projected/cfc3b73f-cb8a-4fda-bd66-774085f8ad2b-kube-api-access-4m8lq\") pod \"perf-node-gather-daemonset-zv6fv\" (UID: \"cfc3b73f-cb8a-4fda-bd66-774085f8ad2b\") " pod="openshift-must-gather-49wdk/perf-node-gather-daemonset-zv6fv"
Jul 22 02:33:35.857420 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:35.857222    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cfc3b73f-cb8a-4fda-bd66-774085f8ad2b-proc\") pod \"perf-node-gather-daemonset-zv6fv\" (UID: \"cfc3b73f-cb8a-4fda-bd66-774085f8ad2b\") " pod="openshift-must-gather-49wdk/perf-node-gather-daemonset-zv6fv"
Jul 22 02:33:35.857420 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:35.857293    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cfc3b73f-cb8a-4fda-bd66-774085f8ad2b-podres\") pod \"perf-node-gather-daemonset-zv6fv\" (UID: \"cfc3b73f-cb8a-4fda-bd66-774085f8ad2b\") " pod="openshift-must-gather-49wdk/perf-node-gather-daemonset-zv6fv"
Jul 22 02:33:35.857420 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:35.857403    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cfc3b73f-cb8a-4fda-bd66-774085f8ad2b-lib-modules\") pod \"perf-node-gather-daemonset-zv6fv\" (UID: \"cfc3b73f-cb8a-4fda-bd66-774085f8ad2b\") " pod="openshift-must-gather-49wdk/perf-node-gather-daemonset-zv6fv"
Jul 22 02:33:35.958661 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:35.958545    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cfc3b73f-cb8a-4fda-bd66-774085f8ad2b-sys\") pod \"perf-node-gather-daemonset-zv6fv\" (UID: \"cfc3b73f-cb8a-4fda-bd66-774085f8ad2b\") " pod="openshift-must-gather-49wdk/perf-node-gather-daemonset-zv6fv"
Jul 22 02:33:35.958661 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:35.958589    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m8lq\" (UniqueName: \"kubernetes.io/projected/cfc3b73f-cb8a-4fda-bd66-774085f8ad2b-kube-api-access-4m8lq\") pod \"perf-node-gather-daemonset-zv6fv\" (UID: \"cfc3b73f-cb8a-4fda-bd66-774085f8ad2b\") " pod="openshift-must-gather-49wdk/perf-node-gather-daemonset-zv6fv"
Jul 22 02:33:35.958661 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:35.958611    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cfc3b73f-cb8a-4fda-bd66-774085f8ad2b-proc\") pod \"perf-node-gather-daemonset-zv6fv\" (UID: \"cfc3b73f-cb8a-4fda-bd66-774085f8ad2b\") " pod="openshift-must-gather-49wdk/perf-node-gather-daemonset-zv6fv"
Jul 22 02:33:35.958661 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:35.958634    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cfc3b73f-cb8a-4fda-bd66-774085f8ad2b-podres\") pod \"perf-node-gather-daemonset-zv6fv\" (UID: \"cfc3b73f-cb8a-4fda-bd66-774085f8ad2b\") " pod="openshift-must-gather-49wdk/perf-node-gather-daemonset-zv6fv"
Jul 22 02:33:35.959036 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:35.958674    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cfc3b73f-cb8a-4fda-bd66-774085f8ad2b-lib-modules\") pod \"perf-node-gather-daemonset-zv6fv\" (UID: \"cfc3b73f-cb8a-4fda-bd66-774085f8ad2b\") " pod="openshift-must-gather-49wdk/perf-node-gather-daemonset-zv6fv"
Jul 22 02:33:35.959036 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:35.958817    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cfc3b73f-cb8a-4fda-bd66-774085f8ad2b-lib-modules\") pod \"perf-node-gather-daemonset-zv6fv\" (UID: \"cfc3b73f-cb8a-4fda-bd66-774085f8ad2b\") " pod="openshift-must-gather-49wdk/perf-node-gather-daemonset-zv6fv"
Jul 22 02:33:35.959036 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:35.958857    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cfc3b73f-cb8a-4fda-bd66-774085f8ad2b-proc\") pod \"perf-node-gather-daemonset-zv6fv\" (UID: \"cfc3b73f-cb8a-4fda-bd66-774085f8ad2b\") " pod="openshift-must-gather-49wdk/perf-node-gather-daemonset-zv6fv"
Jul 22 02:33:35.959036 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:35.958905    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cfc3b73f-cb8a-4fda-bd66-774085f8ad2b-podres\") pod \"perf-node-gather-daemonset-zv6fv\" (UID: \"cfc3b73f-cb8a-4fda-bd66-774085f8ad2b\") " pod="openshift-must-gather-49wdk/perf-node-gather-daemonset-zv6fv"
Jul 22 02:33:35.959036 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:35.959014    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cfc3b73f-cb8a-4fda-bd66-774085f8ad2b-sys\") pod \"perf-node-gather-daemonset-zv6fv\" (UID: \"cfc3b73f-cb8a-4fda-bd66-774085f8ad2b\") " pod="openshift-must-gather-49wdk/perf-node-gather-daemonset-zv6fv"
Jul 22 02:33:35.968413 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:35.968363    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m8lq\" (UniqueName: \"kubernetes.io/projected/cfc3b73f-cb8a-4fda-bd66-774085f8ad2b-kube-api-access-4m8lq\") pod \"perf-node-gather-daemonset-zv6fv\" (UID: \"cfc3b73f-cb8a-4fda-bd66-774085f8ad2b\") " pod="openshift-must-gather-49wdk/perf-node-gather-daemonset-zv6fv"
Jul 22 02:33:36.027517 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:36.027482    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-49wdk/perf-node-gather-daemonset-zv6fv"
Jul 22 02:33:36.231805 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:36.230151    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-49wdk/perf-node-gather-daemonset-zv6fv"]
Jul 22 02:33:36.238094 ip-10-0-135-230 kubenswrapper[2563]: W0722 02:33:36.238019    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podcfc3b73f_cb8a_4fda_bd66_774085f8ad2b.slice/crio-368c7266837eeb719715c0006d511de6d30f7ca70de8746a3a34ea71697a71c9 WatchSource:0}: Error finding container 368c7266837eeb719715c0006d511de6d30f7ca70de8746a3a34ea71697a71c9: Status 404 returned error can't find the container with id 368c7266837eeb719715c0006d511de6d30f7ca70de8746a3a34ea71697a71c9
Jul 22 02:33:36.552946 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:36.552851    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-49wdk/perf-node-gather-daemonset-zv6fv" event={"ID":"cfc3b73f-cb8a-4fda-bd66-774085f8ad2b","Type":"ContainerStarted","Data":"efdbb8aecdf2a5a91ad1b2fb9b8b35b3dab47cccf7f22befcb5a60688037dd8c"}
Jul 22 02:33:36.552946 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:36.552918    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-49wdk/perf-node-gather-daemonset-zv6fv"
Jul 22 02:33:36.552946 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:36.552942    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-49wdk/perf-node-gather-daemonset-zv6fv" event={"ID":"cfc3b73f-cb8a-4fda-bd66-774085f8ad2b","Type":"ContainerStarted","Data":"368c7266837eeb719715c0006d511de6d30f7ca70de8746a3a34ea71697a71c9"}
Jul 22 02:33:36.573736 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:36.573674    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-49wdk/perf-node-gather-daemonset-zv6fv" podStartSLOduration=1.573653181 podStartE2EDuration="1.573653181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:33:36.570681367 +0000 UTC m=+2769.414510508" watchObservedRunningTime="2026-07-22 02:33:36.573653181 +0000 UTC m=+2769.417482313"
Jul 22 02:33:39.296943 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:39.296913    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wkdlw_41d01142-330d-46a2-8625-9e5b9ba1e2bf/dns/0.log"
Jul 22 02:33:39.349970 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:39.349944    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wkdlw_41d01142-330d-46a2-8625-9e5b9ba1e2bf/kube-rbac-proxy/0.log"
Jul 22 02:33:39.390880 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:39.390854    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4xkbt_08637867-465c-4df3-b707-ef1e33b64ed2/dns-node-resolver/0.log"
Jul 22 02:33:40.125164 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:40.125129    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-9d8c7d849-kjrp7_56de72cc-8504-438a-adce-ea7a32dc8bfa/registry/0.log"
Jul 22 02:33:40.252828 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:40.252799    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-drsr9_2393693b-9aef-40c6-ab3f-df34a7500ed8/node-ca/0.log"
Jul 22 02:33:41.654096 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:41.654065    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-75f64fb6dd-xrvcm_179aab03-1b60-45e8-a9a0-1dce658c6971/kube-auth-proxy/0.log"
Jul 22 02:33:41.801420 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:41.801372    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-tcq4d_e71fb6d5-df07-41bc-96fe-95df778f814d/payload-processing/0.log"
Jul 22 02:33:42.571244 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:42.571214    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-49wdk/perf-node-gather-daemonset-zv6fv"
Jul 22 02:33:42.677658 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:42.677631    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mhjk2_6cc54a54-1e35-414c-ac99-ce76586fe402/serve-healthcheck-canary/0.log"
Jul 22 02:33:43.445917 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:43.445888    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jrf2p_3089a472-02e2-4cf4-994d-146d917dd72d/kube-rbac-proxy/0.log"
Jul 22 02:33:43.478901 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:43.478873    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jrf2p_3089a472-02e2-4cf4-994d-146d917dd72d/exporter/0.log"
Jul 22 02:33:43.504467 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:43.504433    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jrf2p_3089a472-02e2-4cf4-994d-146d917dd72d/extractor/0.log"
Jul 22 02:33:46.002588 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:46.002542    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-75f64fb6dd-xrvcm_179aab03-1b60-45e8-a9a0-1dce658c6971/kube-auth-proxy/0.log"
Jul 22 02:33:46.054559 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:46.054531    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-62wkf_958bb2e8-a856-4e85-9482-36370ce716c6/manager/0.log"
Jul 22 02:33:46.114116 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:46.114085    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-tcq4d_e71fb6d5-df07-41bc-96fe-95df778f814d/payload-processing/0.log"
Jul 22 02:33:46.271755 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:46.271509    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-9qxz4_92e08f68-a6f8-482b-ba21-4dab98b2a400/manager/0.log"
Jul 22 02:33:46.628284 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:46.628215    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-lgqv9_250b2d23-5084-4267-acf0-857731a2fe8c/manager/0.log"
Jul 22 02:33:46.894279 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:46.894209    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76db4bc599-kvtx4_f8dd657c-fe0f-4329-87bd-5381cbb98728/manager/0.log"
Jul 22 02:33:46.928835 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:46.928808    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76db4bc599-kvtx4_f8dd657c-fe0f-4329-87bd-5381cbb98728/copy-manifests/0.log"
Jul 22 02:33:47.937856 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:47.937816    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-75f64fb6dd-xrvcm_179aab03-1b60-45e8-a9a0-1dce658c6971/kube-auth-proxy/0.log"
Jul 22 02:33:48.121217 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:48.121184    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-tcq4d_e71fb6d5-df07-41bc-96fe-95df778f814d/payload-processing/0.log"
Jul 22 02:33:48.760134 ip-10-0-135-230 kubenswrapper[2563]: I0722 02:33:48.760086    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-hwnwh_8c950b15-1953-4303-a2eb-d4ba5156b935/openshift-lws-operator/0.log"