Jul 06 22:44:19.979664 ip-10-0-134-116 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 06 22:44:19.979675 ip-10-0-134-116 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 06 22:44:19.979682 ip-10-0-134-116 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 06 22:44:19.979908 ip-10-0-134-116 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 06 22:44:30.012665 ip-10-0-134-116 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 06 22:44:30.012680 ip-10-0-134-116 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot b512a80825c640388f1de43647d693dd --
Jul 06 22:46:27.475442 ip-10-0-134-116 systemd[1]: Starting Kubernetes Kubelet...
Jul 06 22:46:27.968902 ip-10-0-134-116 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 06 22:46:27.968902 ip-10-0-134-116 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 06 22:46:27.968902 ip-10-0-134-116 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 06 22:46:27.968902 ip-10-0-134-116 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 06 22:46:27.968902 ip-10-0-134-116 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 06 22:46:27.968902 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.896653    2569 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 06 22:46:27.968902 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900444    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 22:46:27.968902 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900455    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 22:46:27.968902 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900458    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 22:46:27.968902 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900462    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 22:46:27.968902 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900465    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 22:46:27.968902 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900468    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 22:46:27.968902 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900471    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 22:46:27.968902 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900473    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 22:46:27.968168 ip-10-0-134-116 systemd[1]: Started Kubernetes Kubelet.
Jul 06 22:46:27.983663 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900476    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 22:46:27.983663 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900478    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 22:46:27.983663 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900481    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 22:46:27.983663 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900484    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 22:46:27.983663 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900486    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 22:46:27.983663 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900489    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 22:46:27.983663 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900491    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 22:46:27.983663 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900494    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 22:46:27.983663 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900496    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 22:46:27.983663 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900499    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 22:46:27.983663 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900501    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 22:46:27.983663 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900504    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 22:46:27.983663 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900506    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 22:46:27.983663 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900509    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 22:46:27.983663 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900511    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 22:46:27.983663 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900521    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 22:46:27.983663 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900525    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 22:46:27.983663 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900528    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 22:46:27.983663 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900530    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 22:46:27.983663 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900533    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 22:46:27.985784 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900535    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 22:46:27.985784 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900538    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 22:46:27.985784 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900540    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 22:46:27.985784 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900543    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 22:46:27.985784 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900546    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 22:46:27.985784 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900548    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 22:46:27.985784 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900552    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 22:46:27.985784 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900554    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 22:46:27.985784 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900557    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 22:46:27.985784 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900559    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 22:46:27.985784 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900562    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 22:46:27.985784 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900565    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 22:46:27.985784 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900567    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 22:46:27.985784 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900570    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 22:46:27.985784 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900572    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 22:46:27.985784 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900575    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 22:46:27.985784 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900577    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 22:46:27.985784 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900580    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 22:46:27.985784 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900582    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 22:46:27.988453 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900585    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 22:46:27.988453 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900587    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 22:46:27.988453 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900589    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 22:46:27.988453 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900592    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 22:46:27.988453 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900594    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 22:46:27.988453 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900597    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 22:46:27.988453 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900599    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 22:46:27.988453 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900602    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 22:46:27.988453 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900605    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 22:46:27.988453 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900607    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 22:46:27.988453 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900610    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 22:46:27.988453 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900613    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 22:46:27.988453 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900616    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 22:46:27.988453 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900618    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 22:46:27.988453 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900621    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 22:46:27.988453 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900625    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 22:46:27.988453 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900628    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 22:46:27.988453 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900632    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 22:46:27.988453 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900635    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 22:46:27.990337 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900637    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 22:46:27.990337 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900640    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 22:46:27.990337 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900643    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 22:46:27.990337 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900645    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 22:46:27.990337 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900648    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 22:46:27.990337 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900651    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 22:46:27.990337 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900654    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 22:46:27.990337 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900656    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 22:46:27.990337 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900659    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 22:46:27.990337 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900662    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 22:46:27.990337 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900666    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 22:46:27.990337 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900670    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 22:46:27.990337 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900673    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 22:46:27.990337 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900676    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 22:46:27.990337 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900679    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 22:46:27.990337 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900681    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 22:46:27.990337 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900684    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 22:46:27.990337 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900686    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 22:46:27.990337 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900689    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 22:46:27.990337 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900691    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 22:46:27.991756 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900694    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 22:46:27.991756 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900697    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 22:46:27.991756 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900700    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 22:46:27.991756 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900702    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 22:46:27.991756 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900705    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 22:46:27.991756 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900707    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 22:46:27.991756 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900710    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 22:46:27.991756 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900712    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 22:46:27.991756 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900715    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 22:46:27.991756 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900717    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 22:46:27.991756 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900720    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 22:46:27.991756 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900723    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 22:46:27.991756 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.900725    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 22:46:27.991756 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901429    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 22:46:27.991756 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901437    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 22:46:27.991756 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901440    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 22:46:27.991756 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901443    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 22:46:27.991756 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901447    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 22:46:27.991756 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901450    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 22:46:27.991756 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901453    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 22:46:27.993021 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901455    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 22:46:27.993021 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901458    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 22:46:27.993021 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901460    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 22:46:27.993021 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901463    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 22:46:27.993021 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901466    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 22:46:27.993021 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901468    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 22:46:27.993021 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901471    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 22:46:27.993021 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901473    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 22:46:27.993021 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901476    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 22:46:27.993021 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901479    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 22:46:27.993021 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901481    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 22:46:27.993021 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901484    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 22:46:27.993021 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901486    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 22:46:27.993021 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901489    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 22:46:27.993021 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901491    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 22:46:27.993021 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901494    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 22:46:27.993021 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901496    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 22:46:27.993021 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901499    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 22:46:27.993021 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901501    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 22:46:27.993021 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901505    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 22:46:27.993974 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901507    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 22:46:27.993974 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901510    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 22:46:27.993974 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901513    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 22:46:27.993974 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901516    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 22:46:27.993974 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901519    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 22:46:27.993974 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901521    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 22:46:27.993974 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901524    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 22:46:27.993974 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901526    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 22:46:27.993974 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901529    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 22:46:27.993974 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901532    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 22:46:27.993974 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901534    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 22:46:27.993974 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901538    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 22:46:27.993974 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901542    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 22:46:27.993974 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901546    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 22:46:27.993974 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901549    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 22:46:27.993974 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901552    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 22:46:27.993974 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901554    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 22:46:27.993974 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901558    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 22:46:27.993974 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901562    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 22:46:27.994691 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901565    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 22:46:27.994691 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901568    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 22:46:27.994691 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901571    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 22:46:27.994691 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901574    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 22:46:27.994691 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901577    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 22:46:27.994691 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901580    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 22:46:27.994691 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901584    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 22:46:27.994691 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901586    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 22:46:27.994691 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901589    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 22:46:27.994691 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901592    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 22:46:27.994691 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901594    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 22:46:27.994691 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901597    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 22:46:27.994691 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901599    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 22:46:27.994691 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901602    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 22:46:27.994691 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901604    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 22:46:27.994691 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901607    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 22:46:27.994691 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901610    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 22:46:27.994691 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901612    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 22:46:27.994691 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901615    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 22:46:27.995321 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901617    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 22:46:27.995321 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901619    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 22:46:27.995321 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901622    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 22:46:27.995321 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901625    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 22:46:27.995321 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901627    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 22:46:27.995321 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901630    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 22:46:27.995321 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901633    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 22:46:27.995321 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901635    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 22:46:27.995321 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901638    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 22:46:27.995321 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901640    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 22:46:27.995321 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901643    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 22:46:27.995321 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901645    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 22:46:27.995321 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901648    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 22:46:27.995321 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901651    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 22:46:27.995321 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901653    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 22:46:27.995321 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901656    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 22:46:27.995321 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901658    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 22:46:27.995321 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901661    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 22:46:27.995321 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901663    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 22:46:27.995321 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901665    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 22:46:27.995850 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901668    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 22:46:27.995850 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901670    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 22:46:27.995850 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901673    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 22:46:27.995850 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901677    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 22:46:27.995850 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901679    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 22:46:27.995850 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901682    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 22:46:27.995850 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901685    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 22:46:27.995850 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901687    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 22:46:27.995850 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901690    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 22:46:27.995850 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901692    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 22:46:27.995850 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901695    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 22:46:27.995850 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901697    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 22:46:27.995850 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901700    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 22:46:27.995850 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.901703    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 22:46:27.995850 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902101    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 22:46:27.995850 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902108    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 22:46:27.995850 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902112    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 22:46:27.995850 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902117    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 22:46:27.995850 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902120    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 22:46:27.996428 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902123    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 22:46:27.996428 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902126    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 22:46:27.996428 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902129    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 22:46:27.996428 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902132    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 22:46:27.996428 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902135    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 22:46:27.996428 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902140    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 22:46:27.996428 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902142    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 22:46:27.996428 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902145    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 22:46:27.996428 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902147    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 22:46:27.996428 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902151    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 22:46:27.996428 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902155    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 22:46:27.996428 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902158    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 22:46:27.996428 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902160    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 22:46:27.996428 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902163    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 22:46:27.996428 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902166    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 22:46:27.996428 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902168    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 22:46:27.996428 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902171    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 22:46:27.996428 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902174    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 22:46:27.996428 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902191    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 22:46:27.996428 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902194    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 22:46:27.996954 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902196    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 22:46:27.996954 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902199    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 22:46:27.996954 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902202    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 22:46:27.996954 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902204    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 22:46:27.996954 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902207    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 22:46:27.996954 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902209    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 22:46:27.996954 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902212    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 22:46:27.996954 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902214    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 22:46:27.996954 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902217    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 22:46:27.996954 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902220    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 22:46:27.996954 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902227    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 22:46:27.996954 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902229    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 22:46:27.996954 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902233    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 22:46:27.996954 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902235    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 22:46:27.996954 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902238    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 22:46:27.996954 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902240    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 22:46:27.996954 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902243    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 22:46:27.996954 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902247    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 22:46:27.996954 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902249    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 22:46:27.996954 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902252    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 22:46:27.997626 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902254    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 22:46:27.997626 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902256    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 22:46:27.997626 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902259    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 22:46:27.997626 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902261    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 22:46:27.997626 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902264    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 22:46:27.997626 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902266    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 22:46:27.997626 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902269    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 22:46:27.997626 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902271    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 22:46:27.997626 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902274    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 22:46:27.997626 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902276    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 22:46:27.997626 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902278    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 22:46:27.997626 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902281    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 22:46:27.997626 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902283    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 22:46:27.997626 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902287    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 22:46:27.997626 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902291    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 22:46:27.997626 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902293    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 22:46:27.997626 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902296    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 22:46:27.997626 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902298    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 22:46:27.997626 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902301    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 22:46:27.997626 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902304    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 22:46:27.998349 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902306    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 22:46:27.998349 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902308    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 22:46:27.998349 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902312    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 22:46:27.998349 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902315    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 22:46:27.998349 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902318    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 22:46:27.998349 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902320    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 22:46:27.998349 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902323    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 22:46:27.998349 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902326    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 22:46:27.998349 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902328    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 22:46:27.998349 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902332    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 22:46:27.998349 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902335    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 22:46:27.998349 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902337    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 22:46:27.998349 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902340    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 22:46:27.998349 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902342    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 22:46:27.998349 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902345    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 22:46:27.998349 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902347    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 22:46:27.998349 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902350    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 22:46:27.998349 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902352    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 22:46:27.998349 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902354    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 22:46:27.998349 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902357    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 22:46:27.999238 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902359    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 22:46:27.999238 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902362    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 22:46:27.999238 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902364    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 22:46:27.999238 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902367    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 22:46:27.999238 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902370    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 22:46:27.999238 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902373    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 22:46:27.999238 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902376    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 22:46:27.999238 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902378    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 22:46:27.999238 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902381    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 22:46:27.999238 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902383    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 22:46:27.999238 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902386    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 22:46:27.999238 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902388    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 22:46:27.999238 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902391    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 22:46:27.999238 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.902393    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 22:46:27.999238 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902465    2569 flags.go:64] FLAG: --address="0.0.0.0"
Jul 06 22:46:27.999238 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902472    2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 06 22:46:27.999238 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902480    2569 flags.go:64] FLAG: --anonymous-auth="true"
Jul 06 22:46:27.999238 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902484    2569 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 06 22:46:27.999238 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902489    2569 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 06 22:46:27.999238 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902492    2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 06 22:46:27.999868 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902496    2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 06 22:46:27.999868 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902501    2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 06 22:46:27.999868 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902504    2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 06 22:46:27.999868 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902507    2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 06 22:46:27.999868 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902511    2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 06 22:46:27.999868 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902514    2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 06 22:46:27.999868 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902517    2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 06 22:46:27.999868 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902520    2569 flags.go:64] FLAG: --cgroup-root=""
Jul 06 22:46:27.999868 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902522    2569 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 06 22:46:27.999868 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902525    2569 flags.go:64] FLAG: --client-ca-file=""
Jul 06 22:46:27.999868 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902528    2569 flags.go:64] FLAG: --cloud-provider="external"
Jul 06 22:46:27.999868 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902531    2569 flags.go:64] FLAG: --cluster-dns="[]"
Jul 06 22:46:27.999868 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902534    2569 flags.go:64] FLAG: --cluster-domain=""
Jul 06 22:46:27.999868 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902537    2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 06 22:46:27.999868 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902540    2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 06 22:46:27.999868 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902543    2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 06 22:46:27.999868 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902546    2569 flags.go:64] FLAG: --container-log-max-files="5"
Jul 06 22:46:27.999868 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902549    2569 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 06 22:46:27.999868 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902553    2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 06 22:46:27.999868 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902556    2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 06 22:46:27.999868 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902559    2569 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 06 22:46:27.999868 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902561    2569 flags.go:64] FLAG: --contention-profiling="false"
Jul 06 22:46:27.999868 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902564    2569 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 06 22:46:27.999868 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902567    2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 06 22:46:28.000632 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902570    2569 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 06 22:46:28.000632 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902573    2569 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 06 22:46:28.000632 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902578    2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 06 22:46:28.000632 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902581    2569 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 06 22:46:28.000632 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902584    2569 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 06 22:46:28.000632 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902587    2569 flags.go:64] FLAG: --enable-load-reader="false"
Jul 06 22:46:28.000632 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902590    2569 flags.go:64] FLAG: --enable-server="true"
Jul 06 22:46:28.000632 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902594    2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 06 22:46:28.000632 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902598    2569 flags.go:64] FLAG: --event-burst="100"
Jul 06 22:46:28.000632 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902602    2569 flags.go:64] FLAG: --event-qps="50"
Jul 06 22:46:28.000632 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902605    2569 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 06 22:46:28.000632 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902608    2569 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 06 22:46:28.000632 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902611    2569 flags.go:64] FLAG: --eviction-hard=""
Jul 06 22:46:28.000632 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902614    2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 06 22:46:28.000632 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902617    2569 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 06 22:46:28.000632 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902620    2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 06 22:46:28.000632 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902623    2569 flags.go:64] FLAG: --eviction-soft=""
Jul 06 22:46:28.000632 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902625    2569 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 06 22:46:28.000632 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902628    2569 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 06 22:46:28.000632 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902631    2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 06 22:46:28.000632 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902634    2569 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 06 22:46:28.000632 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902637    2569 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 06 22:46:28.000632 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902640    2569 flags.go:64] FLAG: --fail-swap-on="true"
Jul 06 22:46:28.000632 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902643    2569 flags.go:64] FLAG: --feature-gates=""
Jul 06 22:46:28.000632 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902646    2569 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 06 22:46:28.001393 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902650    2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 06 22:46:28.001393 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902653    2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 06 22:46:28.001393 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902655    2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 06 22:46:28.001393 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902659    2569 flags.go:64] FLAG: --healthz-port="10248"
Jul 06 22:46:28.001393 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902661    2569 flags.go:64] FLAG: --help="false"
Jul 06 22:46:28.001393 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902664    2569 flags.go:64] FLAG: --hostname-override="ip-10-0-134-116.ec2.internal"
Jul 06 22:46:28.001393 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902667    2569 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 06 22:46:28.001393 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902670    2569 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 06 22:46:28.001393 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902673    2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 06 22:46:28.001393 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902677    2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 06 22:46:28.001393 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902680    2569 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 06 22:46:28.001393 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902683    2569 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 06 22:46:28.001393 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902686    2569 flags.go:64] FLAG: --image-service-endpoint=""
Jul 06 22:46:28.001393 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902688    2569 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 06 22:46:28.001393 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902691    2569 flags.go:64] FLAG: --kube-api-burst="100"
Jul 06 22:46:28.001393 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902695    2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 06 22:46:28.001393 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902699    2569 flags.go:64] FLAG: --kube-api-qps="50"
Jul 06 22:46:28.001393 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902702    2569 flags.go:64] FLAG: --kube-reserved=""
Jul 06 22:46:28.001393 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902704    2569 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 06 22:46:28.001393 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902707    2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 06 22:46:28.001393 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902710    2569 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 06 22:46:28.001393 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902713    2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 06 22:46:28.001393 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902715    2569 flags.go:64] FLAG: --lock-file=""
Jul 06 22:46:28.001393 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902718    2569 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 06 22:46:28.002453 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902721    2569 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 06 22:46:28.002453 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902724    2569 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 06 22:46:28.002453 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902729    2569 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 06 22:46:28.002453 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902732    2569 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 06 22:46:28.002453 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902734    2569 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 06 22:46:28.002453 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902737    2569 flags.go:64] FLAG: --logging-format="text"
Jul 06 22:46:28.002453 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902740    2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 06 22:46:28.002453 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902743    2569 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 06 22:46:28.002453 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902746    2569 flags.go:64] FLAG: --manifest-url=""
Jul 06 22:46:28.002453 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902749    2569 flags.go:64] FLAG: --manifest-url-header=""
Jul 06 22:46:28.002453 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902753    2569 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 06 22:46:28.002453 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902755    2569 flags.go:64] FLAG: --max-open-files="1000000"
Jul 06 22:46:28.002453 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902759    2569 flags.go:64] FLAG: --max-pods="110"
Jul 06 22:46:28.002453 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902762    2569 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 06 22:46:28.002453 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902765    2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 06 22:46:28.002453 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902768    2569 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 06 22:46:28.002453 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902771    2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 06 22:46:28.002453 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902775    2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 06 22:46:28.002453 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902778    2569 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 06 22:46:28.002453 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902781    2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 06 22:46:28.002453 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902787    2569 flags.go:64] FLAG: --node-status-max-images="50"
Jul 06 22:46:28.002453 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902790    2569 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 06 22:46:28.002453 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902793    2569 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 06 22:46:28.002453 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902796    2569 flags.go:64] FLAG: --pod-cidr=""
Jul 06 22:46:28.005764 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902800    2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 06 22:46:28.005764 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902806    2569 flags.go:64] FLAG: --pod-manifest-path=""
Jul 06 22:46:28.005764 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902808    2569 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 06 22:46:28.005764 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902811    2569 flags.go:64] FLAG: --pods-per-core="0"
Jul 06 22:46:28.005764 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902814    2569 flags.go:64] FLAG: --port="10250"
Jul 06 22:46:28.005764 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902817    2569 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 06 22:46:28.005764 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902820    2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06521a385dd62b60d"
Jul 06 22:46:28.005764 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902823    2569 flags.go:64] FLAG: --qos-reserved=""
Jul 06 22:46:28.005764 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902826    2569 flags.go:64] FLAG: --read-only-port="10255"
Jul 06 22:46:28.005764 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902829    2569 flags.go:64] FLAG: --register-node="true"
Jul 06 22:46:28.005764 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902831    2569 flags.go:64] FLAG: --register-with-taints=""
Jul 06 22:46:28.005764 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902835    2569 flags.go:64] FLAG: --registry-burst="10"
Jul 06 22:46:28.005764 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902837    2569 flags.go:64] FLAG: --registry-qps="5"
Jul 06 22:46:28.005764 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902840    2569 flags.go:64] FLAG: --reserved-cpus=""
Jul 06 22:46:28.005764 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902843    2569 flags.go:64] FLAG: --reserved-memory=""
Jul 06 22:46:28.005764 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902846    2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 06 22:46:28.005764 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902849    2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 06 22:46:28.005764 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902852    2569 flags.go:64] FLAG: --rotate-certificates="false"
Jul 06 22:46:28.005764 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902855    2569 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 06 22:46:28.005764 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902858    2569 flags.go:64] FLAG: --runonce="false"
Jul 06 22:46:28.005764 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902860    2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 06 22:46:28.005764 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902863    2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 06 22:46:28.005764 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902866    2569 flags.go:64] FLAG: --seccomp-default="false"
Jul 06 22:46:28.005764 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902869    2569 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 06 22:46:28.005764 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902872    2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 06 22:46:28.006595 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902876    2569 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 06 22:46:28.006595 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902878    2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 06 22:46:28.006595 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902881    2569 flags.go:64] FLAG: --storage-driver-password="root"
Jul 06 22:46:28.006595 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902884    2569 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 06 22:46:28.006595 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902887    2569 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 06 22:46:28.006595 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902889    2569 flags.go:64] FLAG: --storage-driver-user="root"
Jul 06 22:46:28.006595 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902892    2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 06 22:46:28.006595 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902895    2569 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 06 22:46:28.006595 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902900    2569 flags.go:64] FLAG: --system-cgroups=""
Jul 06 22:46:28.006595 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902904    2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 06 22:46:28.006595 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902909    2569 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 06 22:46:28.006595 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902911    2569 flags.go:64] FLAG: --tls-cert-file=""
Jul 06 22:46:28.006595 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902915    2569 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 06 22:46:28.006595 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902918    2569 flags.go:64] FLAG: --tls-min-version=""
Jul 06 22:46:28.006595 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902921    2569 flags.go:64] FLAG: --tls-private-key-file=""
Jul 06 22:46:28.006595 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902924    2569 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 06 22:46:28.006595 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902927    2569 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 06 22:46:28.006595 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902930    2569 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 06 22:46:28.006595 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902932    2569 flags.go:64] FLAG: --v="2"
Jul 06 22:46:28.006595 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902936    2569 flags.go:64] FLAG: --version="false"
Jul 06 22:46:28.006595 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902939    2569 flags.go:64] FLAG: --vmodule=""
Jul 06 22:46:28.006595 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902943    2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 06 22:46:28.006595 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.902946    2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 06 22:46:28.006595 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903053    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 22:46:28.006595 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903057    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 22:46:28.120681 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903060    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 22:46:28.120681 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903063    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 22:46:28.120681 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903065    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 22:46:28.120681 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903068    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 22:46:28.120681 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903071    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 22:46:28.120681 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903073    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 22:46:28.120681 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903076    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 22:46:28.120681 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903079    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 22:46:28.120681 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903084    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 22:46:28.120681 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903087    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 22:46:28.120681 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903090    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 22:46:28.120681 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903093    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 22:46:28.120681 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903096    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 22:46:28.120681 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903099    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 22:46:28.120681 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903102    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 22:46:28.120681 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903106    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 22:46:28.120681 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903109    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 22:46:28.120681 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903111    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 22:46:28.120681 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903114    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 22:46:28.140502 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903117    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 22:46:28.140502 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903119    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 22:46:28.140502 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903122    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 22:46:28.140502 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903124    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 22:46:28.140502 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903127    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 22:46:28.140502 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903129    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 22:46:28.140502 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903133    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 22:46:28.140502 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903136    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 22:46:28.140502 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903139    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 22:46:28.140502 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903141    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 22:46:28.140502 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903144    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 22:46:28.140502 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903146    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 22:46:28.140502 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903149    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 22:46:28.140502 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903151    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 22:46:28.140502 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903154    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 22:46:28.140502 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903156    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 22:46:28.140502 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903159    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 22:46:28.140502 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903162    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 22:46:28.140502 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903164    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 22:46:28.140502 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903167    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 22:46:28.141147 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903171    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 22:46:28.141147 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903173    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 22:46:28.141147 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903191    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 22:46:28.141147 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903193    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 22:46:28.141147 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903196    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 22:46:28.141147 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903198    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 22:46:28.141147 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903201    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 22:46:28.141147 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903203    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 22:46:28.141147 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903207    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 22:46:28.141147 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903210    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 22:46:28.141147 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903213    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 22:46:28.141147 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903216    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 22:46:28.141147 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903218    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 22:46:28.141147 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903221    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 22:46:28.141147 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903223    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 22:46:28.141147 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903226    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 22:46:28.141147 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903228    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 22:46:28.141147 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903231    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 22:46:28.141147 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903233    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 22:46:28.141147 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903236    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 22:46:28.141808 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903238    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 22:46:28.141808 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903241    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 22:46:28.141808 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903244    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 22:46:28.141808 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903246    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 22:46:28.141808 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903249    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 22:46:28.141808 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903251    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 22:46:28.141808 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903254    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 22:46:28.141808 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903256    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 22:46:28.141808 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903259    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 22:46:28.141808 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903261    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 22:46:28.141808 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903264    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 22:46:28.141808 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903266    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 22:46:28.141808 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903271    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 22:46:28.141808 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903273    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 22:46:28.141808 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903276    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 22:46:28.141808 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903278    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 22:46:28.141808 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903281    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 22:46:28.141808 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903283    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 22:46:28.141808 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903286    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 22:46:28.141808 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903288    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 22:46:28.142366 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903292    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 22:46:28.142366 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903295    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 22:46:28.142366 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903297    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 22:46:28.142366 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903301    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 22:46:28.142366 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903303    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 22:46:28.142366 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903306    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 22:46:28.142366 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903308    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 22:46:28.142366 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903311    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 22:46:28.142366 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903314    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 22:46:28.142366 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903316    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 22:46:28.142366 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903319    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 22:46:28.142366 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903323    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 22:46:28.142366 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903326    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 22:46:28.142366 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903329    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 22:46:28.142366 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903331    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 22:46:28.142366 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903334    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 22:46:28.142366 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903336    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 22:46:28.142366 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.903339    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 22:46:28.142880 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.904084    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 06 22:46:28.142880 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.910679    2569 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 06 22:46:28.142880 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.910696    2569 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 06 22:46:28.142880 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.911476    2569 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 06 22:46:28.142880 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.911483    2569 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 06 22:46:28.142880 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911562    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 22:46:28.142880 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911567    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 22:46:28.142880 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911572    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 22:46:28.142880 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911577    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 22:46:28.142880 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911580    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 22:46:28.142880 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911583    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 22:46:28.142880 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911586    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 22:46:28.142880 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911588    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 22:46:28.142880 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911591    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 22:46:28.143300 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911593    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 22:46:28.143300 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911596    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 22:46:28.143300 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911599    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 22:46:28.143300 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911602    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 22:46:28.143300 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911604    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 22:46:28.143300 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911607    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 22:46:28.143300 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911609    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 22:46:28.143300 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911612    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 22:46:28.143300 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911615    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 22:46:28.143300 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911617    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 22:46:28.143300 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911620    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 22:46:28.143300 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911622    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 22:46:28.143300 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911625    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 22:46:28.143300 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911627    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 22:46:28.143300 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911630    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 22:46:28.143300 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911633    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 22:46:28.143300 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911636    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 22:46:28.143300 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911638    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 22:46:28.143300 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911641    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 22:46:28.143300 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911643    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 22:46:28.143877 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911652    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 22:46:28.143877 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911655    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 22:46:28.143877 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911657    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 22:46:28.143877 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911660    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 22:46:28.143877 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911662    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 22:46:28.143877 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911665    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 22:46:28.143877 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911667    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 22:46:28.143877 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911670    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 22:46:28.143877 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911672    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 22:46:28.143877 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911675    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 22:46:28.143877 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911677    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 22:46:28.143877 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911680    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 22:46:28.143877 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911682    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 22:46:28.143877 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911685    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 22:46:28.143877 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911687    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 22:46:28.143877 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911690    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 22:46:28.143877 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911692    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 22:46:28.143877 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911695    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 22:46:28.143877 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911699    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 22:46:28.144407 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911703    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 22:46:28.144407 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911706    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 22:46:28.144407 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911709    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 22:46:28.144407 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911712    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 22:46:28.144407 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911715    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 22:46:28.144407 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911718    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 22:46:28.144407 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911721    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 22:46:28.144407 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911723    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 22:46:28.144407 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911726    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 22:46:28.144407 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911729    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 22:46:28.144407 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911731    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 22:46:28.144407 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911734    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 22:46:28.144407 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911736    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 22:46:28.144407 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911739    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 22:46:28.144407 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911747    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 22:46:28.144407 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911749    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 22:46:28.144407 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911752    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 22:46:28.144407 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911754    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 22:46:28.144407 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911757    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 22:46:28.144407 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911759    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 22:46:28.144975 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911762    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 22:46:28.144975 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911764    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 22:46:28.144975 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911767    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 22:46:28.144975 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911770    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 22:46:28.144975 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911772    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 22:46:28.144975 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911775    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 22:46:28.144975 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911778    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 22:46:28.144975 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911780    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 22:46:28.144975 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911783    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 22:46:28.144975 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911785    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 22:46:28.144975 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911788    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 22:46:28.144975 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911790    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 22:46:28.144975 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911792    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 22:46:28.144975 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911797    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 22:46:28.144975 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911800    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 22:46:28.144975 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911802    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 22:46:28.144975 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911805    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 22:46:28.144975 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911807    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 22:46:28.144975 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911810    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 22:46:28.144975 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911813    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 22:46:28.145525 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911815    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 22:46:28.145525 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911818    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 22:46:28.145525 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911820    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 22:46:28.145525 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911823    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 22:46:28.145525 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911825    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 22:46:28.145525 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911830    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 22:46:28.145525 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911833    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 22:46:28.145525 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911841    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 22:46:28.145525 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911843    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 22:46:28.145525 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911846    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 22:46:28.145525 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.911850    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 22:46:28.145525 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.911855    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 06 22:46:28.145525 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912002    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 22:46:28.145525 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912006    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 22:46:28.146025 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912010    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 22:46:28.146025 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912012    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 22:46:28.146025 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912015    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 22:46:28.146025 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912018    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 22:46:28.146025 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912021    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 22:46:28.146025 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912024    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 22:46:28.146025 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912026    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 22:46:28.146025 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912029    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 22:46:28.146025 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912032    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 22:46:28.146025 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912034    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 22:46:28.146025 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912037    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 22:46:28.146025 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912040    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 22:46:28.146025 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912044    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 22:46:28.146025 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912047    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 22:46:28.146025 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912049    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 22:46:28.146025 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912052    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 22:46:28.146025 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912054    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 22:46:28.146025 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912057    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 22:46:28.146025 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912059    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 22:46:28.146025 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912062    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 22:46:28.148754 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912065    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 22:46:28.148754 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912067    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 22:46:28.148754 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912069    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 22:46:28.148754 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912073    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 22:46:28.148754 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912076    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 22:46:28.148754 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912078    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 22:46:28.148754 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912086    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 22:46:28.148754 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912089    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 22:46:28.148754 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912091    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 22:46:28.148754 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912094    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 22:46:28.148754 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912096    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 22:46:28.148754 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912099    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 22:46:28.148754 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912101    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 22:46:28.148754 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912104    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 22:46:28.148754 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912106    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 22:46:28.148754 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912109    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 22:46:28.148754 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912111    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 22:46:28.148754 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912114    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 22:46:28.148754 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912116    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 22:46:28.149372 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912118    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 22:46:28.149372 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912121    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 22:46:28.149372 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912123    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 22:46:28.149372 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912126    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 22:46:28.149372 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912130    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 22:46:28.149372 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912134    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 22:46:28.149372 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912137    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 22:46:28.149372 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912140    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 22:46:28.149372 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912142    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 22:46:28.149372 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912145    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 22:46:28.149372 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912149    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 22:46:28.149372 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912153    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 22:46:28.149372 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912156    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 22:46:28.149372 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912159    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 22:46:28.149372 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912162    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 22:46:28.149372 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912164    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 22:46:28.149372 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912168    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 22:46:28.149372 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912170    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 22:46:28.149372 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912173    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 22:46:28.150009 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912190    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 22:46:28.150009 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912202    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 22:46:28.150009 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912205    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 22:46:28.150009 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912208    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 22:46:28.150009 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912210    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 22:46:28.150009 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912213    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 22:46:28.150009 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912215    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 22:46:28.150009 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912218    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 22:46:28.150009 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912220    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 22:46:28.150009 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912223    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 22:46:28.150009 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912225    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 22:46:28.150009 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912228    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 22:46:28.150009 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912230    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 22:46:28.150009 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912233    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 22:46:28.150009 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912235    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 22:46:28.150009 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912238    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 22:46:28.150009 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912241    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 22:46:28.150009 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912243    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 22:46:28.150009 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912247    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 22:46:28.150635 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912251    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 22:46:28.150635 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912254    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 22:46:28.150635 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912258    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 22:46:28.150635 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912260    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 22:46:28.150635 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912263    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 22:46:28.150635 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912265    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 22:46:28.150635 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912268    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 22:46:28.150635 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912270    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 22:46:28.150635 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912273    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 22:46:28.150635 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912275    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 22:46:28.150635 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912279    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 22:46:28.150635 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912282    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 22:46:28.150635 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912284    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 22:46:28.150635 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912287    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 22:46:28.150635 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912289    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 22:46:28.150635 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912298    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 22:46:28.150635 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912300    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 22:46:28.150635 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912303    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 22:46:28.150635 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912305    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 22:46:28.150635 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:27.912308    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 22:46:28.151480 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.912313    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 06 22:46:28.151480 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.912430    2569 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 06 22:46:28.151480 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.915228    2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 06 22:46:28.151480 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.916592    2569 server.go:1019] "Starting client certificate rotation"
Jul 06 22:46:28.151480 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.916692    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 06 22:46:28.151480 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.916729    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 06 22:46:28.151480 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.919156    2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 06 22:46:28.151480 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.922411    2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 06 22:46:28.151480 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.932519    2569 log.go:25] "Validated CRI v1 runtime API"
Jul 06 22:46:28.151480 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.937065    2569 log.go:25] "Validated CRI v1 image API"
Jul 06 22:46:28.151480 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.937836    2569 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 06 22:46:28.151480 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.940096    2569 fs.go:135] Filesystem UUIDs: map[6d911deb-ebdc-4f3c-b329-86550cb0dcb4:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 91ca08b3-a249-42d3-910d-c0aa32d161a9:/dev/nvme0n1p3]
Jul 06 22:46:28.151892 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.940110    2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 06 22:46:28.152102 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.945377    2569 manager.go:216] Machine: {Timestamp:2026-07-06 22:46:27.944003132 +0000 UTC m=+0.348253028 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098991 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29a1209d539a2cafbffd0d88ee9902 SystemUUID:ec29a120-9d53-9a2c-afbf-fd0d88ee9902 BootID:b512a808-25c6-4038-8f1d-e43647d693dd Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:da:25:6e:60:21 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:da:25:6e:60:21 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f2:b6:54:7e:65:cd Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 06 22:46:28.152102 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.945478    2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 06 22:46:28.152102 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.945546    2569 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 06 22:46:28.152102 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.948827    2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 06 22:46:28.152102 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.948952    2569 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 06 22:46:28.152102 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.948982    2569 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-116.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 06 22:46:28.152102 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.949849    2569 topology_manager.go:138] "Creating topology manager with none policy"
Jul 06 22:46:28.152529 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.949864    2569 container_manager_linux.go:309] "Creating device plugin manager"
Jul 06 22:46:28.152529 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.949878    2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 06 22:46:28.152529 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.950679    2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 06 22:46:28.152529 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.951988    2569 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 06 22:46:28.152529 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.955245    2569 state_mem.go:36] "Initialized new in-memory state store"
Jul 06 22:46:28.152529 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.955377    2569 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 06 22:46:28.152529 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.955428    2569 kubelet.go:486] "Attempting to sync node with API server"
Jul 06 22:46:28.152529 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.955442    2569 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 06 22:46:28.152529 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.955459    2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 06 22:46:28.152529 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.955473    2569 kubelet.go:398] "Adding apiserver pod source"
Jul 06 22:46:28.152529 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.955483    2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 06 22:46:28.152529 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.955836    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 06 22:46:28.152529 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.955855    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 06 22:46:28.152529 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.958307    2569 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 06 22:46:28.152529 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.960197    2569 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 06 22:46:28.152529 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.960224    2569 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 06 22:46:28.152529 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.961558    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 06 22:46:28.152529 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.961571    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 06 22:46:28.152529 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.961577    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 06 22:46:28.152529 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.961582    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 06 22:46:28.152529 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.961593    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 06 22:46:28.152529 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.961599    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 06 22:46:28.152529 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.961607    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 06 22:46:28.152529 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.961612    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 06 22:46:28.152529 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.961618    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 06 22:46:28.152529 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.961623    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 06 22:46:28.152529 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.961631    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 06 22:46:28.152529 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.961638    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 06 22:46:28.152529 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.962517    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 06 22:46:28.152529 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.962524    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 06 22:46:28.152529 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.965494    2569 server.go:1268] "Started kubelet"
Jul 06 22:46:28.152529 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.967687    2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 06 22:46:28.152529 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.967899    2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 06 22:46:28.152529 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.968088    2569 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 06 22:46:28.152529 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.969795    2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-116.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 06 22:46:28.152529 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:27.969957    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-116.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 06 22:46:28.152529 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:27.970149    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 06 22:46:28.152529 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.971452    2569 server.go:310] "Adding debug handlers to kubelet server"
Jul 06 22:46:28.152529 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.972329    2569 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 06 22:46:28.152529 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.972646    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j42hs"
Jul 06 22:46:28.154060 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:27.976100    2569 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 06 22:46:28.154060 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:27.976949    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-116.ec2.internal.18bfd4d4d0eeae08  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-116.ec2.internal,UID:ip-10-0-134-116.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-116.ec2.internal,},FirstTimestamp:2026-07-06 22:46:27.965464072 +0000 UTC m=+0.369713967,LastTimestamp:2026-07-06 22:46:27.965464072 +0000 UTC m=+0.369713967,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-116.ec2.internal,}"
Jul 06 22:46:28.154060 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.979676    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j42hs"
Jul 06 22:46:28.154060 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.980881    2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 06 22:46:28.154060 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.980899    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 06 22:46:28.154060 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:27.980985    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-116.ec2.internal\" not found"
Jul 06 22:46:28.154060 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.981485    2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 06 22:46:28.154060 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.981497    2569 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 06 22:46:28.154060 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.981512    2569 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 06 22:46:28.154060 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.981573    2569 reconstruct.go:97] "Volume reconstruction finished"
Jul 06 22:46:28.154060 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.981581    2569 reconciler.go:29] "Reconciler: start to sync state"
Jul 06 22:46:28.154060 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.981686    2569 factory.go:55] Registering systemd factory
Jul 06 22:46:28.154060 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.981725    2569 factory.go:223] Registration of the systemd container factory successfully
Jul 06 22:46:28.154060 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.981996    2569 factory.go:153] Registering CRI-O factory
Jul 06 22:46:28.154060 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.982011    2569 factory.go:223] Registration of the crio container factory successfully
Jul 06 22:46:28.154060 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.982057    2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 06 22:46:28.154060 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.982077    2569 factory.go:103] Registering Raw factory
Jul 06 22:46:28.154060 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.982273    2569 manager.go:1196] Started watching for new ooms in manager
Jul 06 22:46:28.154060 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.982625    2569 manager.go:349] Starting recovery of all containers
Jul 06 22:46:28.154060 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.991729    2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 22:46:28.154060 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.991779    2569 manager.go:354] Recovery completed
Jul 06 22:46:28.154060 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:27.994285    2569 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-116.ec2.internal\" not found" node="ip-10-0-134-116.ec2.internal"
Jul 06 22:46:28.154060 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.996355    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 22:46:28.154060 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.998022    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-116.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 22:46:28.154060 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.998056    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-116.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 22:46:28.154060 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.998074    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-116.ec2.internal" event="NodeHasSufficientPID"
Jul 06 22:46:28.154060 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.998594    2569 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 06 22:46:28.154060 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.998603    2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 06 22:46:28.154060 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.998617    2569 state_mem.go:36] "Initialized new in-memory state store"
Jul 06 22:46:28.154060 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.998633    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 22:46:28.154060 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.999795    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-116.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 22:46:28.154060 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.999821    2569 policy_none.go:49] "None policy: Start"
Jul 06 22:46:28.154060 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.999823    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-116.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 22:46:28.154060 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.999833    2569 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 06 22:46:28.154060 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.999838    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-116.ec2.internal" event="NodeHasSufficientPID"
Jul 06 22:46:28.155069 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:27.999844    2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 06 22:46:28.155069 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.000785    2569 policy_none.go:47] "Start"
Jul 06 22:46:28.155069 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.059671    2569 manager.go:337] "Starting Device Plugin manager"
Jul 06 22:46:28.155069 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:28.059699    2569 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 06 22:46:28.155069 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.059712    2569 server.go:87] "Starting device plugin registration server"
Jul 06 22:46:28.155069 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.059908    2569 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 06 22:46:28.155069 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.059918    2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 06 22:46:28.155069 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.060033    2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 06 22:46:28.155069 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.060113    2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 06 22:46:28.155069 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.060123    2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 06 22:46:28.155069 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:28.060500    2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 06 22:46:28.155069 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:28.060540    2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-116.ec2.internal\" not found"
Jul 06 22:46:28.278388 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.160598    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 22:46:28.278388 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.161428    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-116.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 22:46:28.278388 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.161453    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-116.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 22:46:28.278388 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.161467    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-116.ec2.internal" event="NodeHasSufficientPID"
Jul 06 22:46:28.278388 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.161496    2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-116.ec2.internal"
Jul 06 22:46:28.278388 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.162492    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 06 22:46:28.278388 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.163693    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 06 22:46:28.278388 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.163733    2569 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 06 22:46:28.278388 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.163757    2569 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 06 22:46:28.278388 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:28.163794    2569 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 06 22:46:28.278388 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.165573    2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 22:46:28.278388 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.169365    2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-116.ec2.internal"
Jul 06 22:46:28.278388 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:28.169384    2569 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-116.ec2.internal\": node \"ip-10-0-134-116.ec2.internal\" not found"
Jul 06 22:46:28.278388 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:28.209122    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-116.ec2.internal\" not found"
Jul 06 22:46:28.278388 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.264204    2569 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-116.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-116.ec2.internal"]
Jul 06 22:46:28.278388 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.264255    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 22:46:28.278388 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.265640    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-116.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 22:46:28.278388 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.265662    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-116.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 22:46:28.278388 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.265676    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-116.ec2.internal" event="NodeHasSufficientPID"
Jul 06 22:46:28.278388 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.266822    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 22:46:28.278388 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.266932    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-116.ec2.internal"
Jul 06 22:46:28.278388 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.266955    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 22:46:28.278388 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.267485    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-116.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 22:46:28.278388 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.267487    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-116.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 22:46:28.278388 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.267529    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-116.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 22:46:28.278388 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.267541    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-116.ec2.internal" event="NodeHasSufficientPID"
Jul 06 22:46:28.278388 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.267509    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-116.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 22:46:28.278388 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.267579    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-116.ec2.internal" event="NodeHasSufficientPID"
Jul 06 22:46:28.278388 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.268662    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-116.ec2.internal"
Jul 06 22:46:28.278388 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.268687    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 22:46:28.278388 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.269335    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-116.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 22:46:28.278388 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.269358    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-116.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 22:46:28.278388 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.269369    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-116.ec2.internal" event="NodeHasSufficientPID"
Jul 06 22:46:28.436481 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.284848    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c10a3b77a8cca715e148a6274757c5ff-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-116.ec2.internal\" (UID: \"c10a3b77a8cca715e148a6274757c5ff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-116.ec2.internal"
Jul 06 22:46:28.436481 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.284867    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c10a3b77a8cca715e148a6274757c5ff-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-116.ec2.internal\" (UID: \"c10a3b77a8cca715e148a6274757c5ff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-116.ec2.internal"
Jul 06 22:46:28.436481 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.284884    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/af57faf3951e9e93eefc26772d49ee48-config\") pod \"kube-apiserver-proxy-ip-10-0-134-116.ec2.internal\" (UID: \"af57faf3951e9e93eefc26772d49ee48\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-116.ec2.internal"
Jul 06 22:46:28.436481 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:28.298698    2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-116.ec2.internal\" not found" node="ip-10-0-134-116.ec2.internal"
Jul 06 22:46:28.436481 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:28.302782    2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-116.ec2.internal\" not found" node="ip-10-0-134-116.ec2.internal"
Jul 06 22:46:28.436481 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:28.309331    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-116.ec2.internal\" not found"
Jul 06 22:46:28.436481 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.385531    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c10a3b77a8cca715e148a6274757c5ff-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-116.ec2.internal\" (UID: \"c10a3b77a8cca715e148a6274757c5ff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-116.ec2.internal"
Jul 06 22:46:28.436481 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.385570    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c10a3b77a8cca715e148a6274757c5ff-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-116.ec2.internal\" (UID: \"c10a3b77a8cca715e148a6274757c5ff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-116.ec2.internal"
Jul 06 22:46:28.436481 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.385585    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/af57faf3951e9e93eefc26772d49ee48-config\") pod \"kube-apiserver-proxy-ip-10-0-134-116.ec2.internal\" (UID: \"af57faf3951e9e93eefc26772d49ee48\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-116.ec2.internal"
Jul 06 22:46:28.436481 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.385635    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c10a3b77a8cca715e148a6274757c5ff-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-116.ec2.internal\" (UID: \"c10a3b77a8cca715e148a6274757c5ff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-116.ec2.internal"
Jul 06 22:46:28.436481 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.385690    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c10a3b77a8cca715e148a6274757c5ff-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-116.ec2.internal\" (UID: \"c10a3b77a8cca715e148a6274757c5ff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-116.ec2.internal"
Jul 06 22:46:28.436481 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.385722    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/af57faf3951e9e93eefc26772d49ee48-config\") pod \"kube-apiserver-proxy-ip-10-0-134-116.ec2.internal\" (UID: \"af57faf3951e9e93eefc26772d49ee48\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-116.ec2.internal"
Jul 06 22:46:28.436481 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:28.409650    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-116.ec2.internal\" not found"
Jul 06 22:46:28.620058 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:28.510951    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-116.ec2.internal\" not found"
Jul 06 22:46:28.620058 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.602171    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-116.ec2.internal"
Jul 06 22:46:28.620058 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.606409    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-116.ec2.internal"
Jul 06 22:46:28.620058 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:28.611873    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-116.ec2.internal\" not found"
Jul 06 22:46:28.761051 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:28.713237    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-116.ec2.internal\" not found"
Jul 06 22:46:28.887783 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:28.813495    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-116.ec2.internal\" not found"
Jul 06 22:46:28.993461 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:28.914226    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-116.ec2.internal\" not found"
Jul 06 22:46:28.993461 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.916550    2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 06 22:46:28.993461 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.916661    2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 06 22:46:28.993461 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.916696    2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 06 22:46:28.993461 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.985456    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 22:41:28 +0000 UTC" deadline="2027-12-06 13:57:52.73555422 +0000 UTC"
Jul 06 22:46:28.993461 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:28.985477    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12423h11m23.750080668s"
Jul 06 22:46:29.082798 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:29.014992    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-116.ec2.internal\" not found"
Jul 06 22:46:29.082798 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.057436    2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 22:46:29.186964 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.088280    2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 22:46:29.186964 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.181937    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-116.ec2.internal"
Jul 06 22:46:29.194096 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.193184    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 06 22:46:29.195195 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.194215    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-116.ec2.internal"
Jul 06 22:46:29.343190 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:29.197980    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc10a3b77a8cca715e148a6274757c5ff.slice/crio-8366442e5802a73970339f16e1a660e1740282d0fe46c64c0354860756a7f30b WatchSource:0}: Error finding container 8366442e5802a73970339f16e1a660e1740282d0fe46c64c0354860756a7f30b: Status 404 returned error can't find the container with id 8366442e5802a73970339f16e1a660e1740282d0fe46c64c0354860756a7f30b
Jul 06 22:46:29.343190 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:29.198751    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf57faf3951e9e93eefc26772d49ee48.slice/crio-5ffbe348e1fe0ae3b61a26ddf2adafab2d190f1332b63a05b717d62f274ba518 WatchSource:0}: Error finding container 5ffbe348e1fe0ae3b61a26ddf2adafab2d190f1332b63a05b717d62f274ba518: Status 404 returned error can't find the container with id 5ffbe348e1fe0ae3b61a26ddf2adafab2d190f1332b63a05b717d62f274ba518
Jul 06 22:46:29.343190 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.204642    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 06 22:46:29.343190 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.206755    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 22:46:29.956552 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.956292    2569 apiserver.go:52] "Watching apiserver"
Jul 06 22:46:29.964987 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.964961    2569 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 06 22:46:29.968446 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.968220    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-mfmlk","kube-system/konnectivity-agent-jtsxh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdmt8","openshift-cluster-node-tuning-operator/tuned-dlqwr","openshift-network-diagnostics/network-check-target-j9csq","openshift-network-operator/iptables-alerter-9sggg","openshift-ovn-kubernetes/ovnkube-node-5bvsb","kube-system/global-pull-secret-syncer-l6fbb","kube-system/kube-apiserver-proxy-ip-10-0-134-116.ec2.internal","openshift-dns/node-resolver-wdchz","openshift-image-registry/node-ca-2zsx8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-116.ec2.internal","openshift-multus/multus-additional-cni-plugins-qgqcr","openshift-multus/multus-hwhh7"]
Jul 06 22:46:29.970130 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.970096    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfmlk"
Jul 06 22:46:29.970250 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:29.970186    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfmlk" podUID="107658f0-a3ae-4883-99dd-ddc62057a8a0"
Jul 06 22:46:29.971315 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.971238    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jtsxh"
Jul 06 22:46:29.972831 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.972809    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdmt8"
Jul 06 22:46:29.974234 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.973934    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-98l7t\""
Jul 06 22:46:29.974234 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.973970    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 06 22:46:29.974234 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.974171    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 06 22:46:29.974711 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.974466    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dlqwr"
Jul 06 22:46:29.975912 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.975329    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 06 22:46:29.975912 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.975427    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 06 22:46:29.975912 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.975588    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 06 22:46:29.975912 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.975705    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-d6skf\""
Jul 06 22:46:29.976279 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.976234    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j9csq"
Jul 06 22:46:29.976609 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:29.976417    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j9csq" podUID="f17325b2-5d96-4911-904b-3a774e4cc8b1"
Jul 06 22:46:29.976943 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.976918    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 06 22:46:29.977174 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.977135    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 06 22:46:29.977288 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.977236    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wzrbj\""
Jul 06 22:46:29.977558 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.977445    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9sggg"
Jul 06 22:46:29.979301 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.979273    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:29.980078 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.979957    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-57t7q\""
Jul 06 22:46:29.980078 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.980019    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 06 22:46:29.980544 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.980413    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 06 22:46:29.980544 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.980432    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 06 22:46:29.980932 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.980911    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6fbb"
Jul 06 22:46:29.981021 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:29.980973    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6fbb" podUID="8cc9b77e-7a6d-4d5b-83a2-8f24359ec453"
Jul 06 22:46:29.982798 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.981363    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 06 22:46:29.982798 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.982300    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 06 22:46:29.982798 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.982593    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 06 22:46:29.982798 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.982600    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jhk4f\""
Jul 06 22:46:29.987230 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.983860    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 06 22:46:29.987230 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.983932    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 06 22:46:29.987230 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.983866    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 06 22:46:29.987230 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.984603    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wdchz"
Jul 06 22:46:29.987505 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.987341    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2zsx8"
Jul 06 22:46:29.988381 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.988357    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 06 22:46:29.988960 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.988743    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bl746\""
Jul 06 22:46:29.988960 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.988770    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 06 22:46:29.988960 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.988774    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 06 22:46:29.990999 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.990658    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qgqcr"
Jul 06 22:46:29.991265 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.991227    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zps9d\""
Jul 06 22:46:29.991615 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.991350    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 06 22:46:29.991615 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.991434    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 06 22:46:29.991615 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.991490    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 06 22:46:29.992720 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.992419    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:29.993274 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.993073    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 06 22:46:29.994193 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.993628    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 06 22:46:29.994193 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.993826    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 06 22:46:29.994193 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.994048    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ll6d9\""
Jul 06 22:46:29.994193 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.994136    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 06 22:46:29.994897 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.994352    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 06 22:46:29.994897 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.994742    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 06 22:46:29.995373 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.995343    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qwg24\""
Jul 06 22:46:29.996084 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:29.996064    2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 06 22:46:30.001856 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.001830    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/04f1bc85-8a29-4e65-bd9d-cd09785fc0dd-etc-sysconfig\") pod \"tuned-dlqwr\" (UID: \"04f1bc85-8a29-4e65-bd9d-cd09785fc0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-dlqwr"
Jul 06 22:46:30.001961 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.001871    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04f1bc85-8a29-4e65-bd9d-cd09785fc0dd-var-lib-kubelet\") pod \"tuned-dlqwr\" (UID: \"04f1bc85-8a29-4e65-bd9d-cd09785fc0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-dlqwr"
Jul 06 22:46:30.001961 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.001899    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-host-run-netns\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.001961 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.001924    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-etc-openvswitch\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.002128 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.001960    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-host-run-ovn-kubernetes\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.002128 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.001986    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/571f407a-249a-4fd9-8c15-53a6541db635-ovn-node-metrics-cert\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.002128 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.002010    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bc58eac9-19c3-4a2c-be0f-2f5211fc5f39-hosts-file\") pod \"node-resolver-wdchz\" (UID: \"bc58eac9-19c3-4a2c-be0f-2f5211fc5f39\") " pod="openshift-dns/node-resolver-wdchz"
Jul 06 22:46:30.002128 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.002032    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/04f1bc85-8a29-4e65-bd9d-cd09785fc0dd-etc-modprobe-d\") pod \"tuned-dlqwr\" (UID: \"04f1bc85-8a29-4e65-bd9d-cd09785fc0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-dlqwr"
Jul 06 22:46:30.002128 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.002057    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5rxc\" (UniqueName: \"kubernetes.io/projected/f17325b2-5d96-4911-904b-3a774e4cc8b1-kube-api-access-h5rxc\") pod \"network-check-target-j9csq\" (UID: \"f17325b2-5d96-4911-904b-3a774e4cc8b1\") " pod="openshift-network-diagnostics/network-check-target-j9csq"
Jul 06 22:46:30.002128 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.002083    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-log-socket\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.002454 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.002131    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.002454 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.002294    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw7bf\" (UniqueName: \"kubernetes.io/projected/107658f0-a3ae-4883-99dd-ddc62057a8a0-kube-api-access-hw7bf\") pod \"network-metrics-daemon-mfmlk\" (UID: \"107658f0-a3ae-4883-99dd-ddc62057a8a0\") " pod="openshift-multus/network-metrics-daemon-mfmlk"
Jul 06 22:46:30.002454 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.002328    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xscbr\" (UniqueName: \"kubernetes.io/projected/bc58eac9-19c3-4a2c-be0f-2f5211fc5f39-kube-api-access-xscbr\") pod \"node-resolver-wdchz\" (UID: \"bc58eac9-19c3-4a2c-be0f-2f5211fc5f39\") " pod="openshift-dns/node-resolver-wdchz"
Jul 06 22:46:30.002454 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.002358    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-run-openvswitch\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.002454 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.002387    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/571f407a-249a-4fd9-8c15-53a6541db635-ovnkube-script-lib\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.002454 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.002413    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgffr\" (UniqueName: \"kubernetes.io/projected/571f407a-249a-4fd9-8c15-53a6541db635-kube-api-access-sgffr\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.002454 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.002439    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dee0d303-a48c-43d1-ac56-f7c2850328b5-iptables-alerter-script\") pod \"iptables-alerter-9sggg\" (UID: \"dee0d303-a48c-43d1-ac56-f7c2850328b5\") " pod="openshift-network-operator/iptables-alerter-9sggg"
Jul 06 22:46:30.002777 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.002462    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04f1bc85-8a29-4e65-bd9d-cd09785fc0dd-etc-kubernetes\") pod \"tuned-dlqwr\" (UID: \"04f1bc85-8a29-4e65-bd9d-cd09785fc0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-dlqwr"
Jul 06 22:46:30.002777 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.002487    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-host-kubelet\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.002777 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.002514    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dee0d303-a48c-43d1-ac56-f7c2850328b5-host-slash\") pod \"iptables-alerter-9sggg\" (UID: \"dee0d303-a48c-43d1-ac56-f7c2850328b5\") " pod="openshift-network-operator/iptables-alerter-9sggg"
Jul 06 22:46:30.002777 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.002538    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc58eac9-19c3-4a2c-be0f-2f5211fc5f39-tmp-dir\") pod \"node-resolver-wdchz\" (UID: \"bc58eac9-19c3-4a2c-be0f-2f5211fc5f39\") " pod="openshift-dns/node-resolver-wdchz"
Jul 06 22:46:30.002777 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.002564    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/04f1bc85-8a29-4e65-bd9d-cd09785fc0dd-etc-sysctl-d\") pod \"tuned-dlqwr\" (UID: \"04f1bc85-8a29-4e65-bd9d-cd09785fc0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-dlqwr"
Jul 06 22:46:30.002777 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.002673    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-run-ovn\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.002777 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.002741    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d762a8a0-e380-4f4b-ac3d-420d6758b924-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cdmt8\" (UID: \"d762a8a0-e380-4f4b-ac3d-420d6758b924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdmt8"
Jul 06 22:46:30.002777 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.002767    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d762a8a0-e380-4f4b-ac3d-420d6758b924-device-dir\") pod \"aws-ebs-csi-driver-node-cdmt8\" (UID: \"d762a8a0-e380-4f4b-ac3d-420d6758b924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdmt8"
Jul 06 22:46:30.003243 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.002790    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d762a8a0-e380-4f4b-ac3d-420d6758b924-sys-fs\") pod \"aws-ebs-csi-driver-node-cdmt8\" (UID: \"d762a8a0-e380-4f4b-ac3d-420d6758b924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdmt8"
Jul 06 22:46:30.003243 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.002816    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-host-cni-bin\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.003243 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.002839    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04f1bc85-8a29-4e65-bd9d-cd09785fc0dd-sys\") pod \"tuned-dlqwr\" (UID: \"04f1bc85-8a29-4e65-bd9d-cd09785fc0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-dlqwr"
Jul 06 22:46:30.003243 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.002863    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d762a8a0-e380-4f4b-ac3d-420d6758b924-registration-dir\") pod \"aws-ebs-csi-driver-node-cdmt8\" (UID: \"d762a8a0-e380-4f4b-ac3d-420d6758b924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdmt8"
Jul 06 22:46:30.003243 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.002907    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4z4j\" (UniqueName: \"kubernetes.io/projected/d762a8a0-e380-4f4b-ac3d-420d6758b924-kube-api-access-s4z4j\") pod \"aws-ebs-csi-driver-node-cdmt8\" (UID: \"d762a8a0-e380-4f4b-ac3d-420d6758b924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdmt8"
Jul 06 22:46:30.003243 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.002938    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/04f1bc85-8a29-4e65-bd9d-cd09785fc0dd-etc-sysctl-conf\") pod \"tuned-dlqwr\" (UID: \"04f1bc85-8a29-4e65-bd9d-cd09785fc0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-dlqwr"
Jul 06 22:46:30.003243 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.003005    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04f1bc85-8a29-4e65-bd9d-cd09785fc0dd-lib-modules\") pod \"tuned-dlqwr\" (UID: \"04f1bc85-8a29-4e65-bd9d-cd09785fc0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-dlqwr"
Jul 06 22:46:30.003243 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.003027    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04f1bc85-8a29-4e65-bd9d-cd09785fc0dd-host\") pod \"tuned-dlqwr\" (UID: \"04f1bc85-8a29-4e65-bd9d-cd09785fc0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-dlqwr"
Jul 06 22:46:30.003243 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.003050    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04f1bc85-8a29-4e65-bd9d-cd09785fc0dd-tmp\") pod \"tuned-dlqwr\" (UID: \"04f1bc85-8a29-4e65-bd9d-cd09785fc0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-dlqwr"
Jul 06 22:46:30.003243 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.003073    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8cc9b77e-7a6d-4d5b-83a2-8f24359ec453-kubelet-config\") pod \"global-pull-secret-syncer-l6fbb\" (UID: \"8cc9b77e-7a6d-4d5b-83a2-8f24359ec453\") " pod="kube-system/global-pull-secret-syncer-l6fbb"
Jul 06 22:46:30.003243 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.003111    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/04f1bc85-8a29-4e65-bd9d-cd09785fc0dd-etc-systemd\") pod \"tuned-dlqwr\" (UID: \"04f1bc85-8a29-4e65-bd9d-cd09785fc0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-dlqwr"
Jul 06 22:46:30.003243 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.003209    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-var-lib-openvswitch\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.003243 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.003245    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8cc9b77e-7a6d-4d5b-83a2-8f24359ec453-dbus\") pod \"global-pull-secret-syncer-l6fbb\" (UID: \"8cc9b77e-7a6d-4d5b-83a2-8f24359ec453\") " pod="kube-system/global-pull-secret-syncer-l6fbb"
Jul 06 22:46:30.003906 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.003292    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8cc9b77e-7a6d-4d5b-83a2-8f24359ec453-original-pull-secret\") pod \"global-pull-secret-syncer-l6fbb\" (UID: \"8cc9b77e-7a6d-4d5b-83a2-8f24359ec453\") " pod="kube-system/global-pull-secret-syncer-l6fbb"
Jul 06 22:46:30.003906 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.003405    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/107658f0-a3ae-4883-99dd-ddc62057a8a0-metrics-certs\") pod \"network-metrics-daemon-mfmlk\" (UID: \"107658f0-a3ae-4883-99dd-ddc62057a8a0\") " pod="openshift-multus/network-metrics-daemon-mfmlk"
Jul 06 22:46:30.003906 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.003433    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-host-slash\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.003906 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.003460    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/04f1bc85-8a29-4e65-bd9d-cd09785fc0dd-etc-tuned\") pod \"tuned-dlqwr\" (UID: \"04f1bc85-8a29-4e65-bd9d-cd09785fc0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-dlqwr"
Jul 06 22:46:30.003906 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.003485    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-node-log\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.003906 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.003564    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/17c837d3-15da-4a3b-97ba-c035de2e4d38-agent-certs\") pod \"konnectivity-agent-jtsxh\" (UID: \"17c837d3-15da-4a3b-97ba-c035de2e4d38\") " pod="kube-system/konnectivity-agent-jtsxh"
Jul 06 22:46:30.003906 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.003589    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-systemd-units\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.003906 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.003688    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/17c837d3-15da-4a3b-97ba-c035de2e4d38-konnectivity-ca\") pod \"konnectivity-agent-jtsxh\" (UID: \"17c837d3-15da-4a3b-97ba-c035de2e4d38\") " pod="kube-system/konnectivity-agent-jtsxh"
Jul 06 22:46:30.003906 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.003725    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d762a8a0-e380-4f4b-ac3d-420d6758b924-socket-dir\") pod \"aws-ebs-csi-driver-node-cdmt8\" (UID: \"d762a8a0-e380-4f4b-ac3d-420d6758b924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdmt8"
Jul 06 22:46:30.003906 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.003750    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/04f1bc85-8a29-4e65-bd9d-cd09785fc0dd-run\") pod \"tuned-dlqwr\" (UID: \"04f1bc85-8a29-4e65-bd9d-cd09785fc0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-dlqwr"
Jul 06 22:46:30.003906 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.003806    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8xrv\" (UniqueName: \"kubernetes.io/projected/04f1bc85-8a29-4e65-bd9d-cd09785fc0dd-kube-api-access-d8xrv\") pod \"tuned-dlqwr\" (UID: \"04f1bc85-8a29-4e65-bd9d-cd09785fc0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-dlqwr"
Jul 06 22:46:30.003906 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.003834    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-run-systemd\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.003906 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.003877    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/571f407a-249a-4fd9-8c15-53a6541db635-env-overrides\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.004475 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.003913    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j8qq\" (UniqueName: \"kubernetes.io/projected/dee0d303-a48c-43d1-ac56-f7c2850328b5-kube-api-access-2j8qq\") pod \"iptables-alerter-9sggg\" (UID: \"dee0d303-a48c-43d1-ac56-f7c2850328b5\") " pod="openshift-network-operator/iptables-alerter-9sggg"
Jul 06 22:46:30.004475 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.003943    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d762a8a0-e380-4f4b-ac3d-420d6758b924-etc-selinux\") pod \"aws-ebs-csi-driver-node-cdmt8\" (UID: \"d762a8a0-e380-4f4b-ac3d-420d6758b924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdmt8"
Jul 06 22:46:30.004475 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.003968    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-host-cni-netd\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.004475 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.003994    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/571f407a-249a-4fd9-8c15-53a6541db635-ovnkube-config\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.019745 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.019720    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rcd67"
Jul 06 22:46:30.025592 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.025567    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rcd67"
Jul 06 22:46:30.082417 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.082391    2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 06 22:46:30.104330 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.104304    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04f1bc85-8a29-4e65-bd9d-cd09785fc0dd-tmp\") pod \"tuned-dlqwr\" (UID: \"04f1bc85-8a29-4e65-bd9d-cd09785fc0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-dlqwr"
Jul 06 22:46:30.104447 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.104345    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8cc9b77e-7a6d-4d5b-83a2-8f24359ec453-kubelet-config\") pod \"global-pull-secret-syncer-l6fbb\" (UID: \"8cc9b77e-7a6d-4d5b-83a2-8f24359ec453\") " pod="kube-system/global-pull-secret-syncer-l6fbb"
Jul 06 22:46:30.104447 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.104378    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-hostroot\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.104447 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.104404    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/04f1bc85-8a29-4e65-bd9d-cd09785fc0dd-etc-systemd\") pod \"tuned-dlqwr\" (UID: \"04f1bc85-8a29-4e65-bd9d-cd09785fc0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-dlqwr"
Jul 06 22:46:30.104447 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.104429    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-var-lib-openvswitch\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.104656 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.104454    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8cc9b77e-7a6d-4d5b-83a2-8f24359ec453-dbus\") pod \"global-pull-secret-syncer-l6fbb\" (UID: \"8cc9b77e-7a6d-4d5b-83a2-8f24359ec453\") " pod="kube-system/global-pull-secret-syncer-l6fbb"
Jul 06 22:46:30.104656 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.104479    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8cc9b77e-7a6d-4d5b-83a2-8f24359ec453-original-pull-secret\") pod \"global-pull-secret-syncer-l6fbb\" (UID: \"8cc9b77e-7a6d-4d5b-83a2-8f24359ec453\") " pod="kube-system/global-pull-secret-syncer-l6fbb"
Jul 06 22:46:30.104656 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.104496    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/107658f0-a3ae-4883-99dd-ddc62057a8a0-metrics-certs\") pod \"network-metrics-daemon-mfmlk\" (UID: \"107658f0-a3ae-4883-99dd-ddc62057a8a0\") " pod="openshift-multus/network-metrics-daemon-mfmlk"
Jul 06 22:46:30.104656 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.104512    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90-cni-binary-copy\") pod \"multus-additional-cni-plugins-qgqcr\" (UID: \"3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90\") " pod="openshift-multus/multus-additional-cni-plugins-qgqcr"
Jul 06 22:46:30.104656 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.104531    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-host-slash\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.104656 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.104557    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6b66febe-a59f-40a7-bff3-d248315b326e-serviceca\") pod \"node-ca-2zsx8\" (UID: \"6b66febe-a59f-40a7-bff3-d248315b326e\") " pod="openshift-image-registry/node-ca-2zsx8"
Jul 06 22:46:30.104656 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.104580    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qgqcr\" (UID: \"3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90\") " pod="openshift-multus/multus-additional-cni-plugins-qgqcr"
Jul 06 22:46:30.104656 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.104621    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-host-run-k8s-cni-cncf-io\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.104656 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.104647    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/04f1bc85-8a29-4e65-bd9d-cd09785fc0dd-etc-tuned\") pod \"tuned-dlqwr\" (UID: \"04f1bc85-8a29-4e65-bd9d-cd09785fc0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-dlqwr"
Jul 06 22:46:30.105136 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.104662    2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 06 22:46:30.105136 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.104721    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/04f1bc85-8a29-4e65-bd9d-cd09785fc0dd-etc-systemd\") pod \"tuned-dlqwr\" (UID: \"04f1bc85-8a29-4e65-bd9d-cd09785fc0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-dlqwr"
Jul 06 22:46:30.105136 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.104671    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8cc9b77e-7a6d-4d5b-83a2-8f24359ec453-kubelet-config\") pod \"global-pull-secret-syncer-l6fbb\" (UID: \"8cc9b77e-7a6d-4d5b-83a2-8f24359ec453\") " pod="kube-system/global-pull-secret-syncer-l6fbb"
Jul 06 22:46:30.105136 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.104762    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-var-lib-openvswitch\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.105136 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.104775    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-node-log\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.105136 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:30.104771    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 22:46:30.105136 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.104784    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8cc9b77e-7a6d-4d5b-83a2-8f24359ec453-dbus\") pod \"global-pull-secret-syncer-l6fbb\" (UID: \"8cc9b77e-7a6d-4d5b-83a2-8f24359ec453\") " pod="kube-system/global-pull-secret-syncer-l6fbb"
Jul 06 22:46:30.105136 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.104831    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-host-slash\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.105136 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:30.104848    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 22:46:30.105136 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.104671    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-node-log\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.105136 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:30.104904    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cc9b77e-7a6d-4d5b-83a2-8f24359ec453-original-pull-secret podName:8cc9b77e-7a6d-4d5b-83a2-8f24359ec453 nodeName:}" failed. No retries permitted until 2026-07-06 22:46:30.604859488 +0000 UTC m=+3.009109394 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8cc9b77e-7a6d-4d5b-83a2-8f24359ec453-original-pull-secret") pod "global-pull-secret-syncer-l6fbb" (UID: "8cc9b77e-7a6d-4d5b-83a2-8f24359ec453") : object "kube-system"/"original-pull-secret" not registered
Jul 06 22:46:30.105136 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:30.104975    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/107658f0-a3ae-4883-99dd-ddc62057a8a0-metrics-certs podName:107658f0-a3ae-4883-99dd-ddc62057a8a0 nodeName:}" failed. No retries permitted until 2026-07-06 22:46:30.604939142 +0000 UTC m=+3.009189039 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/107658f0-a3ae-4883-99dd-ddc62057a8a0-metrics-certs") pod "network-metrics-daemon-mfmlk" (UID: "107658f0-a3ae-4883-99dd-ddc62057a8a0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 22:46:30.105136 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.104999    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/17c837d3-15da-4a3b-97ba-c035de2e4d38-agent-certs\") pod \"konnectivity-agent-jtsxh\" (UID: \"17c837d3-15da-4a3b-97ba-c035de2e4d38\") " pod="kube-system/konnectivity-agent-jtsxh"
Jul 06 22:46:30.105136 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.105029    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b66febe-a59f-40a7-bff3-d248315b326e-host\") pod \"node-ca-2zsx8\" (UID: \"6b66febe-a59f-40a7-bff3-d248315b326e\") " pod="openshift-image-registry/node-ca-2zsx8"
Jul 06 22:46:30.105892 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.105054    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-cnibin\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.105892 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.105079    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-systemd-units\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.105892 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.105119    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/17c837d3-15da-4a3b-97ba-c035de2e4d38-konnectivity-ca\") pod \"konnectivity-agent-jtsxh\" (UID: \"17c837d3-15da-4a3b-97ba-c035de2e4d38\") " pod="kube-system/konnectivity-agent-jtsxh"
Jul 06 22:46:30.105892 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.105150    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d762a8a0-e380-4f4b-ac3d-420d6758b924-socket-dir\") pod \"aws-ebs-csi-driver-node-cdmt8\" (UID: \"d762a8a0-e380-4f4b-ac3d-420d6758b924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdmt8"
Jul 06 22:46:30.105892 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.105147    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-systemd-units\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.105892 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.105181    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qgqcr\" (UID: \"3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90\") " pod="openshift-multus/multus-additional-cni-plugins-qgqcr"
Jul 06 22:46:30.105892 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.105209    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/04f1bc85-8a29-4e65-bd9d-cd09785fc0dd-run\") pod \"tuned-dlqwr\" (UID: \"04f1bc85-8a29-4e65-bd9d-cd09785fc0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-dlqwr"
Jul 06 22:46:30.105892 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.105239    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8xrv\" (UniqueName: \"kubernetes.io/projected/04f1bc85-8a29-4e65-bd9d-cd09785fc0dd-kube-api-access-d8xrv\") pod \"tuned-dlqwr\" (UID: \"04f1bc85-8a29-4e65-bd9d-cd09785fc0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-dlqwr"
Jul 06 22:46:30.105892 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.105290    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d762a8a0-e380-4f4b-ac3d-420d6758b924-socket-dir\") pod \"aws-ebs-csi-driver-node-cdmt8\" (UID: \"d762a8a0-e380-4f4b-ac3d-420d6758b924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdmt8"
Jul 06 22:46:30.105892 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.105293    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-run-systemd\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.105892 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.105336    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/571f407a-249a-4fd9-8c15-53a6541db635-env-overrides\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.105892 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.105340    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-run-systemd\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.105892 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.105365    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j8qq\" (UniqueName: \"kubernetes.io/projected/dee0d303-a48c-43d1-ac56-f7c2850328b5-kube-api-access-2j8qq\") pod \"iptables-alerter-9sggg\" (UID: \"dee0d303-a48c-43d1-ac56-f7c2850328b5\") " pod="openshift-network-operator/iptables-alerter-9sggg"
Jul 06 22:46:30.105892 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.105391    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d762a8a0-e380-4f4b-ac3d-420d6758b924-etc-selinux\") pod \"aws-ebs-csi-driver-node-cdmt8\" (UID: \"d762a8a0-e380-4f4b-ac3d-420d6758b924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdmt8"
Jul 06 22:46:30.106594 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.105411    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/04f1bc85-8a29-4e65-bd9d-cd09785fc0dd-run\") pod \"tuned-dlqwr\" (UID: \"04f1bc85-8a29-4e65-bd9d-cd09785fc0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-dlqwr"
Jul 06 22:46:30.106594 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.105418    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-system-cni-dir\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.106594 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.105445    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-multus-cni-dir\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.106594 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.105469    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-host-cni-netd\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.106594 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.105497    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/571f407a-249a-4fd9-8c15-53a6541db635-ovnkube-config\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.106594 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.105521    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-host-var-lib-kubelet\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.106594 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.105549    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/04f1bc85-8a29-4e65-bd9d-cd09785fc0dd-etc-sysconfig\") pod \"tuned-dlqwr\" (UID: \"04f1bc85-8a29-4e65-bd9d-cd09785fc0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-dlqwr"
Jul 06 22:46:30.106594 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.105574    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04f1bc85-8a29-4e65-bd9d-cd09785fc0dd-var-lib-kubelet\") pod \"tuned-dlqwr\" (UID: \"04f1bc85-8a29-4e65-bd9d-cd09785fc0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-dlqwr"
Jul 06 22:46:30.106594 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.105615    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-host-run-netns\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.106594 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.105642    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-etc-openvswitch\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.106594 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.105667    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-host-run-ovn-kubernetes\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.106594 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.105670    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-host-cni-netd\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.106594 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.105692    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/571f407a-249a-4fd9-8c15-53a6541db635-ovn-node-metrics-cert\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.106594 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.105700    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/17c837d3-15da-4a3b-97ba-c035de2e4d38-konnectivity-ca\") pod \"konnectivity-agent-jtsxh\" (UID: \"17c837d3-15da-4a3b-97ba-c035de2e4d38\") " pod="kube-system/konnectivity-agent-jtsxh"
Jul 06 22:46:30.106594 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.105720    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bc58eac9-19c3-4a2c-be0f-2f5211fc5f39-hosts-file\") pod \"node-resolver-wdchz\" (UID: \"bc58eac9-19c3-4a2c-be0f-2f5211fc5f39\") " pod="openshift-dns/node-resolver-wdchz"
Jul 06 22:46:30.107362 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.105743    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/04f1bc85-8a29-4e65-bd9d-cd09785fc0dd-etc-modprobe-d\") pod \"tuned-dlqwr\" (UID: \"04f1bc85-8a29-4e65-bd9d-cd09785fc0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-dlqwr"
Jul 06 22:46:30.107362 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.105824    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/04f1bc85-8a29-4e65-bd9d-cd09785fc0dd-etc-modprobe-d\") pod \"tuned-dlqwr\" (UID: \"04f1bc85-8a29-4e65-bd9d-cd09785fc0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-dlqwr"
Jul 06 22:46:30.107362 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.105834    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d762a8a0-e380-4f4b-ac3d-420d6758b924-etc-selinux\") pod \"aws-ebs-csi-driver-node-cdmt8\" (UID: \"d762a8a0-e380-4f4b-ac3d-420d6758b924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdmt8"
Jul 06 22:46:30.107362 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.105918    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5rxc\" (UniqueName: \"kubernetes.io/projected/f17325b2-5d96-4911-904b-3a774e4cc8b1-kube-api-access-h5rxc\") pod \"network-check-target-j9csq\" (UID: \"f17325b2-5d96-4911-904b-3a774e4cc8b1\") " pod="openshift-network-diagnostics/network-check-target-j9csq"
Jul 06 22:46:30.107362 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.105948    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-log-socket\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.107362 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.105987    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.107362 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.106042    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw7bf\" (UniqueName: \"kubernetes.io/projected/107658f0-a3ae-4883-99dd-ddc62057a8a0-kube-api-access-hw7bf\") pod \"network-metrics-daemon-mfmlk\" (UID: \"107658f0-a3ae-4883-99dd-ddc62057a8a0\") " pod="openshift-multus/network-metrics-daemon-mfmlk"
Jul 06 22:46:30.107362 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.106098    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/571f407a-249a-4fd9-8c15-53a6541db635-env-overrides\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.107362 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.106107    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xscbr\" (UniqueName: \"kubernetes.io/projected/bc58eac9-19c3-4a2c-be0f-2f5211fc5f39-kube-api-access-xscbr\") pod \"node-resolver-wdchz\" (UID: \"bc58eac9-19c3-4a2c-be0f-2f5211fc5f39\") " pod="openshift-dns/node-resolver-wdchz"
Jul 06 22:46:30.107362 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.106137    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90-cnibin\") pod \"multus-additional-cni-plugins-qgqcr\" (UID: \"3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90\") " pod="openshift-multus/multus-additional-cni-plugins-qgqcr"
Jul 06 22:46:30.107362 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.106169    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-log-socket\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.107362 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.106290    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qgqcr\" (UID: \"3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90\") " pod="openshift-multus/multus-additional-cni-plugins-qgqcr"
Jul 06 22:46:30.107362 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.106326    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/571f407a-249a-4fd9-8c15-53a6541db635-ovnkube-config\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.107362 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.106330    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-run-openvswitch\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.108123 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.106355    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/571f407a-249a-4fd9-8c15-53a6541db635-ovnkube-script-lib\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.108123 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.106400    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/04f1bc85-8a29-4e65-bd9d-cd09785fc0dd-etc-sysconfig\") pod \"tuned-dlqwr\" (UID: \"04f1bc85-8a29-4e65-bd9d-cd09785fc0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-dlqwr"
Jul 06 22:46:30.108123 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.106451    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-host-run-netns\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.108123 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.106497    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-etc-openvswitch\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.108123 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.106534    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-host-run-ovn-kubernetes\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.108123 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.106640    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bc58eac9-19c3-4a2c-be0f-2f5211fc5f39-hosts-file\") pod \"node-resolver-wdchz\" (UID: \"bc58eac9-19c3-4a2c-be0f-2f5211fc5f39\") " pod="openshift-dns/node-resolver-wdchz"
Jul 06 22:46:30.108123 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.106700    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.108123 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.106899    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-run-openvswitch\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.108123 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.106914    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgffr\" (UniqueName: \"kubernetes.io/projected/571f407a-249a-4fd9-8c15-53a6541db635-kube-api-access-sgffr\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.108123 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.106905    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/571f407a-249a-4fd9-8c15-53a6541db635-ovnkube-script-lib\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.108123 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.106949    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04f1bc85-8a29-4e65-bd9d-cd09785fc0dd-var-lib-kubelet\") pod \"tuned-dlqwr\" (UID: \"04f1bc85-8a29-4e65-bd9d-cd09785fc0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-dlqwr"
Jul 06 22:46:30.108123 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.106942    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dee0d303-a48c-43d1-ac56-f7c2850328b5-iptables-alerter-script\") pod \"iptables-alerter-9sggg\" (UID: \"dee0d303-a48c-43d1-ac56-f7c2850328b5\") " pod="openshift-network-operator/iptables-alerter-9sggg"
Jul 06 22:46:30.108123 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.106995    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04f1bc85-8a29-4e65-bd9d-cd09785fc0dd-etc-kubernetes\") pod \"tuned-dlqwr\" (UID: \"04f1bc85-8a29-4e65-bd9d-cd09785fc0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-dlqwr"
Jul 06 22:46:30.108123 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.107029    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmz8r\" (UniqueName: \"kubernetes.io/projected/3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90-kube-api-access-hmz8r\") pod \"multus-additional-cni-plugins-qgqcr\" (UID: \"3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90\") " pod="openshift-multus/multus-additional-cni-plugins-qgqcr"
Jul 06 22:46:30.108811 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.107058    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-multus-socket-dir-parent\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.108811 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.107085    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-host-run-netns\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.108811 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.107105    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04f1bc85-8a29-4e65-bd9d-cd09785fc0dd-etc-kubernetes\") pod \"tuned-dlqwr\" (UID: \"04f1bc85-8a29-4e65-bd9d-cd09785fc0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-dlqwr"
Jul 06 22:46:30.108811 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.107113    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-multus-daemon-config\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.108811 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.107160    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-host-kubelet\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.108811 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.107194    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dee0d303-a48c-43d1-ac56-f7c2850328b5-host-slash\") pod \"iptables-alerter-9sggg\" (UID: \"dee0d303-a48c-43d1-ac56-f7c2850328b5\") " pod="openshift-network-operator/iptables-alerter-9sggg"
Jul 06 22:46:30.108811 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.107240    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dee0d303-a48c-43d1-ac56-f7c2850328b5-host-slash\") pod \"iptables-alerter-9sggg\" (UID: \"dee0d303-a48c-43d1-ac56-f7c2850328b5\") " pod="openshift-network-operator/iptables-alerter-9sggg"
Jul 06 22:46:30.108811 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.107293    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc58eac9-19c3-4a2c-be0f-2f5211fc5f39-tmp-dir\") pod \"node-resolver-wdchz\" (UID: \"bc58eac9-19c3-4a2c-be0f-2f5211fc5f39\") " pod="openshift-dns/node-resolver-wdchz"
Jul 06 22:46:30.108811 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.107306    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-host-kubelet\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.108811 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.107350    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzf7f\" (UniqueName: \"kubernetes.io/projected/6b66febe-a59f-40a7-bff3-d248315b326e-kube-api-access-nzf7f\") pod \"node-ca-2zsx8\" (UID: \"6b66febe-a59f-40a7-bff3-d248315b326e\") " pod="openshift-image-registry/node-ca-2zsx8"
Jul 06 22:46:30.108811 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.107389    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kjg4\" (UniqueName: \"kubernetes.io/projected/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-kube-api-access-5kjg4\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.108811 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.107431    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/04f1bc85-8a29-4e65-bd9d-cd09785fc0dd-etc-sysctl-d\") pod \"tuned-dlqwr\" (UID: \"04f1bc85-8a29-4e65-bd9d-cd09785fc0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-dlqwr"
Jul 06 22:46:30.108811 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.107463    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-run-ovn\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.108811 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.107481    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dee0d303-a48c-43d1-ac56-f7c2850328b5-iptables-alerter-script\") pod \"iptables-alerter-9sggg\" (UID: \"dee0d303-a48c-43d1-ac56-f7c2850328b5\") " pod="openshift-network-operator/iptables-alerter-9sggg"
Jul 06 22:46:30.109574 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.107517    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d762a8a0-e380-4f4b-ac3d-420d6758b924-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cdmt8\" (UID: \"d762a8a0-e380-4f4b-ac3d-420d6758b924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdmt8"
Jul 06 22:46:30.109574 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.107549    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d762a8a0-e380-4f4b-ac3d-420d6758b924-device-dir\") pod \"aws-ebs-csi-driver-node-cdmt8\" (UID: \"d762a8a0-e380-4f4b-ac3d-420d6758b924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdmt8"
Jul 06 22:46:30.109574 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.107551    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-run-ovn\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.109574 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.107590    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc58eac9-19c3-4a2c-be0f-2f5211fc5f39-tmp-dir\") pod \"node-resolver-wdchz\" (UID: \"bc58eac9-19c3-4a2c-be0f-2f5211fc5f39\") " pod="openshift-dns/node-resolver-wdchz"
Jul 06 22:46:30.109574 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.107618    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d762a8a0-e380-4f4b-ac3d-420d6758b924-device-dir\") pod \"aws-ebs-csi-driver-node-cdmt8\" (UID: \"d762a8a0-e380-4f4b-ac3d-420d6758b924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdmt8"
Jul 06 22:46:30.109574 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.107593    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d762a8a0-e380-4f4b-ac3d-420d6758b924-sys-fs\") pod \"aws-ebs-csi-driver-node-cdmt8\" (UID: \"d762a8a0-e380-4f4b-ac3d-420d6758b924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdmt8"
Jul 06 22:46:30.109574 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.107666    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d762a8a0-e380-4f4b-ac3d-420d6758b924-sys-fs\") pod \"aws-ebs-csi-driver-node-cdmt8\" (UID: \"d762a8a0-e380-4f4b-ac3d-420d6758b924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdmt8"
Jul 06 22:46:30.109574 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.107717    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d762a8a0-e380-4f4b-ac3d-420d6758b924-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cdmt8\" (UID: \"d762a8a0-e380-4f4b-ac3d-420d6758b924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdmt8"
Jul 06 22:46:30.109574 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.107741    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-multus-conf-dir\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.109574 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.107778    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-host-cni-bin\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.109574 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.107788    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/04f1bc85-8a29-4e65-bd9d-cd09785fc0dd-etc-sysctl-d\") pod \"tuned-dlqwr\" (UID: \"04f1bc85-8a29-4e65-bd9d-cd09785fc0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-dlqwr"
Jul 06 22:46:30.109574 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.107811    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90-system-cni-dir\") pod \"multus-additional-cni-plugins-qgqcr\" (UID: \"3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90\") " pod="openshift-multus/multus-additional-cni-plugins-qgqcr"
Jul 06 22:46:30.109574 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.107837    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/571f407a-249a-4fd9-8c15-53a6541db635-host-cni-bin\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.109574 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.107889    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90-os-release\") pod \"multus-additional-cni-plugins-qgqcr\" (UID: \"3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90\") " pod="openshift-multus/multus-additional-cni-plugins-qgqcr"
Jul 06 22:46:30.110280 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.107919    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-os-release\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.110280 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.107944    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-cni-binary-copy\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.110280 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.107970    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-host-var-lib-cni-multus\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.110280 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.107995    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-etc-kubernetes\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.110280 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.108025    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04f1bc85-8a29-4e65-bd9d-cd09785fc0dd-sys\") pod \"tuned-dlqwr\" (UID: \"04f1bc85-8a29-4e65-bd9d-cd09785fc0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-dlqwr"
Jul 06 22:46:30.110280 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.108053    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d762a8a0-e380-4f4b-ac3d-420d6758b924-registration-dir\") pod \"aws-ebs-csi-driver-node-cdmt8\" (UID: \"d762a8a0-e380-4f4b-ac3d-420d6758b924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdmt8"
Jul 06 22:46:30.110280 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.108084    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4z4j\" (UniqueName: \"kubernetes.io/projected/d762a8a0-e380-4f4b-ac3d-420d6758b924-kube-api-access-s4z4j\") pod \"aws-ebs-csi-driver-node-cdmt8\" (UID: \"d762a8a0-e380-4f4b-ac3d-420d6758b924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdmt8"
Jul 06 22:46:30.110280 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.108110    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-host-var-lib-cni-bin\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.110280 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.108136    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-host-run-multus-certs\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.110280 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.108164    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/04f1bc85-8a29-4e65-bd9d-cd09785fc0dd-etc-sysctl-conf\") pod \"tuned-dlqwr\" (UID: \"04f1bc85-8a29-4e65-bd9d-cd09785fc0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-dlqwr"
Jul 06 22:46:30.110280 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.108189    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04f1bc85-8a29-4e65-bd9d-cd09785fc0dd-lib-modules\") pod \"tuned-dlqwr\" (UID: \"04f1bc85-8a29-4e65-bd9d-cd09785fc0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-dlqwr"
Jul 06 22:46:30.110280 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.108233    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04f1bc85-8a29-4e65-bd9d-cd09785fc0dd-host\") pod \"tuned-dlqwr\" (UID: \"04f1bc85-8a29-4e65-bd9d-cd09785fc0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-dlqwr"
Jul 06 22:46:30.110280 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.108324    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04f1bc85-8a29-4e65-bd9d-cd09785fc0dd-host\") pod \"tuned-dlqwr\" (UID: \"04f1bc85-8a29-4e65-bd9d-cd09785fc0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-dlqwr"
Jul 06 22:46:30.110280 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.108377    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04f1bc85-8a29-4e65-bd9d-cd09785fc0dd-sys\") pod \"tuned-dlqwr\" (UID: \"04f1bc85-8a29-4e65-bd9d-cd09785fc0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-dlqwr"
Jul 06 22:46:30.110946 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.108433    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d762a8a0-e380-4f4b-ac3d-420d6758b924-registration-dir\") pod \"aws-ebs-csi-driver-node-cdmt8\" (UID: \"d762a8a0-e380-4f4b-ac3d-420d6758b924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdmt8"
Jul 06 22:46:30.110946 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.108720    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/04f1bc85-8a29-4e65-bd9d-cd09785fc0dd-etc-sysctl-conf\") pod \"tuned-dlqwr\" (UID: \"04f1bc85-8a29-4e65-bd9d-cd09785fc0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-dlqwr"
Jul 06 22:46:30.110946 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.108723    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04f1bc85-8a29-4e65-bd9d-cd09785fc0dd-lib-modules\") pod \"tuned-dlqwr\" (UID: \"04f1bc85-8a29-4e65-bd9d-cd09785fc0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-dlqwr"
Jul 06 22:46:30.110946 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.109187    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04f1bc85-8a29-4e65-bd9d-cd09785fc0dd-tmp\") pod \"tuned-dlqwr\" (UID: \"04f1bc85-8a29-4e65-bd9d-cd09785fc0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-dlqwr"
Jul 06 22:46:30.110946 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.109226    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/17c837d3-15da-4a3b-97ba-c035de2e4d38-agent-certs\") pod \"konnectivity-agent-jtsxh\" (UID: \"17c837d3-15da-4a3b-97ba-c035de2e4d38\") " pod="kube-system/konnectivity-agent-jtsxh"
Jul 06 22:46:30.111199 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.111112    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/571f407a-249a-4fd9-8c15-53a6541db635-ovn-node-metrics-cert\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.112148 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:30.112129    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 22:46:30.112241 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:30.112157    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 22:46:30.112241 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:30.112172    2569 projected.go:196] Error preparing data for projected volume kube-api-access-h5rxc for pod openshift-network-diagnostics/network-check-target-j9csq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 22:46:30.112241 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:30.112225    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f17325b2-5d96-4911-904b-3a774e4cc8b1-kube-api-access-h5rxc podName:f17325b2-5d96-4911-904b-3a774e4cc8b1 nodeName:}" failed. No retries permitted until 2026-07-06 22:46:30.612208095 +0000 UTC m=+3.016457999 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-h5rxc" (UniqueName: "kubernetes.io/projected/f17325b2-5d96-4911-904b-3a774e4cc8b1-kube-api-access-h5rxc") pod "network-check-target-j9csq" (UID: "f17325b2-5d96-4911-904b-3a774e4cc8b1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 22:46:30.112575 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.112552    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/04f1bc85-8a29-4e65-bd9d-cd09785fc0dd-etc-tuned\") pod \"tuned-dlqwr\" (UID: \"04f1bc85-8a29-4e65-bd9d-cd09785fc0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-dlqwr"
Jul 06 22:46:30.115814 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.115753    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j8qq\" (UniqueName: \"kubernetes.io/projected/dee0d303-a48c-43d1-ac56-f7c2850328b5-kube-api-access-2j8qq\") pod \"iptables-alerter-9sggg\" (UID: \"dee0d303-a48c-43d1-ac56-f7c2850328b5\") " pod="openshift-network-operator/iptables-alerter-9sggg"
Jul 06 22:46:30.116353 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.116308    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgffr\" (UniqueName: \"kubernetes.io/projected/571f407a-249a-4fd9-8c15-53a6541db635-kube-api-access-sgffr\") pod \"ovnkube-node-5bvsb\" (UID: \"571f407a-249a-4fd9-8c15-53a6541db635\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.117070 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.117007    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw7bf\" (UniqueName: \"kubernetes.io/projected/107658f0-a3ae-4883-99dd-ddc62057a8a0-kube-api-access-hw7bf\") pod \"network-metrics-daemon-mfmlk\" (UID: \"107658f0-a3ae-4883-99dd-ddc62057a8a0\") " pod="openshift-multus/network-metrics-daemon-mfmlk"
Jul 06 22:46:30.119477 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.119401    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4z4j\" (UniqueName: \"kubernetes.io/projected/d762a8a0-e380-4f4b-ac3d-420d6758b924-kube-api-access-s4z4j\") pod \"aws-ebs-csi-driver-node-cdmt8\" (UID: \"d762a8a0-e380-4f4b-ac3d-420d6758b924\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdmt8"
Jul 06 22:46:30.120211 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.120186    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8xrv\" (UniqueName: \"kubernetes.io/projected/04f1bc85-8a29-4e65-bd9d-cd09785fc0dd-kube-api-access-d8xrv\") pod \"tuned-dlqwr\" (UID: \"04f1bc85-8a29-4e65-bd9d-cd09785fc0dd\") " pod="openshift-cluster-node-tuning-operator/tuned-dlqwr"
Jul 06 22:46:30.121511 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.121487    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xscbr\" (UniqueName: \"kubernetes.io/projected/bc58eac9-19c3-4a2c-be0f-2f5211fc5f39-kube-api-access-xscbr\") pod \"node-resolver-wdchz\" (UID: \"bc58eac9-19c3-4a2c-be0f-2f5211fc5f39\") " pod="openshift-dns/node-resolver-wdchz"
Jul 06 22:46:30.168920 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.168871    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-116.ec2.internal" event={"ID":"c10a3b77a8cca715e148a6274757c5ff","Type":"ContainerStarted","Data":"8366442e5802a73970339f16e1a660e1740282d0fe46c64c0354860756a7f30b"}
Jul 06 22:46:30.170171 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.170123    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-116.ec2.internal" event={"ID":"af57faf3951e9e93eefc26772d49ee48","Type":"ContainerStarted","Data":"5ffbe348e1fe0ae3b61a26ddf2adafab2d190f1332b63a05b717d62f274ba518"}
Jul 06 22:46:30.209349 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.209321    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qgqcr\" (UID: \"3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90\") " pod="openshift-multus/multus-additional-cni-plugins-qgqcr"
Jul 06 22:46:30.209465 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.209362    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmz8r\" (UniqueName: \"kubernetes.io/projected/3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90-kube-api-access-hmz8r\") pod \"multus-additional-cni-plugins-qgqcr\" (UID: \"3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90\") " pod="openshift-multus/multus-additional-cni-plugins-qgqcr"
Jul 06 22:46:30.209465 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.209388    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-multus-socket-dir-parent\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.209465 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.209434    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-host-run-netns\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.209636 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.209460    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-multus-daemon-config\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.209636 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.209511    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzf7f\" (UniqueName: \"kubernetes.io/projected/6b66febe-a59f-40a7-bff3-d248315b326e-kube-api-access-nzf7f\") pod \"node-ca-2zsx8\" (UID: \"6b66febe-a59f-40a7-bff3-d248315b326e\") " pod="openshift-image-registry/node-ca-2zsx8"
Jul 06 22:46:30.209636 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.209537    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kjg4\" (UniqueName: \"kubernetes.io/projected/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-kube-api-access-5kjg4\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.209636 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.209550    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qgqcr\" (UID: \"3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90\") " pod="openshift-multus/multus-additional-cni-plugins-qgqcr"
Jul 06 22:46:30.209636 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.209560    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-multus-socket-dir-parent\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.209636 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.209570    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-multus-conf-dir\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.209636 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.209618    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-multus-conf-dir\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.209636 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.209624    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90-system-cni-dir\") pod \"multus-additional-cni-plugins-qgqcr\" (UID: \"3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90\") " pod="openshift-multus/multus-additional-cni-plugins-qgqcr"
Jul 06 22:46:30.210016 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.209668    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90-system-cni-dir\") pod \"multus-additional-cni-plugins-qgqcr\" (UID: \"3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90\") " pod="openshift-multus/multus-additional-cni-plugins-qgqcr"
Jul 06 22:46:30.210016 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.209617    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-host-run-netns\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.210016 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.209723    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90-os-release\") pod \"multus-additional-cni-plugins-qgqcr\" (UID: \"3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90\") " pod="openshift-multus/multus-additional-cni-plugins-qgqcr"
Jul 06 22:46:30.210016 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.209755    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-os-release\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.210016 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.209782    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-cni-binary-copy\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.210016 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.209808    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-host-var-lib-cni-multus\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.210016 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.209813    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90-os-release\") pod \"multus-additional-cni-plugins-qgqcr\" (UID: \"3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90\") " pod="openshift-multus/multus-additional-cni-plugins-qgqcr"
Jul 06 22:46:30.210016 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.209837    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-etc-kubernetes\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.210016 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.209868    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-host-var-lib-cni-bin\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.210016 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.209896    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-host-run-multus-certs\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.210016 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.209918    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-os-release\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.210016 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.209927    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-hostroot\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.210016 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.209969    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-host-var-lib-cni-bin\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.210016 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.209982    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90-cni-binary-copy\") pod \"multus-additional-cni-plugins-qgqcr\" (UID: \"3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90\") " pod="openshift-multus/multus-additional-cni-plugins-qgqcr"
Jul 06 22:46:30.210016 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.210007    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-host-var-lib-cni-multus\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.210757 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.210010    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6b66febe-a59f-40a7-bff3-d248315b326e-serviceca\") pod \"node-ca-2zsx8\" (UID: \"6b66febe-a59f-40a7-bff3-d248315b326e\") " pod="openshift-image-registry/node-ca-2zsx8"
Jul 06 22:46:30.210757 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.210046    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qgqcr\" (UID: \"3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90\") " pod="openshift-multus/multus-additional-cni-plugins-qgqcr"
Jul 06 22:46:30.210757 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.210063    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-host-run-multus-certs\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.210757 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.210074    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-host-run-k8s-cni-cncf-io\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.210757 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.210103    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b66febe-a59f-40a7-bff3-d248315b326e-host\") pod \"node-ca-2zsx8\" (UID: \"6b66febe-a59f-40a7-bff3-d248315b326e\") " pod="openshift-image-registry/node-ca-2zsx8"
Jul 06 22:46:30.210757 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.210108    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-hostroot\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.210757 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.210127    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-cnibin\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.210757 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.210156    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qgqcr\" (UID: \"3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90\") " pod="openshift-multus/multus-additional-cni-plugins-qgqcr"
Jul 06 22:46:30.210757 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.210190    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-system-cni-dir\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.210757 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.210197    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-multus-daemon-config\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.210757 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.210213    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-multus-cni-dir\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.210757 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.210240    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-cni-binary-copy\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.210757 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.210251    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-host-var-lib-kubelet\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.210757 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.210288    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b66febe-a59f-40a7-bff3-d248315b326e-host\") pod \"node-ca-2zsx8\" (UID: \"6b66febe-a59f-40a7-bff3-d248315b326e\") " pod="openshift-image-registry/node-ca-2zsx8"
Jul 06 22:46:30.210757 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.210296    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-multus-cni-dir\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.211433 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.210321    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-etc-kubernetes\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.211433 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.210359    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-cnibin\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.211433 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.210330    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-host-var-lib-kubelet\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.211433 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.210414    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90-cnibin\") pod \"multus-additional-cni-plugins-qgqcr\" (UID: \"3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90\") " pod="openshift-multus/multus-additional-cni-plugins-qgqcr"
Jul 06 22:46:30.211433 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.210502    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90-cnibin\") pod \"multus-additional-cni-plugins-qgqcr\" (UID: \"3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90\") " pod="openshift-multus/multus-additional-cni-plugins-qgqcr"
Jul 06 22:46:30.211433 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.210556    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-system-cni-dir\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.211433 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.210620    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-host-run-k8s-cni-cncf-io\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.211433 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.210684    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6b66febe-a59f-40a7-bff3-d248315b326e-serviceca\") pod \"node-ca-2zsx8\" (UID: \"6b66febe-a59f-40a7-bff3-d248315b326e\") " pod="openshift-image-registry/node-ca-2zsx8"
Jul 06 22:46:30.211433 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.210784    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90-cni-binary-copy\") pod \"multus-additional-cni-plugins-qgqcr\" (UID: \"3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90\") " pod="openshift-multus/multus-additional-cni-plugins-qgqcr"
Jul 06 22:46:30.211433 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.210991    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qgqcr\" (UID: \"3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90\") " pod="openshift-multus/multus-additional-cni-plugins-qgqcr"
Jul 06 22:46:30.211433 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.211090    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qgqcr\" (UID: \"3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90\") " pod="openshift-multus/multus-additional-cni-plugins-qgqcr"
Jul 06 22:46:30.217736 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.217715    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzf7f\" (UniqueName: \"kubernetes.io/projected/6b66febe-a59f-40a7-bff3-d248315b326e-kube-api-access-nzf7f\") pod \"node-ca-2zsx8\" (UID: \"6b66febe-a59f-40a7-bff3-d248315b326e\") " pod="openshift-image-registry/node-ca-2zsx8"
Jul 06 22:46:30.217869 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.217852    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmz8r\" (UniqueName: \"kubernetes.io/projected/3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90-kube-api-access-hmz8r\") pod \"multus-additional-cni-plugins-qgqcr\" (UID: \"3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90\") " pod="openshift-multus/multus-additional-cni-plugins-qgqcr"
Jul 06 22:46:30.218308 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.218291    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kjg4\" (UniqueName: \"kubernetes.io/projected/345357e4-a764-47c9-8ef9-acfc5b4e8cf6-kube-api-access-5kjg4\") pod \"multus-hwhh7\" (UID: \"345357e4-a764-47c9-8ef9-acfc5b4e8cf6\") " pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.291434 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.291361    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jtsxh"
Jul 06 22:46:30.297991 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.297971    2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 22:46:30.303348 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.303329    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9sggg"
Jul 06 22:46:30.313746 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.313725    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdmt8"
Jul 06 22:46:30.319981 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.319962    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dlqwr"
Jul 06 22:46:30.334901 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.334881    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:30.343519 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.343500    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wdchz"
Jul 06 22:46:30.351544 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.351521    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2zsx8"
Jul 06 22:46:30.359534 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.359511    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qgqcr"
Jul 06 22:46:30.365569 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.365553    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hwhh7"
Jul 06 22:46:30.398046 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.398028    2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 22:46:30.614422 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.614343    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5rxc\" (UniqueName: \"kubernetes.io/projected/f17325b2-5d96-4911-904b-3a774e4cc8b1-kube-api-access-h5rxc\") pod \"network-check-target-j9csq\" (UID: \"f17325b2-5d96-4911-904b-3a774e4cc8b1\") " pod="openshift-network-diagnostics/network-check-target-j9csq"
Jul 06 22:46:30.614422 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.614410    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8cc9b77e-7a6d-4d5b-83a2-8f24359ec453-original-pull-secret\") pod \"global-pull-secret-syncer-l6fbb\" (UID: \"8cc9b77e-7a6d-4d5b-83a2-8f24359ec453\") " pod="kube-system/global-pull-secret-syncer-l6fbb"
Jul 06 22:46:30.614635 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:30.614441    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/107658f0-a3ae-4883-99dd-ddc62057a8a0-metrics-certs\") pod \"network-metrics-daemon-mfmlk\" (UID: \"107658f0-a3ae-4883-99dd-ddc62057a8a0\") " pod="openshift-multus/network-metrics-daemon-mfmlk"
Jul 06 22:46:30.614635 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:30.614522    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 22:46:30.614635 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:30.614541    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 22:46:30.614635 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:30.614559    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 22:46:30.614635 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:30.614583    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 22:46:30.614635 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:30.614590    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cc9b77e-7a6d-4d5b-83a2-8f24359ec453-original-pull-secret podName:8cc9b77e-7a6d-4d5b-83a2-8f24359ec453 nodeName:}" failed. No retries permitted until 2026-07-06 22:46:31.614570453 +0000 UTC m=+4.018820349 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8cc9b77e-7a6d-4d5b-83a2-8f24359ec453-original-pull-secret") pod "global-pull-secret-syncer-l6fbb" (UID: "8cc9b77e-7a6d-4d5b-83a2-8f24359ec453") : object "kube-system"/"original-pull-secret" not registered
Jul 06 22:46:30.614635 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:30.614594    2569 projected.go:196] Error preparing data for projected volume kube-api-access-h5rxc for pod openshift-network-diagnostics/network-check-target-j9csq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 22:46:30.614635 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:30.614605    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/107658f0-a3ae-4883-99dd-ddc62057a8a0-metrics-certs podName:107658f0-a3ae-4883-99dd-ddc62057a8a0 nodeName:}" failed. No retries permitted until 2026-07-06 22:46:31.614598244 +0000 UTC m=+4.018848133 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/107658f0-a3ae-4883-99dd-ddc62057a8a0-metrics-certs") pod "network-metrics-daemon-mfmlk" (UID: "107658f0-a3ae-4883-99dd-ddc62057a8a0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 22:46:30.614949 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:30.614654    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f17325b2-5d96-4911-904b-3a774e4cc8b1-kube-api-access-h5rxc podName:f17325b2-5d96-4911-904b-3a774e4cc8b1 nodeName:}" failed. No retries permitted until 2026-07-06 22:46:31.614638291 +0000 UTC m=+4.018888175 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-h5rxc" (UniqueName: "kubernetes.io/projected/f17325b2-5d96-4911-904b-3a774e4cc8b1-kube-api-access-h5rxc") pod "network-check-target-j9csq" (UID: "f17325b2-5d96-4911-904b-3a774e4cc8b1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 22:46:31.027969 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:31.027853    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 22:41:30 +0000 UTC" deadline="2027-12-17 03:06:36.460986184 +0000 UTC"
Jul 06 22:46:31.027969 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:31.027919    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12676h20m5.433070275s"
Jul 06 22:46:31.164669 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:31.164637    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfmlk"
Jul 06 22:46:31.164852 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:31.164798    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfmlk" podUID="107658f0-a3ae-4883-99dd-ddc62057a8a0"
Jul 06 22:46:31.280438 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:31.280412    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04f1bc85_8a29_4e65_bd9d_cd09785fc0dd.slice/crio-2d4c59b31c7a71d0b4523cd35a3df00085f4db867cf8a99b255963376ede8467 WatchSource:0}: Error finding container 2d4c59b31c7a71d0b4523cd35a3df00085f4db867cf8a99b255963376ede8467: Status 404 returned error can't find the container with id 2d4c59b31c7a71d0b4523cd35a3df00085f4db867cf8a99b255963376ede8467
Jul 06 22:46:31.281794 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:31.281763    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd762a8a0_e380_4f4b_ac3d_420d6758b924.slice/crio-779de283d3ef5788f8d02c1a54641ea0c5cd49a074772d0a2e99e0c5e317e31a WatchSource:0}: Error finding container 779de283d3ef5788f8d02c1a54641ea0c5cd49a074772d0a2e99e0c5e317e31a: Status 404 returned error can't find the container with id 779de283d3ef5788f8d02c1a54641ea0c5cd49a074772d0a2e99e0c5e317e31a
Jul 06 22:46:31.283008 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:31.282824    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b66febe_a59f_40a7_bff3_d248315b326e.slice/crio-9d7e4914f8d9576e087198b2a326f571c83e4fc2c2f1b09ef43e1a9049171421 WatchSource:0}: Error finding container 9d7e4914f8d9576e087198b2a326f571c83e4fc2c2f1b09ef43e1a9049171421: Status 404 returned error can't find the container with id 9d7e4914f8d9576e087198b2a326f571c83e4fc2c2f1b09ef43e1a9049171421
Jul 06 22:46:31.283522 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:31.283496    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e2c05e5_ad3c_4568_99a3_3cfb9b42cd90.slice/crio-a83aab842fde2a07a0357b2eade03b75a83d25f79311aac5324db433be501a3d WatchSource:0}: Error finding container a83aab842fde2a07a0357b2eade03b75a83d25f79311aac5324db433be501a3d: Status 404 returned error can't find the container with id a83aab842fde2a07a0357b2eade03b75a83d25f79311aac5324db433be501a3d
Jul 06 22:46:31.285045 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:31.285016    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddee0d303_a48c_43d1_ac56_f7c2850328b5.slice/crio-b1af34794ae859e9892fd58fdada81be2af7436315c34337a0bbee6631ba770f WatchSource:0}: Error finding container b1af34794ae859e9892fd58fdada81be2af7436315c34337a0bbee6631ba770f: Status 404 returned error can't find the container with id b1af34794ae859e9892fd58fdada81be2af7436315c34337a0bbee6631ba770f
Jul 06 22:46:31.288114 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:31.288095    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc58eac9_19c3_4a2c_be0f_2f5211fc5f39.slice/crio-b6e04b17de62c1a48a73b91c86ff48608acb5abfcdc4fde227fc69bd00c8bc34 WatchSource:0}: Error finding container b6e04b17de62c1a48a73b91c86ff48608acb5abfcdc4fde227fc69bd00c8bc34: Status 404 returned error can't find the container with id b6e04b17de62c1a48a73b91c86ff48608acb5abfcdc4fde227fc69bd00c8bc34
Jul 06 22:46:31.289915 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:31.289576    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17c837d3_15da_4a3b_97ba_c035de2e4d38.slice/crio-07080d6421850c164463159e18fa0dc9cb5adb12c29b84959dc35c32cdd1fa81 WatchSource:0}: Error finding container 07080d6421850c164463159e18fa0dc9cb5adb12c29b84959dc35c32cdd1fa81: Status 404 returned error can't find the container with id 07080d6421850c164463159e18fa0dc9cb5adb12c29b84959dc35c32cdd1fa81
Jul 06 22:46:31.290555 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:31.290509    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod345357e4_a764_47c9_8ef9_acfc5b4e8cf6.slice/crio-6f2b42ee15a3c76d366f44dc2a3f374ba9dcbb84b697811948eef22cbbe69d1e WatchSource:0}: Error finding container 6f2b42ee15a3c76d366f44dc2a3f374ba9dcbb84b697811948eef22cbbe69d1e: Status 404 returned error can't find the container with id 6f2b42ee15a3c76d366f44dc2a3f374ba9dcbb84b697811948eef22cbbe69d1e
Jul 06 22:46:31.292487 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:46:31.292176    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod571f407a_249a_4fd9_8c15_53a6541db635.slice/crio-eac69e771cc18ddd3b93dc8d9014f4878c742178e1d9d531d9cf137fd04d7a41 WatchSource:0}: Error finding container eac69e771cc18ddd3b93dc8d9014f4878c742178e1d9d531d9cf137fd04d7a41: Status 404 returned error can't find the container with id eac69e771cc18ddd3b93dc8d9014f4878c742178e1d9d531d9cf137fd04d7a41
Jul 06 22:46:31.621727 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:31.621700    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8cc9b77e-7a6d-4d5b-83a2-8f24359ec453-original-pull-secret\") pod \"global-pull-secret-syncer-l6fbb\" (UID: \"8cc9b77e-7a6d-4d5b-83a2-8f24359ec453\") " pod="kube-system/global-pull-secret-syncer-l6fbb"
Jul 06 22:46:31.621827 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:31.621802    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/107658f0-a3ae-4883-99dd-ddc62057a8a0-metrics-certs\") pod \"network-metrics-daemon-mfmlk\" (UID: \"107658f0-a3ae-4883-99dd-ddc62057a8a0\") " pod="openshift-multus/network-metrics-daemon-mfmlk"
Jul 06 22:46:31.621882 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:31.621854    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5rxc\" (UniqueName: \"kubernetes.io/projected/f17325b2-5d96-4911-904b-3a774e4cc8b1-kube-api-access-h5rxc\") pod \"network-check-target-j9csq\" (UID: \"f17325b2-5d96-4911-904b-3a774e4cc8b1\") " pod="openshift-network-diagnostics/network-check-target-j9csq"
Jul 06 22:46:31.621931 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:31.621810    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 22:46:31.621997 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:31.621935    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 22:46:31.621997 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:31.621950    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cc9b77e-7a6d-4d5b-83a2-8f24359ec453-original-pull-secret podName:8cc9b77e-7a6d-4d5b-83a2-8f24359ec453 nodeName:}" failed. No retries permitted until 2026-07-06 22:46:33.621930232 +0000 UTC m=+6.026180137 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8cc9b77e-7a6d-4d5b-83a2-8f24359ec453-original-pull-secret") pod "global-pull-secret-syncer-l6fbb" (UID: "8cc9b77e-7a6d-4d5b-83a2-8f24359ec453") : object "kube-system"/"original-pull-secret" not registered
Jul 06 22:46:31.621997 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:31.621967    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 22:46:31.621997 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:31.621985    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 22:46:31.621997 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:31.621997    2569 projected.go:196] Error preparing data for projected volume kube-api-access-h5rxc for pod openshift-network-diagnostics/network-check-target-j9csq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 22:46:31.621997 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:31.621999    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/107658f0-a3ae-4883-99dd-ddc62057a8a0-metrics-certs podName:107658f0-a3ae-4883-99dd-ddc62057a8a0 nodeName:}" failed. No retries permitted until 2026-07-06 22:46:33.62197881 +0000 UTC m=+6.026228708 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/107658f0-a3ae-4883-99dd-ddc62057a8a0-metrics-certs") pod "network-metrics-daemon-mfmlk" (UID: "107658f0-a3ae-4883-99dd-ddc62057a8a0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 22:46:31.622199 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:31.622034    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f17325b2-5d96-4911-904b-3a774e4cc8b1-kube-api-access-h5rxc podName:f17325b2-5d96-4911-904b-3a774e4cc8b1 nodeName:}" failed. No retries permitted until 2026-07-06 22:46:33.622023029 +0000 UTC m=+6.026272934 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-h5rxc" (UniqueName: "kubernetes.io/projected/f17325b2-5d96-4911-904b-3a774e4cc8b1-kube-api-access-h5rxc") pod "network-check-target-j9csq" (UID: "f17325b2-5d96-4911-904b-3a774e4cc8b1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 22:46:32.028719 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:32.028607    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 22:41:30 +0000 UTC" deadline="2028-01-25 00:10:39.304878334 +0000 UTC"
Jul 06 22:46:32.028719 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:32.028644    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13609h24m7.276239018s"
Jul 06 22:46:32.165199 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:32.164600    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j9csq"
Jul 06 22:46:32.165199 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:32.164775    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j9csq" podUID="f17325b2-5d96-4911-904b-3a774e4cc8b1"
Jul 06 22:46:32.165766 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:32.165584    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6fbb"
Jul 06 22:46:32.165766 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:32.165709    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6fbb" podUID="8cc9b77e-7a6d-4d5b-83a2-8f24359ec453"
Jul 06 22:46:32.193058 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:32.192381    2569 generic.go:356] "Generic (PLEG): container finished" podID="c10a3b77a8cca715e148a6274757c5ff" containerID="8370118c1d06036eaf6c27dae50a8d7025f99f0dbb1e7b2ef61bcad9a51d40ad" exitCode=0
Jul 06 22:46:32.193058 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:32.192473    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-116.ec2.internal" event={"ID":"c10a3b77a8cca715e148a6274757c5ff","Type":"ContainerDied","Data":"8370118c1d06036eaf6c27dae50a8d7025f99f0dbb1e7b2ef61bcad9a51d40ad"}
Jul 06 22:46:32.227404 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:32.227371    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb" event={"ID":"571f407a-249a-4fd9-8c15-53a6541db635","Type":"ContainerStarted","Data":"eac69e771cc18ddd3b93dc8d9014f4878c742178e1d9d531d9cf137fd04d7a41"}
Jul 06 22:46:32.236748 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:32.236668    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hwhh7" event={"ID":"345357e4-a764-47c9-8ef9-acfc5b4e8cf6","Type":"ContainerStarted","Data":"6f2b42ee15a3c76d366f44dc2a3f374ba9dcbb84b697811948eef22cbbe69d1e"}
Jul 06 22:46:32.248542 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:32.248504    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2zsx8" event={"ID":"6b66febe-a59f-40a7-bff3-d248315b326e","Type":"ContainerStarted","Data":"9d7e4914f8d9576e087198b2a326f571c83e4fc2c2f1b09ef43e1a9049171421"}
Jul 06 22:46:32.252004 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:32.251975    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdmt8" event={"ID":"d762a8a0-e380-4f4b-ac3d-420d6758b924","Type":"ContainerStarted","Data":"779de283d3ef5788f8d02c1a54641ea0c5cd49a074772d0a2e99e0c5e317e31a"}
Jul 06 22:46:32.283434 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:32.283370    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-116.ec2.internal" event={"ID":"af57faf3951e9e93eefc26772d49ee48","Type":"ContainerStarted","Data":"53c40282aa25c34dda7c18643a2735428e0330dca0646d4454b699ed9bc2c405"}
Jul 06 22:46:32.297170 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:32.297140    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jtsxh" event={"ID":"17c837d3-15da-4a3b-97ba-c035de2e4d38","Type":"ContainerStarted","Data":"07080d6421850c164463159e18fa0dc9cb5adb12c29b84959dc35c32cdd1fa81"}
Jul 06 22:46:32.298471 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:32.298039    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-116.ec2.internal" podStartSLOduration=3.298020246 podStartE2EDuration="3.298020246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:46:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:46:32.297548184 +0000 UTC m=+4.701798092" watchObservedRunningTime="2026-07-06 22:46:32.298020246 +0000 UTC m=+4.702270156"
Jul 06 22:46:32.315935 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:32.315908    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wdchz" event={"ID":"bc58eac9-19c3-4a2c-be0f-2f5211fc5f39","Type":"ContainerStarted","Data":"b6e04b17de62c1a48a73b91c86ff48608acb5abfcdc4fde227fc69bd00c8bc34"}
Jul 06 22:46:32.333298 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:32.328331    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9sggg" event={"ID":"dee0d303-a48c-43d1-ac56-f7c2850328b5","Type":"ContainerStarted","Data":"b1af34794ae859e9892fd58fdada81be2af7436315c34337a0bbee6631ba770f"}
Jul 06 22:46:32.333298 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:32.332244    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qgqcr" event={"ID":"3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90","Type":"ContainerStarted","Data":"a83aab842fde2a07a0357b2eade03b75a83d25f79311aac5324db433be501a3d"}
Jul 06 22:46:32.335039 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:32.334986    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dlqwr" event={"ID":"04f1bc85-8a29-4e65-bd9d-cd09785fc0dd","Type":"ContainerStarted","Data":"2d4c59b31c7a71d0b4523cd35a3df00085f4db867cf8a99b255963376ede8467"}
Jul 06 22:46:33.164123 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:33.164092    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfmlk"
Jul 06 22:46:33.164608 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:33.164246    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfmlk" podUID="107658f0-a3ae-4883-99dd-ddc62057a8a0"
Jul 06 22:46:33.359384 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:33.359312    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-116.ec2.internal" event={"ID":"c10a3b77a8cca715e148a6274757c5ff","Type":"ContainerStarted","Data":"961ce539d0bbe3435b1526af3a79531e61bc1d781c9fef337257c8794734697f"}
Jul 06 22:46:33.638524 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:33.637506    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8cc9b77e-7a6d-4d5b-83a2-8f24359ec453-original-pull-secret\") pod \"global-pull-secret-syncer-l6fbb\" (UID: \"8cc9b77e-7a6d-4d5b-83a2-8f24359ec453\") " pod="kube-system/global-pull-secret-syncer-l6fbb"
Jul 06 22:46:33.638524 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:33.637553    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/107658f0-a3ae-4883-99dd-ddc62057a8a0-metrics-certs\") pod \"network-metrics-daemon-mfmlk\" (UID: \"107658f0-a3ae-4883-99dd-ddc62057a8a0\") " pod="openshift-multus/network-metrics-daemon-mfmlk"
Jul 06 22:46:33.638524 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:33.637603    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5rxc\" (UniqueName: \"kubernetes.io/projected/f17325b2-5d96-4911-904b-3a774e4cc8b1-kube-api-access-h5rxc\") pod \"network-check-target-j9csq\" (UID: \"f17325b2-5d96-4911-904b-3a774e4cc8b1\") " pod="openshift-network-diagnostics/network-check-target-j9csq"
Jul 06 22:46:33.638524 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:33.637761    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 22:46:33.638524 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:33.637783    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 22:46:33.638524 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:33.637796    2569 projected.go:196] Error preparing data for projected volume kube-api-access-h5rxc for pod openshift-network-diagnostics/network-check-target-j9csq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 22:46:33.638524 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:33.637857    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f17325b2-5d96-4911-904b-3a774e4cc8b1-kube-api-access-h5rxc podName:f17325b2-5d96-4911-904b-3a774e4cc8b1 nodeName:}" failed. No retries permitted until 2026-07-06 22:46:37.637837785 +0000 UTC m=+10.042087691 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-h5rxc" (UniqueName: "kubernetes.io/projected/f17325b2-5d96-4911-904b-3a774e4cc8b1-kube-api-access-h5rxc") pod "network-check-target-j9csq" (UID: "f17325b2-5d96-4911-904b-3a774e4cc8b1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 22:46:33.638524 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:33.638269    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 22:46:33.638524 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:33.638322    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cc9b77e-7a6d-4d5b-83a2-8f24359ec453-original-pull-secret podName:8cc9b77e-7a6d-4d5b-83a2-8f24359ec453 nodeName:}" failed. No retries permitted until 2026-07-06 22:46:37.638306364 +0000 UTC m=+10.042556266 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8cc9b77e-7a6d-4d5b-83a2-8f24359ec453-original-pull-secret") pod "global-pull-secret-syncer-l6fbb" (UID: "8cc9b77e-7a6d-4d5b-83a2-8f24359ec453") : object "kube-system"/"original-pull-secret" not registered
Jul 06 22:46:33.638524 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:33.638402    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 22:46:33.638524 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:33.638439    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/107658f0-a3ae-4883-99dd-ddc62057a8a0-metrics-certs podName:107658f0-a3ae-4883-99dd-ddc62057a8a0 nodeName:}" failed. No retries permitted until 2026-07-06 22:46:37.63842723 +0000 UTC m=+10.042677117 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/107658f0-a3ae-4883-99dd-ddc62057a8a0-metrics-certs") pod "network-metrics-daemon-mfmlk" (UID: "107658f0-a3ae-4883-99dd-ddc62057a8a0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 22:46:34.167004 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:34.166970    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j9csq"
Jul 06 22:46:34.167594 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:34.167107    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j9csq" podUID="f17325b2-5d96-4911-904b-3a774e4cc8b1"
Jul 06 22:46:34.167594 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:34.167533    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6fbb"
Jul 06 22:46:34.167725 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:34.167653    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6fbb" podUID="8cc9b77e-7a6d-4d5b-83a2-8f24359ec453"
Jul 06 22:46:35.163984 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:35.163947    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfmlk"
Jul 06 22:46:35.164191 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:35.164095    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfmlk" podUID="107658f0-a3ae-4883-99dd-ddc62057a8a0"
Jul 06 22:46:36.165029 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:36.164991    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j9csq"
Jul 06 22:46:36.165537 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:36.165132    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j9csq" podUID="f17325b2-5d96-4911-904b-3a774e4cc8b1"
Jul 06 22:46:36.165537 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:36.165006    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6fbb"
Jul 06 22:46:36.165664 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:36.165636    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6fbb" podUID="8cc9b77e-7a6d-4d5b-83a2-8f24359ec453"
Jul 06 22:46:37.164503 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:37.164455    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfmlk"
Jul 06 22:46:37.164714 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:37.164616    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfmlk" podUID="107658f0-a3ae-4883-99dd-ddc62057a8a0"
Jul 06 22:46:37.675775 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:37.675512    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8cc9b77e-7a6d-4d5b-83a2-8f24359ec453-original-pull-secret\") pod \"global-pull-secret-syncer-l6fbb\" (UID: \"8cc9b77e-7a6d-4d5b-83a2-8f24359ec453\") " pod="kube-system/global-pull-secret-syncer-l6fbb"
Jul 06 22:46:37.675775 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:37.675566    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/107658f0-a3ae-4883-99dd-ddc62057a8a0-metrics-certs\") pod \"network-metrics-daemon-mfmlk\" (UID: \"107658f0-a3ae-4883-99dd-ddc62057a8a0\") " pod="openshift-multus/network-metrics-daemon-mfmlk"
Jul 06 22:46:37.675775 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:37.675617    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5rxc\" (UniqueName: \"kubernetes.io/projected/f17325b2-5d96-4911-904b-3a774e4cc8b1-kube-api-access-h5rxc\") pod \"network-check-target-j9csq\" (UID: \"f17325b2-5d96-4911-904b-3a774e4cc8b1\") " pod="openshift-network-diagnostics/network-check-target-j9csq"
Jul 06 22:46:37.675775 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:37.675687    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 22:46:37.675775 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:37.675770    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cc9b77e-7a6d-4d5b-83a2-8f24359ec453-original-pull-secret podName:8cc9b77e-7a6d-4d5b-83a2-8f24359ec453 nodeName:}" failed. No retries permitted until 2026-07-06 22:46:45.675744601 +0000 UTC m=+18.079994488 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8cc9b77e-7a6d-4d5b-83a2-8f24359ec453-original-pull-secret") pod "global-pull-secret-syncer-l6fbb" (UID: "8cc9b77e-7a6d-4d5b-83a2-8f24359ec453") : object "kube-system"/"original-pull-secret" not registered
Jul 06 22:46:37.675775 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:37.675779    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 22:46:37.676635 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:37.675803    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 22:46:37.676635 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:37.675816    2569 projected.go:196] Error preparing data for projected volume kube-api-access-h5rxc for pod openshift-network-diagnostics/network-check-target-j9csq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 22:46:37.676635 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:37.675869    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f17325b2-5d96-4911-904b-3a774e4cc8b1-kube-api-access-h5rxc podName:f17325b2-5d96-4911-904b-3a774e4cc8b1 nodeName:}" failed. No retries permitted until 2026-07-06 22:46:45.675852386 +0000 UTC m=+18.080102291 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-h5rxc" (UniqueName: "kubernetes.io/projected/f17325b2-5d96-4911-904b-3a774e4cc8b1-kube-api-access-h5rxc") pod "network-check-target-j9csq" (UID: "f17325b2-5d96-4911-904b-3a774e4cc8b1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 22:46:37.676635 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:37.675940    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 22:46:37.676635 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:37.675976    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/107658f0-a3ae-4883-99dd-ddc62057a8a0-metrics-certs podName:107658f0-a3ae-4883-99dd-ddc62057a8a0 nodeName:}" failed. No retries permitted until 2026-07-06 22:46:45.675963804 +0000 UTC m=+18.080213689 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/107658f0-a3ae-4883-99dd-ddc62057a8a0-metrics-certs") pod "network-metrics-daemon-mfmlk" (UID: "107658f0-a3ae-4883-99dd-ddc62057a8a0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 22:46:38.166562 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:38.165824    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j9csq"
Jul 06 22:46:38.166562 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:38.165942    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j9csq" podUID="f17325b2-5d96-4911-904b-3a774e4cc8b1"
Jul 06 22:46:38.166562 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:38.166332    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6fbb"
Jul 06 22:46:38.166562 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:38.166433    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6fbb" podUID="8cc9b77e-7a6d-4d5b-83a2-8f24359ec453"
Jul 06 22:46:39.164521 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:39.164480    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfmlk"
Jul 06 22:46:39.164970 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:39.164640    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfmlk" podUID="107658f0-a3ae-4883-99dd-ddc62057a8a0"
Jul 06 22:46:40.164172 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:40.164106    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6fbb"
Jul 06 22:46:40.164172 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:40.164106    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j9csq"
Jul 06 22:46:40.164172 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:40.164240    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6fbb" podUID="8cc9b77e-7a6d-4d5b-83a2-8f24359ec453"
Jul 06 22:46:40.164172 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:40.164285    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j9csq" podUID="f17325b2-5d96-4911-904b-3a774e4cc8b1"
Jul 06 22:46:41.164748 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:41.164714    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfmlk"
Jul 06 22:46:41.165312 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:41.164875    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfmlk" podUID="107658f0-a3ae-4883-99dd-ddc62057a8a0"
Jul 06 22:46:42.164288 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:42.164251    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6fbb"
Jul 06 22:46:42.164490 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:42.164383    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6fbb" podUID="8cc9b77e-7a6d-4d5b-83a2-8f24359ec453"
Jul 06 22:46:42.164490 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:42.164421    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j9csq"
Jul 06 22:46:42.164585 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:42.164500    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j9csq" podUID="f17325b2-5d96-4911-904b-3a774e4cc8b1"
Jul 06 22:46:43.164092 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:43.164056    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfmlk"
Jul 06 22:46:43.164547 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:43.164191    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfmlk" podUID="107658f0-a3ae-4883-99dd-ddc62057a8a0"
Jul 06 22:46:44.164593 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:44.164558    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j9csq"
Jul 06 22:46:44.165022 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:44.164558    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6fbb"
Jul 06 22:46:44.165022 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:44.164667    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j9csq" podUID="f17325b2-5d96-4911-904b-3a774e4cc8b1"
Jul 06 22:46:44.165022 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:44.164784    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6fbb" podUID="8cc9b77e-7a6d-4d5b-83a2-8f24359ec453"
Jul 06 22:46:45.164813 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:45.164779    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfmlk"
Jul 06 22:46:45.165300 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:45.164916    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfmlk" podUID="107658f0-a3ae-4883-99dd-ddc62057a8a0"
Jul 06 22:46:45.740999 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:45.740957    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5rxc\" (UniqueName: \"kubernetes.io/projected/f17325b2-5d96-4911-904b-3a774e4cc8b1-kube-api-access-h5rxc\") pod \"network-check-target-j9csq\" (UID: \"f17325b2-5d96-4911-904b-3a774e4cc8b1\") " pod="openshift-network-diagnostics/network-check-target-j9csq"
Jul 06 22:46:45.741167 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:45.741032    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8cc9b77e-7a6d-4d5b-83a2-8f24359ec453-original-pull-secret\") pod \"global-pull-secret-syncer-l6fbb\" (UID: \"8cc9b77e-7a6d-4d5b-83a2-8f24359ec453\") " pod="kube-system/global-pull-secret-syncer-l6fbb"
Jul 06 22:46:45.741167 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:45.741059    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/107658f0-a3ae-4883-99dd-ddc62057a8a0-metrics-certs\") pod \"network-metrics-daemon-mfmlk\" (UID: \"107658f0-a3ae-4883-99dd-ddc62057a8a0\") " pod="openshift-multus/network-metrics-daemon-mfmlk"
Jul 06 22:46:45.741167 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:45.741134    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 22:46:45.741167 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:45.741158    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 22:46:45.741167 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:45.741161    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 22:46:45.741167 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:45.741171    2569 projected.go:196] Error preparing data for projected volume kube-api-access-h5rxc for pod openshift-network-diagnostics/network-check-target-j9csq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 22:46:45.741423 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:45.741190    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 22:46:45.741423 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:45.741240    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/107658f0-a3ae-4883-99dd-ddc62057a8a0-metrics-certs podName:107658f0-a3ae-4883-99dd-ddc62057a8a0 nodeName:}" failed. No retries permitted until 2026-07-06 22:47:01.74122247 +0000 UTC m=+34.145472355 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/107658f0-a3ae-4883-99dd-ddc62057a8a0-metrics-certs") pod "network-metrics-daemon-mfmlk" (UID: "107658f0-a3ae-4883-99dd-ddc62057a8a0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 22:46:45.741423 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:45.741259    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f17325b2-5d96-4911-904b-3a774e4cc8b1-kube-api-access-h5rxc podName:f17325b2-5d96-4911-904b-3a774e4cc8b1 nodeName:}" failed. No retries permitted until 2026-07-06 22:47:01.741248944 +0000 UTC m=+34.145498831 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-h5rxc" (UniqueName: "kubernetes.io/projected/f17325b2-5d96-4911-904b-3a774e4cc8b1-kube-api-access-h5rxc") pod "network-check-target-j9csq" (UID: "f17325b2-5d96-4911-904b-3a774e4cc8b1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 22:46:45.741423 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:45.741277    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cc9b77e-7a6d-4d5b-83a2-8f24359ec453-original-pull-secret podName:8cc9b77e-7a6d-4d5b-83a2-8f24359ec453 nodeName:}" failed. No retries permitted until 2026-07-06 22:47:01.741265795 +0000 UTC m=+34.145515682 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8cc9b77e-7a6d-4d5b-83a2-8f24359ec453-original-pull-secret") pod "global-pull-secret-syncer-l6fbb" (UID: "8cc9b77e-7a6d-4d5b-83a2-8f24359ec453") : object "kube-system"/"original-pull-secret" not registered
Jul 06 22:46:46.164656 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:46.164590    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j9csq"
Jul 06 22:46:46.164785 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:46.164707    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j9csq" podUID="f17325b2-5d96-4911-904b-3a774e4cc8b1"
Jul 06 22:46:46.164785 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:46.164757    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6fbb"
Jul 06 22:46:46.165114 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:46.164840    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6fbb" podUID="8cc9b77e-7a6d-4d5b-83a2-8f24359ec453"
Jul 06 22:46:47.163929 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:47.163899    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfmlk"
Jul 06 22:46:47.164125 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:47.164013    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfmlk" podUID="107658f0-a3ae-4883-99dd-ddc62057a8a0"
Jul 06 22:46:48.165501 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:48.165464    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6fbb"
Jul 06 22:46:48.165961 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:48.165581    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6fbb" podUID="8cc9b77e-7a6d-4d5b-83a2-8f24359ec453"
Jul 06 22:46:48.165961 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:48.165642    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j9csq"
Jul 06 22:46:48.165961 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:48.165750    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j9csq" podUID="f17325b2-5d96-4911-904b-3a774e4cc8b1"
Jul 06 22:46:49.164587 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:49.164555    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfmlk"
Jul 06 22:46:49.164789 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:49.164694    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfmlk" podUID="107658f0-a3ae-4883-99dd-ddc62057a8a0"
Jul 06 22:46:50.164910 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:50.164824    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j9csq"
Jul 06 22:46:50.165381 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:50.164962    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j9csq" podUID="f17325b2-5d96-4911-904b-3a774e4cc8b1"
Jul 06 22:46:50.165381 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:50.165007    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6fbb"
Jul 06 22:46:50.165381 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:50.165106    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6fbb" podUID="8cc9b77e-7a6d-4d5b-83a2-8f24359ec453"
Jul 06 22:46:51.164925 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:51.164699    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfmlk"
Jul 06 22:46:51.165474 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:51.165015    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfmlk" podUID="107658f0-a3ae-4883-99dd-ddc62057a8a0"
Jul 06 22:46:51.404246 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:51.404209    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdmt8" event={"ID":"d762a8a0-e380-4f4b-ac3d-420d6758b924","Type":"ContainerStarted","Data":"0b0595c4dacc561c9960d5b04c168a10b7957066289e22193dc69a255efa9f57"}
Jul 06 22:46:51.405814 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:51.405783    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jtsxh" event={"ID":"17c837d3-15da-4a3b-97ba-c035de2e4d38","Type":"ContainerStarted","Data":"12af13bad84fea361e9e05767a3887f21dc874bd5a9814b53f8c37a1480fb127"}
Jul 06 22:46:51.407587 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:51.407559    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wdchz" event={"ID":"bc58eac9-19c3-4a2c-be0f-2f5211fc5f39","Type":"ContainerStarted","Data":"9bdc2a8ab0fe60c9cb0f4637132cc91d64fd73f658265f4c49e6713eedc07685"}
Jul 06 22:46:51.409232 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:51.409207    2569 generic.go:356] "Generic (PLEG): container finished" podID="3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90" containerID="24fdad3a35371e97c5b3dc7185565f9c939293735923e5a9ae1ac3414837954a" exitCode=0
Jul 06 22:46:51.409325 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:51.409288    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qgqcr" event={"ID":"3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90","Type":"ContainerDied","Data":"24fdad3a35371e97c5b3dc7185565f9c939293735923e5a9ae1ac3414837954a"}
Jul 06 22:46:51.411093 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:51.411065    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dlqwr" event={"ID":"04f1bc85-8a29-4e65-bd9d-cd09785fc0dd","Type":"ContainerStarted","Data":"71d7c248a9ac87cf18b3f7e578a357a70377f73b00cb022271748822a55b9bea"}
Jul 06 22:46:51.414355 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:51.414337    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bvsb_571f407a-249a-4fd9-8c15-53a6541db635/ovn-acl-logging/0.log"
Jul 06 22:46:51.414709 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:51.414691    2569 generic.go:356] "Generic (PLEG): container finished" podID="571f407a-249a-4fd9-8c15-53a6541db635" containerID="946dafe0427775a9b9276420d2e7c36a79d8bf2e8e94d75a67741f1938365394" exitCode=1
Jul 06 22:46:51.414792 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:51.414755    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb" event={"ID":"571f407a-249a-4fd9-8c15-53a6541db635","Type":"ContainerStarted","Data":"c0b1aa64445f6b8152f04b670d4374855d693f0835a01153694a2eaeea679757"}
Jul 06 22:46:51.414844 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:51.414833    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb" event={"ID":"571f407a-249a-4fd9-8c15-53a6541db635","Type":"ContainerStarted","Data":"9e140aeaa048449e96d3d023e78290e3c98fac29130472234c5578eed4109bd6"}
Jul 06 22:46:51.414896 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:51.414882    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb" event={"ID":"571f407a-249a-4fd9-8c15-53a6541db635","Type":"ContainerStarted","Data":"fa1a271aa1db66013580364bdd7ad58f615beb66592da41e59de0cae5ffce8ae"}
Jul 06 22:46:51.414933 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:51.414920    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb" event={"ID":"571f407a-249a-4fd9-8c15-53a6541db635","Type":"ContainerStarted","Data":"46166e59952e7aef26378a192c6fd6b3be5d620a935d767de6c5d387272b88c1"}
Jul 06 22:46:51.414977 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:51.414955    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb" event={"ID":"571f407a-249a-4fd9-8c15-53a6541db635","Type":"ContainerDied","Data":"946dafe0427775a9b9276420d2e7c36a79d8bf2e8e94d75a67741f1938365394"}
Jul 06 22:46:51.415032 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:51.414984    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb" event={"ID":"571f407a-249a-4fd9-8c15-53a6541db635","Type":"ContainerStarted","Data":"c911c38ab482a8b72ad36e3cecea665a76cf76a772f69c1d24ab50ecb3665ffc"}
Jul 06 22:46:51.416305 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:51.416268    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hwhh7" event={"ID":"345357e4-a764-47c9-8ef9-acfc5b4e8cf6","Type":"ContainerStarted","Data":"9f8b61ee5a13d8f3ed30c8edd0875032d3e8a9bae5dc36935dbeff7934fa299c"}
Jul 06 22:46:51.417896 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:51.417872    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2zsx8" event={"ID":"6b66febe-a59f-40a7-bff3-d248315b326e","Type":"ContainerStarted","Data":"b925f99ebbd856e493e71669ff621d748b6f9d9761a62266cb63a03ca920749d"}
Jul 06 22:46:51.423806 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:51.423743    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-116.ec2.internal" podStartSLOduration=22.423731718 podStartE2EDuration="22.423731718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:46:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:46:33.378799587 +0000 UTC m=+5.783049496" watchObservedRunningTime="2026-07-06 22:46:51.423731718 +0000 UTC m=+23.827981626"
Jul 06 22:46:51.424089 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:51.424050    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jtsxh" podStartSLOduration=3.227586866 podStartE2EDuration="22.424042912s" totalImagesPullingTime="19.196456046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:46:51.423468877 +0000 UTC m=+23.827718780" watchObservedRunningTime="2026-07-06 22:46:51.424042912 +0000 UTC m=+23.828292816"
Jul 06 22:46:51.440456 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:51.440425    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dlqwr" podStartSLOduration=3.204624267 podStartE2EDuration="22.440416687s" totalImagesPullingTime="19.23579242s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:46:51.440249141 +0000 UTC m=+23.844499045" watchObservedRunningTime="2026-07-06 22:46:51.440416687 +0000 UTC m=+23.844666591"
Jul 06 22:46:51.456343 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:51.456304    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-wdchz" podStartSLOduration=3.248977098 podStartE2EDuration="22.45628876s" totalImagesPullingTime="19.207311662s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:46:51.455568484 +0000 UTC m=+23.859818387" watchObservedRunningTime="2026-07-06 22:46:51.45628876 +0000 UTC m=+23.860538665"
Jul 06 22:46:51.496955 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:51.496906    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-hwhh7" podStartSLOduration=3.237690949 podStartE2EDuration="22.49688791s" totalImagesPullingTime="19.259196961s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:46:51.496326643 +0000 UTC m=+23.900576550" watchObservedRunningTime="2026-07-06 22:46:51.49688791 +0000 UTC m=+23.901137845"
Jul 06 22:46:51.510982 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:51.510940    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2zsx8" podStartSLOduration=3.298635589 podStartE2EDuration="22.510924999s" totalImagesPullingTime="19.21228941s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:46:51.510885523 +0000 UTC m=+23.915135429" watchObservedRunningTime="2026-07-06 22:46:51.510924999 +0000 UTC m=+23.915174904"
Jul 06 22:46:52.015723 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:52.015698    2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 06 22:46:52.098060 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:52.097960    2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T22:46:52.01572026Z","UUID":"e9a6804f-68a0-4944-a202-724ae5cc34bf","Handler":null,"Name":"","Endpoint":""}
Jul 06 22:46:52.099620 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:52.099600    2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 06 22:46:52.099726 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:52.099637    2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 06 22:46:52.164166 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:52.164101    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6fbb"
Jul 06 22:46:52.164303 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:52.164107    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j9csq"
Jul 06 22:46:52.164303 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:52.164224    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6fbb" podUID="8cc9b77e-7a6d-4d5b-83a2-8f24359ec453"
Jul 06 22:46:52.164303 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:52.164294    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j9csq" podUID="f17325b2-5d96-4911-904b-3a774e4cc8b1"
Jul 06 22:46:52.422871 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:52.422793    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdmt8" event={"ID":"d762a8a0-e380-4f4b-ac3d-420d6758b924","Type":"ContainerStarted","Data":"c269504933dadabc8d5692f5d219fba2ee620f9af18d2b0c8283e8fbf303e0df"}
Jul 06 22:46:52.424379 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:52.424349    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9sggg" event={"ID":"dee0d303-a48c-43d1-ac56-f7c2850328b5","Type":"ContainerStarted","Data":"8adc0c52062eeb67a3f35df1e943c181cfddb16eab8a01f17c41c4b6e892dece"}
Jul 06 22:46:52.438561 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:52.438509    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9sggg" podStartSLOduration=4.22704177 podStartE2EDuration="23.438492217s" totalImagesPullingTime="19.211450447s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:46:52.438247124 +0000 UTC m=+24.842497027" watchObservedRunningTime="2026-07-06 22:46:52.438492217 +0000 UTC m=+24.842742146"
Jul 06 22:46:53.164752 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:53.164723    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfmlk"
Jul 06 22:46:53.164938 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:53.164867    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfmlk" podUID="107658f0-a3ae-4883-99dd-ddc62057a8a0"
Jul 06 22:46:53.429566 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:53.429389    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdmt8" event={"ID":"d762a8a0-e380-4f4b-ac3d-420d6758b924","Type":"ContainerStarted","Data":"f4739ecd87108f11eee8b939275487334af2d9ac177b30f1da38b4504c6cca10"}
Jul 06 22:46:53.446303 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:53.446263    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cdmt8" podStartSLOduration=3.1155419540000002 podStartE2EDuration="24.446245103s" totalImagesPullingTime="21.330703149s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:46:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:46:53.445995829 +0000 UTC m=+25.850245733" watchObservedRunningTime="2026-07-06 22:46:53.446245103 +0000 UTC m=+25.850495008"
Jul 06 22:46:54.164836 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:54.164804    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6fbb"
Jul 06 22:46:54.165035 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:54.164804    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j9csq"
Jul 06 22:46:54.165035 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:54.164927    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6fbb" podUID="8cc9b77e-7a6d-4d5b-83a2-8f24359ec453"
Jul 06 22:46:54.165035 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:54.165018    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j9csq" podUID="f17325b2-5d96-4911-904b-3a774e4cc8b1"
Jul 06 22:46:54.435553 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:54.435530    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bvsb_571f407a-249a-4fd9-8c15-53a6541db635/ovn-acl-logging/0.log"
Jul 06 22:46:54.436038 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:54.435924    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb" event={"ID":"571f407a-249a-4fd9-8c15-53a6541db635","Type":"ContainerStarted","Data":"0adca480735a7cb908c1a0d0e5edf9860b3773d83f9466dd4f67e087306e27c7"}
Jul 06 22:46:55.164588 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:55.164561    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfmlk"
Jul 06 22:46:55.164755 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:55.164682    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfmlk" podUID="107658f0-a3ae-4883-99dd-ddc62057a8a0"
Jul 06 22:46:55.181959 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:55.181929    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jtsxh"
Jul 06 22:46:55.182503 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:55.182482    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jtsxh"
Jul 06 22:46:55.438006 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:55.437983    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jtsxh"
Jul 06 22:46:55.438580 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:55.438563    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jtsxh"
Jul 06 22:46:56.164567 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:56.164508    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j9csq"
Jul 06 22:46:56.164674 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:56.164514    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6fbb"
Jul 06 22:46:56.164674 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:56.164622    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j9csq" podUID="f17325b2-5d96-4911-904b-3a774e4cc8b1"
Jul 06 22:46:56.164757 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:56.164676    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6fbb" podUID="8cc9b77e-7a6d-4d5b-83a2-8f24359ec453"
Jul 06 22:46:56.441738 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:56.441706    2569 generic.go:356] "Generic (PLEG): container finished" podID="3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90" containerID="16101e92a2b9b7874563f14a98c09bfc8e55d0764ee167c270f77ccc4865d762" exitCode=0
Jul 06 22:46:56.442321 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:56.441800    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qgqcr" event={"ID":"3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90","Type":"ContainerDied","Data":"16101e92a2b9b7874563f14a98c09bfc8e55d0764ee167c270f77ccc4865d762"}
Jul 06 22:46:56.445710 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:56.445692    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bvsb_571f407a-249a-4fd9-8c15-53a6541db635/ovn-acl-logging/0.log"
Jul 06 22:46:56.446147 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:56.446121    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb" event={"ID":"571f407a-249a-4fd9-8c15-53a6541db635","Type":"ContainerStarted","Data":"bd566964310e0a19e56886124a3dfbee4dbfe46dfb71b320a43c31c4b579353d"}
Jul 06 22:46:56.446376 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:56.446361    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:56.446520 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:56.446501    2569 scope.go:117] "RemoveContainer" containerID="946dafe0427775a9b9276420d2e7c36a79d8bf2e8e94d75a67741f1938365394"
Jul 06 22:46:56.461828 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:56.461811    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:57.164334 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:57.164303    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfmlk"
Jul 06 22:46:57.164565 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:57.164437    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfmlk" podUID="107658f0-a3ae-4883-99dd-ddc62057a8a0"
Jul 06 22:46:57.452129 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:57.452106    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bvsb_571f407a-249a-4fd9-8c15-53a6541db635/ovn-acl-logging/0.log"
Jul 06 22:46:57.452532 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:57.452507    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb" event={"ID":"571f407a-249a-4fd9-8c15-53a6541db635","Type":"ContainerStarted","Data":"c6f09937350e04e0f2cd10d4626ba0062e14f3b945286b81a6d36d868c663b95"}
Jul 06 22:46:57.452726 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:57.452700    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:57.452855 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:57.452771    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:57.467434 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:57.467301    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:46:57.481585 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:57.481548    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb" podStartSLOduration=9.220421357 podStartE2EDuration="28.481536806s" totalImagesPullingTime="19.261115449s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:46:57.480329359 +0000 UTC m=+29.884579282" watchObservedRunningTime="2026-07-06 22:46:57.481536806 +0000 UTC m=+29.885786740"
Jul 06 22:46:57.527342 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:57.527322    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l6fbb"]
Jul 06 22:46:57.527437 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:57.527408    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6fbb"
Jul 06 22:46:57.527509 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:57.527491    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6fbb" podUID="8cc9b77e-7a6d-4d5b-83a2-8f24359ec453"
Jul 06 22:46:57.532593 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:57.532561    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j9csq"]
Jul 06 22:46:57.532710 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:57.532664    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j9csq"
Jul 06 22:46:57.532798 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:57.532741    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j9csq" podUID="f17325b2-5d96-4911-904b-3a774e4cc8b1"
Jul 06 22:46:57.533340 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:57.533319    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mfmlk"]
Jul 06 22:46:57.533421 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:57.533403    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfmlk"
Jul 06 22:46:57.533508 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:57.533492    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfmlk" podUID="107658f0-a3ae-4883-99dd-ddc62057a8a0"
Jul 06 22:46:58.459468 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:58.459433    2569 generic.go:356] "Generic (PLEG): container finished" podID="3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90" containerID="2804093ecd748ac473919c0e76ccc460917fb44027e88f500e6120a67a68b006" exitCode=0
Jul 06 22:46:58.459939 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:58.459517    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qgqcr" event={"ID":"3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90","Type":"ContainerDied","Data":"2804093ecd748ac473919c0e76ccc460917fb44027e88f500e6120a67a68b006"}
Jul 06 22:46:59.164352 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:59.164316    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfmlk"
Jul 06 22:46:59.164511 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:59.164362    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6fbb"
Jul 06 22:46:59.164511 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:59.164334    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j9csq"
Jul 06 22:46:59.164511 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:59.164438    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfmlk" podUID="107658f0-a3ae-4883-99dd-ddc62057a8a0"
Jul 06 22:46:59.164511 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:59.164505    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j9csq" podUID="f17325b2-5d96-4911-904b-3a774e4cc8b1"
Jul 06 22:46:59.164647 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:46:59.164558    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6fbb" podUID="8cc9b77e-7a6d-4d5b-83a2-8f24359ec453"
Jul 06 22:46:59.463813 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:46:59.463779    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qgqcr" event={"ID":"3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90","Type":"ContainerStarted","Data":"c862e9443a29e6d38d0df180e90b234ea6044ce8df4d015ed7c83d0eb07148aa"}
Jul 06 22:47:00.468527 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:00.468459    2569 generic.go:356] "Generic (PLEG): container finished" podID="3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90" containerID="c862e9443a29e6d38d0df180e90b234ea6044ce8df4d015ed7c83d0eb07148aa" exitCode=0
Jul 06 22:47:00.468527 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:00.468518    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qgqcr" event={"ID":"3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90","Type":"ContainerDied","Data":"c862e9443a29e6d38d0df180e90b234ea6044ce8df4d015ed7c83d0eb07148aa"}
Jul 06 22:47:01.164370 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:01.164338    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6fbb"
Jul 06 22:47:01.164535 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:01.164340    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j9csq"
Jul 06 22:47:01.164535 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:01.164454    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6fbb" podUID="8cc9b77e-7a6d-4d5b-83a2-8f24359ec453"
Jul 06 22:47:01.164535 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:01.164343    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfmlk"
Jul 06 22:47:01.164714 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:01.164544    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j9csq" podUID="f17325b2-5d96-4911-904b-3a774e4cc8b1"
Jul 06 22:47:01.164714 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:01.164667    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfmlk" podUID="107658f0-a3ae-4883-99dd-ddc62057a8a0"
Jul 06 22:47:01.763937 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:01.763899    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8cc9b77e-7a6d-4d5b-83a2-8f24359ec453-original-pull-secret\") pod \"global-pull-secret-syncer-l6fbb\" (UID: \"8cc9b77e-7a6d-4d5b-83a2-8f24359ec453\") " pod="kube-system/global-pull-secret-syncer-l6fbb"
Jul 06 22:47:01.764406 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:01.763944    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/107658f0-a3ae-4883-99dd-ddc62057a8a0-metrics-certs\") pod \"network-metrics-daemon-mfmlk\" (UID: \"107658f0-a3ae-4883-99dd-ddc62057a8a0\") " pod="openshift-multus/network-metrics-daemon-mfmlk"
Jul 06 22:47:01.764406 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:01.763978    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5rxc\" (UniqueName: \"kubernetes.io/projected/f17325b2-5d96-4911-904b-3a774e4cc8b1-kube-api-access-h5rxc\") pod \"network-check-target-j9csq\" (UID: \"f17325b2-5d96-4911-904b-3a774e4cc8b1\") " pod="openshift-network-diagnostics/network-check-target-j9csq"
Jul 06 22:47:01.764406 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:01.764068    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 22:47:01.764406 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:01.764104    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 22:47:01.764406 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:01.764132    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 22:47:01.764406 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:01.764148    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 22:47:01.764406 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:01.764161    2569 projected.go:196] Error preparing data for projected volume kube-api-access-h5rxc for pod openshift-network-diagnostics/network-check-target-j9csq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 22:47:01.764406 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:01.764148    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cc9b77e-7a6d-4d5b-83a2-8f24359ec453-original-pull-secret podName:8cc9b77e-7a6d-4d5b-83a2-8f24359ec453 nodeName:}" failed. No retries permitted until 2026-07-06 22:47:33.764125415 +0000 UTC m=+66.168375299 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8cc9b77e-7a6d-4d5b-83a2-8f24359ec453-original-pull-secret") pod "global-pull-secret-syncer-l6fbb" (UID: "8cc9b77e-7a6d-4d5b-83a2-8f24359ec453") : object "kube-system"/"original-pull-secret" not registered
Jul 06 22:47:01.764406 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:01.764227    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/107658f0-a3ae-4883-99dd-ddc62057a8a0-metrics-certs podName:107658f0-a3ae-4883-99dd-ddc62057a8a0 nodeName:}" failed. No retries permitted until 2026-07-06 22:47:33.764213767 +0000 UTC m=+66.168463650 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/107658f0-a3ae-4883-99dd-ddc62057a8a0-metrics-certs") pod "network-metrics-daemon-mfmlk" (UID: "107658f0-a3ae-4883-99dd-ddc62057a8a0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 22:47:01.764406 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:01.764240    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f17325b2-5d96-4911-904b-3a774e4cc8b1-kube-api-access-h5rxc podName:f17325b2-5d96-4911-904b-3a774e4cc8b1 nodeName:}" failed. No retries permitted until 2026-07-06 22:47:33.764232826 +0000 UTC m=+66.168482707 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-h5rxc" (UniqueName: "kubernetes.io/projected/f17325b2-5d96-4911-904b-3a774e4cc8b1-kube-api-access-h5rxc") pod "network-check-target-j9csq" (UID: "f17325b2-5d96-4911-904b-3a774e4cc8b1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 22:47:03.164064 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:03.163891    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfmlk"
Jul 06 22:47:03.164527 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:03.163891    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j9csq"
Jul 06 22:47:03.164527 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:03.163897    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6fbb"
Jul 06 22:47:03.164527 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:03.164171    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mfmlk" podUID="107658f0-a3ae-4883-99dd-ddc62057a8a0"
Jul 06 22:47:03.164527 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:03.164290    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j9csq" podUID="f17325b2-5d96-4911-904b-3a774e4cc8b1"
Jul 06 22:47:03.164527 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:03.164356    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l6fbb" podUID="8cc9b77e-7a6d-4d5b-83a2-8f24359ec453"
Jul 06 22:47:03.406288 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:03.406260    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-116.ec2.internal" event="NodeReady"
Jul 06 22:47:03.406473 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:03.406383    2569 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 06 22:47:03.448825 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:03.448791    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2xhr2"]
Jul 06 22:47:03.451489 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:03.451463    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2xhr2"
Jul 06 22:47:03.454610 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:03.454587    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 06 22:47:03.454713 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:03.454630    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 06 22:47:03.454713 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:03.454633    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jx42h\""
Jul 06 22:47:03.455060 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:03.455037    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dwnzz"]
Jul 06 22:47:03.456739 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:03.456712    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dwnzz"
Jul 06 22:47:03.458917 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:03.458897    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-d66nl\""
Jul 06 22:47:03.459010 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:03.458985    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 06 22:47:03.459173 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:03.459156    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 06 22:47:03.459329 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:03.459294    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 06 22:47:03.461261 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:03.461240    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2xhr2"]
Jul 06 22:47:03.465122 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:03.465103    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dwnzz"]
Jul 06 22:47:03.476944 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:03.476919    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8624c1c4-bd8a-4e14-9445-3c7352c1cd60-cert\") pod \"ingress-canary-dwnzz\" (UID: \"8624c1c4-bd8a-4e14-9445-3c7352c1cd60\") " pod="openshift-ingress-canary/ingress-canary-dwnzz"
Jul 06 22:47:03.477066 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:03.476961    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82hqd\" (UniqueName: \"kubernetes.io/projected/8624c1c4-bd8a-4e14-9445-3c7352c1cd60-kube-api-access-82hqd\") pod \"ingress-canary-dwnzz\" (UID: \"8624c1c4-bd8a-4e14-9445-3c7352c1cd60\") " pod="openshift-ingress-canary/ingress-canary-dwnzz"
Jul 06 22:47:03.577849 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:03.577815    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a674edfd-a962-4327-b319-320fb62922c0-tmp-dir\") pod \"dns-default-2xhr2\" (UID: \"a674edfd-a962-4327-b319-320fb62922c0\") " pod="openshift-dns/dns-default-2xhr2"
Jul 06 22:47:03.578007 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:03.577862    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a674edfd-a962-4327-b319-320fb62922c0-metrics-tls\") pod \"dns-default-2xhr2\" (UID: \"a674edfd-a962-4327-b319-320fb62922c0\") " pod="openshift-dns/dns-default-2xhr2"
Jul 06 22:47:03.578007 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:03.577922    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8624c1c4-bd8a-4e14-9445-3c7352c1cd60-cert\") pod \"ingress-canary-dwnzz\" (UID: \"8624c1c4-bd8a-4e14-9445-3c7352c1cd60\") " pod="openshift-ingress-canary/ingress-canary-dwnzz"
Jul 06 22:47:03.578007 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:03.577951    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82hqd\" (UniqueName: \"kubernetes.io/projected/8624c1c4-bd8a-4e14-9445-3c7352c1cd60-kube-api-access-82hqd\") pod \"ingress-canary-dwnzz\" (UID: \"8624c1c4-bd8a-4e14-9445-3c7352c1cd60\") " pod="openshift-ingress-canary/ingress-canary-dwnzz"
Jul 06 22:47:03.578007 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:03.577975    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a674edfd-a962-4327-b319-320fb62922c0-config-volume\") pod \"dns-default-2xhr2\" (UID: \"a674edfd-a962-4327-b319-320fb62922c0\") " pod="openshift-dns/dns-default-2xhr2"
Jul 06 22:47:03.578224 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:03.578026    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2kdg\" (UniqueName: \"kubernetes.io/projected/a674edfd-a962-4327-b319-320fb62922c0-kube-api-access-p2kdg\") pod \"dns-default-2xhr2\" (UID: \"a674edfd-a962-4327-b319-320fb62922c0\") " pod="openshift-dns/dns-default-2xhr2"
Jul 06 22:47:03.582520 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:03.582492    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8624c1c4-bd8a-4e14-9445-3c7352c1cd60-cert\") pod \"ingress-canary-dwnzz\" (UID: \"8624c1c4-bd8a-4e14-9445-3c7352c1cd60\") " pod="openshift-ingress-canary/ingress-canary-dwnzz"
Jul 06 22:47:03.586102 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:03.586048    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82hqd\" (UniqueName: \"kubernetes.io/projected/8624c1c4-bd8a-4e14-9445-3c7352c1cd60-kube-api-access-82hqd\") pod \"ingress-canary-dwnzz\" (UID: \"8624c1c4-bd8a-4e14-9445-3c7352c1cd60\") " pod="openshift-ingress-canary/ingress-canary-dwnzz"
Jul 06 22:47:03.678965 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:03.678934    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a674edfd-a962-4327-b319-320fb62922c0-metrics-tls\") pod \"dns-default-2xhr2\" (UID: \"a674edfd-a962-4327-b319-320fb62922c0\") " pod="openshift-dns/dns-default-2xhr2"
Jul 06 22:47:03.679090 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:03.678986    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a674edfd-a962-4327-b319-320fb62922c0-config-volume\") pod \"dns-default-2xhr2\" (UID: \"a674edfd-a962-4327-b319-320fb62922c0\") " pod="openshift-dns/dns-default-2xhr2"
Jul 06 22:47:03.679145 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:03.679100    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 22:47:03.679232 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:03.679207    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a674edfd-a962-4327-b319-320fb62922c0-metrics-tls podName:a674edfd-a962-4327-b319-320fb62922c0 nodeName:}" failed. No retries permitted until 2026-07-06 22:47:04.179167709 +0000 UTC m=+36.583417594 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a674edfd-a962-4327-b319-320fb62922c0-metrics-tls") pod "dns-default-2xhr2" (UID: "a674edfd-a962-4327-b319-320fb62922c0") : secret "dns-default-metrics-tls" not found
Jul 06 22:47:03.679308 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:03.679244    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2kdg\" (UniqueName: \"kubernetes.io/projected/a674edfd-a962-4327-b319-320fb62922c0-kube-api-access-p2kdg\") pod \"dns-default-2xhr2\" (UID: \"a674edfd-a962-4327-b319-320fb62922c0\") " pod="openshift-dns/dns-default-2xhr2"
Jul 06 22:47:03.679367 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:03.679321    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a674edfd-a962-4327-b319-320fb62922c0-tmp-dir\") pod \"dns-default-2xhr2\" (UID: \"a674edfd-a962-4327-b319-320fb62922c0\") " pod="openshift-dns/dns-default-2xhr2"
Jul 06 22:47:03.679588 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:03.679557    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a674edfd-a962-4327-b319-320fb62922c0-config-volume\") pod \"dns-default-2xhr2\" (UID: \"a674edfd-a962-4327-b319-320fb62922c0\") " pod="openshift-dns/dns-default-2xhr2"
Jul 06 22:47:03.679700 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:03.679641    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a674edfd-a962-4327-b319-320fb62922c0-tmp-dir\") pod \"dns-default-2xhr2\" (UID: \"a674edfd-a962-4327-b319-320fb62922c0\") " pod="openshift-dns/dns-default-2xhr2"
Jul 06 22:47:03.687919 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:03.687899    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2kdg\" (UniqueName: \"kubernetes.io/projected/a674edfd-a962-4327-b319-320fb62922c0-kube-api-access-p2kdg\") pod \"dns-default-2xhr2\" (UID: \"a674edfd-a962-4327-b319-320fb62922c0\") " pod="openshift-dns/dns-default-2xhr2"
Jul 06 22:47:03.773989 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:03.773914    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dwnzz"
Jul 06 22:47:04.182637 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.182610    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a674edfd-a962-4327-b319-320fb62922c0-metrics-tls\") pod \"dns-default-2xhr2\" (UID: \"a674edfd-a962-4327-b319-320fb62922c0\") " pod="openshift-dns/dns-default-2xhr2"
Jul 06 22:47:04.183045 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:04.182730    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 22:47:04.183045 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:04.182795    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a674edfd-a962-4327-b319-320fb62922c0-metrics-tls podName:a674edfd-a962-4327-b319-320fb62922c0 nodeName:}" failed. No retries permitted until 2026-07-06 22:47:05.182781493 +0000 UTC m=+37.587031375 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a674edfd-a962-4327-b319-320fb62922c0-metrics-tls") pod "dns-default-2xhr2" (UID: "a674edfd-a962-4327-b319-320fb62922c0") : secret "dns-default-metrics-tls" not found
Jul 06 22:47:04.444853 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.444797    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpdfh"]
Jul 06 22:47:04.447451 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.447428    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpdfh"
Jul 06 22:47:04.449715 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.449666    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 06 22:47:04.449979 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.449961    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 06 22:47:04.453280 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.452898    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-vzg2t\""
Jul 06 22:47:04.453280 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.453139    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 06 22:47:04.453280 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.453151    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 06 22:47:04.458107 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.458088    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpdfh"]
Jul 06 22:47:04.481494 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.481470    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dwnzz"]
Jul 06 22:47:04.486027 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.485994    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ad24049a-6893-4d5e-bb79-0e176fc731c3-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-dpdfh\" (UID: \"ad24049a-6893-4d5e-bb79-0e176fc731c3\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpdfh"
Jul 06 22:47:04.486126 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.486036    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad24049a-6893-4d5e-bb79-0e176fc731c3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-dpdfh\" (UID: \"ad24049a-6893-4d5e-bb79-0e176fc731c3\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpdfh"
Jul 06 22:47:04.486219 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:47:04.486199    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8624c1c4_bd8a_4e14_9445_3c7352c1cd60.slice/crio-735fe1c801d7614b243de97376a0d7237cd37f1ad947c7699fc556181f1ca719 WatchSource:0}: Error finding container 735fe1c801d7614b243de97376a0d7237cd37f1ad947c7699fc556181f1ca719: Status 404 returned error can't find the container with id 735fe1c801d7614b243de97376a0d7237cd37f1ad947c7699fc556181f1ca719
Jul 06 22:47:04.486268 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.486224    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjv7w\" (UniqueName: \"kubernetes.io/projected/ad24049a-6893-4d5e-bb79-0e176fc731c3-kube-api-access-fjv7w\") pod \"cluster-monitoring-operator-64786d9cbb-dpdfh\" (UID: \"ad24049a-6893-4d5e-bb79-0e176fc731c3\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpdfh"
Jul 06 22:47:04.569947 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.569917    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-z62hq"]
Jul 06 22:47:04.571975 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.571956    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-z62hq"
Jul 06 22:47:04.574845 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.574824    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 06 22:47:04.575217 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.575199    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 06 22:47:04.575418 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.575400    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 06 22:47:04.575490 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.575459    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 06 22:47:04.575718 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.575703    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-2w4nb\""
Jul 06 22:47:04.586694 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.586537    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ad24049a-6893-4d5e-bb79-0e176fc731c3-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-dpdfh\" (UID: \"ad24049a-6893-4d5e-bb79-0e176fc731c3\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpdfh"
Jul 06 22:47:04.586791 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.586708    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad24049a-6893-4d5e-bb79-0e176fc731c3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-dpdfh\" (UID: \"ad24049a-6893-4d5e-bb79-0e176fc731c3\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpdfh"
Jul 06 22:47:04.586791 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.586734    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e00d4ca-8ce5-4ec1-8895-fe762a2d9e4c-tmp\") pod \"insights-operator-5c847dcfc4-z62hq\" (UID: \"6e00d4ca-8ce5-4ec1-8895-fe762a2d9e4c\") " pod="openshift-insights/insights-operator-5c847dcfc4-z62hq"
Jul 06 22:47:04.586878 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:04.586836    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 06 22:47:04.586878 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.586859    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6e00d4ca-8ce5-4ec1-8895-fe762a2d9e4c-snapshots\") pod \"insights-operator-5c847dcfc4-z62hq\" (UID: \"6e00d4ca-8ce5-4ec1-8895-fe762a2d9e4c\") " pod="openshift-insights/insights-operator-5c847dcfc4-z62hq"
Jul 06 22:47:04.586956 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:04.586901    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad24049a-6893-4d5e-bb79-0e176fc731c3-cluster-monitoring-operator-tls podName:ad24049a-6893-4d5e-bb79-0e176fc731c3 nodeName:}" failed. No retries permitted until 2026-07-06 22:47:05.086884548 +0000 UTC m=+37.491134432 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ad24049a-6893-4d5e-bb79-0e176fc731c3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-dpdfh" (UID: "ad24049a-6893-4d5e-bb79-0e176fc731c3") : secret "cluster-monitoring-operator-tls" not found
Jul 06 22:47:04.586956 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.586917    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e00d4ca-8ce5-4ec1-8895-fe762a2d9e4c-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-z62hq\" (UID: \"6e00d4ca-8ce5-4ec1-8895-fe762a2d9e4c\") " pod="openshift-insights/insights-operator-5c847dcfc4-z62hq"
Jul 06 22:47:04.586956 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.586938    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjm4q\" (UniqueName: \"kubernetes.io/projected/6e00d4ca-8ce5-4ec1-8895-fe762a2d9e4c-kube-api-access-mjm4q\") pod \"insights-operator-5c847dcfc4-z62hq\" (UID: \"6e00d4ca-8ce5-4ec1-8895-fe762a2d9e4c\") " pod="openshift-insights/insights-operator-5c847dcfc4-z62hq"
Jul 06 22:47:04.587144 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.586993    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6e00d4ca-8ce5-4ec1-8895-fe762a2d9e4c-serving-cert\") pod \"insights-operator-5c847dcfc4-z62hq\" (UID: \"6e00d4ca-8ce5-4ec1-8895-fe762a2d9e4c\") " pod="openshift-insights/insights-operator-5c847dcfc4-z62hq"
Jul 06 22:47:04.587144 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.587018    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e00d4ca-8ce5-4ec1-8895-fe762a2d9e4c-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-z62hq\" (UID: \"6e00d4ca-8ce5-4ec1-8895-fe762a2d9e4c\") " pod="openshift-insights/insights-operator-5c847dcfc4-z62hq"
Jul 06 22:47:04.587144 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.587046    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjv7w\" (UniqueName: \"kubernetes.io/projected/ad24049a-6893-4d5e-bb79-0e176fc731c3-kube-api-access-fjv7w\") pod \"cluster-monitoring-operator-64786d9cbb-dpdfh\" (UID: \"ad24049a-6893-4d5e-bb79-0e176fc731c3\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpdfh"
Jul 06 22:47:04.587569 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.587549    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ad24049a-6893-4d5e-bb79-0e176fc731c3-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-dpdfh\" (UID: \"ad24049a-6893-4d5e-bb79-0e176fc731c3\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpdfh"
Jul 06 22:47:04.587927 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.587908    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 06 22:47:04.592321 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.592303    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-z62hq"]
Jul 06 22:47:04.603396 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.603372    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjv7w\" (UniqueName: \"kubernetes.io/projected/ad24049a-6893-4d5e-bb79-0e176fc731c3-kube-api-access-fjv7w\") pod \"cluster-monitoring-operator-64786d9cbb-dpdfh\" (UID: \"ad24049a-6893-4d5e-bb79-0e176fc731c3\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpdfh"
Jul 06 22:47:04.659945 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.659922    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-65b4d9bcf5-kkhtq"]
Jul 06 22:47:04.661661 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.661644    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-65b4d9bcf5-kkhtq"
Jul 06 22:47:04.666300 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.666279    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 06 22:47:04.666940 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.666922    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 06 22:47:04.667027 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.666931    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 06 22:47:04.667630 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.667608    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 06 22:47:04.667735 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.667646    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-5td4z\""
Jul 06 22:47:04.667844 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.667828    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 06 22:47:04.667912 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.667852    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 06 22:47:04.681044 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.681023    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-65b4d9bcf5-kkhtq"]
Jul 06 22:47:04.687633 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.687614    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6e00d4ca-8ce5-4ec1-8895-fe762a2d9e4c-serving-cert\") pod \"insights-operator-5c847dcfc4-z62hq\" (UID: \"6e00d4ca-8ce5-4ec1-8895-fe762a2d9e4c\") " pod="openshift-insights/insights-operator-5c847dcfc4-z62hq"
Jul 06 22:47:04.687727 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.687642    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e6a1480-5c80-484b-87e0-78d5a855594e-metrics-certs\") pod \"router-default-65b4d9bcf5-kkhtq\" (UID: \"3e6a1480-5c80-484b-87e0-78d5a855594e\") " pod="openshift-ingress/router-default-65b4d9bcf5-kkhtq"
Jul 06 22:47:04.687727 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.687669    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e00d4ca-8ce5-4ec1-8895-fe762a2d9e4c-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-z62hq\" (UID: \"6e00d4ca-8ce5-4ec1-8895-fe762a2d9e4c\") " pod="openshift-insights/insights-operator-5c847dcfc4-z62hq"
Jul 06 22:47:04.687727 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.687695    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3e6a1480-5c80-484b-87e0-78d5a855594e-default-certificate\") pod \"router-default-65b4d9bcf5-kkhtq\" (UID: \"3e6a1480-5c80-484b-87e0-78d5a855594e\") " pod="openshift-ingress/router-default-65b4d9bcf5-kkhtq"
Jul 06 22:47:04.687727 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.687722    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3e6a1480-5c80-484b-87e0-78d5a855594e-stats-auth\") pod \"router-default-65b4d9bcf5-kkhtq\" (UID: \"3e6a1480-5c80-484b-87e0-78d5a855594e\") " pod="openshift-ingress/router-default-65b4d9bcf5-kkhtq"
Jul 06 22:47:04.687903 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.687747    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e6a1480-5c80-484b-87e0-78d5a855594e-service-ca-bundle\") pod \"router-default-65b4d9bcf5-kkhtq\" (UID: \"3e6a1480-5c80-484b-87e0-78d5a855594e\") " pod="openshift-ingress/router-default-65b4d9bcf5-kkhtq"
Jul 06 22:47:04.687903 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.687821    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e00d4ca-8ce5-4ec1-8895-fe762a2d9e4c-tmp\") pod \"insights-operator-5c847dcfc4-z62hq\" (UID: \"6e00d4ca-8ce5-4ec1-8895-fe762a2d9e4c\") " pod="openshift-insights/insights-operator-5c847dcfc4-z62hq"
Jul 06 22:47:04.687903 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.687868    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6e00d4ca-8ce5-4ec1-8895-fe762a2d9e4c-snapshots\") pod \"insights-operator-5c847dcfc4-z62hq\" (UID: \"6e00d4ca-8ce5-4ec1-8895-fe762a2d9e4c\") " pod="openshift-insights/insights-operator-5c847dcfc4-z62hq"
Jul 06 22:47:04.687903 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.687887    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e00d4ca-8ce5-4ec1-8895-fe762a2d9e4c-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-z62hq\" (UID: \"6e00d4ca-8ce5-4ec1-8895-fe762a2d9e4c\") " pod="openshift-insights/insights-operator-5c847dcfc4-z62hq"
Jul 06 22:47:04.688091 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.687906    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjm4q\" (UniqueName: \"kubernetes.io/projected/6e00d4ca-8ce5-4ec1-8895-fe762a2d9e4c-kube-api-access-mjm4q\") pod \"insights-operator-5c847dcfc4-z62hq\" (UID: \"6e00d4ca-8ce5-4ec1-8895-fe762a2d9e4c\") " pod="openshift-insights/insights-operator-5c847dcfc4-z62hq"
Jul 06 22:47:04.688091 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.687930    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khr2s\" (UniqueName: \"kubernetes.io/projected/3e6a1480-5c80-484b-87e0-78d5a855594e-kube-api-access-khr2s\") pod \"router-default-65b4d9bcf5-kkhtq\" (UID: \"3e6a1480-5c80-484b-87e0-78d5a855594e\") " pod="openshift-ingress/router-default-65b4d9bcf5-kkhtq"
Jul 06 22:47:04.688343 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.688305    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e00d4ca-8ce5-4ec1-8895-fe762a2d9e4c-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-z62hq\" (UID: \"6e00d4ca-8ce5-4ec1-8895-fe762a2d9e4c\") " pod="openshift-insights/insights-operator-5c847dcfc4-z62hq"
Jul 06 22:47:04.688471 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.688448    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e00d4ca-8ce5-4ec1-8895-fe762a2d9e4c-tmp\") pod \"insights-operator-5c847dcfc4-z62hq\" (UID: \"6e00d4ca-8ce5-4ec1-8895-fe762a2d9e4c\") " pod="openshift-insights/insights-operator-5c847dcfc4-z62hq"
Jul 06 22:47:04.688543 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.688520    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6e00d4ca-8ce5-4ec1-8895-fe762a2d9e4c-snapshots\") pod \"insights-operator-5c847dcfc4-z62hq\" (UID: \"6e00d4ca-8ce5-4ec1-8895-fe762a2d9e4c\") " pod="openshift-insights/insights-operator-5c847dcfc4-z62hq"
Jul 06 22:47:04.688781 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.688765    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e00d4ca-8ce5-4ec1-8895-fe762a2d9e4c-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-z62hq\" (UID: \"6e00d4ca-8ce5-4ec1-8895-fe762a2d9e4c\") " pod="openshift-insights/insights-operator-5c847dcfc4-z62hq"
Jul 06 22:47:04.690136 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.690113    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6e00d4ca-8ce5-4ec1-8895-fe762a2d9e4c-serving-cert\") pod \"insights-operator-5c847dcfc4-z62hq\" (UID: \"6e00d4ca-8ce5-4ec1-8895-fe762a2d9e4c\") " pod="openshift-insights/insights-operator-5c847dcfc4-z62hq"
Jul 06 22:47:04.703468 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.703419    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjm4q\" (UniqueName: \"kubernetes.io/projected/6e00d4ca-8ce5-4ec1-8895-fe762a2d9e4c-kube-api-access-mjm4q\") pod \"insights-operator-5c847dcfc4-z62hq\" (UID: \"6e00d4ca-8ce5-4ec1-8895-fe762a2d9e4c\") " pod="openshift-insights/insights-operator-5c847dcfc4-z62hq"
Jul 06 22:47:04.788609 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.788581    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3e6a1480-5c80-484b-87e0-78d5a855594e-default-certificate\") pod \"router-default-65b4d9bcf5-kkhtq\" (UID: \"3e6a1480-5c80-484b-87e0-78d5a855594e\") " pod="openshift-ingress/router-default-65b4d9bcf5-kkhtq"
Jul 06 22:47:04.788715 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.788614    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3e6a1480-5c80-484b-87e0-78d5a855594e-stats-auth\") pod \"router-default-65b4d9bcf5-kkhtq\" (UID: \"3e6a1480-5c80-484b-87e0-78d5a855594e\") " pod="openshift-ingress/router-default-65b4d9bcf5-kkhtq"
Jul 06 22:47:04.788715 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.788636    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e6a1480-5c80-484b-87e0-78d5a855594e-service-ca-bundle\") pod \"router-default-65b4d9bcf5-kkhtq\" (UID: \"3e6a1480-5c80-484b-87e0-78d5a855594e\") " pod="openshift-ingress/router-default-65b4d9bcf5-kkhtq"
Jul 06 22:47:04.788715 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.788709    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khr2s\" (UniqueName: \"kubernetes.io/projected/3e6a1480-5c80-484b-87e0-78d5a855594e-kube-api-access-khr2s\") pod \"router-default-65b4d9bcf5-kkhtq\" (UID: \"3e6a1480-5c80-484b-87e0-78d5a855594e\") " pod="openshift-ingress/router-default-65b4d9bcf5-kkhtq"
Jul 06 22:47:04.788942 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.788750    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e6a1480-5c80-484b-87e0-78d5a855594e-metrics-certs\") pod \"router-default-65b4d9bcf5-kkhtq\" (UID: \"3e6a1480-5c80-484b-87e0-78d5a855594e\") " pod="openshift-ingress/router-default-65b4d9bcf5-kkhtq"
Jul 06 22:47:04.788942 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:04.788848    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 06 22:47:04.788942 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:04.788875    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3e6a1480-5c80-484b-87e0-78d5a855594e-service-ca-bundle podName:3e6a1480-5c80-484b-87e0-78d5a855594e nodeName:}" failed. No retries permitted until 2026-07-06 22:47:05.288856025 +0000 UTC m=+37.693105923 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3e6a1480-5c80-484b-87e0-78d5a855594e-service-ca-bundle") pod "router-default-65b4d9bcf5-kkhtq" (UID: "3e6a1480-5c80-484b-87e0-78d5a855594e") : configmap references non-existent config key: service-ca.crt
Jul 06 22:47:04.788942 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:04.788896    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e6a1480-5c80-484b-87e0-78d5a855594e-metrics-certs podName:3e6a1480-5c80-484b-87e0-78d5a855594e nodeName:}" failed. No retries permitted until 2026-07-06 22:47:05.288885668 +0000 UTC m=+37.693135553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3e6a1480-5c80-484b-87e0-78d5a855594e-metrics-certs") pod "router-default-65b4d9bcf5-kkhtq" (UID: "3e6a1480-5c80-484b-87e0-78d5a855594e") : secret "router-metrics-certs-default" not found
Jul 06 22:47:04.791075 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.791053    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3e6a1480-5c80-484b-87e0-78d5a855594e-default-certificate\") pod \"router-default-65b4d9bcf5-kkhtq\" (UID: \"3e6a1480-5c80-484b-87e0-78d5a855594e\") " pod="openshift-ingress/router-default-65b4d9bcf5-kkhtq"
Jul 06 22:47:04.791455 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.791436    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3e6a1480-5c80-484b-87e0-78d5a855594e-stats-auth\") pod \"router-default-65b4d9bcf5-kkhtq\" (UID: \"3e6a1480-5c80-484b-87e0-78d5a855594e\") " pod="openshift-ingress/router-default-65b4d9bcf5-kkhtq"
Jul 06 22:47:04.800033 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.800013    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khr2s\" (UniqueName: \"kubernetes.io/projected/3e6a1480-5c80-484b-87e0-78d5a855594e-kube-api-access-khr2s\") pod \"router-default-65b4d9bcf5-kkhtq\" (UID: \"3e6a1480-5c80-484b-87e0-78d5a855594e\") " pod="openshift-ingress/router-default-65b4d9bcf5-kkhtq"
Jul 06 22:47:04.936426 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:04.936398    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-z62hq"
Jul 06 22:47:05.066861 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:05.066777    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-z62hq"]
Jul 06 22:47:05.070564 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:47:05.070538    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e00d4ca_8ce5_4ec1_8895_fe762a2d9e4c.slice/crio-46682512c761af64f7652c51e15d457446370015566ef305e3a3aaf06d5c736f WatchSource:0}: Error finding container 46682512c761af64f7652c51e15d457446370015566ef305e3a3aaf06d5c736f: Status 404 returned error can't find the container with id 46682512c761af64f7652c51e15d457446370015566ef305e3a3aaf06d5c736f
Jul 06 22:47:05.092280 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:05.092253    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad24049a-6893-4d5e-bb79-0e176fc731c3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-dpdfh\" (UID: \"ad24049a-6893-4d5e-bb79-0e176fc731c3\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpdfh"
Jul 06 22:47:05.092441 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:05.092420    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 06 22:47:05.092510 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:05.092495    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad24049a-6893-4d5e-bb79-0e176fc731c3-cluster-monitoring-operator-tls podName:ad24049a-6893-4d5e-bb79-0e176fc731c3 nodeName:}" failed. No retries permitted until 2026-07-06 22:47:06.092475217 +0000 UTC m=+38.496725118 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ad24049a-6893-4d5e-bb79-0e176fc731c3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-dpdfh" (UID: "ad24049a-6893-4d5e-bb79-0e176fc731c3") : secret "cluster-monitoring-operator-tls" not found
Jul 06 22:47:05.164753 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:05.164728    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j9csq"
Jul 06 22:47:05.164867 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:05.164728    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfmlk"
Jul 06 22:47:05.164867 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:05.164728    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6fbb"
Jul 06 22:47:05.167846 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:05.167550    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 06 22:47:05.167846 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:05.167642    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 06 22:47:05.167846 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:05.167651    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 06 22:47:05.167846 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:05.167773    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 06 22:47:05.167846 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:05.167804    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lkkl7\""
Jul 06 22:47:05.167846 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:05.167809    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5j2wj\""
Jul 06 22:47:05.193137 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:05.193114    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a674edfd-a962-4327-b319-320fb62922c0-metrics-tls\") pod \"dns-default-2xhr2\" (UID: \"a674edfd-a962-4327-b319-320fb62922c0\") " pod="openshift-dns/dns-default-2xhr2"
Jul 06 22:47:05.193685 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:05.193266    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 22:47:05.193685 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:05.193326    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a674edfd-a962-4327-b319-320fb62922c0-metrics-tls podName:a674edfd-a962-4327-b319-320fb62922c0 nodeName:}" failed. No retries permitted until 2026-07-06 22:47:07.193308748 +0000 UTC m=+39.597558634 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a674edfd-a962-4327-b319-320fb62922c0-metrics-tls") pod "dns-default-2xhr2" (UID: "a674edfd-a962-4327-b319-320fb62922c0") : secret "dns-default-metrics-tls" not found
Jul 06 22:47:05.294042 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:05.293979    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e6a1480-5c80-484b-87e0-78d5a855594e-metrics-certs\") pod \"router-default-65b4d9bcf5-kkhtq\" (UID: \"3e6a1480-5c80-484b-87e0-78d5a855594e\") " pod="openshift-ingress/router-default-65b4d9bcf5-kkhtq"
Jul 06 22:47:05.294042 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:05.294023    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e6a1480-5c80-484b-87e0-78d5a855594e-service-ca-bundle\") pod \"router-default-65b4d9bcf5-kkhtq\" (UID: \"3e6a1480-5c80-484b-87e0-78d5a855594e\") " pod="openshift-ingress/router-default-65b4d9bcf5-kkhtq"
Jul 06 22:47:05.294228 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:05.294131    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 06 22:47:05.294228 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:05.294210    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e6a1480-5c80-484b-87e0-78d5a855594e-metrics-certs podName:3e6a1480-5c80-484b-87e0-78d5a855594e nodeName:}" failed. No retries permitted until 2026-07-06 22:47:06.294190434 +0000 UTC m=+38.698440339 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3e6a1480-5c80-484b-87e0-78d5a855594e-metrics-certs") pod "router-default-65b4d9bcf5-kkhtq" (UID: "3e6a1480-5c80-484b-87e0-78d5a855594e") : secret "router-metrics-certs-default" not found
Jul 06 22:47:05.294341 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:05.294241    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3e6a1480-5c80-484b-87e0-78d5a855594e-service-ca-bundle podName:3e6a1480-5c80-484b-87e0-78d5a855594e nodeName:}" failed. No retries permitted until 2026-07-06 22:47:06.294222295 +0000 UTC m=+38.698472184 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3e6a1480-5c80-484b-87e0-78d5a855594e-service-ca-bundle") pod "router-default-65b4d9bcf5-kkhtq" (UID: "3e6a1480-5c80-484b-87e0-78d5a855594e") : configmap references non-existent config key: service-ca.crt
Jul 06 22:47:05.479902 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:05.479866    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-z62hq" event={"ID":"6e00d4ca-8ce5-4ec1-8895-fe762a2d9e4c","Type":"ContainerStarted","Data":"46682512c761af64f7652c51e15d457446370015566ef305e3a3aaf06d5c736f"}
Jul 06 22:47:05.481050 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:05.481018    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dwnzz" event={"ID":"8624c1c4-bd8a-4e14-9445-3c7352c1cd60","Type":"ContainerStarted","Data":"735fe1c801d7614b243de97376a0d7237cd37f1ad947c7699fc556181f1ca719"}
Jul 06 22:47:05.484121 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:05.484096    2569 generic.go:356] "Generic (PLEG): container finished" podID="3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90" containerID="6b7331014165060f28c2dd5b66103a4b1814e2ae6df3c9f5063ed492fc526811" exitCode=0
Jul 06 22:47:05.484252 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:05.484135    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qgqcr" event={"ID":"3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90","Type":"ContainerDied","Data":"6b7331014165060f28c2dd5b66103a4b1814e2ae6df3c9f5063ed492fc526811"}
Jul 06 22:47:06.102265 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:06.102233    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad24049a-6893-4d5e-bb79-0e176fc731c3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-dpdfh\" (UID: \"ad24049a-6893-4d5e-bb79-0e176fc731c3\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpdfh"
Jul 06 22:47:06.102439 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:06.102378    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 06 22:47:06.102502 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:06.102445    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad24049a-6893-4d5e-bb79-0e176fc731c3-cluster-monitoring-operator-tls podName:ad24049a-6893-4d5e-bb79-0e176fc731c3 nodeName:}" failed. No retries permitted until 2026-07-06 22:47:08.102424475 +0000 UTC m=+40.506674370 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ad24049a-6893-4d5e-bb79-0e176fc731c3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-dpdfh" (UID: "ad24049a-6893-4d5e-bb79-0e176fc731c3") : secret "cluster-monitoring-operator-tls" not found
Jul 06 22:47:06.303893 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:06.303855    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e6a1480-5c80-484b-87e0-78d5a855594e-metrics-certs\") pod \"router-default-65b4d9bcf5-kkhtq\" (UID: \"3e6a1480-5c80-484b-87e0-78d5a855594e\") " pod="openshift-ingress/router-default-65b4d9bcf5-kkhtq"
Jul 06 22:47:06.304414 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:06.303906    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e6a1480-5c80-484b-87e0-78d5a855594e-service-ca-bundle\") pod \"router-default-65b4d9bcf5-kkhtq\" (UID: \"3e6a1480-5c80-484b-87e0-78d5a855594e\") " pod="openshift-ingress/router-default-65b4d9bcf5-kkhtq"
Jul 06 22:47:06.304414 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:06.304033    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 06 22:47:06.304414 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:06.304040    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3e6a1480-5c80-484b-87e0-78d5a855594e-service-ca-bundle podName:3e6a1480-5c80-484b-87e0-78d5a855594e nodeName:}" failed. No retries permitted until 2026-07-06 22:47:08.30402629 +0000 UTC m=+40.708276172 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3e6a1480-5c80-484b-87e0-78d5a855594e-service-ca-bundle") pod "router-default-65b4d9bcf5-kkhtq" (UID: "3e6a1480-5c80-484b-87e0-78d5a855594e") : configmap references non-existent config key: service-ca.crt
Jul 06 22:47:06.304414 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:06.304200    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e6a1480-5c80-484b-87e0-78d5a855594e-metrics-certs podName:3e6a1480-5c80-484b-87e0-78d5a855594e nodeName:}" failed. No retries permitted until 2026-07-06 22:47:08.304166099 +0000 UTC m=+40.708415986 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3e6a1480-5c80-484b-87e0-78d5a855594e-metrics-certs") pod "router-default-65b4d9bcf5-kkhtq" (UID: "3e6a1480-5c80-484b-87e0-78d5a855594e") : secret "router-metrics-certs-default" not found
Jul 06 22:47:06.491516 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:06.491396    2569 generic.go:356] "Generic (PLEG): container finished" podID="3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90" containerID="a936a2194dd6898c1957d5f911195a08f0859028191cc28a12e5ac9ad77ea1ba" exitCode=0
Jul 06 22:47:06.491516 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:06.491446    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qgqcr" event={"ID":"3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90","Type":"ContainerDied","Data":"a936a2194dd6898c1957d5f911195a08f0859028191cc28a12e5ac9ad77ea1ba"}
Jul 06 22:47:07.211446 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:07.211404    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a674edfd-a962-4327-b319-320fb62922c0-metrics-tls\") pod \"dns-default-2xhr2\" (UID: \"a674edfd-a962-4327-b319-320fb62922c0\") " pod="openshift-dns/dns-default-2xhr2"
Jul 06 22:47:07.211626 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:07.211546    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 22:47:07.211690 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:07.211628    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a674edfd-a962-4327-b319-320fb62922c0-metrics-tls podName:a674edfd-a962-4327-b319-320fb62922c0 nodeName:}" failed. No retries permitted until 2026-07-06 22:47:11.211606273 +0000 UTC m=+43.615856156 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a674edfd-a962-4327-b319-320fb62922c0-metrics-tls") pod "dns-default-2xhr2" (UID: "a674edfd-a962-4327-b319-320fb62922c0") : secret "dns-default-metrics-tls" not found
Jul 06 22:47:07.495499 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:07.495296    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dwnzz" event={"ID":"8624c1c4-bd8a-4e14-9445-3c7352c1cd60","Type":"ContainerStarted","Data":"99396b0de7e2712333b1c53dee78beba1990a52c3305ecf086f3fed76dd019f5"}
Jul 06 22:47:07.498984 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:07.498957    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qgqcr" event={"ID":"3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90","Type":"ContainerStarted","Data":"1ae609be55f23ece9ecf38b55f23e8a7aec7786c271ed2fa35e06f738023f87a"}
Jul 06 22:47:07.513210 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:07.513147    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dwnzz" podStartSLOduration=1.623105459 podStartE2EDuration="4.513131968s" totalImagesPullingTime="2.890026509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:47:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:47:07.511751686 +0000 UTC m=+39.916001591" watchObservedRunningTime="2026-07-06 22:47:07.513131968 +0000 UTC m=+39.917381871"
Jul 06 22:47:07.535720 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:07.535673    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qgqcr" podStartSLOduration=9.10048267 podStartE2EDuration="38.535660612s" totalImagesPullingTime="29.435177942s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:46:29 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:47:07.534605293 +0000 UTC m=+39.938855195" watchObservedRunningTime="2026-07-06 22:47:07.535660612 +0000 UTC m=+39.939910516"
Jul 06 22:47:08.117341 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:08.117266    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad24049a-6893-4d5e-bb79-0e176fc731c3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-dpdfh\" (UID: \"ad24049a-6893-4d5e-bb79-0e176fc731c3\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpdfh"
Jul 06 22:47:08.117491 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:08.117404    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 06 22:47:08.117491 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:08.117475    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad24049a-6893-4d5e-bb79-0e176fc731c3-cluster-monitoring-operator-tls podName:ad24049a-6893-4d5e-bb79-0e176fc731c3 nodeName:}" failed. No retries permitted until 2026-07-06 22:47:12.117454738 +0000 UTC m=+44.521704621 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ad24049a-6893-4d5e-bb79-0e176fc731c3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-dpdfh" (UID: "ad24049a-6893-4d5e-bb79-0e176fc731c3") : secret "cluster-monitoring-operator-tls" not found
Jul 06 22:47:08.318477 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:08.318443    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e6a1480-5c80-484b-87e0-78d5a855594e-metrics-certs\") pod \"router-default-65b4d9bcf5-kkhtq\" (UID: \"3e6a1480-5c80-484b-87e0-78d5a855594e\") " pod="openshift-ingress/router-default-65b4d9bcf5-kkhtq"
Jul 06 22:47:08.318622 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:08.318487    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e6a1480-5c80-484b-87e0-78d5a855594e-service-ca-bundle\") pod \"router-default-65b4d9bcf5-kkhtq\" (UID: \"3e6a1480-5c80-484b-87e0-78d5a855594e\") " pod="openshift-ingress/router-default-65b4d9bcf5-kkhtq"
Jul 06 22:47:08.318622 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:08.318587    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 06 22:47:08.318685 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:08.318639    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3e6a1480-5c80-484b-87e0-78d5a855594e-service-ca-bundle podName:3e6a1480-5c80-484b-87e0-78d5a855594e nodeName:}" failed. No retries permitted until 2026-07-06 22:47:12.318624048 +0000 UTC m=+44.722873931 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3e6a1480-5c80-484b-87e0-78d5a855594e-service-ca-bundle") pod "router-default-65b4d9bcf5-kkhtq" (UID: "3e6a1480-5c80-484b-87e0-78d5a855594e") : configmap references non-existent config key: service-ca.crt
Jul 06 22:47:08.318685 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:08.318657    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e6a1480-5c80-484b-87e0-78d5a855594e-metrics-certs podName:3e6a1480-5c80-484b-87e0-78d5a855594e nodeName:}" failed. No retries permitted until 2026-07-06 22:47:12.318650199 +0000 UTC m=+44.722900082 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3e6a1480-5c80-484b-87e0-78d5a855594e-metrics-certs") pod "router-default-65b4d9bcf5-kkhtq" (UID: "3e6a1480-5c80-484b-87e0-78d5a855594e") : secret "router-metrics-certs-default" not found
Jul 06 22:47:08.503155 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:08.503119    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-z62hq" event={"ID":"6e00d4ca-8ce5-4ec1-8895-fe762a2d9e4c","Type":"ContainerStarted","Data":"a4b850c3205b71271f1637054cf94b30681ed5c094f132f9126c5b423bd33dac"}
Jul 06 22:47:08.520349 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:08.520305    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-z62hq" podStartSLOduration=1.873325015 podStartE2EDuration="4.520290545s" totalImagesPullingTime="2.64696553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:47:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:47:08.519355678 +0000 UTC m=+40.923605580" watchObservedRunningTime="2026-07-06 22:47:08.520290545 +0000 UTC m=+40.924540448"
Jul 06 22:47:09.224619 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:09.224586    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8bd6t"]
Jul 06 22:47:09.254160 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:09.254132    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8bd6t"]
Jul 06 22:47:09.254319 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:09.254252    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8bd6t"
Jul 06 22:47:09.256901 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:09.256873    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 06 22:47:09.256901 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:09.256891    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4p5jh\""
Jul 06 22:47:09.257218 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:09.257171    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 06 22:47:09.326989 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:09.326966    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c13e679b-e3c2-432e-b81d-619f3e346e92-data-volume\") pod \"insights-runtime-extractor-8bd6t\" (UID: \"c13e679b-e3c2-432e-b81d-619f3e346e92\") " pod="openshift-insights/insights-runtime-extractor-8bd6t"
Jul 06 22:47:09.327093 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:09.327028    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c13e679b-e3c2-432e-b81d-619f3e346e92-crio-socket\") pod \"insights-runtime-extractor-8bd6t\" (UID: \"c13e679b-e3c2-432e-b81d-619f3e346e92\") " pod="openshift-insights/insights-runtime-extractor-8bd6t"
Jul 06 22:47:09.327093 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:09.327053    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ksc8\" (UniqueName: \"kubernetes.io/projected/c13e679b-e3c2-432e-b81d-619f3e346e92-kube-api-access-5ksc8\") pod \"insights-runtime-extractor-8bd6t\" (UID: \"c13e679b-e3c2-432e-b81d-619f3e346e92\") " pod="openshift-insights/insights-runtime-extractor-8bd6t"
Jul 06 22:47:09.327093 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:09.327084    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c13e679b-e3c2-432e-b81d-619f3e346e92-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8bd6t\" (UID: \"c13e679b-e3c2-432e-b81d-619f3e346e92\") " pod="openshift-insights/insights-runtime-extractor-8bd6t"
Jul 06 22:47:09.327250 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:09.327173    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c13e679b-e3c2-432e-b81d-619f3e346e92-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8bd6t\" (UID: \"c13e679b-e3c2-432e-b81d-619f3e346e92\") " pod="openshift-insights/insights-runtime-extractor-8bd6t"
Jul 06 22:47:09.427764 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:09.427737    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c13e679b-e3c2-432e-b81d-619f3e346e92-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8bd6t\" (UID: \"c13e679b-e3c2-432e-b81d-619f3e346e92\") " pod="openshift-insights/insights-runtime-extractor-8bd6t"
Jul 06 22:47:09.427869 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:09.427766    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c13e679b-e3c2-432e-b81d-619f3e346e92-data-volume\") pod \"insights-runtime-extractor-8bd6t\" (UID: \"c13e679b-e3c2-432e-b81d-619f3e346e92\") " pod="openshift-insights/insights-runtime-extractor-8bd6t"
Jul 06 22:47:09.427869 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:09.427810    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c13e679b-e3c2-432e-b81d-619f3e346e92-crio-socket\") pod \"insights-runtime-extractor-8bd6t\" (UID: \"c13e679b-e3c2-432e-b81d-619f3e346e92\") " pod="openshift-insights/insights-runtime-extractor-8bd6t"
Jul 06 22:47:09.427869 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:09.427836    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ksc8\" (UniqueName: \"kubernetes.io/projected/c13e679b-e3c2-432e-b81d-619f3e346e92-kube-api-access-5ksc8\") pod \"insights-runtime-extractor-8bd6t\" (UID: \"c13e679b-e3c2-432e-b81d-619f3e346e92\") " pod="openshift-insights/insights-runtime-extractor-8bd6t"
Jul 06 22:47:09.427869 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:09.427854    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c13e679b-e3c2-432e-b81d-619f3e346e92-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8bd6t\" (UID: \"c13e679b-e3c2-432e-b81d-619f3e346e92\") " pod="openshift-insights/insights-runtime-extractor-8bd6t"
Jul 06 22:47:09.428068 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:09.427897    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 22:47:09.428068 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:09.427937    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 22:47:09.428068 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:09.427972    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c13e679b-e3c2-432e-b81d-619f3e346e92-insights-runtime-extractor-tls podName:c13e679b-e3c2-432e-b81d-619f3e346e92 nodeName:}" failed. No retries permitted until 2026-07-06 22:47:09.927950196 +0000 UTC m=+42.332200102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c13e679b-e3c2-432e-b81d-619f3e346e92-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8bd6t" (UID: "c13e679b-e3c2-432e-b81d-619f3e346e92") : secret "insights-runtime-extractor-tls" not found
Jul 06 22:47:09.428068 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:09.427991    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c13e679b-e3c2-432e-b81d-619f3e346e92-kube-rbac-proxy-cm podName:c13e679b-e3c2-432e-b81d-619f3e346e92 nodeName:}" failed. No retries permitted until 2026-07-06 22:47:09.927981131 +0000 UTC m=+42.332231015 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c13e679b-e3c2-432e-b81d-619f3e346e92-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8bd6t" (UID: "c13e679b-e3c2-432e-b81d-619f3e346e92") : configmap "kube-rbac-proxy" not found
Jul 06 22:47:09.428068 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:09.428032    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c13e679b-e3c2-432e-b81d-619f3e346e92-data-volume\") pod \"insights-runtime-extractor-8bd6t\" (UID: \"c13e679b-e3c2-432e-b81d-619f3e346e92\") " pod="openshift-insights/insights-runtime-extractor-8bd6t"
Jul 06 22:47:09.428068 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:09.428039    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c13e679b-e3c2-432e-b81d-619f3e346e92-crio-socket\") pod \"insights-runtime-extractor-8bd6t\" (UID: \"c13e679b-e3c2-432e-b81d-619f3e346e92\") " pod="openshift-insights/insights-runtime-extractor-8bd6t"
Jul 06 22:47:09.437129 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:09.437111    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ksc8\" (UniqueName: \"kubernetes.io/projected/c13e679b-e3c2-432e-b81d-619f3e346e92-kube-api-access-5ksc8\") pod \"insights-runtime-extractor-8bd6t\" (UID: \"c13e679b-e3c2-432e-b81d-619f3e346e92\") " pod="openshift-insights/insights-runtime-extractor-8bd6t"
Jul 06 22:47:09.931161 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:09.931129    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c13e679b-e3c2-432e-b81d-619f3e346e92-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8bd6t\" (UID: \"c13e679b-e3c2-432e-b81d-619f3e346e92\") " pod="openshift-insights/insights-runtime-extractor-8bd6t"
Jul 06 22:47:09.931536 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:09.931237    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c13e679b-e3c2-432e-b81d-619f3e346e92-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8bd6t\" (UID: \"c13e679b-e3c2-432e-b81d-619f3e346e92\") " pod="openshift-insights/insights-runtime-extractor-8bd6t"
Jul 06 22:47:09.931536 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:09.931262    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 22:47:09.931536 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:09.931323    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c13e679b-e3c2-432e-b81d-619f3e346e92-kube-rbac-proxy-cm podName:c13e679b-e3c2-432e-b81d-619f3e346e92 nodeName:}" failed. No retries permitted until 2026-07-06 22:47:10.931305073 +0000 UTC m=+43.335554968 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c13e679b-e3c2-432e-b81d-619f3e346e92-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8bd6t" (UID: "c13e679b-e3c2-432e-b81d-619f3e346e92") : configmap "kube-rbac-proxy" not found
Jul 06 22:47:09.931536 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:09.931370    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 22:47:09.931536 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:09.931425    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c13e679b-e3c2-432e-b81d-619f3e346e92-insights-runtime-extractor-tls podName:c13e679b-e3c2-432e-b81d-619f3e346e92 nodeName:}" failed. No retries permitted until 2026-07-06 22:47:10.931410203 +0000 UTC m=+43.335660084 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c13e679b-e3c2-432e-b81d-619f3e346e92-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8bd6t" (UID: "c13e679b-e3c2-432e-b81d-619f3e346e92") : secret "insights-runtime-extractor-tls" not found
Jul 06 22:47:10.937944 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:10.937912    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c13e679b-e3c2-432e-b81d-619f3e346e92-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8bd6t\" (UID: \"c13e679b-e3c2-432e-b81d-619f3e346e92\") " pod="openshift-insights/insights-runtime-extractor-8bd6t"
Jul 06 22:47:10.938320 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:10.937987    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c13e679b-e3c2-432e-b81d-619f3e346e92-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8bd6t\" (UID: \"c13e679b-e3c2-432e-b81d-619f3e346e92\") " pod="openshift-insights/insights-runtime-extractor-8bd6t"
Jul 06 22:47:10.938320 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:10.938034    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 22:47:10.938320 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:10.938079    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 22:47:10.938320 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:10.938103    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c13e679b-e3c2-432e-b81d-619f3e346e92-kube-rbac-proxy-cm podName:c13e679b-e3c2-432e-b81d-619f3e346e92 nodeName:}" failed. No retries permitted until 2026-07-06 22:47:12.938083762 +0000 UTC m=+45.342333644 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c13e679b-e3c2-432e-b81d-619f3e346e92-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8bd6t" (UID: "c13e679b-e3c2-432e-b81d-619f3e346e92") : configmap "kube-rbac-proxy" not found
Jul 06 22:47:10.938320 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:10.938122    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c13e679b-e3c2-432e-b81d-619f3e346e92-insights-runtime-extractor-tls podName:c13e679b-e3c2-432e-b81d-619f3e346e92 nodeName:}" failed. No retries permitted until 2026-07-06 22:47:12.938110256 +0000 UTC m=+45.342360137 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c13e679b-e3c2-432e-b81d-619f3e346e92-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8bd6t" (UID: "c13e679b-e3c2-432e-b81d-619f3e346e92") : secret "insights-runtime-extractor-tls" not found
Jul 06 22:47:11.192825 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:11.192763    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wdchz_bc58eac9-19c3-4a2c-be0f-2f5211fc5f39/dns-node-resolver/0.log"
Jul 06 22:47:11.240160 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:11.240134    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a674edfd-a962-4327-b319-320fb62922c0-metrics-tls\") pod \"dns-default-2xhr2\" (UID: \"a674edfd-a962-4327-b319-320fb62922c0\") " pod="openshift-dns/dns-default-2xhr2"
Jul 06 22:47:11.240297 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:11.240285    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 22:47:11.240342 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:11.240337    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a674edfd-a962-4327-b319-320fb62922c0-metrics-tls podName:a674edfd-a962-4327-b319-320fb62922c0 nodeName:}" failed. No retries permitted until 2026-07-06 22:47:19.240321107 +0000 UTC m=+51.644571001 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a674edfd-a962-4327-b319-320fb62922c0-metrics-tls") pod "dns-default-2xhr2" (UID: "a674edfd-a962-4327-b319-320fb62922c0") : secret "dns-default-metrics-tls" not found
Jul 06 22:47:11.873444 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:11.873415    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-v2lzx"]
Jul 06 22:47:11.892548 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:11.892527    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-v2lzx"]
Jul 06 22:47:11.892676 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:11.892620    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-v2lzx"
Jul 06 22:47:11.895566 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:11.895542    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 06 22:47:11.895665 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:11.895640    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 06 22:47:11.904903 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:11.904885    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-wcf67\""
Jul 06 22:47:11.944791 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:11.944765    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vxs9\" (UniqueName: \"kubernetes.io/projected/6ab91bcb-c3c8-47ef-a1b8-2fb8cafac398-kube-api-access-8vxs9\") pod \"migrator-d57d58cc9-v2lzx\" (UID: \"6ab91bcb-c3c8-47ef-a1b8-2fb8cafac398\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-v2lzx"
Jul 06 22:47:11.995075 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:11.995058    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2zsx8_6b66febe-a59f-40a7-bff3-d248315b326e/node-ca/0.log"
Jul 06 22:47:12.046084 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:12.046063    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vxs9\" (UniqueName: \"kubernetes.io/projected/6ab91bcb-c3c8-47ef-a1b8-2fb8cafac398-kube-api-access-8vxs9\") pod \"migrator-d57d58cc9-v2lzx\" (UID: \"6ab91bcb-c3c8-47ef-a1b8-2fb8cafac398\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-v2lzx"
Jul 06 22:47:12.056005 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:12.055982    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vxs9\" (UniqueName: \"kubernetes.io/projected/6ab91bcb-c3c8-47ef-a1b8-2fb8cafac398-kube-api-access-8vxs9\") pod \"migrator-d57d58cc9-v2lzx\" (UID: \"6ab91bcb-c3c8-47ef-a1b8-2fb8cafac398\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-v2lzx"
Jul 06 22:47:12.146726 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:12.146668    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad24049a-6893-4d5e-bb79-0e176fc731c3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-dpdfh\" (UID: \"ad24049a-6893-4d5e-bb79-0e176fc731c3\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpdfh"
Jul 06 22:47:12.146826 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:12.146809    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 06 22:47:12.146880 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:12.146869    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad24049a-6893-4d5e-bb79-0e176fc731c3-cluster-monitoring-operator-tls podName:ad24049a-6893-4d5e-bb79-0e176fc731c3 nodeName:}" failed. No retries permitted until 2026-07-06 22:47:20.146855459 +0000 UTC m=+52.551105341 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ad24049a-6893-4d5e-bb79-0e176fc731c3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-dpdfh" (UID: "ad24049a-6893-4d5e-bb79-0e176fc731c3") : secret "cluster-monitoring-operator-tls" not found
Jul 06 22:47:12.202147 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:12.202116    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-v2lzx"
Jul 06 22:47:12.348812 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:12.348782    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e6a1480-5c80-484b-87e0-78d5a855594e-service-ca-bundle\") pod \"router-default-65b4d9bcf5-kkhtq\" (UID: \"3e6a1480-5c80-484b-87e0-78d5a855594e\") " pod="openshift-ingress/router-default-65b4d9bcf5-kkhtq"
Jul 06 22:47:12.348971 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:12.348952    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3e6a1480-5c80-484b-87e0-78d5a855594e-service-ca-bundle podName:3e6a1480-5c80-484b-87e0-78d5a855594e nodeName:}" failed. No retries permitted until 2026-07-06 22:47:20.348934488 +0000 UTC m=+52.753184370 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3e6a1480-5c80-484b-87e0-78d5a855594e-service-ca-bundle") pod "router-default-65b4d9bcf5-kkhtq" (UID: "3e6a1480-5c80-484b-87e0-78d5a855594e") : configmap references non-existent config key: service-ca.crt
Jul 06 22:47:12.349046 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:12.349031    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e6a1480-5c80-484b-87e0-78d5a855594e-metrics-certs\") pod \"router-default-65b4d9bcf5-kkhtq\" (UID: \"3e6a1480-5c80-484b-87e0-78d5a855594e\") " pod="openshift-ingress/router-default-65b4d9bcf5-kkhtq"
Jul 06 22:47:12.349144 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:12.349134    2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 06 22:47:12.349194 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:12.349171    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e6a1480-5c80-484b-87e0-78d5a855594e-metrics-certs podName:3e6a1480-5c80-484b-87e0-78d5a855594e nodeName:}" failed. No retries permitted until 2026-07-06 22:47:20.349159592 +0000 UTC m=+52.753409474 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3e6a1480-5c80-484b-87e0-78d5a855594e-metrics-certs") pod "router-default-65b4d9bcf5-kkhtq" (UID: "3e6a1480-5c80-484b-87e0-78d5a855594e") : secret "router-metrics-certs-default" not found
Jul 06 22:47:12.356657 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:12.356571    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-v2lzx"]
Jul 06 22:47:12.359145 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:47:12.359121    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab91bcb_c3c8_47ef_a1b8_2fb8cafac398.slice/crio-c96baf2c399dd189171c9a86479963d57f3ffa8fb4aab7123a45cf519fda7886 WatchSource:0}: Error finding container c96baf2c399dd189171c9a86479963d57f3ffa8fb4aab7123a45cf519fda7886: Status 404 returned error can't find the container with id c96baf2c399dd189171c9a86479963d57f3ffa8fb4aab7123a45cf519fda7886
Jul 06 22:47:12.513969 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:12.513943    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-v2lzx" event={"ID":"6ab91bcb-c3c8-47ef-a1b8-2fb8cafac398","Type":"ContainerStarted","Data":"c96baf2c399dd189171c9a86479963d57f3ffa8fb4aab7123a45cf519fda7886"}
Jul 06 22:47:12.792149 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:12.792090    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dwnzz_8624c1c4-bd8a-4e14-9445-3c7352c1cd60/serve-healthcheck-canary/0.log"
Jul 06 22:47:12.954082 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:12.954056    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c13e679b-e3c2-432e-b81d-619f3e346e92-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8bd6t\" (UID: \"c13e679b-e3c2-432e-b81d-619f3e346e92\") " pod="openshift-insights/insights-runtime-extractor-8bd6t"
Jul 06 22:47:12.954505 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:12.954244    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 22:47:12.954505 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:12.954285    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c13e679b-e3c2-432e-b81d-619f3e346e92-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8bd6t\" (UID: \"c13e679b-e3c2-432e-b81d-619f3e346e92\") " pod="openshift-insights/insights-runtime-extractor-8bd6t"
Jul 06 22:47:12.954505 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:12.954312    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c13e679b-e3c2-432e-b81d-619f3e346e92-insights-runtime-extractor-tls podName:c13e679b-e3c2-432e-b81d-619f3e346e92 nodeName:}" failed. No retries permitted until 2026-07-06 22:47:16.954292701 +0000 UTC m=+49.358542587 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c13e679b-e3c2-432e-b81d-619f3e346e92-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8bd6t" (UID: "c13e679b-e3c2-432e-b81d-619f3e346e92") : secret "insights-runtime-extractor-tls" not found
Jul 06 22:47:12.954505 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:12.954358    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 22:47:12.954505 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:12.954417    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c13e679b-e3c2-432e-b81d-619f3e346e92-kube-rbac-proxy-cm podName:c13e679b-e3c2-432e-b81d-619f3e346e92 nodeName:}" failed. No retries permitted until 2026-07-06 22:47:16.954401936 +0000 UTC m=+49.358651837 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c13e679b-e3c2-432e-b81d-619f3e346e92-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8bd6t" (UID: "c13e679b-e3c2-432e-b81d-619f3e346e92") : configmap "kube-rbac-proxy" not found
Jul 06 22:47:14.520863 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:14.520832    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-v2lzx" event={"ID":"6ab91bcb-c3c8-47ef-a1b8-2fb8cafac398","Type":"ContainerStarted","Data":"366c4e47eee11dd2e3868eea39d84f0a68ac1eec070972e9aa3a1d81c6a743ae"}
Jul 06 22:47:14.521318 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:14.520888    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-v2lzx" event={"ID":"6ab91bcb-c3c8-47ef-a1b8-2fb8cafac398","Type":"ContainerStarted","Data":"7d124e768cdf7be402c26c806f063cdb411a059f29fe2866ca1420479586ed00"}
Jul 06 22:47:14.540503 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:14.540456    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-v2lzx" podStartSLOduration=2.004712761 podStartE2EDuration="3.540440711s" totalImagesPullingTime="1.53572795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:47:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:47:14.539865776 +0000 UTC m=+46.944115680" watchObservedRunningTime="2026-07-06 22:47:14.540440711 +0000 UTC m=+46.944690615"
Jul 06 22:47:16.981167 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:16.981132    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c13e679b-e3c2-432e-b81d-619f3e346e92-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8bd6t\" (UID: \"c13e679b-e3c2-432e-b81d-619f3e346e92\") " pod="openshift-insights/insights-runtime-extractor-8bd6t"
Jul 06 22:47:16.981595 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:16.981251    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c13e679b-e3c2-432e-b81d-619f3e346e92-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8bd6t\" (UID: \"c13e679b-e3c2-432e-b81d-619f3e346e92\") " pod="openshift-insights/insights-runtime-extractor-8bd6t"
Jul 06 22:47:16.981595 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:16.981275    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 22:47:16.981595 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:16.981335    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c13e679b-e3c2-432e-b81d-619f3e346e92-insights-runtime-extractor-tls podName:c13e679b-e3c2-432e-b81d-619f3e346e92 nodeName:}" failed. No retries permitted until 2026-07-06 22:47:24.981319099 +0000 UTC m=+57.385568981 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c13e679b-e3c2-432e-b81d-619f3e346e92-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8bd6t" (UID: "c13e679b-e3c2-432e-b81d-619f3e346e92") : secret "insights-runtime-extractor-tls" not found
Jul 06 22:47:16.981595 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:16.981363    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 22:47:16.981595 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:16.981414    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c13e679b-e3c2-432e-b81d-619f3e346e92-kube-rbac-proxy-cm podName:c13e679b-e3c2-432e-b81d-619f3e346e92 nodeName:}" failed. No retries permitted until 2026-07-06 22:47:24.98140086 +0000 UTC m=+57.385650748 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c13e679b-e3c2-432e-b81d-619f3e346e92-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8bd6t" (UID: "c13e679b-e3c2-432e-b81d-619f3e346e92") : configmap "kube-rbac-proxy" not found
Jul 06 22:47:19.301409 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:19.301374    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a674edfd-a962-4327-b319-320fb62922c0-metrics-tls\") pod \"dns-default-2xhr2\" (UID: \"a674edfd-a962-4327-b319-320fb62922c0\") " pod="openshift-dns/dns-default-2xhr2"
Jul 06 22:47:19.301785 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:19.301523    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 22:47:19.301785 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:19.301587    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a674edfd-a962-4327-b319-320fb62922c0-metrics-tls podName:a674edfd-a962-4327-b319-320fb62922c0 nodeName:}" failed. No retries permitted until 2026-07-06 22:47:35.30157159 +0000 UTC m=+67.705821471 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a674edfd-a962-4327-b319-320fb62922c0-metrics-tls") pod "dns-default-2xhr2" (UID: "a674edfd-a962-4327-b319-320fb62922c0") : secret "dns-default-metrics-tls" not found
Jul 06 22:47:20.208035 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:20.208005    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad24049a-6893-4d5e-bb79-0e176fc731c3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-dpdfh\" (UID: \"ad24049a-6893-4d5e-bb79-0e176fc731c3\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpdfh"
Jul 06 22:47:20.208237 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:20.208149    2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 06 22:47:20.208287 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:20.208250    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad24049a-6893-4d5e-bb79-0e176fc731c3-cluster-monitoring-operator-tls podName:ad24049a-6893-4d5e-bb79-0e176fc731c3 nodeName:}" failed. No retries permitted until 2026-07-06 22:47:36.208234705 +0000 UTC m=+68.612484590 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ad24049a-6893-4d5e-bb79-0e176fc731c3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-dpdfh" (UID: "ad24049a-6893-4d5e-bb79-0e176fc731c3") : secret "cluster-monitoring-operator-tls" not found
Jul 06 22:47:20.409524 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:20.409495    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e6a1480-5c80-484b-87e0-78d5a855594e-metrics-certs\") pod \"router-default-65b4d9bcf5-kkhtq\" (UID: \"3e6a1480-5c80-484b-87e0-78d5a855594e\") " pod="openshift-ingress/router-default-65b4d9bcf5-kkhtq"
Jul 06 22:47:20.409858 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:20.409535    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e6a1480-5c80-484b-87e0-78d5a855594e-service-ca-bundle\") pod \"router-default-65b4d9bcf5-kkhtq\" (UID: \"3e6a1480-5c80-484b-87e0-78d5a855594e\") " pod="openshift-ingress/router-default-65b4d9bcf5-kkhtq"
Jul 06 22:47:20.409858 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:20.409670    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3e6a1480-5c80-484b-87e0-78d5a855594e-service-ca-bundle podName:3e6a1480-5c80-484b-87e0-78d5a855594e nodeName:}" failed. No retries permitted until 2026-07-06 22:47:36.409656031 +0000 UTC m=+68.813905913 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3e6a1480-5c80-484b-87e0-78d5a855594e-service-ca-bundle") pod "router-default-65b4d9bcf5-kkhtq" (UID: "3e6a1480-5c80-484b-87e0-78d5a855594e") : configmap references non-existent config key: service-ca.crt
Jul 06 22:47:20.411905 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:20.411881    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e6a1480-5c80-484b-87e0-78d5a855594e-metrics-certs\") pod \"router-default-65b4d9bcf5-kkhtq\" (UID: \"3e6a1480-5c80-484b-87e0-78d5a855594e\") " pod="openshift-ingress/router-default-65b4d9bcf5-kkhtq"
Jul 06 22:47:25.047407 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:25.047360    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c13e679b-e3c2-432e-b81d-619f3e346e92-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8bd6t\" (UID: \"c13e679b-e3c2-432e-b81d-619f3e346e92\") " pod="openshift-insights/insights-runtime-extractor-8bd6t"
Jul 06 22:47:25.048034 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:25.047444    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c13e679b-e3c2-432e-b81d-619f3e346e92-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8bd6t\" (UID: \"c13e679b-e3c2-432e-b81d-619f3e346e92\") " pod="openshift-insights/insights-runtime-extractor-8bd6t"
Jul 06 22:47:25.048034 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:25.047530    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 22:47:25.048034 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:25.047620    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c13e679b-e3c2-432e-b81d-619f3e346e92-insights-runtime-extractor-tls podName:c13e679b-e3c2-432e-b81d-619f3e346e92 nodeName:}" failed. No retries permitted until 2026-07-06 22:47:41.047601431 +0000 UTC m=+73.451851313 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c13e679b-e3c2-432e-b81d-619f3e346e92-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8bd6t" (UID: "c13e679b-e3c2-432e-b81d-619f3e346e92") : secret "insights-runtime-extractor-tls" not found
Jul 06 22:47:25.048034 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:25.047968    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c13e679b-e3c2-432e-b81d-619f3e346e92-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8bd6t\" (UID: \"c13e679b-e3c2-432e-b81d-619f3e346e92\") " pod="openshift-insights/insights-runtime-extractor-8bd6t"
Jul 06 22:47:29.474511 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:29.474485    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5bvsb"
Jul 06 22:47:33.812287 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:33.812239    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5rxc\" (UniqueName: \"kubernetes.io/projected/f17325b2-5d96-4911-904b-3a774e4cc8b1-kube-api-access-h5rxc\") pod \"network-check-target-j9csq\" (UID: \"f17325b2-5d96-4911-904b-3a774e4cc8b1\") " pod="openshift-network-diagnostics/network-check-target-j9csq"
Jul 06 22:47:33.812751 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:33.812334    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8cc9b77e-7a6d-4d5b-83a2-8f24359ec453-original-pull-secret\") pod \"global-pull-secret-syncer-l6fbb\" (UID: \"8cc9b77e-7a6d-4d5b-83a2-8f24359ec453\") " pod="kube-system/global-pull-secret-syncer-l6fbb"
Jul 06 22:47:33.812751 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:33.812352    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/107658f0-a3ae-4883-99dd-ddc62057a8a0-metrics-certs\") pod \"network-metrics-daemon-mfmlk\" (UID: \"107658f0-a3ae-4883-99dd-ddc62057a8a0\") " pod="openshift-multus/network-metrics-daemon-mfmlk"
Jul 06 22:47:33.815248 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:33.815229    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 06 22:47:33.816109 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:33.816089    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 06 22:47:33.816229 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:33.816091    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 06 22:47:33.824632 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:33.824607    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 06 22:47:33.824774 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:33.824754    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8cc9b77e-7a6d-4d5b-83a2-8f24359ec453-original-pull-secret\") pod \"global-pull-secret-syncer-l6fbb\" (UID: \"8cc9b77e-7a6d-4d5b-83a2-8f24359ec453\") " pod="kube-system/global-pull-secret-syncer-l6fbb"
Jul 06 22:47:33.824821 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:33.824805    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/107658f0-a3ae-4883-99dd-ddc62057a8a0-metrics-certs\") pod \"network-metrics-daemon-mfmlk\" (UID: \"107658f0-a3ae-4883-99dd-ddc62057a8a0\") " pod="openshift-multus/network-metrics-daemon-mfmlk"
Jul 06 22:47:33.834654 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:33.834624    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5rxc\" (UniqueName: \"kubernetes.io/projected/f17325b2-5d96-4911-904b-3a774e4cc8b1-kube-api-access-h5rxc\") pod \"network-check-target-j9csq\" (UID: \"f17325b2-5d96-4911-904b-3a774e4cc8b1\") " pod="openshift-network-diagnostics/network-check-target-j9csq"
Jul 06 22:47:33.980099 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:33.980074    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lkkl7\""
Jul 06 22:47:33.986795 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:33.986776    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5j2wj\""
Jul 06 22:47:33.987807 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:33.987793    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j9csq"
Jul 06 22:47:33.989827 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:33.989811    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l6fbb"
Jul 06 22:47:33.994799 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:33.994782    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mfmlk"
Jul 06 22:47:34.130141 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:34.130108    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mfmlk"]
Jul 06 22:47:34.133422 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:47:34.133383    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod107658f0_a3ae_4883_99dd_ddc62057a8a0.slice/crio-edd20287801e331ae75cbde54fe99155fce623533e74112d2418b76a098dd83f WatchSource:0}: Error finding container edd20287801e331ae75cbde54fe99155fce623533e74112d2418b76a098dd83f: Status 404 returned error can't find the container with id edd20287801e331ae75cbde54fe99155fce623533e74112d2418b76a098dd83f
Jul 06 22:47:34.353602 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:34.353531    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l6fbb"]
Jul 06 22:47:34.356242 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:34.356214    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j9csq"]
Jul 06 22:47:34.357257 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:47:34.357237    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cc9b77e_7a6d_4d5b_83a2_8f24359ec453.slice/crio-195e0b19d2263e33d095fc9a358baee72d6b97dfebf06854db1ac2f2a7aa9dbc WatchSource:0}: Error finding container 195e0b19d2263e33d095fc9a358baee72d6b97dfebf06854db1ac2f2a7aa9dbc: Status 404 returned error can't find the container with id 195e0b19d2263e33d095fc9a358baee72d6b97dfebf06854db1ac2f2a7aa9dbc
Jul 06 22:47:34.358419 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:47:34.358396    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf17325b2_5d96_4911_904b_3a774e4cc8b1.slice/crio-63bdb27f5eabf73e2fa0f6769e0b3ef1290c6bfb6dbbe1726abac6537ed3b009 WatchSource:0}: Error finding container 63bdb27f5eabf73e2fa0f6769e0b3ef1290c6bfb6dbbe1726abac6537ed3b009: Status 404 returned error can't find the container with id 63bdb27f5eabf73e2fa0f6769e0b3ef1290c6bfb6dbbe1726abac6537ed3b009
Jul 06 22:47:34.568900 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:34.568864    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-l6fbb" event={"ID":"8cc9b77e-7a6d-4d5b-83a2-8f24359ec453","Type":"ContainerStarted","Data":"195e0b19d2263e33d095fc9a358baee72d6b97dfebf06854db1ac2f2a7aa9dbc"}
Jul 06 22:47:34.570426 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:34.570397    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j9csq" event={"ID":"f17325b2-5d96-4911-904b-3a774e4cc8b1","Type":"ContainerStarted","Data":"63bdb27f5eabf73e2fa0f6769e0b3ef1290c6bfb6dbbe1726abac6537ed3b009"}
Jul 06 22:47:34.571700 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:34.571668    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mfmlk" event={"ID":"107658f0-a3ae-4883-99dd-ddc62057a8a0","Type":"ContainerStarted","Data":"edd20287801e331ae75cbde54fe99155fce623533e74112d2418b76a098dd83f"}
Jul 06 22:47:35.147924 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:35.147891    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798b7d449c-tddgk"]
Jul 06 22:47:35.163149 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:35.163056    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798b7d449c-tddgk"
Jul 06 22:47:35.168282 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:35.168258    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798b7d449c-tddgk"]
Jul 06 22:47:35.170705 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:35.170583    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 06 22:47:35.170705 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:35.170664    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 06 22:47:35.170971 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:35.170871    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 06 22:47:35.170971 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:35.170951    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 06 22:47:35.171118 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:35.171020    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 06 22:47:35.171118 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:35.171047    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 06 22:47:35.171563 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:35.171544    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 06 22:47:35.225049 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:35.225026    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/473acfc2-87c6-4f5b-80aa-a3b02e6027b3-ca\") pod \"cluster-proxy-proxy-agent-798b7d449c-tddgk\" (UID: \"473acfc2-87c6-4f5b-80aa-a3b02e6027b3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798b7d449c-tddgk"
Jul 06 22:47:35.225201 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:35.225099    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/473acfc2-87c6-4f5b-80aa-a3b02e6027b3-additional-service-ca\") pod \"cluster-proxy-proxy-agent-798b7d449c-tddgk\" (UID: \"473acfc2-87c6-4f5b-80aa-a3b02e6027b3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798b7d449c-tddgk"
Jul 06 22:47:35.225201 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:35.225127    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk4cq\" (UniqueName: \"kubernetes.io/projected/473acfc2-87c6-4f5b-80aa-a3b02e6027b3-kube-api-access-lk4cq\") pod \"cluster-proxy-proxy-agent-798b7d449c-tddgk\" (UID: \"473acfc2-87c6-4f5b-80aa-a3b02e6027b3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798b7d449c-tddgk"
Jul 06 22:47:35.225201 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:35.225152    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/473acfc2-87c6-4f5b-80aa-a3b02e6027b3-hub\") pod \"cluster-proxy-proxy-agent-798b7d449c-tddgk\" (UID: \"473acfc2-87c6-4f5b-80aa-a3b02e6027b3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798b7d449c-tddgk"
Jul 06 22:47:35.225368 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:35.225215    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/473acfc2-87c6-4f5b-80aa-a3b02e6027b3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-798b7d449c-tddgk\" (UID: \"473acfc2-87c6-4f5b-80aa-a3b02e6027b3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798b7d449c-tddgk"
Jul 06 22:47:35.225368 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:35.225261    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/473acfc2-87c6-4f5b-80aa-a3b02e6027b3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-798b7d449c-tddgk\" (UID: \"473acfc2-87c6-4f5b-80aa-a3b02e6027b3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798b7d449c-tddgk"
Jul 06 22:47:35.326489 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:35.326462    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/473acfc2-87c6-4f5b-80aa-a3b02e6027b3-ca\") pod \"cluster-proxy-proxy-agent-798b7d449c-tddgk\" (UID: \"473acfc2-87c6-4f5b-80aa-a3b02e6027b3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798b7d449c-tddgk"
Jul 06 22:47:35.326622 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:35.326515    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a674edfd-a962-4327-b319-320fb62922c0-metrics-tls\") pod \"dns-default-2xhr2\" (UID: \"a674edfd-a962-4327-b319-320fb62922c0\") " pod="openshift-dns/dns-default-2xhr2"
Jul 06 22:47:35.326622 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:35.326537    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/473acfc2-87c6-4f5b-80aa-a3b02e6027b3-additional-service-ca\") pod \"cluster-proxy-proxy-agent-798b7d449c-tddgk\" (UID: \"473acfc2-87c6-4f5b-80aa-a3b02e6027b3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798b7d449c-tddgk"
Jul 06 22:47:35.326622 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:35.326558    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk4cq\" (UniqueName: \"kubernetes.io/projected/473acfc2-87c6-4f5b-80aa-a3b02e6027b3-kube-api-access-lk4cq\") pod \"cluster-proxy-proxy-agent-798b7d449c-tddgk\" (UID: \"473acfc2-87c6-4f5b-80aa-a3b02e6027b3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798b7d449c-tddgk"
Jul 06 22:47:35.326622 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:35.326575    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/473acfc2-87c6-4f5b-80aa-a3b02e6027b3-hub\") pod \"cluster-proxy-proxy-agent-798b7d449c-tddgk\" (UID: \"473acfc2-87c6-4f5b-80aa-a3b02e6027b3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798b7d449c-tddgk"
Jul 06 22:47:35.326622 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:35.326604    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/473acfc2-87c6-4f5b-80aa-a3b02e6027b3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-798b7d449c-tddgk\" (UID: \"473acfc2-87c6-4f5b-80aa-a3b02e6027b3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798b7d449c-tddgk"
Jul 06 22:47:35.326883 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:35.326642    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/473acfc2-87c6-4f5b-80aa-a3b02e6027b3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-798b7d449c-tddgk\" (UID: \"473acfc2-87c6-4f5b-80aa-a3b02e6027b3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798b7d449c-tddgk"
Jul 06 22:47:35.327746 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:35.327692    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/473acfc2-87c6-4f5b-80aa-a3b02e6027b3-additional-service-ca\") pod \"cluster-proxy-proxy-agent-798b7d449c-tddgk\" (UID: \"473acfc2-87c6-4f5b-80aa-a3b02e6027b3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798b7d449c-tddgk"
Jul 06 22:47:35.330059 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:35.330020    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/473acfc2-87c6-4f5b-80aa-a3b02e6027b3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-798b7d449c-tddgk\" (UID: \"473acfc2-87c6-4f5b-80aa-a3b02e6027b3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798b7d449c-tddgk"
Jul 06 22:47:35.330059 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:35.330033    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/473acfc2-87c6-4f5b-80aa-a3b02e6027b3-hub\") pod \"cluster-proxy-proxy-agent-798b7d449c-tddgk\" (UID: \"473acfc2-87c6-4f5b-80aa-a3b02e6027b3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798b7d449c-tddgk"
Jul 06 22:47:35.330264 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:35.330081    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/473acfc2-87c6-4f5b-80aa-a3b02e6027b3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-798b7d449c-tddgk\" (UID: \"473acfc2-87c6-4f5b-80aa-a3b02e6027b3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798b7d449c-tddgk"
Jul 06 22:47:35.330783 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:35.330740    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/473acfc2-87c6-4f5b-80aa-a3b02e6027b3-ca\") pod \"cluster-proxy-proxy-agent-798b7d449c-tddgk\" (UID: \"473acfc2-87c6-4f5b-80aa-a3b02e6027b3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798b7d449c-tddgk"
Jul 06 22:47:35.331074 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:35.331054    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a674edfd-a962-4327-b319-320fb62922c0-metrics-tls\") pod \"dns-default-2xhr2\" (UID: \"a674edfd-a962-4327-b319-320fb62922c0\") " pod="openshift-dns/dns-default-2xhr2"
Jul 06 22:47:35.336114 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:35.336091    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk4cq\" (UniqueName: \"kubernetes.io/projected/473acfc2-87c6-4f5b-80aa-a3b02e6027b3-kube-api-access-lk4cq\") pod \"cluster-proxy-proxy-agent-798b7d449c-tddgk\" (UID: \"473acfc2-87c6-4f5b-80aa-a3b02e6027b3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798b7d449c-tddgk"
Jul 06 22:47:35.486308 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:35.486275    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798b7d449c-tddgk"
Jul 06 22:47:35.568308 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:35.568082    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jx42h\""
Jul 06 22:47:35.575385 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:35.575206    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2xhr2"
Jul 06 22:47:35.681776 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:35.681739    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798b7d449c-tddgk"]
Jul 06 22:47:35.686028 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:47:35.685982    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod473acfc2_87c6_4f5b_80aa_a3b02e6027b3.slice/crio-41c1d920808b18e98389d60f2c9e40de63ddd02e03492126e28a5a26f527e7d3 WatchSource:0}: Error finding container 41c1d920808b18e98389d60f2c9e40de63ddd02e03492126e28a5a26f527e7d3: Status 404 returned error can't find the container with id 41c1d920808b18e98389d60f2c9e40de63ddd02e03492126e28a5a26f527e7d3
Jul 06 22:47:35.747000 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:35.746946    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2xhr2"]
Jul 06 22:47:35.752628 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:47:35.752599    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda674edfd_a962_4327_b319_320fb62922c0.slice/crio-f66a42199b3f18ad3aea926016ce1d92ee4231ed50f0b04cb139ed44a0c5b087 WatchSource:0}: Error finding container f66a42199b3f18ad3aea926016ce1d92ee4231ed50f0b04cb139ed44a0c5b087: Status 404 returned error can't find the container with id f66a42199b3f18ad3aea926016ce1d92ee4231ed50f0b04cb139ed44a0c5b087
Jul 06 22:47:36.235542 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:36.235335    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad24049a-6893-4d5e-bb79-0e176fc731c3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-dpdfh\" (UID: \"ad24049a-6893-4d5e-bb79-0e176fc731c3\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpdfh"
Jul 06 22:47:36.238575 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:36.238546    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad24049a-6893-4d5e-bb79-0e176fc731c3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-dpdfh\" (UID: \"ad24049a-6893-4d5e-bb79-0e176fc731c3\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpdfh"
Jul 06 22:47:36.261223 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:36.261004    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-vzg2t\""
Jul 06 22:47:36.268457 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:36.268426    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpdfh"
Jul 06 22:47:36.437419 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:36.437387    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e6a1480-5c80-484b-87e0-78d5a855594e-service-ca-bundle\") pod \"router-default-65b4d9bcf5-kkhtq\" (UID: \"3e6a1480-5c80-484b-87e0-78d5a855594e\") " pod="openshift-ingress/router-default-65b4d9bcf5-kkhtq"
Jul 06 22:47:36.438213 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:36.438172    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e6a1480-5c80-484b-87e0-78d5a855594e-service-ca-bundle\") pod \"router-default-65b4d9bcf5-kkhtq\" (UID: \"3e6a1480-5c80-484b-87e0-78d5a855594e\") " pod="openshift-ingress/router-default-65b4d9bcf5-kkhtq"
Jul 06 22:47:36.446491 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:36.446424    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpdfh"]
Jul 06 22:47:36.473514 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:36.473311    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-5td4z\""
Jul 06 22:47:36.480931 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:36.480745    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-65b4d9bcf5-kkhtq"
Jul 06 22:47:36.587173 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:36.587141    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpdfh" event={"ID":"ad24049a-6893-4d5e-bb79-0e176fc731c3","Type":"ContainerStarted","Data":"e2d3c36434d9dda2e25b2f58d2f716213e8c34ed4259877395aae165dabdb608"}
Jul 06 22:47:36.588948 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:36.588885    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798b7d449c-tddgk" event={"ID":"473acfc2-87c6-4f5b-80aa-a3b02e6027b3","Type":"ContainerStarted","Data":"41c1d920808b18e98389d60f2c9e40de63ddd02e03492126e28a5a26f527e7d3"}
Jul 06 22:47:36.592901 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:36.592202    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mfmlk" event={"ID":"107658f0-a3ae-4883-99dd-ddc62057a8a0","Type":"ContainerStarted","Data":"5f05e0011d3e55a31a24529ca5913b370597f2ca017603fadf276ccc0fbae21a"}
Jul 06 22:47:36.592901 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:36.592255    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mfmlk" event={"ID":"107658f0-a3ae-4883-99dd-ddc62057a8a0","Type":"ContainerStarted","Data":"02ab1186eaa4a1886942034ba0a9f252243d2ce8703df1bda14bf60b0f79cdcd"}
Jul 06 22:47:36.595719 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:36.595690    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2xhr2" event={"ID":"a674edfd-a962-4327-b319-320fb62922c0","Type":"ContainerStarted","Data":"f66a42199b3f18ad3aea926016ce1d92ee4231ed50f0b04cb139ed44a0c5b087"}
Jul 06 22:47:36.641463 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:36.640455    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mfmlk" podStartSLOduration=66.247042687 podStartE2EDuration="1m7.640431911s" totalImagesPullingTime="1.393389224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:47:36.639613306 +0000 UTC m=+69.043863210" watchObservedRunningTime="2026-07-06 22:47:36.640431911 +0000 UTC m=+69.044681817"
Jul 06 22:47:36.683711 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:36.683406    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-65b4d9bcf5-kkhtq"]
Jul 06 22:47:38.056003 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:47:38.055972    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e6a1480_5c80_484b_87e0_78d5a855594e.slice/crio-c0a60258004e9a9e41e515b76437db5882930d0844ec256b83dc04e274ac43ca WatchSource:0}: Error finding container c0a60258004e9a9e41e515b76437db5882930d0844ec256b83dc04e274ac43ca: Status 404 returned error can't find the container with id c0a60258004e9a9e41e515b76437db5882930d0844ec256b83dc04e274ac43ca
Jul 06 22:47:38.607323 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:38.607286    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-65b4d9bcf5-kkhtq" event={"ID":"3e6a1480-5c80-484b-87e0-78d5a855594e","Type":"ContainerStarted","Data":"c0a60258004e9a9e41e515b76437db5882930d0844ec256b83dc04e274ac43ca"}
Jul 06 22:47:41.075295 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:41.075220    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c13e679b-e3c2-432e-b81d-619f3e346e92-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8bd6t\" (UID: \"c13e679b-e3c2-432e-b81d-619f3e346e92\") " pod="openshift-insights/insights-runtime-extractor-8bd6t"
Jul 06 22:47:41.077615 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:41.077584    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c13e679b-e3c2-432e-b81d-619f3e346e92-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8bd6t\" (UID: \"c13e679b-e3c2-432e-b81d-619f3e346e92\") " pod="openshift-insights/insights-runtime-extractor-8bd6t"
Jul 06 22:47:41.366405 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:41.366333    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4p5jh\""
Jul 06 22:47:41.373965 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:41.373939    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8bd6t"
Jul 06 22:47:42.000845 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:42.000809    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8bd6t"]
Jul 06 22:47:42.384618 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:47:42.384582    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13e679b_e3c2_432e_b81d_619f3e346e92.slice/crio-02530544c168c50c389e184f14ab5e665f84b4891052fc04bc6e24fc9e1c79fc WatchSource:0}: Error finding container 02530544c168c50c389e184f14ab5e665f84b4891052fc04bc6e24fc9e1c79fc: Status 404 returned error can't find the container with id 02530544c168c50c389e184f14ab5e665f84b4891052fc04bc6e24fc9e1c79fc
Jul 06 22:47:42.628485 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:42.627748    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2xhr2" event={"ID":"a674edfd-a962-4327-b319-320fb62922c0","Type":"ContainerStarted","Data":"ac7543fff1bee8e8f3e96a84c700c49a656e33bf8b2ec4fe5048e4f3c367e27a"}
Jul 06 22:47:42.628485 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:42.627843    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2xhr2" event={"ID":"a674edfd-a962-4327-b319-320fb62922c0","Type":"ContainerStarted","Data":"c3407098e79b3bb058cbd0406050259aedb734d7ffd1728e6b9a4c8c0abf44f0"}
Jul 06 22:47:42.629192 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:42.629151    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8bd6t" event={"ID":"c13e679b-e3c2-432e-b81d-619f3e346e92","Type":"ContainerStarted","Data":"02530544c168c50c389e184f14ab5e665f84b4891052fc04bc6e24fc9e1c79fc"}
Jul 06 22:47:42.630778 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:42.630754    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-65b4d9bcf5-kkhtq" event={"ID":"3e6a1480-5c80-484b-87e0-78d5a855594e","Type":"ContainerStarted","Data":"27c2e7730eba89c47d495f1e11ce2c7b9b412295a16f43afc39fa1695dfd69f9"}
Jul 06 22:47:42.632755 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:42.632733    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-l6fbb" event={"ID":"8cc9b77e-7a6d-4d5b-83a2-8f24359ec453","Type":"ContainerStarted","Data":"87455091b7f847f926693cfa29a2e55fff8e71d5cdef9d80ab678941be7d3d3d"}
Jul 06 22:47:42.638955 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:42.636165    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpdfh" event={"ID":"ad24049a-6893-4d5e-bb79-0e176fc731c3","Type":"ContainerStarted","Data":"e602e50da67d4b37c5350bc8afe9055f29438db2f7e8e302ec763ea8e8fdb26e"}
Jul 06 22:47:42.639807 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:42.639722    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798b7d449c-tddgk" event={"ID":"473acfc2-87c6-4f5b-80aa-a3b02e6027b3","Type":"ContainerStarted","Data":"f1e39992eeeeb6d08685317bdd0c4557207588d11c5856ec32afd17969031cba"}
Jul 06 22:47:42.639807 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:42.639761    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798b7d449c-tddgk" event={"ID":"473acfc2-87c6-4f5b-80aa-a3b02e6027b3","Type":"ContainerStarted","Data":"7b92dc853c09afe039649314fbee29ae22859a455cde90e5d59236b040f014c8"}
Jul 06 22:47:42.639807 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:42.639783    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798b7d449c-tddgk" event={"ID":"473acfc2-87c6-4f5b-80aa-a3b02e6027b3","Type":"ContainerStarted","Data":"816a22064d6955ab74f6c0ca309ba5e57bd3680e482607c7bbd8514abafd9f72"}
Jul 06 22:47:42.643016 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:42.642833    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j9csq" event={"ID":"f17325b2-5d96-4911-904b-3a774e4cc8b1","Type":"ContainerStarted","Data":"3a82d1004615960e5bdd3d1fdd2ecfe1e5a9ed879bbdde56f4c7761a9cd96556"}
Jul 06 22:47:42.643295 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:42.643245    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-j9csq"
Jul 06 22:47:42.649894 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:42.649851    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-65b4d9bcf5-kkhtq" podStartSLOduration=38.649834963 podStartE2EDuration="38.649834963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:47:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:47:42.649079822 +0000 UTC m=+75.053329764" watchObservedRunningTime="2026-07-06 22:47:42.649834963 +0000 UTC m=+75.054084867"
Jul 06 22:47:42.680890 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:42.680849    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-798b7d449c-tddgk" podStartSLOduration=0.98452355 podStartE2EDuration="7.680836505s" totalImagesPullingTime="6.696312955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:47:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:47:42.680130985 +0000 UTC m=+75.084380888" watchObservedRunningTime="2026-07-06 22:47:42.680836505 +0000 UTC m=+75.085086409"
Jul 06 22:47:42.680984 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:42.680944    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-j9csq" podStartSLOduration=66.183602184 podStartE2EDuration="1m13.680937285s" totalImagesPullingTime="7.497335101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:47:42.664029916 +0000 UTC m=+75.068279821" watchObservedRunningTime="2026-07-06 22:47:42.680937285 +0000 UTC m=+75.085187190"
Jul 06 22:47:42.693757 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:42.693710    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-l6fbb" podStartSLOduration=65.66734723 podStartE2EDuration="1m13.693694022s" totalImagesPullingTime="8.026346792s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:47:42.693441516 +0000 UTC m=+75.097691421" watchObservedRunningTime="2026-07-06 22:47:42.693694022 +0000 UTC m=+75.097943928"
Jul 06 22:47:42.710444 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:42.709669    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpdfh" podStartSLOduration=32.742242542 podStartE2EDuration="38.709651847s" totalImagesPullingTime="5.967409305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:47:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:47:42.709210913 +0000 UTC m=+75.113460814" watchObservedRunningTime="2026-07-06 22:47:42.709651847 +0000 UTC m=+75.113901751"
Jul 06 22:47:43.481778 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:43.481741    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-65b4d9bcf5-kkhtq"
Jul 06 22:47:43.484923 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:43.484901    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-65b4d9bcf5-kkhtq"
Jul 06 22:47:43.647578 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:43.647550    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-65b4d9bcf5-kkhtq"
Jul 06 22:47:43.648912 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:43.648890    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-65b4d9bcf5-kkhtq"
Jul 06 22:47:43.663438 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:43.663392    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-2xhr2" podStartSLOduration=34.53955777 podStartE2EDuration="40.663379138s" totalImagesPullingTime="6.123821368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:47:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:47:43.662902255 +0000 UTC m=+76.067152160" watchObservedRunningTime="2026-07-06 22:47:43.663379138 +0000 UTC m=+76.067629041"
Jul 06 22:47:44.650956 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:44.650904    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2xhr2"
Jul 06 22:47:46.662688 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:46.662659    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8bd6t" event={"ID":"c13e679b-e3c2-432e-b81d-619f3e346e92","Type":"ContainerStarted","Data":"55a76f170a09727f18401934a264a9a1b4464ad4ca6be8f7367f82f51c24531c"}
Jul 06 22:47:46.662954 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:46.662712    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8bd6t" event={"ID":"c13e679b-e3c2-432e-b81d-619f3e346e92","Type":"ContainerStarted","Data":"4f090c5399265f3a3fd97b4e85443d4780ae2a198068aecc9896603b896f3d61"}
Jul 06 22:47:49.679300 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:49.679231    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8bd6t" event={"ID":"c13e679b-e3c2-432e-b81d-619f3e346e92","Type":"ContainerStarted","Data":"1a0fa34f45fbb6d53eb9fe11234bce1c637d5fa86bbd5c5c0fc0c2618bc39344"}
Jul 06 22:47:49.697987 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:49.697943    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8bd6t" podStartSLOduration=33.808108497 podStartE2EDuration="40.697929677s" totalImagesPullingTime="6.88982118s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:47:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:47:49.697289392 +0000 UTC m=+82.101539299" watchObservedRunningTime="2026-07-06 22:47:49.697929677 +0000 UTC m=+82.102179580"
Jul 06 22:47:51.353907 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:51.353806    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nvtn5"]
Jul 06 22:47:51.357672 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:51.357655    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nvtn5"
Jul 06 22:47:51.361237 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:51.361211    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 06 22:47:51.361708 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:51.361687    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-v6465\""
Jul 06 22:47:51.361796 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:51.361690    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 06 22:47:51.362504 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:51.362488    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 06 22:47:51.362565 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:51.362514    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 06 22:47:51.450396 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:51.450364    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h2sb\" (UniqueName: \"kubernetes.io/projected/97cb756f-f621-419b-8f56-1b144ccdade3-kube-api-access-4h2sb\") pod \"node-exporter-nvtn5\" (UID: \"97cb756f-f621-419b-8f56-1b144ccdade3\") " pod="openshift-monitoring/node-exporter-nvtn5"
Jul 06 22:47:51.450519 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:51.450430    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/97cb756f-f621-419b-8f56-1b144ccdade3-node-exporter-accelerators-collector-config\") pod \"node-exporter-nvtn5\" (UID: \"97cb756f-f621-419b-8f56-1b144ccdade3\") " pod="openshift-monitoring/node-exporter-nvtn5"
Jul 06 22:47:51.450519 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:51.450472    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/97cb756f-f621-419b-8f56-1b144ccdade3-node-exporter-textfile\") pod \"node-exporter-nvtn5\" (UID: \"97cb756f-f621-419b-8f56-1b144ccdade3\") " pod="openshift-monitoring/node-exporter-nvtn5"
Jul 06 22:47:51.450519 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:51.450499    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97cb756f-f621-419b-8f56-1b144ccdade3-sys\") pod \"node-exporter-nvtn5\" (UID: \"97cb756f-f621-419b-8f56-1b144ccdade3\") " pod="openshift-monitoring/node-exporter-nvtn5"
Jul 06 22:47:51.450659 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:51.450547    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97cb756f-f621-419b-8f56-1b144ccdade3-metrics-client-ca\") pod \"node-exporter-nvtn5\" (UID: \"97cb756f-f621-419b-8f56-1b144ccdade3\") " pod="openshift-monitoring/node-exporter-nvtn5"
Jul 06 22:47:51.450659 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:51.450585    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/97cb756f-f621-419b-8f56-1b144ccdade3-node-exporter-tls\") pod \"node-exporter-nvtn5\" (UID: \"97cb756f-f621-419b-8f56-1b144ccdade3\") " pod="openshift-monitoring/node-exporter-nvtn5"
Jul 06 22:47:51.450659 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:51.450608    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/97cb756f-f621-419b-8f56-1b144ccdade3-node-exporter-wtmp\") pod \"node-exporter-nvtn5\" (UID: \"97cb756f-f621-419b-8f56-1b144ccdade3\") " pod="openshift-monitoring/node-exporter-nvtn5"
Jul 06 22:47:51.450659 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:51.450637    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/97cb756f-f621-419b-8f56-1b144ccdade3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nvtn5\" (UID: \"97cb756f-f621-419b-8f56-1b144ccdade3\") " pod="openshift-monitoring/node-exporter-nvtn5"
Jul 06 22:47:51.450816 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:51.450687    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/97cb756f-f621-419b-8f56-1b144ccdade3-root\") pod \"node-exporter-nvtn5\" (UID: \"97cb756f-f621-419b-8f56-1b144ccdade3\") " pod="openshift-monitoring/node-exporter-nvtn5"
Jul 06 22:47:51.551915 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:51.551891    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/97cb756f-f621-419b-8f56-1b144ccdade3-node-exporter-tls\") pod \"node-exporter-nvtn5\" (UID: \"97cb756f-f621-419b-8f56-1b144ccdade3\") " pod="openshift-monitoring/node-exporter-nvtn5"
Jul 06 22:47:51.551915 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:51.551918    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/97cb756f-f621-419b-8f56-1b144ccdade3-node-exporter-wtmp\") pod \"node-exporter-nvtn5\" (UID: \"97cb756f-f621-419b-8f56-1b144ccdade3\") " pod="openshift-monitoring/node-exporter-nvtn5"
Jul 06 22:47:51.552100 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:51.551936    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/97cb756f-f621-419b-8f56-1b144ccdade3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nvtn5\" (UID: \"97cb756f-f621-419b-8f56-1b144ccdade3\") " pod="openshift-monitoring/node-exporter-nvtn5"
Jul 06 22:47:51.552100 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:51.551955    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/97cb756f-f621-419b-8f56-1b144ccdade3-root\") pod \"node-exporter-nvtn5\" (UID: \"97cb756f-f621-419b-8f56-1b144ccdade3\") " pod="openshift-monitoring/node-exporter-nvtn5"
Jul 06 22:47:51.552100 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:51.551981    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h2sb\" (UniqueName: \"kubernetes.io/projected/97cb756f-f621-419b-8f56-1b144ccdade3-kube-api-access-4h2sb\") pod \"node-exporter-nvtn5\" (UID: \"97cb756f-f621-419b-8f56-1b144ccdade3\") " pod="openshift-monitoring/node-exporter-nvtn5"
Jul 06 22:47:51.552100 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:51.552009    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/97cb756f-f621-419b-8f56-1b144ccdade3-node-exporter-accelerators-collector-config\") pod \"node-exporter-nvtn5\" (UID: \"97cb756f-f621-419b-8f56-1b144ccdade3\") " pod="openshift-monitoring/node-exporter-nvtn5"
Jul 06 22:47:51.552100 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:51.552032    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/97cb756f-f621-419b-8f56-1b144ccdade3-node-exporter-textfile\") pod \"node-exporter-nvtn5\" (UID: \"97cb756f-f621-419b-8f56-1b144ccdade3\") " pod="openshift-monitoring/node-exporter-nvtn5"
Jul 06 22:47:51.552100 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:51.552043    2569 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 06 22:47:51.552100 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:51.552079    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97cb756f-f621-419b-8f56-1b144ccdade3-sys\") pod \"node-exporter-nvtn5\" (UID: \"97cb756f-f621-419b-8f56-1b144ccdade3\") " pod="openshift-monitoring/node-exporter-nvtn5"
Jul 06 22:47:51.552100 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:51.552045    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/97cb756f-f621-419b-8f56-1b144ccdade3-root\") pod \"node-exporter-nvtn5\" (UID: \"97cb756f-f621-419b-8f56-1b144ccdade3\") " pod="openshift-monitoring/node-exporter-nvtn5"
Jul 06 22:47:51.552100 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:51.552093    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/97cb756f-f621-419b-8f56-1b144ccdade3-node-exporter-wtmp\") pod \"node-exporter-nvtn5\" (UID: \"97cb756f-f621-419b-8f56-1b144ccdade3\") " pod="openshift-monitoring/node-exporter-nvtn5"
Jul 06 22:47:51.552509 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:51.552046    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97cb756f-f621-419b-8f56-1b144ccdade3-sys\") pod \"node-exporter-nvtn5\" (UID: \"97cb756f-f621-419b-8f56-1b144ccdade3\") " pod="openshift-monitoring/node-exporter-nvtn5"
Jul 06 22:47:51.552509 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:47:51.552113    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97cb756f-f621-419b-8f56-1b144ccdade3-node-exporter-tls podName:97cb756f-f621-419b-8f56-1b144ccdade3 nodeName:}" failed. No retries permitted until 2026-07-06 22:47:52.052091235 +0000 UTC m=+84.456341120 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/97cb756f-f621-419b-8f56-1b144ccdade3-node-exporter-tls") pod "node-exporter-nvtn5" (UID: "97cb756f-f621-419b-8f56-1b144ccdade3") : secret "node-exporter-tls" not found
Jul 06 22:47:51.552509 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:51.552165    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97cb756f-f621-419b-8f56-1b144ccdade3-metrics-client-ca\") pod \"node-exporter-nvtn5\" (UID: \"97cb756f-f621-419b-8f56-1b144ccdade3\") " pod="openshift-monitoring/node-exporter-nvtn5"
Jul 06 22:47:51.552509 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:51.552416    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/97cb756f-f621-419b-8f56-1b144ccdade3-node-exporter-textfile\") pod \"node-exporter-nvtn5\" (UID: \"97cb756f-f621-419b-8f56-1b144ccdade3\") " pod="openshift-monitoring/node-exporter-nvtn5"
Jul 06 22:47:51.552786 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:51.552764    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97cb756f-f621-419b-8f56-1b144ccdade3-metrics-client-ca\") pod \"node-exporter-nvtn5\" (UID: \"97cb756f-f621-419b-8f56-1b144ccdade3\") " pod="openshift-monitoring/node-exporter-nvtn5"
Jul 06 22:47:51.552822 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:51.552772    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/97cb756f-f621-419b-8f56-1b144ccdade3-node-exporter-accelerators-collector-config\") pod \"node-exporter-nvtn5\" (UID: \"97cb756f-f621-419b-8f56-1b144ccdade3\") " pod="openshift-monitoring/node-exporter-nvtn5"
Jul 06 22:47:51.554173 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:51.554156    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/97cb756f-f621-419b-8f56-1b144ccdade3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nvtn5\" (UID: \"97cb756f-f621-419b-8f56-1b144ccdade3\") " pod="openshift-monitoring/node-exporter-nvtn5"
Jul 06 22:47:51.560437 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:51.560411    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h2sb\" (UniqueName: \"kubernetes.io/projected/97cb756f-f621-419b-8f56-1b144ccdade3-kube-api-access-4h2sb\") pod \"node-exporter-nvtn5\" (UID: \"97cb756f-f621-419b-8f56-1b144ccdade3\") " pod="openshift-monitoring/node-exporter-nvtn5"
Jul 06 22:47:52.056389 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:52.056357    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/97cb756f-f621-419b-8f56-1b144ccdade3-node-exporter-tls\") pod \"node-exporter-nvtn5\" (UID: \"97cb756f-f621-419b-8f56-1b144ccdade3\") " pod="openshift-monitoring/node-exporter-nvtn5"
Jul 06 22:47:52.058528 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:52.058507    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/97cb756f-f621-419b-8f56-1b144ccdade3-node-exporter-tls\") pod \"node-exporter-nvtn5\" (UID: \"97cb756f-f621-419b-8f56-1b144ccdade3\") " pod="openshift-monitoring/node-exporter-nvtn5"
Jul 06 22:47:52.266453 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:52.266427    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nvtn5"
Jul 06 22:47:52.277980 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:47:52.277956    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97cb756f_f621_419b_8f56_1b144ccdade3.slice/crio-e57974cd0e9265d3b157fa37e390ab19454fad5bd4ad27421c9ce8b85a3491cb WatchSource:0}: Error finding container e57974cd0e9265d3b157fa37e390ab19454fad5bd4ad27421c9ce8b85a3491cb: Status 404 returned error can't find the container with id e57974cd0e9265d3b157fa37e390ab19454fad5bd4ad27421c9ce8b85a3491cb
Jul 06 22:47:52.690764 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:52.690733    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nvtn5" event={"ID":"97cb756f-f621-419b-8f56-1b144ccdade3","Type":"ContainerStarted","Data":"e57974cd0e9265d3b157fa37e390ab19454fad5bd4ad27421c9ce8b85a3491cb"}
Jul 06 22:47:53.699605 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:53.699572    2569 generic.go:356] "Generic (PLEG): container finished" podID="97cb756f-f621-419b-8f56-1b144ccdade3" containerID="8fc66232afaeba530c5acae546a1013f049c232863ace1bdbec6f2ac8071b37c" exitCode=0
Jul 06 22:47:53.699928 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:53.699628    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nvtn5" event={"ID":"97cb756f-f621-419b-8f56-1b144ccdade3","Type":"ContainerDied","Data":"8fc66232afaeba530c5acae546a1013f049c232863ace1bdbec6f2ac8071b37c"}
Jul 06 22:47:54.659684 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:54.659655    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2xhr2"
Jul 06 22:47:54.705585 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:54.705553    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nvtn5" event={"ID":"97cb756f-f621-419b-8f56-1b144ccdade3","Type":"ContainerStarted","Data":"4dfcbad6750041ce58ad7ec6d5b1d0e653f8b71f55542ce32509a048f337213d"}
Jul 06 22:47:54.706038 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:54.705612    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nvtn5" event={"ID":"97cb756f-f621-419b-8f56-1b144ccdade3","Type":"ContainerStarted","Data":"e1a20d07fa1309faae16fc31ef4996113f3a2cde65eb8299be51b94f179ee339"}
Jul 06 22:47:54.725137 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:47:54.725082    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nvtn5" podStartSLOduration=2.5881633429999997 podStartE2EDuration="3.725064493s" totalImagesPullingTime="1.076101451s" totalInitContainerRuntime="60.799699ms" isStatefulPod=true podCreationTimestamp="2026-07-06 22:47:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:47:54.724669246 +0000 UTC m=+87.128919150" watchObservedRunningTime="2026-07-06 22:47:54.725064493 +0000 UTC m=+87.129314398"
Jul 06 22:48:13.648853 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:48:13.648809    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-j9csq"
Jul 06 22:48:24.807390 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:48:24.807351    2569 generic.go:356] "Generic (PLEG): container finished" podID="6e00d4ca-8ce5-4ec1-8895-fe762a2d9e4c" containerID="a4b850c3205b71271f1637054cf94b30681ed5c094f132f9126c5b423bd33dac" exitCode=0
Jul 06 22:48:24.807874 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:48:24.807425    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-z62hq" event={"ID":"6e00d4ca-8ce5-4ec1-8895-fe762a2d9e4c","Type":"ContainerDied","Data":"a4b850c3205b71271f1637054cf94b30681ed5c094f132f9126c5b423bd33dac"}
Jul 06 22:48:24.807874 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:48:24.807726    2569 scope.go:117] "RemoveContainer" containerID="a4b850c3205b71271f1637054cf94b30681ed5c094f132f9126c5b423bd33dac"
Jul 06 22:48:25.812632 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:48:25.812601    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-z62hq" event={"ID":"6e00d4ca-8ce5-4ec1-8895-fe762a2d9e4c","Type":"ContainerStarted","Data":"abea2503ea81cc1e5e750af999811e2727d472c4e5d284b9b9ca83cdde8145a7"}
Jul 06 22:51:27.990597 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:51:27.990566    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bvsb_571f407a-249a-4fd9-8c15-53a6541db635/ovn-acl-logging/0.log"
Jul 06 22:51:28.004053 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:51:28.004010    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bvsb_571f407a-249a-4fd9-8c15-53a6541db635/ovn-acl-logging/0.log"
Jul 06 22:51:28.005088 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:51:28.005066    2569 kubelet.go:1648] "Image garbage collection succeeded"
Jul 06 22:52:11.741583 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:11.741552    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z6drs"]
Jul 06 22:52:11.743460 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:11.743445    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z6drs"
Jul 06 22:52:11.746278 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:11.746257    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 06 22:52:11.747490 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:11.747472    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-d6gx8\""
Jul 06 22:52:11.747549 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:11.747472    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 06 22:52:11.758661 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:11.758639    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z6drs"]
Jul 06 22:52:11.836601 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:11.836579    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvs6d\" (UniqueName: \"kubernetes.io/projected/448200ad-4175-4440-b1c1-dd87168693c6-kube-api-access-qvs6d\") pod \"openshift-lws-operator-fd8ccff4c-z6drs\" (UID: \"448200ad-4175-4440-b1c1-dd87168693c6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z6drs"
Jul 06 22:52:11.836707 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:11.836612    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/448200ad-4175-4440-b1c1-dd87168693c6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-z6drs\" (UID: \"448200ad-4175-4440-b1c1-dd87168693c6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z6drs"
Jul 06 22:52:11.937513 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:11.937484    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvs6d\" (UniqueName: \"kubernetes.io/projected/448200ad-4175-4440-b1c1-dd87168693c6-kube-api-access-qvs6d\") pod \"openshift-lws-operator-fd8ccff4c-z6drs\" (UID: \"448200ad-4175-4440-b1c1-dd87168693c6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z6drs"
Jul 06 22:52:11.937623 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:11.937531    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/448200ad-4175-4440-b1c1-dd87168693c6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-z6drs\" (UID: \"448200ad-4175-4440-b1c1-dd87168693c6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z6drs"
Jul 06 22:52:11.937883 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:11.937862    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/448200ad-4175-4440-b1c1-dd87168693c6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-z6drs\" (UID: \"448200ad-4175-4440-b1c1-dd87168693c6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z6drs"
Jul 06 22:52:11.945694 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:11.945673    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvs6d\" (UniqueName: \"kubernetes.io/projected/448200ad-4175-4440-b1c1-dd87168693c6-kube-api-access-qvs6d\") pod \"openshift-lws-operator-fd8ccff4c-z6drs\" (UID: \"448200ad-4175-4440-b1c1-dd87168693c6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z6drs"
Jul 06 22:52:12.051852 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:12.051795    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z6drs"
Jul 06 22:52:12.172227 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:12.172126    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z6drs"]
Jul 06 22:52:12.174280 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:52:12.174253    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod448200ad_4175_4440_b1c1_dd87168693c6.slice/crio-35e6903a250ab25132a34424655aa1ae11f6f2784f7fd9c41db95c04b43af267 WatchSource:0}: Error finding container 35e6903a250ab25132a34424655aa1ae11f6f2784f7fd9c41db95c04b43af267: Status 404 returned error can't find the container with id 35e6903a250ab25132a34424655aa1ae11f6f2784f7fd9c41db95c04b43af267
Jul 06 22:52:12.175601 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:12.175587    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 22:52:12.565204 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:12.565157    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z6drs" event={"ID":"448200ad-4175-4440-b1c1-dd87168693c6","Type":"ContainerStarted","Data":"35e6903a250ab25132a34424655aa1ae11f6f2784f7fd9c41db95c04b43af267"}
Jul 06 22:52:15.578784 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:15.578739    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z6drs" event={"ID":"448200ad-4175-4440-b1c1-dd87168693c6","Type":"ContainerStarted","Data":"9dc11bd27c37f4456942d7acc1b91db6238e016a7c2d07f4c42468eb0c4fc553"}
Jul 06 22:52:15.595405 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:15.595338    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z6drs" podStartSLOduration=2.118300156 podStartE2EDuration="4.595322616s" totalImagesPullingTime="2.47702246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:52:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:52:15.594662784 +0000 UTC m=+347.998912688" watchObservedRunningTime="2026-07-06 22:52:15.595322616 +0000 UTC m=+347.999572518"
Jul 06 22:52:37.590203 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:37.590108    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b8c6df956-qbpzs"]
Jul 06 22:52:37.592501 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:37.592483    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6b8c6df956-qbpzs"
Jul 06 22:52:37.596155 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:37.596133    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 06 22:52:37.596295 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:37.596269    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 06 22:52:37.596533 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:37.596509    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8grvv\""
Jul 06 22:52:37.596638 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:37.596621    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 06 22:52:37.596871 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:37.596855    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 06 22:52:37.615932 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:37.615905    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b8c6df956-qbpzs"]
Jul 06 22:52:37.700555 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:37.700530    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c906c6f8-114c-4e69-a05f-db2147e435ff-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b8c6df956-qbpzs\" (UID: \"c906c6f8-114c-4e69-a05f-db2147e435ff\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8c6df956-qbpzs"
Jul 06 22:52:37.700704 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:37.700564    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c906c6f8-114c-4e69-a05f-db2147e435ff-manifests\") pod \"opendatahub-operator-controller-manager-6b8c6df956-qbpzs\" (UID: \"c906c6f8-114c-4e69-a05f-db2147e435ff\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8c6df956-qbpzs"
Jul 06 22:52:37.700704 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:37.700591    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c906c6f8-114c-4e69-a05f-db2147e435ff-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b8c6df956-qbpzs\" (UID: \"c906c6f8-114c-4e69-a05f-db2147e435ff\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8c6df956-qbpzs"
Jul 06 22:52:37.700704 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:37.700639    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c906c6f8-114c-4e69-a05f-db2147e435ff-tmp\") pod \"opendatahub-operator-controller-manager-6b8c6df956-qbpzs\" (UID: \"c906c6f8-114c-4e69-a05f-db2147e435ff\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8c6df956-qbpzs"
Jul 06 22:52:37.700704 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:37.700659    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs9pk\" (UniqueName: \"kubernetes.io/projected/c906c6f8-114c-4e69-a05f-db2147e435ff-kube-api-access-gs9pk\") pod \"opendatahub-operator-controller-manager-6b8c6df956-qbpzs\" (UID: \"c906c6f8-114c-4e69-a05f-db2147e435ff\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8c6df956-qbpzs"
Jul 06 22:52:37.801305 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:37.801276    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c906c6f8-114c-4e69-a05f-db2147e435ff-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b8c6df956-qbpzs\" (UID: \"c906c6f8-114c-4e69-a05f-db2147e435ff\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8c6df956-qbpzs"
Jul 06 22:52:37.801440 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:37.801310    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c906c6f8-114c-4e69-a05f-db2147e435ff-manifests\") pod \"opendatahub-operator-controller-manager-6b8c6df956-qbpzs\" (UID: \"c906c6f8-114c-4e69-a05f-db2147e435ff\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8c6df956-qbpzs"
Jul 06 22:52:37.801440 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:37.801331    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c906c6f8-114c-4e69-a05f-db2147e435ff-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b8c6df956-qbpzs\" (UID: \"c906c6f8-114c-4e69-a05f-db2147e435ff\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8c6df956-qbpzs"
Jul 06 22:52:37.801440 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:37.801351    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c906c6f8-114c-4e69-a05f-db2147e435ff-tmp\") pod \"opendatahub-operator-controller-manager-6b8c6df956-qbpzs\" (UID: \"c906c6f8-114c-4e69-a05f-db2147e435ff\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8c6df956-qbpzs"
Jul 06 22:52:37.801546 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:37.801460    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs9pk\" (UniqueName: \"kubernetes.io/projected/c906c6f8-114c-4e69-a05f-db2147e435ff-kube-api-access-gs9pk\") pod \"opendatahub-operator-controller-manager-6b8c6df956-qbpzs\" (UID: \"c906c6f8-114c-4e69-a05f-db2147e435ff\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8c6df956-qbpzs"
Jul 06 22:52:37.801709 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:37.801678    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c906c6f8-114c-4e69-a05f-db2147e435ff-manifests\") pod \"opendatahub-operator-controller-manager-6b8c6df956-qbpzs\" (UID: \"c906c6f8-114c-4e69-a05f-db2147e435ff\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8c6df956-qbpzs"
Jul 06 22:52:37.803547 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:37.803525    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c906c6f8-114c-4e69-a05f-db2147e435ff-tmp\") pod \"opendatahub-operator-controller-manager-6b8c6df956-qbpzs\" (UID: \"c906c6f8-114c-4e69-a05f-db2147e435ff\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8c6df956-qbpzs"
Jul 06 22:52:37.803819 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:37.803802    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c906c6f8-114c-4e69-a05f-db2147e435ff-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b8c6df956-qbpzs\" (UID: \"c906c6f8-114c-4e69-a05f-db2147e435ff\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8c6df956-qbpzs"
Jul 06 22:52:37.803869 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:37.803851    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c906c6f8-114c-4e69-a05f-db2147e435ff-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b8c6df956-qbpzs\" (UID: \"c906c6f8-114c-4e69-a05f-db2147e435ff\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8c6df956-qbpzs"
Jul 06 22:52:37.810585 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:37.810563    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs9pk\" (UniqueName: \"kubernetes.io/projected/c906c6f8-114c-4e69-a05f-db2147e435ff-kube-api-access-gs9pk\") pod \"opendatahub-operator-controller-manager-6b8c6df956-qbpzs\" (UID: \"c906c6f8-114c-4e69-a05f-db2147e435ff\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8c6df956-qbpzs"
Jul 06 22:52:37.902342 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:37.902279    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6b8c6df956-qbpzs"
Jul 06 22:52:38.028405 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:38.028267    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b8c6df956-qbpzs"]
Jul 06 22:52:38.031287 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:52:38.031250    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc906c6f8_114c_4e69_a05f_db2147e435ff.slice/crio-10defb75b7bdf06cfe619cd38d49d9565143783dbcf40157338d9fc1ca76cf88 WatchSource:0}: Error finding container 10defb75b7bdf06cfe619cd38d49d9565143783dbcf40157338d9fc1ca76cf88: Status 404 returned error can't find the container with id 10defb75b7bdf06cfe619cd38d49d9565143783dbcf40157338d9fc1ca76cf88
Jul 06 22:52:38.659632 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:38.659589    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b8c6df956-qbpzs" event={"ID":"c906c6f8-114c-4e69-a05f-db2147e435ff","Type":"ContainerStarted","Data":"10defb75b7bdf06cfe619cd38d49d9565143783dbcf40157338d9fc1ca76cf88"}
Jul 06 22:52:41.673451 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:41.673416    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b8c6df956-qbpzs" event={"ID":"c906c6f8-114c-4e69-a05f-db2147e435ff","Type":"ContainerStarted","Data":"2d8ac44a82af4189a5bd737b2a0223ea12b8cf823f6624a8ef8af81ccfc4edbc"}
Jul 06 22:52:42.678784 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:42.678751    2569 generic.go:356] "Generic (PLEG): container finished" podID="c906c6f8-114c-4e69-a05f-db2147e435ff" containerID="2d8ac44a82af4189a5bd737b2a0223ea12b8cf823f6624a8ef8af81ccfc4edbc" exitCode=0
Jul 06 22:52:42.679159 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:42.678795    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b8c6df956-qbpzs" event={"ID":"c906c6f8-114c-4e69-a05f-db2147e435ff","Type":"ContainerDied","Data":"2d8ac44a82af4189a5bd737b2a0223ea12b8cf823f6624a8ef8af81ccfc4edbc"}
Jul 06 22:52:43.685545 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:43.685508    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b8c6df956-qbpzs" event={"ID":"c906c6f8-114c-4e69-a05f-db2147e435ff","Type":"ContainerStarted","Data":"9b81ab1fb4089d454f7093b176b27364ac8724c2bdc02f53cc51592088da31cd"}
Jul 06 22:52:43.685930 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:43.685654    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6b8c6df956-qbpzs"
Jul 06 22:52:43.727533 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:43.727485    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6b8c6df956-qbpzs" podStartSLOduration=2.359835721 podStartE2EDuration="6.727472368s" totalImagesPullingTime="3.287332666s" totalInitContainerRuntime="1.080303981s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:52:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:52:43.727036713 +0000 UTC m=+376.131286631" watchObservedRunningTime="2026-07-06 22:52:43.727472368 +0000 UTC m=+376.131722295"
Jul 06 22:52:54.691462 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:54.691435    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6b8c6df956-qbpzs"
Jul 06 22:52:57.640266 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:57.640234    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-698448cf7f-ll8cq"]
Jul 06 22:52:57.648361 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:57.648330    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-698448cf7f-ll8cq"
Jul 06 22:52:57.652282 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:57.652257    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 06 22:52:57.652405 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:57.652368    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 06 22:52:57.679083 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:57.679063    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-698448cf7f-ll8cq"]
Jul 06 22:52:57.733811 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:57.733790    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73f21974-debc-40ab-8b7d-18ca71bd3082-tmp\") pod \"kube-auth-proxy-698448cf7f-ll8cq\" (UID: \"73f21974-debc-40ab-8b7d-18ca71bd3082\") " pod="openshift-ingress/kube-auth-proxy-698448cf7f-ll8cq"
Jul 06 22:52:57.733955 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:57.733818    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73f21974-debc-40ab-8b7d-18ca71bd3082-tls-certs\") pod \"kube-auth-proxy-698448cf7f-ll8cq\" (UID: \"73f21974-debc-40ab-8b7d-18ca71bd3082\") " pod="openshift-ingress/kube-auth-proxy-698448cf7f-ll8cq"
Jul 06 22:52:57.733955 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:57.733835    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6lj9\" (UniqueName: \"kubernetes.io/projected/73f21974-debc-40ab-8b7d-18ca71bd3082-kube-api-access-p6lj9\") pod \"kube-auth-proxy-698448cf7f-ll8cq\" (UID: \"73f21974-debc-40ab-8b7d-18ca71bd3082\") " pod="openshift-ingress/kube-auth-proxy-698448cf7f-ll8cq"
Jul 06 22:52:57.834613 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:57.834589    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73f21974-debc-40ab-8b7d-18ca71bd3082-tmp\") pod \"kube-auth-proxy-698448cf7f-ll8cq\" (UID: \"73f21974-debc-40ab-8b7d-18ca71bd3082\") " pod="openshift-ingress/kube-auth-proxy-698448cf7f-ll8cq"
Jul 06 22:52:57.834741 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:57.834619    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73f21974-debc-40ab-8b7d-18ca71bd3082-tls-certs\") pod \"kube-auth-proxy-698448cf7f-ll8cq\" (UID: \"73f21974-debc-40ab-8b7d-18ca71bd3082\") " pod="openshift-ingress/kube-auth-proxy-698448cf7f-ll8cq"
Jul 06 22:52:57.834741 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:57.834635    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6lj9\" (UniqueName: \"kubernetes.io/projected/73f21974-debc-40ab-8b7d-18ca71bd3082-kube-api-access-p6lj9\") pod \"kube-auth-proxy-698448cf7f-ll8cq\" (UID: \"73f21974-debc-40ab-8b7d-18ca71bd3082\") " pod="openshift-ingress/kube-auth-proxy-698448cf7f-ll8cq"
Jul 06 22:52:57.836971 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:57.836947    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73f21974-debc-40ab-8b7d-18ca71bd3082-tmp\") pod \"kube-auth-proxy-698448cf7f-ll8cq\" (UID: \"73f21974-debc-40ab-8b7d-18ca71bd3082\") " pod="openshift-ingress/kube-auth-proxy-698448cf7f-ll8cq"
Jul 06 22:52:57.837205 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:57.837169    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73f21974-debc-40ab-8b7d-18ca71bd3082-tls-certs\") pod \"kube-auth-proxy-698448cf7f-ll8cq\" (UID: \"73f21974-debc-40ab-8b7d-18ca71bd3082\") " pod="openshift-ingress/kube-auth-proxy-698448cf7f-ll8cq"
Jul 06 22:52:57.859939 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:57.859918    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6lj9\" (UniqueName: \"kubernetes.io/projected/73f21974-debc-40ab-8b7d-18ca71bd3082-kube-api-access-p6lj9\") pod \"kube-auth-proxy-698448cf7f-ll8cq\" (UID: \"73f21974-debc-40ab-8b7d-18ca71bd3082\") " pod="openshift-ingress/kube-auth-proxy-698448cf7f-ll8cq"
Jul 06 22:52:57.957973 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:57.957945    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-698448cf7f-ll8cq"
Jul 06 22:52:58.150862 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:58.150841    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-698448cf7f-ll8cq"]
Jul 06 22:52:58.153412 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:52:58.153384    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73f21974_debc_40ab_8b7d_18ca71bd3082.slice/crio-9651683262b2b3dd3279a7a3a526f2aeed996f363d8381294042164572e9cc66 WatchSource:0}: Error finding container 9651683262b2b3dd3279a7a3a526f2aeed996f363d8381294042164572e9cc66: Status 404 returned error can't find the container with id 9651683262b2b3dd3279a7a3a526f2aeed996f363d8381294042164572e9cc66
Jul 06 22:52:58.743024 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:58.742981    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-698448cf7f-ll8cq" event={"ID":"73f21974-debc-40ab-8b7d-18ca71bd3082","Type":"ContainerStarted","Data":"9651683262b2b3dd3279a7a3a526f2aeed996f363d8381294042164572e9cc66"}
Jul 06 22:52:59.433581 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:59.433551    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vqgqh"]
Jul 06 22:52:59.436355 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:59.436313    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-vqgqh"
Jul 06 22:52:59.439211 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:59.439170    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 06 22:52:59.453051 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:59.453030    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vqgqh"]
Jul 06 22:52:59.545701 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:59.545672    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1cfb9e6-e14a-4ba6-a715-3af91a5f38d6-tls-certs\") pod \"model-serving-api-54d89bd5fc-vqgqh\" (UID: \"d1cfb9e6-e14a-4ba6-a715-3af91a5f38d6\") " pod="opendatahub/model-serving-api-54d89bd5fc-vqgqh"
Jul 06 22:52:59.545896 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:59.545710    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxz9r\" (UniqueName: \"kubernetes.io/projected/d1cfb9e6-e14a-4ba6-a715-3af91a5f38d6-kube-api-access-wxz9r\") pod \"model-serving-api-54d89bd5fc-vqgqh\" (UID: \"d1cfb9e6-e14a-4ba6-a715-3af91a5f38d6\") " pod="opendatahub/model-serving-api-54d89bd5fc-vqgqh"
Jul 06 22:52:59.646996 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:59.646956    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1cfb9e6-e14a-4ba6-a715-3af91a5f38d6-tls-certs\") pod \"model-serving-api-54d89bd5fc-vqgqh\" (UID: \"d1cfb9e6-e14a-4ba6-a715-3af91a5f38d6\") " pod="opendatahub/model-serving-api-54d89bd5fc-vqgqh"
Jul 06 22:52:59.646996 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:59.646995    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxz9r\" (UniqueName: \"kubernetes.io/projected/d1cfb9e6-e14a-4ba6-a715-3af91a5f38d6-kube-api-access-wxz9r\") pod \"model-serving-api-54d89bd5fc-vqgqh\" (UID: \"d1cfb9e6-e14a-4ba6-a715-3af91a5f38d6\") " pod="opendatahub/model-serving-api-54d89bd5fc-vqgqh"
Jul 06 22:52:59.649814 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:59.649778    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1cfb9e6-e14a-4ba6-a715-3af91a5f38d6-tls-certs\") pod \"model-serving-api-54d89bd5fc-vqgqh\" (UID: \"d1cfb9e6-e14a-4ba6-a715-3af91a5f38d6\") " pod="opendatahub/model-serving-api-54d89bd5fc-vqgqh"
Jul 06 22:52:59.656676 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:59.656643    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxz9r\" (UniqueName: \"kubernetes.io/projected/d1cfb9e6-e14a-4ba6-a715-3af91a5f38d6-kube-api-access-wxz9r\") pod \"model-serving-api-54d89bd5fc-vqgqh\" (UID: \"d1cfb9e6-e14a-4ba6-a715-3af91a5f38d6\") " pod="opendatahub/model-serving-api-54d89bd5fc-vqgqh"
Jul 06 22:52:59.748508 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:59.748429    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-vqgqh"
Jul 06 22:52:59.812620 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:59.812587    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-vcslh"]
Jul 06 22:52:59.815856 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:59.815834    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vcslh"
Jul 06 22:52:59.820267 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:59.820213    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 06 22:52:59.820267 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:59.820265    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 06 22:52:59.820454 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:59.820318    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-8m8wg\""
Jul 06 22:52:59.820853 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:59.820686    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 06 22:52:59.820944 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:59.820858    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 06 22:52:59.833954 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:59.833932    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-vcslh"]
Jul 06 22:52:59.847746 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:59.847718    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/63f261c5-e338-4bd7-ac23-3cbfbae0a1ea-cacerts\") pod \"istiod-openshift-gateway-7574459656-vcslh\" (UID: \"63f261c5-e338-4bd7-ac23-3cbfbae0a1ea\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vcslh"
Jul 06 22:52:59.847840 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:59.847755    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/63f261c5-e338-4bd7-ac23-3cbfbae0a1ea-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-vcslh\" (UID: \"63f261c5-e338-4bd7-ac23-3cbfbae0a1ea\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vcslh"
Jul 06 22:52:59.847840 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:59.847775    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63f261c5-e338-4bd7-ac23-3cbfbae0a1ea-istio-token\") pod \"istiod-openshift-gateway-7574459656-vcslh\" (UID: \"63f261c5-e338-4bd7-ac23-3cbfbae0a1ea\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vcslh"
Jul 06 22:52:59.847840 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:59.847813    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/63f261c5-e338-4bd7-ac23-3cbfbae0a1ea-local-certs\") pod \"istiod-openshift-gateway-7574459656-vcslh\" (UID: \"63f261c5-e338-4bd7-ac23-3cbfbae0a1ea\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vcslh"
Jul 06 22:52:59.847840 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:59.847832    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9rcz\" (UniqueName: \"kubernetes.io/projected/63f261c5-e338-4bd7-ac23-3cbfbae0a1ea-kube-api-access-s9rcz\") pod \"istiod-openshift-gateway-7574459656-vcslh\" (UID: \"63f261c5-e338-4bd7-ac23-3cbfbae0a1ea\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vcslh"
Jul 06 22:52:59.848057 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:59.847868    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/63f261c5-e338-4bd7-ac23-3cbfbae0a1ea-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-vcslh\" (UID: \"63f261c5-e338-4bd7-ac23-3cbfbae0a1ea\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vcslh"
Jul 06 22:52:59.848057 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:59.847911    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/63f261c5-e338-4bd7-ac23-3cbfbae0a1ea-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-vcslh\" (UID: \"63f261c5-e338-4bd7-ac23-3cbfbae0a1ea\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vcslh"
Jul 06 22:52:59.948406 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:59.948374    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/63f261c5-e338-4bd7-ac23-3cbfbae0a1ea-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-vcslh\" (UID: \"63f261c5-e338-4bd7-ac23-3cbfbae0a1ea\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vcslh"
Jul 06 22:52:59.948582 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:59.948415    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63f261c5-e338-4bd7-ac23-3cbfbae0a1ea-istio-token\") pod \"istiod-openshift-gateway-7574459656-vcslh\" (UID: \"63f261c5-e338-4bd7-ac23-3cbfbae0a1ea\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vcslh"
Jul 06 22:52:59.948582 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:59.948442    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/63f261c5-e338-4bd7-ac23-3cbfbae0a1ea-local-certs\") pod \"istiod-openshift-gateway-7574459656-vcslh\" (UID: \"63f261c5-e338-4bd7-ac23-3cbfbae0a1ea\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vcslh"
Jul 06 22:52:59.948582 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:59.948468    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9rcz\" (UniqueName: \"kubernetes.io/projected/63f261c5-e338-4bd7-ac23-3cbfbae0a1ea-kube-api-access-s9rcz\") pod \"istiod-openshift-gateway-7574459656-vcslh\" (UID: \"63f261c5-e338-4bd7-ac23-3cbfbae0a1ea\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vcslh"
Jul 06 22:52:59.948582 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:59.948504    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/63f261c5-e338-4bd7-ac23-3cbfbae0a1ea-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-vcslh\" (UID: \"63f261c5-e338-4bd7-ac23-3cbfbae0a1ea\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vcslh"
Jul 06 22:52:59.948582 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:59.948547    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/63f261c5-e338-4bd7-ac23-3cbfbae0a1ea-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-vcslh\" (UID: \"63f261c5-e338-4bd7-ac23-3cbfbae0a1ea\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vcslh"
Jul 06 22:52:59.948837 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:59.948637    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/63f261c5-e338-4bd7-ac23-3cbfbae0a1ea-cacerts\") pod \"istiod-openshift-gateway-7574459656-vcslh\" (UID: \"63f261c5-e338-4bd7-ac23-3cbfbae0a1ea\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vcslh"
Jul 06 22:52:59.949553 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:59.949524    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/63f261c5-e338-4bd7-ac23-3cbfbae0a1ea-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-vcslh\" (UID: \"63f261c5-e338-4bd7-ac23-3cbfbae0a1ea\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vcslh"
Jul 06 22:52:59.951523 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:59.951495    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/63f261c5-e338-4bd7-ac23-3cbfbae0a1ea-cacerts\") pod \"istiod-openshift-gateway-7574459656-vcslh\" (UID: \"63f261c5-e338-4bd7-ac23-3cbfbae0a1ea\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vcslh"
Jul 06 22:52:59.951697 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:59.951677    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/63f261c5-e338-4bd7-ac23-3cbfbae0a1ea-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-vcslh\" (UID: \"63f261c5-e338-4bd7-ac23-3cbfbae0a1ea\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vcslh"
Jul 06 22:52:59.951745 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:59.951727    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/63f261c5-e338-4bd7-ac23-3cbfbae0a1ea-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-vcslh\" (UID: \"63f261c5-e338-4bd7-ac23-3cbfbae0a1ea\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vcslh"
Jul 06 22:52:59.951791 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:59.951771    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/63f261c5-e338-4bd7-ac23-3cbfbae0a1ea-local-certs\") pod \"istiod-openshift-gateway-7574459656-vcslh\" (UID: \"63f261c5-e338-4bd7-ac23-3cbfbae0a1ea\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vcslh"
Jul 06 22:52:59.961561 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:59.961537    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63f261c5-e338-4bd7-ac23-3cbfbae0a1ea-istio-token\") pod \"istiod-openshift-gateway-7574459656-vcslh\" (UID: \"63f261c5-e338-4bd7-ac23-3cbfbae0a1ea\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vcslh"
Jul 06 22:52:59.971847 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:52:59.971817    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9rcz\" (UniqueName: \"kubernetes.io/projected/63f261c5-e338-4bd7-ac23-3cbfbae0a1ea-kube-api-access-s9rcz\") pod \"istiod-openshift-gateway-7574459656-vcslh\" (UID: \"63f261c5-e338-4bd7-ac23-3cbfbae0a1ea\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vcslh"
Jul 06 22:53:00.127893 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:00.127818    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vcslh"
Jul 06 22:53:00.275325 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:00.275300    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vqgqh"]
Jul 06 22:53:00.279403 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:53:00.279377    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1cfb9e6_e14a_4ba6_a715_3af91a5f38d6.slice/crio-cf254ff2701348027b6c3327cae61200e4ce61904acc90613e02d5ed6d3b4322 WatchSource:0}: Error finding container cf254ff2701348027b6c3327cae61200e4ce61904acc90613e02d5ed6d3b4322: Status 404 returned error can't find the container with id cf254ff2701348027b6c3327cae61200e4ce61904acc90613e02d5ed6d3b4322
Jul 06 22:53:00.297198 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:00.297141    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-vcslh"]
Jul 06 22:53:00.299754 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:53:00.299729    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63f261c5_e338_4bd7_ac23_3cbfbae0a1ea.slice/crio-dc2a4c4f74f5b80268e330b0cf28004d6fb582e4c5ed5405c59feaca7caf8d8c WatchSource:0}: Error finding container dc2a4c4f74f5b80268e330b0cf28004d6fb582e4c5ed5405c59feaca7caf8d8c: Status 404 returned error can't find the container with id dc2a4c4f74f5b80268e330b0cf28004d6fb582e4c5ed5405c59feaca7caf8d8c
Jul 06 22:53:00.754251 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:00.754207    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-vqgqh" event={"ID":"d1cfb9e6-e14a-4ba6-a715-3af91a5f38d6","Type":"ContainerStarted","Data":"cf254ff2701348027b6c3327cae61200e4ce61904acc90613e02d5ed6d3b4322"}
Jul 06 22:53:00.755565 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:00.755538    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vcslh" event={"ID":"63f261c5-e338-4bd7-ac23-3cbfbae0a1ea","Type":"ContainerStarted","Data":"dc2a4c4f74f5b80268e330b0cf28004d6fb582e4c5ed5405c59feaca7caf8d8c"}
Jul 06 22:53:00.757110 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:00.757082    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-698448cf7f-ll8cq" event={"ID":"73f21974-debc-40ab-8b7d-18ca71bd3082","Type":"ContainerStarted","Data":"1972e50d6bcb62aed7bb83e14744bd85569b0749a3b360292d2c56854de10dee"}
Jul 06 22:53:00.776453 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:00.776404    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-698448cf7f-ll8cq" podStartSLOduration=1.7443094719999999 podStartE2EDuration="3.776386037s" totalImagesPullingTime="2.032076565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:52:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:53:00.774578059 +0000 UTC m=+393.178827972" watchObservedRunningTime="2026-07-06 22:53:00.776386037 +0000 UTC m=+393.180635942"
Jul 06 22:53:02.767254 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:02.767211    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-vqgqh" event={"ID":"d1cfb9e6-e14a-4ba6-a715-3af91a5f38d6","Type":"ContainerStarted","Data":"3e55fbd23bfeb4a65471aaa9e9f179c305765595713c83f8ef3fcf4b0365e98e"}
Jul 06 22:53:02.767729 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:02.767340    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-vqgqh"
Jul 06 22:53:02.785558 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:02.785512    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-vqgqh" podStartSLOduration=2.306346401 podStartE2EDuration="3.785496564s" totalImagesPullingTime="1.479150163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:52:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:53:02.783418861 +0000 UTC m=+395.187668780" watchObservedRunningTime="2026-07-06 22:53:02.785496564 +0000 UTC m=+395.189746475"
Jul 06 22:53:03.707400 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:03.707361    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 22:53:03.707515 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:03.707431    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 22:53:04.779417 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:04.779385    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vcslh" event={"ID":"63f261c5-e338-4bd7-ac23-3cbfbae0a1ea","Type":"ContainerStarted","Data":"1a5e9f084c4b2d810755dbe6d428d8fe22a215405d1a7b49a411762cdddbce89"}
Jul 06 22:53:04.779788 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:04.779563    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vcslh"
Jul 06 22:53:04.781128 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:04.781092    2569 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-vcslh container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 06 22:53:04.781263 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:04.781147    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vcslh" podUID="63f261c5-e338-4bd7-ac23-3cbfbae0a1ea" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 22:53:04.819786 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:04.819744    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vcslh" podStartSLOduration=2.414489212 podStartE2EDuration="5.819730841s" totalImagesPullingTime="3.405241629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:52:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:53:04.818730093 +0000 UTC m=+397.222979998" watchObservedRunningTime="2026-07-06 22:53:04.819730841 +0000 UTC m=+397.223980745"
Jul 06 22:53:05.784837 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:05.784809    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vcslh"
Jul 06 22:53:10.921192 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:10.921148    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-dhcx5"]
Jul 06 22:53:10.929034 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:10.929015    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-dhcx5"
Jul 06 22:53:10.933246 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:10.933230    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 06 22:53:10.933778 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:10.933763    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-gjb9c\""
Jul 06 22:53:10.936503 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:10.936485    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-dmnld"]
Jul 06 22:53:10.939836 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:10.939822    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dmnld"
Jul 06 22:53:10.955107 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:10.955089    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 06 22:53:10.955216 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:10.955127    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-w5z97\""
Jul 06 22:53:10.972937 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:10.972917    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-dmnld"]
Jul 06 22:53:11.035528 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:11.035491    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5bzk\" (UniqueName: \"kubernetes.io/projected/1753a9aa-4104-4f17-b295-abeea36a2262-kube-api-access-l5bzk\") pod \"llmisvc-controller-manager-777c6bd7d5-dmnld\" (UID: \"1753a9aa-4104-4f17-b295-abeea36a2262\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dmnld"
Jul 06 22:53:11.035528 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:11.035523    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ec333e7-07df-46b2-825f-91b729228c0b-cert\") pod \"kserve-controller-manager-d6f4b65b8-dhcx5\" (UID: \"4ec333e7-07df-46b2-825f-91b729228c0b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-dhcx5"
Jul 06 22:53:11.035684 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:11.035605    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6tff\" (UniqueName: \"kubernetes.io/projected/4ec333e7-07df-46b2-825f-91b729228c0b-kube-api-access-w6tff\") pod \"kserve-controller-manager-d6f4b65b8-dhcx5\" (UID: \"4ec333e7-07df-46b2-825f-91b729228c0b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-dhcx5"
Jul 06 22:53:11.035684 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:11.035676    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1753a9aa-4104-4f17-b295-abeea36a2262-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-dmnld\" (UID: \"1753a9aa-4104-4f17-b295-abeea36a2262\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dmnld"
Jul 06 22:53:11.036666 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:11.036644    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-dhcx5"]
Jul 06 22:53:11.136579 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:11.136552    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ec333e7-07df-46b2-825f-91b729228c0b-cert\") pod \"kserve-controller-manager-d6f4b65b8-dhcx5\" (UID: \"4ec333e7-07df-46b2-825f-91b729228c0b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-dhcx5"
Jul 06 22:53:11.136727 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:11.136619    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6tff\" (UniqueName: \"kubernetes.io/projected/4ec333e7-07df-46b2-825f-91b729228c0b-kube-api-access-w6tff\") pod \"kserve-controller-manager-d6f4b65b8-dhcx5\" (UID: \"4ec333e7-07df-46b2-825f-91b729228c0b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-dhcx5"
Jul 06 22:53:11.136727 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:11.136670    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1753a9aa-4104-4f17-b295-abeea36a2262-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-dmnld\" (UID: \"1753a9aa-4104-4f17-b295-abeea36a2262\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dmnld"
Jul 06 22:53:11.136727 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:11.136708    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5bzk\" (UniqueName: \"kubernetes.io/projected/1753a9aa-4104-4f17-b295-abeea36a2262-kube-api-access-l5bzk\") pod \"llmisvc-controller-manager-777c6bd7d5-dmnld\" (UID: \"1753a9aa-4104-4f17-b295-abeea36a2262\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dmnld"
Jul 06 22:53:11.139191 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:11.139154    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ec333e7-07df-46b2-825f-91b729228c0b-cert\") pod \"kserve-controller-manager-d6f4b65b8-dhcx5\" (UID: \"4ec333e7-07df-46b2-825f-91b729228c0b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-dhcx5"
Jul 06 22:53:11.139297 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:11.139253    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1753a9aa-4104-4f17-b295-abeea36a2262-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-dmnld\" (UID: \"1753a9aa-4104-4f17-b295-abeea36a2262\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dmnld"
Jul 06 22:53:11.159052 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:11.159017    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5bzk\" (UniqueName: \"kubernetes.io/projected/1753a9aa-4104-4f17-b295-abeea36a2262-kube-api-access-l5bzk\") pod \"llmisvc-controller-manager-777c6bd7d5-dmnld\" (UID: \"1753a9aa-4104-4f17-b295-abeea36a2262\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dmnld"
Jul 06 22:53:11.159931 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:11.159911    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6tff\" (UniqueName: \"kubernetes.io/projected/4ec333e7-07df-46b2-825f-91b729228c0b-kube-api-access-w6tff\") pod \"kserve-controller-manager-d6f4b65b8-dhcx5\" (UID: \"4ec333e7-07df-46b2-825f-91b729228c0b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-dhcx5"
Jul 06 22:53:11.239210 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:11.239172    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-dhcx5"
Jul 06 22:53:11.247382 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:11.247360    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dmnld"
Jul 06 22:53:11.419775 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:11.419749    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-dhcx5"]
Jul 06 22:53:11.422125 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:53:11.422100    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ec333e7_07df_46b2_825f_91b729228c0b.slice/crio-71f06ffae5c350962c12d2a2877a34e8eb3be8ab356aec38d4887a3fe88a3560 WatchSource:0}: Error finding container 71f06ffae5c350962c12d2a2877a34e8eb3be8ab356aec38d4887a3fe88a3560: Status 404 returned error can't find the container with id 71f06ffae5c350962c12d2a2877a34e8eb3be8ab356aec38d4887a3fe88a3560
Jul 06 22:53:11.444263 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:11.444239    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-dmnld"]
Jul 06 22:53:11.446403 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:53:11.446379    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1753a9aa_4104_4f17_b295_abeea36a2262.slice/crio-c5befd5e65505e58bac1fdf7b16310fd7722a890bd0382b663efee529d6ab17b WatchSource:0}: Error finding container c5befd5e65505e58bac1fdf7b16310fd7722a890bd0382b663efee529d6ab17b: Status 404 returned error can't find the container with id c5befd5e65505e58bac1fdf7b16310fd7722a890bd0382b663efee529d6ab17b
Jul 06 22:53:11.808465 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:11.808433    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-dhcx5" event={"ID":"4ec333e7-07df-46b2-825f-91b729228c0b","Type":"ContainerStarted","Data":"71f06ffae5c350962c12d2a2877a34e8eb3be8ab356aec38d4887a3fe88a3560"}
Jul 06 22:53:11.809511 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:11.809488    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dmnld" event={"ID":"1753a9aa-4104-4f17-b295-abeea36a2262","Type":"ContainerStarted","Data":"c5befd5e65505e58bac1fdf7b16310fd7722a890bd0382b663efee529d6ab17b"}
Jul 06 22:53:13.662232 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:13.662139    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-vcslh"]
Jul 06 22:53:13.662659 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:13.662416    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vcslh" podUID="63f261c5-e338-4bd7-ac23-3cbfbae0a1ea" containerName="discovery" containerID="cri-o://1a5e9f084c4b2d810755dbe6d428d8fe22a215405d1a7b49a411762cdddbce89" gracePeriod=30
Jul 06 22:53:13.778679 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:13.778645    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-vqgqh"
Jul 06 22:53:14.087293 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:14.087274    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vcslh"
Jul 06 22:53:14.162908 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:14.162881    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9rcz\" (UniqueName: \"kubernetes.io/projected/63f261c5-e338-4bd7-ac23-3cbfbae0a1ea-kube-api-access-s9rcz\") pod \"63f261c5-e338-4bd7-ac23-3cbfbae0a1ea\" (UID: \"63f261c5-e338-4bd7-ac23-3cbfbae0a1ea\") "
Jul 06 22:53:14.163038 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:14.162914    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/63f261c5-e338-4bd7-ac23-3cbfbae0a1ea-local-certs\") pod \"63f261c5-e338-4bd7-ac23-3cbfbae0a1ea\" (UID: \"63f261c5-e338-4bd7-ac23-3cbfbae0a1ea\") "
Jul 06 22:53:14.163038 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:14.162935    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/63f261c5-e338-4bd7-ac23-3cbfbae0a1ea-istio-csr-ca-configmap\") pod \"63f261c5-e338-4bd7-ac23-3cbfbae0a1ea\" (UID: \"63f261c5-e338-4bd7-ac23-3cbfbae0a1ea\") "
Jul 06 22:53:14.163038 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:14.162962    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/63f261c5-e338-4bd7-ac23-3cbfbae0a1ea-istio-kubeconfig\") pod \"63f261c5-e338-4bd7-ac23-3cbfbae0a1ea\" (UID: \"63f261c5-e338-4bd7-ac23-3cbfbae0a1ea\") "
Jul 06 22:53:14.163038 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:14.163013    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/63f261c5-e338-4bd7-ac23-3cbfbae0a1ea-cacerts\") pod \"63f261c5-e338-4bd7-ac23-3cbfbae0a1ea\" (UID: \"63f261c5-e338-4bd7-ac23-3cbfbae0a1ea\") "
Jul 06 22:53:14.163288 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:14.163049    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63f261c5-e338-4bd7-ac23-3cbfbae0a1ea-istio-token\") pod \"63f261c5-e338-4bd7-ac23-3cbfbae0a1ea\" (UID: \"63f261c5-e338-4bd7-ac23-3cbfbae0a1ea\") "
Jul 06 22:53:14.163288 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:14.163085    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/63f261c5-e338-4bd7-ac23-3cbfbae0a1ea-istio-csr-dns-cert\") pod \"63f261c5-e338-4bd7-ac23-3cbfbae0a1ea\" (UID: \"63f261c5-e338-4bd7-ac23-3cbfbae0a1ea\") "
Jul 06 22:53:14.163405 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:14.163330    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63f261c5-e338-4bd7-ac23-3cbfbae0a1ea-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "63f261c5-e338-4bd7-ac23-3cbfbae0a1ea" (UID: "63f261c5-e338-4bd7-ac23-3cbfbae0a1ea"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 22:53:14.165167 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:14.165142    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63f261c5-e338-4bd7-ac23-3cbfbae0a1ea-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "63f261c5-e338-4bd7-ac23-3cbfbae0a1ea" (UID: "63f261c5-e338-4bd7-ac23-3cbfbae0a1ea"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 22:53:14.165411 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:14.165393    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63f261c5-e338-4bd7-ac23-3cbfbae0a1ea-local-certs" (OuterVolumeSpecName: "local-certs") pod "63f261c5-e338-4bd7-ac23-3cbfbae0a1ea" (UID: "63f261c5-e338-4bd7-ac23-3cbfbae0a1ea"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:53:14.165569 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:14.165548    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63f261c5-e338-4bd7-ac23-3cbfbae0a1ea-cacerts" (OuterVolumeSpecName: "cacerts") pod "63f261c5-e338-4bd7-ac23-3cbfbae0a1ea" (UID: "63f261c5-e338-4bd7-ac23-3cbfbae0a1ea"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 22:53:14.165569 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:14.165557    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63f261c5-e338-4bd7-ac23-3cbfbae0a1ea-istio-token" (OuterVolumeSpecName: "istio-token") pod "63f261c5-e338-4bd7-ac23-3cbfbae0a1ea" (UID: "63f261c5-e338-4bd7-ac23-3cbfbae0a1ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 22:53:14.165692 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:14.165575    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63f261c5-e338-4bd7-ac23-3cbfbae0a1ea-kube-api-access-s9rcz" (OuterVolumeSpecName: "kube-api-access-s9rcz") pod "63f261c5-e338-4bd7-ac23-3cbfbae0a1ea" (UID: "63f261c5-e338-4bd7-ac23-3cbfbae0a1ea"). InnerVolumeSpecName "kube-api-access-s9rcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 22:53:14.165692 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:14.165596    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63f261c5-e338-4bd7-ac23-3cbfbae0a1ea-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "63f261c5-e338-4bd7-ac23-3cbfbae0a1ea" (UID: "63f261c5-e338-4bd7-ac23-3cbfbae0a1ea"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 22:53:14.264569 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:14.264512    2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63f261c5-e338-4bd7-ac23-3cbfbae0a1ea-istio-token\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 22:53:14.264569 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:14.264538    2569 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/63f261c5-e338-4bd7-ac23-3cbfbae0a1ea-istio-csr-dns-cert\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 22:53:14.264569 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:14.264553    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9rcz\" (UniqueName: \"kubernetes.io/projected/63f261c5-e338-4bd7-ac23-3cbfbae0a1ea-kube-api-access-s9rcz\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 22:53:14.264569 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:14.264566    2569 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/63f261c5-e338-4bd7-ac23-3cbfbae0a1ea-local-certs\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 22:53:14.264752 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:14.264580    2569 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/63f261c5-e338-4bd7-ac23-3cbfbae0a1ea-istio-csr-ca-configmap\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 22:53:14.264752 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:14.264592    2569 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/63f261c5-e338-4bd7-ac23-3cbfbae0a1ea-istio-kubeconfig\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 22:53:14.264752 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:14.264604    2569 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/63f261c5-e338-4bd7-ac23-3cbfbae0a1ea-cacerts\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 22:53:14.825223 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:14.825170    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dmnld" event={"ID":"1753a9aa-4104-4f17-b295-abeea36a2262","Type":"ContainerStarted","Data":"45a398119c7877316d5496c1addaa4f49835cd05dd7b5e5cdc0d34d188ea7cea"}
Jul 06 22:53:14.825658 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:14.825281    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dmnld"
Jul 06 22:53:14.826433 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:14.826412    2569 generic.go:356] "Generic (PLEG): container finished" podID="63f261c5-e338-4bd7-ac23-3cbfbae0a1ea" containerID="1a5e9f084c4b2d810755dbe6d428d8fe22a215405d1a7b49a411762cdddbce89" exitCode=0
Jul 06 22:53:14.826537 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:14.826436    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vcslh" event={"ID":"63f261c5-e338-4bd7-ac23-3cbfbae0a1ea","Type":"ContainerDied","Data":"1a5e9f084c4b2d810755dbe6d428d8fe22a215405d1a7b49a411762cdddbce89"}
Jul 06 22:53:14.826537 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:14.826450    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vcslh"
Jul 06 22:53:14.826537 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:14.826474    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vcslh" event={"ID":"63f261c5-e338-4bd7-ac23-3cbfbae0a1ea","Type":"ContainerDied","Data":"dc2a4c4f74f5b80268e330b0cf28004d6fb582e4c5ed5405c59feaca7caf8d8c"}
Jul 06 22:53:14.826537 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:14.826503    2569 scope.go:117] "RemoveContainer" containerID="1a5e9f084c4b2d810755dbe6d428d8fe22a215405d1a7b49a411762cdddbce89"
Jul 06 22:53:14.828142 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:14.828121    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-dhcx5" event={"ID":"4ec333e7-07df-46b2-825f-91b729228c0b","Type":"ContainerStarted","Data":"ca7d2bff5d02ccb876fb20f172a5087d6b540675176a4459f0727c8945b53c5d"}
Jul 06 22:53:14.828278 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:14.828225    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-dhcx5"
Jul 06 22:53:14.834683 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:14.834667    2569 scope.go:117] "RemoveContainer" containerID="1a5e9f084c4b2d810755dbe6d428d8fe22a215405d1a7b49a411762cdddbce89"
Jul 06 22:53:14.834940 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:53:14.834914    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a5e9f084c4b2d810755dbe6d428d8fe22a215405d1a7b49a411762cdddbce89\": container with ID starting with 1a5e9f084c4b2d810755dbe6d428d8fe22a215405d1a7b49a411762cdddbce89 not found: ID does not exist" containerID="1a5e9f084c4b2d810755dbe6d428d8fe22a215405d1a7b49a411762cdddbce89"
Jul 06 22:53:14.834996 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:14.834948    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a5e9f084c4b2d810755dbe6d428d8fe22a215405d1a7b49a411762cdddbce89"} err="failed to get container status \"1a5e9f084c4b2d810755dbe6d428d8fe22a215405d1a7b49a411762cdddbce89\": rpc error: code = NotFound desc = could not find container \"1a5e9f084c4b2d810755dbe6d428d8fe22a215405d1a7b49a411762cdddbce89\": container with ID starting with 1a5e9f084c4b2d810755dbe6d428d8fe22a215405d1a7b49a411762cdddbce89 not found: ID does not exist"
Jul 06 22:53:14.841649 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:14.841613    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dmnld" podStartSLOduration=2.291880471 podStartE2EDuration="4.841600514s" totalImagesPullingTime="2.549720043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:53:14.841006289 +0000 UTC m=+407.245256194" watchObservedRunningTime="2026-07-06 22:53:14.841600514 +0000 UTC m=+407.245850469"
Jul 06 22:53:14.854678 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:14.854657    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-vcslh"]
Jul 06 22:53:14.859858 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:14.859841    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-vcslh"]
Jul 06 22:53:14.875694 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:14.875657    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-dhcx5" podStartSLOduration=2.353651832 podStartE2EDuration="4.875641915s" totalImagesPullingTime="2.521990083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:53:14.874401185 +0000 UTC m=+407.278651090" watchObservedRunningTime="2026-07-06 22:53:14.875641915 +0000 UTC m=+407.279891851"
Jul 06 22:53:16.174903 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:16.174865    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63f261c5-e338-4bd7-ac23-3cbfbae0a1ea" path="/var/lib/kubelet/pods/63f261c5-e338-4bd7-ac23-3cbfbae0a1ea/volumes"
Jul 06 22:53:45.836314 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:45.836281    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dmnld"
Jul 06 22:53:45.839482 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:45.839459    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-dhcx5"
Jul 06 22:53:58.702367 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:58.702289    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-v74kz"]
Jul 06 22:53:58.702722 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:58.702612    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63f261c5-e338-4bd7-ac23-3cbfbae0a1ea" containerName="discovery"
Jul 06 22:53:58.702722 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:58.702625    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="63f261c5-e338-4bd7-ac23-3cbfbae0a1ea" containerName="discovery"
Jul 06 22:53:58.702722 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:58.702688    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="63f261c5-e338-4bd7-ac23-3cbfbae0a1ea" containerName="discovery"
Jul 06 22:53:58.705435 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:58.705414    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-v74kz"
Jul 06 22:53:58.708282 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:58.708263    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 06 22:53:58.708432 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:58.708283    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 06 22:53:58.708583 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:58.708567    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-22nh6\""
Jul 06 22:53:58.716832 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:58.716809    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-v74kz"]
Jul 06 22:53:58.880079 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:58.880053    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz24m\" (UniqueName: \"kubernetes.io/projected/37ea3904-cf9a-44af-8a84-6b5b6383805a-kube-api-access-vz24m\") pod \"kuadrant-operator-catalog-v74kz\" (UID: \"37ea3904-cf9a-44af-8a84-6b5b6383805a\") " pod="kuadrant-system/kuadrant-operator-catalog-v74kz"
Jul 06 22:53:58.980603 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:58.980542    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz24m\" (UniqueName: \"kubernetes.io/projected/37ea3904-cf9a-44af-8a84-6b5b6383805a-kube-api-access-vz24m\") pod \"kuadrant-operator-catalog-v74kz\" (UID: \"37ea3904-cf9a-44af-8a84-6b5b6383805a\") " pod="kuadrant-system/kuadrant-operator-catalog-v74kz"
Jul 06 22:53:58.989043 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:58.989022    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz24m\" (UniqueName: \"kubernetes.io/projected/37ea3904-cf9a-44af-8a84-6b5b6383805a-kube-api-access-vz24m\") pod \"kuadrant-operator-catalog-v74kz\" (UID: \"37ea3904-cf9a-44af-8a84-6b5b6383805a\") " pod="kuadrant-system/kuadrant-operator-catalog-v74kz"
Jul 06 22:53:59.015090 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:59.015068    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-v74kz"
Jul 06 22:53:59.071540 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:59.071502    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-v74kz"]
Jul 06 22:53:59.157994 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:59.157967    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-v74kz"]
Jul 06 22:53:59.160755 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:53:59.160728    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37ea3904_cf9a_44af_8a84_6b5b6383805a.slice/crio-98667b63e31d81f38f3f02aa3d44de656b6b89312473c5e2e998f3b29be3dd49 WatchSource:0}: Error finding container 98667b63e31d81f38f3f02aa3d44de656b6b89312473c5e2e998f3b29be3dd49: Status 404 returned error can't find the container with id 98667b63e31d81f38f3f02aa3d44de656b6b89312473c5e2e998f3b29be3dd49
Jul 06 22:53:59.284402 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:59.284344    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jkmbd"]
Jul 06 22:53:59.288545 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:59.288527    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-jkmbd"
Jul 06 22:53:59.298389 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:59.298367    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jkmbd"]
Jul 06 22:53:59.384974 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:59.384952    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tx72\" (UniqueName: \"kubernetes.io/projected/3559ecd8-3e89-4132-812a-461d37c05b2e-kube-api-access-8tx72\") pod \"kuadrant-operator-catalog-jkmbd\" (UID: \"3559ecd8-3e89-4132-812a-461d37c05b2e\") " pod="kuadrant-system/kuadrant-operator-catalog-jkmbd"
Jul 06 22:53:59.485890 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:59.485869    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tx72\" (UniqueName: \"kubernetes.io/projected/3559ecd8-3e89-4132-812a-461d37c05b2e-kube-api-access-8tx72\") pod \"kuadrant-operator-catalog-jkmbd\" (UID: \"3559ecd8-3e89-4132-812a-461d37c05b2e\") " pod="kuadrant-system/kuadrant-operator-catalog-jkmbd"
Jul 06 22:53:59.494788 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:59.494766    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tx72\" (UniqueName: \"kubernetes.io/projected/3559ecd8-3e89-4132-812a-461d37c05b2e-kube-api-access-8tx72\") pod \"kuadrant-operator-catalog-jkmbd\" (UID: \"3559ecd8-3e89-4132-812a-461d37c05b2e\") " pod="kuadrant-system/kuadrant-operator-catalog-jkmbd"
Jul 06 22:53:59.597685 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:59.597624    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-jkmbd"
Jul 06 22:53:59.803961 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:53:59.803908    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jkmbd"]
Jul 06 22:53:59.953703 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:53:59.953668    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3559ecd8_3e89_4132_812a_461d37c05b2e.slice/crio-2a15554c7a89ce2ab9c64bfa98f8e3c831907474e6b56263bd756d5bed358832 WatchSource:0}: Error finding container 2a15554c7a89ce2ab9c64bfa98f8e3c831907474e6b56263bd756d5bed358832: Status 404 returned error can't find the container with id 2a15554c7a89ce2ab9c64bfa98f8e3c831907474e6b56263bd756d5bed358832
Jul 06 22:54:00.014637 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:00.014601    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-jkmbd" event={"ID":"3559ecd8-3e89-4132-812a-461d37c05b2e","Type":"ContainerStarted","Data":"2a15554c7a89ce2ab9c64bfa98f8e3c831907474e6b56263bd756d5bed358832"}
Jul 06 22:54:00.016016 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:00.015985    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-v74kz" event={"ID":"37ea3904-cf9a-44af-8a84-6b5b6383805a","Type":"ContainerStarted","Data":"98667b63e31d81f38f3f02aa3d44de656b6b89312473c5e2e998f3b29be3dd49"}
Jul 06 22:54:01.022829 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:01.022799    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-jkmbd" event={"ID":"3559ecd8-3e89-4132-812a-461d37c05b2e","Type":"ContainerStarted","Data":"2465225ce25e0c7d0b961c47e370be37886a6321294f8fc585830198447c282f"}
Jul 06 22:54:01.025351 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:01.025315    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-v74kz" event={"ID":"37ea3904-cf9a-44af-8a84-6b5b6383805a","Type":"ContainerStarted","Data":"4e21b4368e355ab250342ed533b34cfd61920f87d9637180ba29ae290ae2fb9d"}
Jul 06 22:54:01.025456 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:01.025405    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-v74kz" podUID="37ea3904-cf9a-44af-8a84-6b5b6383805a" containerName="registry-server" containerID="cri-o://4e21b4368e355ab250342ed533b34cfd61920f87d9637180ba29ae290ae2fb9d" gracePeriod=2
Jul 06 22:54:01.039670 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:01.039622    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-jkmbd" podStartSLOduration=1.048741555 podStartE2EDuration="2.03960497s" totalImagesPullingTime="990.863415ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:53:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:54:01.039586216 +0000 UTC m=+453.443836120" watchObservedRunningTime="2026-07-06 22:54:01.03960497 +0000 UTC m=+453.443854876"
Jul 06 22:54:01.058456 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:01.058416    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-v74kz" podStartSLOduration=1.275097825 podStartE2EDuration="3.058401997s" totalImagesPullingTime="1.783304172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:53:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:54:01.057313643 +0000 UTC m=+453.461563546" watchObservedRunningTime="2026-07-06 22:54:01.058401997 +0000 UTC m=+453.462651901"
Jul 06 22:54:01.255057 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:01.255037    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-v74kz_37ea3904-cf9a-44af-8a84-6b5b6383805a/registry-server/0.log"
Jul 06 22:54:01.257436 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:01.257420    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-v74kz"
Jul 06 22:54:01.401934 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:01.401909    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vz24m\" (UniqueName: \"kubernetes.io/projected/37ea3904-cf9a-44af-8a84-6b5b6383805a-kube-api-access-vz24m\") pod \"37ea3904-cf9a-44af-8a84-6b5b6383805a\" (UID: \"37ea3904-cf9a-44af-8a84-6b5b6383805a\") "
Jul 06 22:54:01.403789 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:01.403763    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37ea3904-cf9a-44af-8a84-6b5b6383805a-kube-api-access-vz24m" (OuterVolumeSpecName: "kube-api-access-vz24m") pod "37ea3904-cf9a-44af-8a84-6b5b6383805a" (UID: "37ea3904-cf9a-44af-8a84-6b5b6383805a"). InnerVolumeSpecName "kube-api-access-vz24m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 22:54:01.502682 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:01.502609    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz24m\" (UniqueName: \"kubernetes.io/projected/37ea3904-cf9a-44af-8a84-6b5b6383805a-kube-api-access-vz24m\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 22:54:02.030428 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:02.030400    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-v74kz_37ea3904-cf9a-44af-8a84-6b5b6383805a/registry-server/0.log"
Jul 06 22:54:02.030827 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:02.030443    2569 generic.go:356] "Generic (PLEG): container finished" podID="37ea3904-cf9a-44af-8a84-6b5b6383805a" containerID="4e21b4368e355ab250342ed533b34cfd61920f87d9637180ba29ae290ae2fb9d" exitCode=2
Jul 06 22:54:02.030827 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:02.030496    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-v74kz"
Jul 06 22:54:02.030827 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:02.030531    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-v74kz" event={"ID":"37ea3904-cf9a-44af-8a84-6b5b6383805a","Type":"ContainerDied","Data":"4e21b4368e355ab250342ed533b34cfd61920f87d9637180ba29ae290ae2fb9d"}
Jul 06 22:54:02.030827 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:02.030610    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-v74kz" event={"ID":"37ea3904-cf9a-44af-8a84-6b5b6383805a","Type":"ContainerDied","Data":"98667b63e31d81f38f3f02aa3d44de656b6b89312473c5e2e998f3b29be3dd49"}
Jul 06 22:54:02.030827 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:02.030633    2569 scope.go:117] "RemoveContainer" containerID="4e21b4368e355ab250342ed533b34cfd61920f87d9637180ba29ae290ae2fb9d"
Jul 06 22:54:02.039411 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:02.039390    2569 scope.go:117] "RemoveContainer" containerID="4e21b4368e355ab250342ed533b34cfd61920f87d9637180ba29ae290ae2fb9d"
Jul 06 22:54:02.039681 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:54:02.039664    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e21b4368e355ab250342ed533b34cfd61920f87d9637180ba29ae290ae2fb9d\": container with ID starting with 4e21b4368e355ab250342ed533b34cfd61920f87d9637180ba29ae290ae2fb9d not found: ID does not exist" containerID="4e21b4368e355ab250342ed533b34cfd61920f87d9637180ba29ae290ae2fb9d"
Jul 06 22:54:02.039736 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:02.039690    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e21b4368e355ab250342ed533b34cfd61920f87d9637180ba29ae290ae2fb9d"} err="failed to get container status \"4e21b4368e355ab250342ed533b34cfd61920f87d9637180ba29ae290ae2fb9d\": rpc error: code = NotFound desc = could not find container \"4e21b4368e355ab250342ed533b34cfd61920f87d9637180ba29ae290ae2fb9d\": container with ID starting with 4e21b4368e355ab250342ed533b34cfd61920f87d9637180ba29ae290ae2fb9d not found: ID does not exist"
Jul 06 22:54:02.051372 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:02.051344    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-v74kz"]
Jul 06 22:54:02.057084 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:02.057064    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-v74kz"]
Jul 06 22:54:02.169499 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:02.169479    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37ea3904-cf9a-44af-8a84-6b5b6383805a" path="/var/lib/kubelet/pods/37ea3904-cf9a-44af-8a84-6b5b6383805a/volumes"
Jul 06 22:54:09.598413 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:09.598373    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-jkmbd"
Jul 06 22:54:09.598845 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:09.598436    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-jkmbd"
Jul 06 22:54:09.620535 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:09.620513    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-jkmbd"
Jul 06 22:54:10.088366 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:10.088338    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-jkmbd"
Jul 06 22:54:31.117659 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:31.117627    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-bvl49"]
Jul 06 22:54:31.118042 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:31.117901    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37ea3904-cf9a-44af-8a84-6b5b6383805a" containerName="registry-server"
Jul 06 22:54:31.118042 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:31.117911    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="37ea3904-cf9a-44af-8a84-6b5b6383805a" containerName="registry-server"
Jul 06 22:54:31.118042 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:31.117964    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="37ea3904-cf9a-44af-8a84-6b5b6383805a" containerName="registry-server"
Jul 06 22:54:31.123718 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:31.123702    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bvl49"
Jul 06 22:54:31.126534 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:31.126511    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-kpvbw\""
Jul 06 22:54:31.137094 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:31.137070    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-bvl49"]
Jul 06 22:54:31.196362 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:31.196333    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27z9t\" (UniqueName: \"kubernetes.io/projected/eac3b1e8-3fdc-4930-84f4-bf118dbc353d-kube-api-access-27z9t\") pod \"authorino-operator-75cc8d5cb7-bvl49\" (UID: \"eac3b1e8-3fdc-4930-84f4-bf118dbc353d\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-bvl49"
Jul 06 22:54:31.296628 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:31.296602    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27z9t\" (UniqueName: \"kubernetes.io/projected/eac3b1e8-3fdc-4930-84f4-bf118dbc353d-kube-api-access-27z9t\") pod \"authorino-operator-75cc8d5cb7-bvl49\" (UID: \"eac3b1e8-3fdc-4930-84f4-bf118dbc353d\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-bvl49"
Jul 06 22:54:31.307945 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:31.307916    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27z9t\" (UniqueName: \"kubernetes.io/projected/eac3b1e8-3fdc-4930-84f4-bf118dbc353d-kube-api-access-27z9t\") pod \"authorino-operator-75cc8d5cb7-bvl49\" (UID: \"eac3b1e8-3fdc-4930-84f4-bf118dbc353d\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-bvl49"
Jul 06 22:54:31.433928 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:31.433907    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bvl49"
Jul 06 22:54:31.563669 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:31.563605    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-bvl49"]
Jul 06 22:54:31.565978 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:54:31.565950    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeac3b1e8_3fdc_4930_84f4_bf118dbc353d.slice/crio-8ca2644bac2513a0e0f949b74df04c613d975678f60010451c441fda659dc4ba WatchSource:0}: Error finding container 8ca2644bac2513a0e0f949b74df04c613d975678f60010451c441fda659dc4ba: Status 404 returned error can't find the container with id 8ca2644bac2513a0e0f949b74df04c613d975678f60010451c441fda659dc4ba
Jul 06 22:54:32.161431 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:32.161389    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bvl49" event={"ID":"eac3b1e8-3fdc-4930-84f4-bf118dbc353d","Type":"ContainerStarted","Data":"8ca2644bac2513a0e0f949b74df04c613d975678f60010451c441fda659dc4ba"}
Jul 06 22:54:34.171586 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:34.171551    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bvl49" event={"ID":"eac3b1e8-3fdc-4930-84f4-bf118dbc353d","Type":"ContainerStarted","Data":"1e11e06764bb3e9057f8e1e7f1e3036ca263b3692e067b235eda99d65b364ffa"}
Jul 06 22:54:34.171999 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:34.171626    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bvl49"
Jul 06 22:54:34.202609 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:34.202578    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-5frkz"]
Jul 06 22:54:34.205598 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:34.205584    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5frkz"
Jul 06 22:54:34.208769 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:34.208751    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-trw6f\""
Jul 06 22:54:34.209028 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:34.209013    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 06 22:54:34.217982 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:34.217965    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8ssb\" (UniqueName: \"kubernetes.io/projected/5313328e-7735-45c7-ad8f-e72407ff30df-kube-api-access-m8ssb\") pod \"dns-operator-controller-manager-864dd78c4c-5frkz\" (UID: \"5313328e-7735-45c7-ad8f-e72407ff30df\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5frkz"
Jul 06 22:54:34.223230 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:34.223172    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bvl49" podStartSLOduration=1.139171341 podStartE2EDuration="3.223159276s" totalImagesPullingTime="2.083987935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:54:34.222766827 +0000 UTC m=+486.627016733" watchObservedRunningTime="2026-07-06 22:54:34.223159276 +0000 UTC m=+486.627409181"
Jul 06 22:54:34.241709 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:34.241685    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-5frkz"]
Jul 06 22:54:34.319292 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:34.319270    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8ssb\" (UniqueName: \"kubernetes.io/projected/5313328e-7735-45c7-ad8f-e72407ff30df-kube-api-access-m8ssb\") pod \"dns-operator-controller-manager-864dd78c4c-5frkz\" (UID: \"5313328e-7735-45c7-ad8f-e72407ff30df\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5frkz"
Jul 06 22:54:34.332931 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:34.332906    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8ssb\" (UniqueName: \"kubernetes.io/projected/5313328e-7735-45c7-ad8f-e72407ff30df-kube-api-access-m8ssb\") pod \"dns-operator-controller-manager-864dd78c4c-5frkz\" (UID: \"5313328e-7735-45c7-ad8f-e72407ff30df\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5frkz"
Jul 06 22:54:34.515881 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:34.515855    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5frkz"
Jul 06 22:54:34.644353 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:34.644330    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-5frkz"]
Jul 06 22:54:34.646687 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:54:34.646662    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5313328e_7735_45c7_ad8f_e72407ff30df.slice/crio-d13eca3db82900a1c2361e26f663874272897ebb18e5aaebd4b813c50d416491 WatchSource:0}: Error finding container d13eca3db82900a1c2361e26f663874272897ebb18e5aaebd4b813c50d416491: Status 404 returned error can't find the container with id d13eca3db82900a1c2361e26f663874272897ebb18e5aaebd4b813c50d416491
Jul 06 22:54:35.177337 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:35.177301    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5frkz" event={"ID":"5313328e-7735-45c7-ad8f-e72407ff30df","Type":"ContainerStarted","Data":"d13eca3db82900a1c2361e26f663874272897ebb18e5aaebd4b813c50d416491"}
Jul 06 22:54:36.814039 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:36.813958    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kt6ct"]
Jul 06 22:54:36.816315 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:36.816293    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kt6ct"
Jul 06 22:54:36.819229 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:36.819203    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-m4wfp\""
Jul 06 22:54:36.829004 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:36.828982    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kt6ct"]
Jul 06 22:54:36.838393 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:36.838373    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5k7g\" (UniqueName: \"kubernetes.io/projected/21082097-f3b1-44c0-b98b-3df031e99412-kube-api-access-p5k7g\") pod \"limitador-operator-controller-manager-859d7b78bd-kt6ct\" (UID: \"21082097-f3b1-44c0-b98b-3df031e99412\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kt6ct"
Jul 06 22:54:36.939365 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:36.939338    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5k7g\" (UniqueName: \"kubernetes.io/projected/21082097-f3b1-44c0-b98b-3df031e99412-kube-api-access-p5k7g\") pod \"limitador-operator-controller-manager-859d7b78bd-kt6ct\" (UID: \"21082097-f3b1-44c0-b98b-3df031e99412\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kt6ct"
Jul 06 22:54:36.950030 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:36.950003    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5k7g\" (UniqueName: \"kubernetes.io/projected/21082097-f3b1-44c0-b98b-3df031e99412-kube-api-access-p5k7g\") pod \"limitador-operator-controller-manager-859d7b78bd-kt6ct\" (UID: \"21082097-f3b1-44c0-b98b-3df031e99412\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kt6ct"
Jul 06 22:54:37.126426 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:37.126367    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kt6ct"
Jul 06 22:54:37.187962 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:37.187928    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5frkz" event={"ID":"5313328e-7735-45c7-ad8f-e72407ff30df","Type":"ContainerStarted","Data":"66784c03d74ea429f27c8f904c6a1ad86c1275198b6c878a22454d154d86695b"}
Jul 06 22:54:37.188081 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:37.188014    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5frkz"
Jul 06 22:54:37.216622 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:37.215508    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5frkz" podStartSLOduration=1.399162877 podStartE2EDuration="3.215484856s" totalImagesPullingTime="1.816321979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:54:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:54:37.214190397 +0000 UTC m=+489.618440300" watchObservedRunningTime="2026-07-06 22:54:37.215484856 +0000 UTC m=+489.619734762"
Jul 06 22:54:37.276625 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:37.276604    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kt6ct"]
Jul 06 22:54:37.278807 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:54:37.278778    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21082097_f3b1_44c0_b98b_3df031e99412.slice/crio-1072c7fe157b34c51c0dabc732190aac3c758a8195681fdbee284179e7e1a463 WatchSource:0}: Error finding container 1072c7fe157b34c51c0dabc732190aac3c758a8195681fdbee284179e7e1a463: Status 404 returned error can't find the container with id 1072c7fe157b34c51c0dabc732190aac3c758a8195681fdbee284179e7e1a463
Jul 06 22:54:38.193555 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:38.193518    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kt6ct" event={"ID":"21082097-f3b1-44c0-b98b-3df031e99412","Type":"ContainerStarted","Data":"1072c7fe157b34c51c0dabc732190aac3c758a8195681fdbee284179e7e1a463"}
Jul 06 22:54:39.199371 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:39.199331    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kt6ct" event={"ID":"21082097-f3b1-44c0-b98b-3df031e99412","Type":"ContainerStarted","Data":"ce68d228881af05bbea1cde43ef02f3755d5f835ffc96848de71910a98bb0ba7"}
Jul 06 22:54:39.199837 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:39.199424    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kt6ct"
Jul 06 22:54:41.120197 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:41.120132    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kt6ct" podStartSLOduration=3.547900346 podStartE2EDuration="5.120114483s" totalImagesPullingTime="1.572214137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:54:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:54:39.216600276 +0000 UTC m=+491.620850191" watchObservedRunningTime="2026-07-06 22:54:41.120114483 +0000 UTC m=+493.524364365"
Jul 06 22:54:41.121247 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:41.121224    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tmcgg"]
Jul 06 22:54:41.124337 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:41.124319    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tmcgg"
Jul 06 22:54:41.127082 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:41.127063    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-nsvrd\""
Jul 06 22:54:41.143340 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:41.143313    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tmcgg"]
Jul 06 22:54:41.173466 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:41.173444    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtvst\" (UniqueName: \"kubernetes.io/projected/b8b4c880-eb04-43eb-a16a-7c086e94fddd-kube-api-access-mtvst\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tmcgg\" (UID: \"b8b4c880-eb04-43eb-a16a-7c086e94fddd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tmcgg"
Jul 06 22:54:41.173584 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:41.173484    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8b4c880-eb04-43eb-a16a-7c086e94fddd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tmcgg\" (UID: \"b8b4c880-eb04-43eb-a16a-7c086e94fddd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tmcgg"
Jul 06 22:54:41.273965 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:41.273939    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtvst\" (UniqueName: \"kubernetes.io/projected/b8b4c880-eb04-43eb-a16a-7c086e94fddd-kube-api-access-mtvst\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tmcgg\" (UID: \"b8b4c880-eb04-43eb-a16a-7c086e94fddd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tmcgg"
Jul 06 22:54:41.274078 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:41.273977    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8b4c880-eb04-43eb-a16a-7c086e94fddd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tmcgg\" (UID: \"b8b4c880-eb04-43eb-a16a-7c086e94fddd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tmcgg"
Jul 06 22:54:41.274365 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:41.274346    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8b4c880-eb04-43eb-a16a-7c086e94fddd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tmcgg\" (UID: \"b8b4c880-eb04-43eb-a16a-7c086e94fddd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tmcgg"
Jul 06 22:54:41.282316 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:41.282297    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtvst\" (UniqueName: \"kubernetes.io/projected/b8b4c880-eb04-43eb-a16a-7c086e94fddd-kube-api-access-mtvst\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tmcgg\" (UID: \"b8b4c880-eb04-43eb-a16a-7c086e94fddd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tmcgg"
Jul 06 22:54:41.433775 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:41.433750    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tmcgg"
Jul 06 22:54:41.563970 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:41.563935    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tmcgg"]
Jul 06 22:54:41.567216 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:54:41.567171    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8b4c880_eb04_43eb_a16a_7c086e94fddd.slice/crio-325bf8eb181a4d3e5ab1b354f3be5044271432b7a81b5db4b2044237565af351 WatchSource:0}: Error finding container 325bf8eb181a4d3e5ab1b354f3be5044271432b7a81b5db4b2044237565af351: Status 404 returned error can't find the container with id 325bf8eb181a4d3e5ab1b354f3be5044271432b7a81b5db4b2044237565af351
Jul 06 22:54:42.213102 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:42.213067    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tmcgg" event={"ID":"b8b4c880-eb04-43eb-a16a-7c086e94fddd","Type":"ContainerStarted","Data":"325bf8eb181a4d3e5ab1b354f3be5044271432b7a81b5db4b2044237565af351"}
Jul 06 22:54:45.180439 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:45.180400    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bvl49"
Jul 06 22:54:46.234622 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:46.234586    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tmcgg" event={"ID":"b8b4c880-eb04-43eb-a16a-7c086e94fddd","Type":"ContainerStarted","Data":"1d06c1dc71b9712972626c5d9f2df6c222bcc0c330f325cddaad3db5a49275f9"}
Jul 06 22:54:46.234985 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:46.234739    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tmcgg"
Jul 06 22:54:46.256658 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:46.256613    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tmcgg" podStartSLOduration=1.19079969 podStartE2EDuration="5.256599581s" totalImagesPullingTime="4.065799891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:54:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:54:46.254086572 +0000 UTC m=+498.658336489" watchObservedRunningTime="2026-07-06 22:54:46.256599581 +0000 UTC m=+498.660849512"
Jul 06 22:54:48.196267 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:48.196237    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5frkz"
Jul 06 22:54:48.216413 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:48.216384    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-k5xcs"]
Jul 06 22:54:48.219549 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:48.219533    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-k5xcs"
Jul 06 22:54:48.222461 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:48.222442    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 06 22:54:48.222556 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:48.222445    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 06 22:54:48.222556 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:48.222469    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wn2q7\""
Jul 06 22:54:48.231057 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:48.231036    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-k5xcs"]
Jul 06 22:54:48.327829 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:48.327801    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fe3069f0-f612-4413-bc74-2cfd710ccf6b-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-k5xcs\" (UID: \"fe3069f0-f612-4413-bc74-2cfd710ccf6b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-k5xcs"
Jul 06 22:54:48.327969 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:48.327844    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe3069f0-f612-4413-bc74-2cfd710ccf6b-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-k5xcs\" (UID: \"fe3069f0-f612-4413-bc74-2cfd710ccf6b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-k5xcs"
Jul 06 22:54:48.327969 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:48.327869    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjn7c\" (UniqueName: \"kubernetes.io/projected/fe3069f0-f612-4413-bc74-2cfd710ccf6b-kube-api-access-pjn7c\") pod \"kuadrant-console-plugin-66b5774985-k5xcs\" (UID: \"fe3069f0-f612-4413-bc74-2cfd710ccf6b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-k5xcs"
Jul 06 22:54:48.428493 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:48.428467    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe3069f0-f612-4413-bc74-2cfd710ccf6b-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-k5xcs\" (UID: \"fe3069f0-f612-4413-bc74-2cfd710ccf6b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-k5xcs"
Jul 06 22:54:48.428619 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:48.428499    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjn7c\" (UniqueName: \"kubernetes.io/projected/fe3069f0-f612-4413-bc74-2cfd710ccf6b-kube-api-access-pjn7c\") pod \"kuadrant-console-plugin-66b5774985-k5xcs\" (UID: \"fe3069f0-f612-4413-bc74-2cfd710ccf6b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-k5xcs"
Jul 06 22:54:48.428619 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:48.428543    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fe3069f0-f612-4413-bc74-2cfd710ccf6b-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-k5xcs\" (UID: \"fe3069f0-f612-4413-bc74-2cfd710ccf6b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-k5xcs"
Jul 06 22:54:48.429067 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:48.429050    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fe3069f0-f612-4413-bc74-2cfd710ccf6b-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-k5xcs\" (UID: \"fe3069f0-f612-4413-bc74-2cfd710ccf6b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-k5xcs"
Jul 06 22:54:48.430905 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:48.430889    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe3069f0-f612-4413-bc74-2cfd710ccf6b-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-k5xcs\" (UID: \"fe3069f0-f612-4413-bc74-2cfd710ccf6b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-k5xcs"
Jul 06 22:54:48.441915 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:48.441892    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjn7c\" (UniqueName: \"kubernetes.io/projected/fe3069f0-f612-4413-bc74-2cfd710ccf6b-kube-api-access-pjn7c\") pod \"kuadrant-console-plugin-66b5774985-k5xcs\" (UID: \"fe3069f0-f612-4413-bc74-2cfd710ccf6b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-k5xcs"
Jul 06 22:54:48.528863 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:48.528796    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-k5xcs"
Jul 06 22:54:48.668074 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:48.668054    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-k5xcs"]
Jul 06 22:54:48.670080 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:54:48.670048    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe3069f0_f612_4413_bc74_2cfd710ccf6b.slice/crio-6e2ee1b21ab301a58839a36fc848959e21a6cb0fd19ce01d920c4ccecce69144 WatchSource:0}: Error finding container 6e2ee1b21ab301a58839a36fc848959e21a6cb0fd19ce01d920c4ccecce69144: Status 404 returned error can't find the container with id 6e2ee1b21ab301a58839a36fc848959e21a6cb0fd19ce01d920c4ccecce69144
Jul 06 22:54:49.251094 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:49.251055    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-k5xcs" event={"ID":"fe3069f0-f612-4413-bc74-2cfd710ccf6b","Type":"ContainerStarted","Data":"6e2ee1b21ab301a58839a36fc848959e21a6cb0fd19ce01d920c4ccecce69144"}
Jul 06 22:54:50.206054 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:50.205967    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kt6ct"
Jul 06 22:54:57.241682 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:57.241646    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tmcgg"
Jul 06 22:54:59.158549 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:59.158517    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tmcgg"]
Jul 06 22:54:59.159058 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:59.158754    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tmcgg" podUID="b8b4c880-eb04-43eb-a16a-7c086e94fddd" containerName="manager" containerID="cri-o://1d06c1dc71b9712972626c5d9f2df6c222bcc0c330f325cddaad3db5a49275f9" gracePeriod=2
Jul 06 22:54:59.174467 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:59.174427    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tmcgg"]
Jul 06 22:54:59.188998 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:54:59.188919    2569 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"b8b4c880-eb04-43eb-a16a-7c086e94fddd\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-tmcgg\": pods \"kuadrant-operator-controller-manager-8684c9ff49-tmcgg\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tmcgg"
Jul 06 22:54:59.193092 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:59.193059    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kt6ct"]
Jul 06 22:54:59.196401 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:59.195837    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kt6ct" podUID="21082097-f3b1-44c0-b98b-3df031e99412" containerName="manager" containerID="cri-o://ce68d228881af05bbea1cde43ef02f3755d5f835ffc96848de71910a98bb0ba7" gracePeriod=2
Jul 06 22:54:59.197943 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:59.197923    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hnq49"]
Jul 06 22:54:59.198442 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:59.198418    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8b4c880-eb04-43eb-a16a-7c086e94fddd" containerName="manager"
Jul 06 22:54:59.198442 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:59.198443    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8b4c880-eb04-43eb-a16a-7c086e94fddd" containerName="manager"
Jul 06 22:54:59.198593 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:59.198548    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8b4c880-eb04-43eb-a16a-7c086e94fddd" containerName="manager"
Jul 06 22:54:59.199061 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:54:59.198918    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-tmcgg\" is forbidden: User \"system:node:ip-10-0-134-116.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-116.ec2.internal' and this object" podUID="b8b4c880-eb04-43eb-a16a-7c086e94fddd" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tmcgg"
Jul 06 22:54:59.232224 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:59.231809    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kt6ct"]
Jul 06 22:54:59.232224 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:59.231858    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hnq49"]
Jul 06 22:54:59.232224 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:59.231891    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lf6v7"]
Jul 06 22:54:59.233302 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:59.232425    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hnq49"
Jul 06 22:54:59.233756 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:59.233731    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21082097-f3b1-44c0-b98b-3df031e99412" containerName="manager"
Jul 06 22:54:59.233756 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:59.233757    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="21082097-f3b1-44c0-b98b-3df031e99412" containerName="manager"
Jul 06 22:54:59.233935 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:59.233854    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="21082097-f3b1-44c0-b98b-3df031e99412" containerName="manager"
Jul 06 22:54:59.236344 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:54:59.236031    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-tmcgg\" is forbidden: User \"system:node:ip-10-0-134-116.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-116.ec2.internal' and this object" podUID="b8b4c880-eb04-43eb-a16a-7c086e94fddd" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tmcgg"
Jul 06 22:54:59.238789 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:59.238747    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lf6v7"
Jul 06 22:54:59.238948 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:59.238904    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lf6v7"]
Jul 06 22:54:59.267729 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:54:59.267694    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kt6ct\" is forbidden: User \"system:node:ip-10-0-134-116.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-116.ec2.internal' and this object" podUID="21082097-f3b1-44c0-b98b-3df031e99412" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kt6ct"
Jul 06 22:54:59.290421 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:54:59.290388    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-tmcgg\" is forbidden: User \"system:node:ip-10-0-134-116.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-116.ec2.internal' and this object" podUID="b8b4c880-eb04-43eb-a16a-7c086e94fddd" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tmcgg"
Jul 06 22:54:59.292702 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:54:59.292673    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kt6ct\" is forbidden: User \"system:node:ip-10-0-134-116.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-116.ec2.internal' and this object" podUID="21082097-f3b1-44c0-b98b-3df031e99412" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kt6ct"
Jul 06 22:54:59.322979 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:59.322947    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxz8t\" (UniqueName: \"kubernetes.io/projected/eef1a6cf-89f8-4e69-a5b6-a54fdda4c483-kube-api-access-lxz8t\") pod \"limitador-operator-controller-manager-859d7b78bd-lf6v7\" (UID: \"eef1a6cf-89f8-4e69-a5b6-a54fdda4c483\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lf6v7"
Jul 06 22:54:59.323113 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:59.322996    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrv5w\" (UniqueName: \"kubernetes.io/projected/f689fe0e-4605-4b38-ab39-166ea6bdb9a7-kube-api-access-xrv5w\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hnq49\" (UID: \"f689fe0e-4605-4b38-ab39-166ea6bdb9a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hnq49"
Jul 06 22:54:59.323113 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:59.323048    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f689fe0e-4605-4b38-ab39-166ea6bdb9a7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hnq49\" (UID: \"f689fe0e-4605-4b38-ab39-166ea6bdb9a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hnq49"
Jul 06 22:54:59.424062 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:59.423996    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxz8t\" (UniqueName: \"kubernetes.io/projected/eef1a6cf-89f8-4e69-a5b6-a54fdda4c483-kube-api-access-lxz8t\") pod \"limitador-operator-controller-manager-859d7b78bd-lf6v7\" (UID: \"eef1a6cf-89f8-4e69-a5b6-a54fdda4c483\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lf6v7"
Jul 06 22:54:59.424062 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:59.424035    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrv5w\" (UniqueName: \"kubernetes.io/projected/f689fe0e-4605-4b38-ab39-166ea6bdb9a7-kube-api-access-xrv5w\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hnq49\" (UID: \"f689fe0e-4605-4b38-ab39-166ea6bdb9a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hnq49"
Jul 06 22:54:59.424062 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:59.424057    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f689fe0e-4605-4b38-ab39-166ea6bdb9a7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hnq49\" (UID: \"f689fe0e-4605-4b38-ab39-166ea6bdb9a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hnq49"
Jul 06 22:54:59.424472 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:59.424454    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f689fe0e-4605-4b38-ab39-166ea6bdb9a7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hnq49\" (UID: \"f689fe0e-4605-4b38-ab39-166ea6bdb9a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hnq49"
Jul 06 22:54:59.438008 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:59.437986    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrv5w\" (UniqueName: \"kubernetes.io/projected/f689fe0e-4605-4b38-ab39-166ea6bdb9a7-kube-api-access-xrv5w\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hnq49\" (UID: \"f689fe0e-4605-4b38-ab39-166ea6bdb9a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hnq49"
Jul 06 22:54:59.438140 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:59.438097    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxz8t\" (UniqueName: \"kubernetes.io/projected/eef1a6cf-89f8-4e69-a5b6-a54fdda4c483-kube-api-access-lxz8t\") pod \"limitador-operator-controller-manager-859d7b78bd-lf6v7\" (UID: \"eef1a6cf-89f8-4e69-a5b6-a54fdda4c483\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lf6v7"
Jul 06 22:54:59.641026 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:59.640995    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hnq49"
Jul 06 22:54:59.648540 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:54:59.648510    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lf6v7"
Jul 06 22:55:09.071326 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:09.071255    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hnq49"]
Jul 06 22:55:09.071728 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:09.071681    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tmcgg"
Jul 06 22:55:09.074046 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:55:09.074017    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-tmcgg\" is forbidden: User \"system:node:ip-10-0-134-116.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-116.ec2.internal' and this object" podUID="b8b4c880-eb04-43eb-a16a-7c086e94fddd" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tmcgg"
Jul 06 22:55:09.075062 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:09.075043    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kt6ct"
Jul 06 22:55:09.075350 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:55:09.075315    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf689fe0e_4605_4b38_ab39_166ea6bdb9a7.slice/crio-aa8401221df7e7051ba3e3cff3f26467a629ed35bcc494a18d82b07a8ee0f6b4 WatchSource:0}: Error finding container aa8401221df7e7051ba3e3cff3f26467a629ed35bcc494a18d82b07a8ee0f6b4: Status 404 returned error can't find the container with id aa8401221df7e7051ba3e3cff3f26467a629ed35bcc494a18d82b07a8ee0f6b4
Jul 06 22:55:09.077211 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:55:09.077169    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-tmcgg\" is forbidden: User \"system:node:ip-10-0-134-116.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-116.ec2.internal' and this object" podUID="b8b4c880-eb04-43eb-a16a-7c086e94fddd" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tmcgg"
Jul 06 22:55:09.079240 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:55:09.079215    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kt6ct\" is forbidden: User \"system:node:ip-10-0-134-116.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-116.ec2.internal' and this object" podUID="21082097-f3b1-44c0-b98b-3df031e99412" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kt6ct"
Jul 06 22:55:09.097443 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:09.097381    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lf6v7"]
Jul 06 22:55:09.110289 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:55:09.110264    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeef1a6cf_89f8_4e69_a5b6_a54fdda4c483.slice/crio-823eabd9371b1a42c311a6523ffb29fe9876d9eaaeebd4280ee835ca1e0fa0b0 WatchSource:0}: Error finding container 823eabd9371b1a42c311a6523ffb29fe9876d9eaaeebd4280ee835ca1e0fa0b0: Status 404 returned error can't find the container with id 823eabd9371b1a42c311a6523ffb29fe9876d9eaaeebd4280ee835ca1e0fa0b0
Jul 06 22:55:09.205788 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:09.205761    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8b4c880-eb04-43eb-a16a-7c086e94fddd-extensions-socket-volume\") pod \"b8b4c880-eb04-43eb-a16a-7c086e94fddd\" (UID: \"b8b4c880-eb04-43eb-a16a-7c086e94fddd\") "
Jul 06 22:55:09.205962 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:09.205899    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtvst\" (UniqueName: \"kubernetes.io/projected/b8b4c880-eb04-43eb-a16a-7c086e94fddd-kube-api-access-mtvst\") pod \"b8b4c880-eb04-43eb-a16a-7c086e94fddd\" (UID: \"b8b4c880-eb04-43eb-a16a-7c086e94fddd\") "
Jul 06 22:55:09.205962 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:09.205922    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5k7g\" (UniqueName: \"kubernetes.io/projected/21082097-f3b1-44c0-b98b-3df031e99412-kube-api-access-p5k7g\") pod \"21082097-f3b1-44c0-b98b-3df031e99412\" (UID: \"21082097-f3b1-44c0-b98b-3df031e99412\") "
Jul 06 22:55:09.206378 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:09.206342    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b4c880-eb04-43eb-a16a-7c086e94fddd-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b8b4c880-eb04-43eb-a16a-7c086e94fddd" (UID: "b8b4c880-eb04-43eb-a16a-7c086e94fddd"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:55:09.208014 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:09.207990    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21082097-f3b1-44c0-b98b-3df031e99412-kube-api-access-p5k7g" (OuterVolumeSpecName: "kube-api-access-p5k7g") pod "21082097-f3b1-44c0-b98b-3df031e99412" (UID: "21082097-f3b1-44c0-b98b-3df031e99412"). InnerVolumeSpecName "kube-api-access-p5k7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 22:55:09.208099 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:09.208031    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b4c880-eb04-43eb-a16a-7c086e94fddd-kube-api-access-mtvst" (OuterVolumeSpecName: "kube-api-access-mtvst") pod "b8b4c880-eb04-43eb-a16a-7c086e94fddd" (UID: "b8b4c880-eb04-43eb-a16a-7c086e94fddd"). InnerVolumeSpecName "kube-api-access-mtvst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 22:55:09.306555 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:09.306518    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtvst\" (UniqueName: \"kubernetes.io/projected/b8b4c880-eb04-43eb-a16a-7c086e94fddd-kube-api-access-mtvst\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 22:55:09.306555 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:09.306553    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5k7g\" (UniqueName: \"kubernetes.io/projected/21082097-f3b1-44c0-b98b-3df031e99412-kube-api-access-p5k7g\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 22:55:09.306722 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:09.306568    2569 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8b4c880-eb04-43eb-a16a-7c086e94fddd-extensions-socket-volume\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 22:55:09.354746 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:09.354717    2569 generic.go:356] "Generic (PLEG): container finished" podID="21082097-f3b1-44c0-b98b-3df031e99412" containerID="ce68d228881af05bbea1cde43ef02f3755d5f835ffc96848de71910a98bb0ba7" exitCode=0
Jul 06 22:55:09.354855 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:09.354765    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kt6ct"
Jul 06 22:55:09.354855 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:09.354798    2569 scope.go:117] "RemoveContainer" containerID="ce68d228881af05bbea1cde43ef02f3755d5f835ffc96848de71910a98bb0ba7"
Jul 06 22:55:09.356569 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:09.356544    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lf6v7" event={"ID":"eef1a6cf-89f8-4e69-a5b6-a54fdda4c483","Type":"ContainerStarted","Data":"7692df5099042c08908257a529a6765ede2e0cceae2e809dba2dc5903605aa81"}
Jul 06 22:55:09.356795 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:09.356588    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lf6v7" event={"ID":"eef1a6cf-89f8-4e69-a5b6-a54fdda4c483","Type":"ContainerStarted","Data":"823eabd9371b1a42c311a6523ffb29fe9876d9eaaeebd4280ee835ca1e0fa0b0"}
Jul 06 22:55:09.356795 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:09.356657    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lf6v7"
Jul 06 22:55:09.357301 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:55:09.357270    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kt6ct\" is forbidden: User \"system:node:ip-10-0-134-116.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-116.ec2.internal' and this object" podUID="21082097-f3b1-44c0-b98b-3df031e99412" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kt6ct"
Jul 06 22:55:09.358390 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:09.358368    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-k5xcs" event={"ID":"fe3069f0-f612-4413-bc74-2cfd710ccf6b","Type":"ContainerStarted","Data":"f29172dd6e29652ce00e873f04e89d124edaa1eb0d36ebe6ed125aff3068b136"}
Jul 06 22:55:09.359432 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:55:09.359403    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-tmcgg\" is forbidden: User \"system:node:ip-10-0-134-116.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-116.ec2.internal' and this object" podUID="b8b4c880-eb04-43eb-a16a-7c086e94fddd" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tmcgg"
Jul 06 22:55:09.360142 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:09.360121    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hnq49" event={"ID":"f689fe0e-4605-4b38-ab39-166ea6bdb9a7","Type":"ContainerStarted","Data":"06c01aecbc609c194412a7d36f60fb951164014c35214dfff9ceceed3b2b098a"}
Jul 06 22:55:09.360255 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:09.360165    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hnq49" event={"ID":"f689fe0e-4605-4b38-ab39-166ea6bdb9a7","Type":"ContainerStarted","Data":"aa8401221df7e7051ba3e3cff3f26467a629ed35bcc494a18d82b07a8ee0f6b4"}
Jul 06 22:55:09.360255 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:09.360212    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hnq49"
Jul 06 22:55:09.361470 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:09.361448    2569 generic.go:356] "Generic (PLEG): container finished" podID="b8b4c880-eb04-43eb-a16a-7c086e94fddd" containerID="1d06c1dc71b9712972626c5d9f2df6c222bcc0c330f325cddaad3db5a49275f9" exitCode=0
Jul 06 22:55:09.361552 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:09.361479    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tmcgg"
Jul 06 22:55:09.364958 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:09.364940    2569 scope.go:117] "RemoveContainer" containerID="ce68d228881af05bbea1cde43ef02f3755d5f835ffc96848de71910a98bb0ba7"
Jul 06 22:55:09.365248 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:55:09.365227    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce68d228881af05bbea1cde43ef02f3755d5f835ffc96848de71910a98bb0ba7\": container with ID starting with ce68d228881af05bbea1cde43ef02f3755d5f835ffc96848de71910a98bb0ba7 not found: ID does not exist" containerID="ce68d228881af05bbea1cde43ef02f3755d5f835ffc96848de71910a98bb0ba7"
Jul 06 22:55:09.365309 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:09.365257    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce68d228881af05bbea1cde43ef02f3755d5f835ffc96848de71910a98bb0ba7"} err="failed to get container status \"ce68d228881af05bbea1cde43ef02f3755d5f835ffc96848de71910a98bb0ba7\": rpc error: code = NotFound desc = could not find container \"ce68d228881af05bbea1cde43ef02f3755d5f835ffc96848de71910a98bb0ba7\": container with ID starting with ce68d228881af05bbea1cde43ef02f3755d5f835ffc96848de71910a98bb0ba7 not found: ID does not exist"
Jul 06 22:55:09.365309 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:09.365273    2569 scope.go:117] "RemoveContainer" containerID="1d06c1dc71b9712972626c5d9f2df6c222bcc0c330f325cddaad3db5a49275f9"
Jul 06 22:55:09.373881 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:09.373839    2569 scope.go:117] "RemoveContainer" containerID="1d06c1dc71b9712972626c5d9f2df6c222bcc0c330f325cddaad3db5a49275f9"
Jul 06 22:55:09.374239 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:55:09.374216    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d06c1dc71b9712972626c5d9f2df6c222bcc0c330f325cddaad3db5a49275f9\": container with ID starting with 1d06c1dc71b9712972626c5d9f2df6c222bcc0c330f325cddaad3db5a49275f9 not found: ID does not exist" containerID="1d06c1dc71b9712972626c5d9f2df6c222bcc0c330f325cddaad3db5a49275f9"
Jul 06 22:55:09.374344 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:09.374245    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d06c1dc71b9712972626c5d9f2df6c222bcc0c330f325cddaad3db5a49275f9"} err="failed to get container status \"1d06c1dc71b9712972626c5d9f2df6c222bcc0c330f325cddaad3db5a49275f9\": rpc error: code = NotFound desc = could not find container \"1d06c1dc71b9712972626c5d9f2df6c222bcc0c330f325cddaad3db5a49275f9\": container with ID starting with 1d06c1dc71b9712972626c5d9f2df6c222bcc0c330f325cddaad3db5a49275f9 not found: ID does not exist"
Jul 06 22:55:09.378612 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:09.378565    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lf6v7" podStartSLOduration=10.378549436 podStartE2EDuration="10.378549436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:55:09.377996397 +0000 UTC m=+521.782246301" watchObservedRunningTime="2026-07-06 22:55:09.378549436 +0000 UTC m=+521.782799342"
Jul 06 22:55:09.397007 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:55:09.396979    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-tmcgg\" is forbidden: User \"system:node:ip-10-0-134-116.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-116.ec2.internal' and this object" podUID="b8b4c880-eb04-43eb-a16a-7c086e94fddd" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tmcgg"
Jul 06 22:55:09.397107 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:09.396975    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-k5xcs" podStartSLOduration=1.080562356 podStartE2EDuration="21.396962392s" totalImagesPullingTime="20.316400036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:54:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:55:09.394792789 +0000 UTC m=+521.799042693" watchObservedRunningTime="2026-07-06 22:55:09.396962392 +0000 UTC m=+521.801212295"
Jul 06 22:55:09.399079 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:55:09.399056    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kt6ct\" is forbidden: User \"system:node:ip-10-0-134-116.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-116.ec2.internal' and this object" podUID="21082097-f3b1-44c0-b98b-3df031e99412" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kt6ct"
Jul 06 22:55:09.401189 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:55:09.401161    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-tmcgg\" is forbidden: User \"system:node:ip-10-0-134-116.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-116.ec2.internal' and this object" podUID="b8b4c880-eb04-43eb-a16a-7c086e94fddd" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tmcgg"
Jul 06 22:55:09.422117 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:09.422081    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hnq49" podStartSLOduration=10.422070395 podStartE2EDuration="10.422070395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:55:09.421171497 +0000 UTC m=+521.825421401" watchObservedRunningTime="2026-07-06 22:55:09.422070395 +0000 UTC m=+521.826320299"
Jul 06 22:55:09.423315 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:55:09.423289    2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kt6ct\" is forbidden: User \"system:node:ip-10-0-134-116.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-116.ec2.internal' and this object" podUID="21082097-f3b1-44c0-b98b-3df031e99412" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kt6ct"
Jul 06 22:55:10.170072 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:10.170002    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21082097-f3b1-44c0-b98b-3df031e99412" path="/var/lib/kubelet/pods/21082097-f3b1-44c0-b98b-3df031e99412/volumes"
Jul 06 22:55:10.170487 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:10.170343    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8b4c880-eb04-43eb-a16a-7c086e94fddd" path="/var/lib/kubelet/pods/b8b4c880-eb04-43eb-a16a-7c086e94fddd/volumes"
Jul 06 22:55:15.472150 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:15.472106    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hnq49"]
Jul 06 22:55:15.472588 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:15.472342    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hnq49" podUID="f689fe0e-4605-4b38-ab39-166ea6bdb9a7" containerName="manager" containerID="cri-o://06c01aecbc609c194412a7d36f60fb951164014c35214dfff9ceceed3b2b098a" gracePeriod=10
Jul 06 22:55:15.474166 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:15.474140    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hnq49"
Jul 06 22:55:15.714935 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:15.714906    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hnq49"
Jul 06 22:55:15.854473 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:15.854396    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f689fe0e-4605-4b38-ab39-166ea6bdb9a7-extensions-socket-volume\") pod \"f689fe0e-4605-4b38-ab39-166ea6bdb9a7\" (UID: \"f689fe0e-4605-4b38-ab39-166ea6bdb9a7\") "
Jul 06 22:55:15.854473 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:15.854438    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrv5w\" (UniqueName: \"kubernetes.io/projected/f689fe0e-4605-4b38-ab39-166ea6bdb9a7-kube-api-access-xrv5w\") pod \"f689fe0e-4605-4b38-ab39-166ea6bdb9a7\" (UID: \"f689fe0e-4605-4b38-ab39-166ea6bdb9a7\") "
Jul 06 22:55:15.854741 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:15.854719    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f689fe0e-4605-4b38-ab39-166ea6bdb9a7-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f689fe0e-4605-4b38-ab39-166ea6bdb9a7" (UID: "f689fe0e-4605-4b38-ab39-166ea6bdb9a7"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 22:55:15.856540 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:15.856521    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f689fe0e-4605-4b38-ab39-166ea6bdb9a7-kube-api-access-xrv5w" (OuterVolumeSpecName: "kube-api-access-xrv5w") pod "f689fe0e-4605-4b38-ab39-166ea6bdb9a7" (UID: "f689fe0e-4605-4b38-ab39-166ea6bdb9a7"). InnerVolumeSpecName "kube-api-access-xrv5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 22:55:15.955743 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:15.955709    2569 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f689fe0e-4605-4b38-ab39-166ea6bdb9a7-extensions-socket-volume\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 22:55:15.955743 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:15.955741    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrv5w\" (UniqueName: \"kubernetes.io/projected/f689fe0e-4605-4b38-ab39-166ea6bdb9a7-kube-api-access-xrv5w\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 22:55:16.394924 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:16.394894    2569 generic.go:356] "Generic (PLEG): container finished" podID="f689fe0e-4605-4b38-ab39-166ea6bdb9a7" containerID="06c01aecbc609c194412a7d36f60fb951164014c35214dfff9ceceed3b2b098a" exitCode=0
Jul 06 22:55:16.395063 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:16.394943    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hnq49"
Jul 06 22:55:16.395063 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:16.394958    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hnq49" event={"ID":"f689fe0e-4605-4b38-ab39-166ea6bdb9a7","Type":"ContainerDied","Data":"06c01aecbc609c194412a7d36f60fb951164014c35214dfff9ceceed3b2b098a"}
Jul 06 22:55:16.395063 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:16.394996    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hnq49" event={"ID":"f689fe0e-4605-4b38-ab39-166ea6bdb9a7","Type":"ContainerDied","Data":"aa8401221df7e7051ba3e3cff3f26467a629ed35bcc494a18d82b07a8ee0f6b4"}
Jul 06 22:55:16.395063 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:16.395021    2569 scope.go:117] "RemoveContainer" containerID="06c01aecbc609c194412a7d36f60fb951164014c35214dfff9ceceed3b2b098a"
Jul 06 22:55:16.403739 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:16.403718    2569 scope.go:117] "RemoveContainer" containerID="06c01aecbc609c194412a7d36f60fb951164014c35214dfff9ceceed3b2b098a"
Jul 06 22:55:16.404008 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:55:16.403987    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06c01aecbc609c194412a7d36f60fb951164014c35214dfff9ceceed3b2b098a\": container with ID starting with 06c01aecbc609c194412a7d36f60fb951164014c35214dfff9ceceed3b2b098a not found: ID does not exist" containerID="06c01aecbc609c194412a7d36f60fb951164014c35214dfff9ceceed3b2b098a"
Jul 06 22:55:16.404072 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:16.404016    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06c01aecbc609c194412a7d36f60fb951164014c35214dfff9ceceed3b2b098a"} err="failed to get container status \"06c01aecbc609c194412a7d36f60fb951164014c35214dfff9ceceed3b2b098a\": rpc error: code = NotFound desc = could not find container \"06c01aecbc609c194412a7d36f60fb951164014c35214dfff9ceceed3b2b098a\": container with ID starting with 06c01aecbc609c194412a7d36f60fb951164014c35214dfff9ceceed3b2b098a not found: ID does not exist"
Jul 06 22:55:16.417683 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:16.417662    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hnq49"]
Jul 06 22:55:16.422330 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:16.422309    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hnq49"]
Jul 06 22:55:18.170283 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:18.170248    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f689fe0e-4605-4b38-ab39-166ea6bdb9a7" path="/var/lib/kubelet/pods/f689fe0e-4605-4b38-ab39-166ea6bdb9a7/volumes"
Jul 06 22:55:20.370098 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:20.370022    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lf6v7"
Jul 06 22:55:31.769588 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:31.769552    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6"]
Jul 06 22:55:31.770058 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:31.769874    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f689fe0e-4605-4b38-ab39-166ea6bdb9a7" containerName="manager"
Jul 06 22:55:31.770058 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:31.769887    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f689fe0e-4605-4b38-ab39-166ea6bdb9a7" containerName="manager"
Jul 06 22:55:31.770058 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:31.769960    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f689fe0e-4605-4b38-ab39-166ea6bdb9a7" containerName="manager"
Jul 06 22:55:31.876620 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:31.876592    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6"]
Jul 06 22:55:31.876761 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:31.876687    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6"
Jul 06 22:55:31.879605 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:31.879583    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 06 22:55:31.879731 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:31.879613    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-h697s\""
Jul 06 22:55:31.961370 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:31.961344    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/404a2601-0b38-4fe9-81dc-23ae8fe26c6d-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tzqx6\" (UID: \"404a2601-0b38-4fe9-81dc-23ae8fe26c6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6"
Jul 06 22:55:31.961500 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:31.961375    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/404a2601-0b38-4fe9-81dc-23ae8fe26c6d-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tzqx6\" (UID: \"404a2601-0b38-4fe9-81dc-23ae8fe26c6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6"
Jul 06 22:55:31.961500 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:31.961393    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/404a2601-0b38-4fe9-81dc-23ae8fe26c6d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tzqx6\" (UID: \"404a2601-0b38-4fe9-81dc-23ae8fe26c6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6"
Jul 06 22:55:31.961500 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:31.961408    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/404a2601-0b38-4fe9-81dc-23ae8fe26c6d-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tzqx6\" (UID: \"404a2601-0b38-4fe9-81dc-23ae8fe26c6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6"
Jul 06 22:55:31.961500 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:31.961432    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/404a2601-0b38-4fe9-81dc-23ae8fe26c6d-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tzqx6\" (UID: \"404a2601-0b38-4fe9-81dc-23ae8fe26c6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6"
Jul 06 22:55:31.961500 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:31.961457    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/404a2601-0b38-4fe9-81dc-23ae8fe26c6d-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tzqx6\" (UID: \"404a2601-0b38-4fe9-81dc-23ae8fe26c6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6"
Jul 06 22:55:31.961500 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:31.961486    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/404a2601-0b38-4fe9-81dc-23ae8fe26c6d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tzqx6\" (UID: \"404a2601-0b38-4fe9-81dc-23ae8fe26c6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6"
Jul 06 22:55:31.961730 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:31.961507    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/404a2601-0b38-4fe9-81dc-23ae8fe26c6d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tzqx6\" (UID: \"404a2601-0b38-4fe9-81dc-23ae8fe26c6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6"
Jul 06 22:55:31.961730 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:31.961563    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbdql\" (UniqueName: \"kubernetes.io/projected/404a2601-0b38-4fe9-81dc-23ae8fe26c6d-kube-api-access-dbdql\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tzqx6\" (UID: \"404a2601-0b38-4fe9-81dc-23ae8fe26c6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6"
Jul 06 22:55:32.062341 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:32.062287    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/404a2601-0b38-4fe9-81dc-23ae8fe26c6d-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tzqx6\" (UID: \"404a2601-0b38-4fe9-81dc-23ae8fe26c6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6"
Jul 06 22:55:32.062341 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:32.062315    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/404a2601-0b38-4fe9-81dc-23ae8fe26c6d-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tzqx6\" (UID: \"404a2601-0b38-4fe9-81dc-23ae8fe26c6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6"
Jul 06 22:55:32.062341 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:32.062334    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/404a2601-0b38-4fe9-81dc-23ae8fe26c6d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tzqx6\" (UID: \"404a2601-0b38-4fe9-81dc-23ae8fe26c6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6"
Jul 06 22:55:32.062575 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:32.062348    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/404a2601-0b38-4fe9-81dc-23ae8fe26c6d-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tzqx6\" (UID: \"404a2601-0b38-4fe9-81dc-23ae8fe26c6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6"
Jul 06 22:55:32.062575 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:32.062367    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/404a2601-0b38-4fe9-81dc-23ae8fe26c6d-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tzqx6\" (UID: \"404a2601-0b38-4fe9-81dc-23ae8fe26c6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6"
Jul 06 22:55:32.062575 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:32.062402    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/404a2601-0b38-4fe9-81dc-23ae8fe26c6d-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tzqx6\" (UID: \"404a2601-0b38-4fe9-81dc-23ae8fe26c6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6"
Jul 06 22:55:32.062575 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:32.062435    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/404a2601-0b38-4fe9-81dc-23ae8fe26c6d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tzqx6\" (UID: \"404a2601-0b38-4fe9-81dc-23ae8fe26c6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6"
Jul 06 22:55:32.062575 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:32.062458    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/404a2601-0b38-4fe9-81dc-23ae8fe26c6d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tzqx6\" (UID: \"404a2601-0b38-4fe9-81dc-23ae8fe26c6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6"
Jul 06 22:55:32.062575 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:32.062510    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbdql\" (UniqueName: \"kubernetes.io/projected/404a2601-0b38-4fe9-81dc-23ae8fe26c6d-kube-api-access-dbdql\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tzqx6\" (UID: \"404a2601-0b38-4fe9-81dc-23ae8fe26c6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6"
Jul 06 22:55:32.062891 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:32.062752    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/404a2601-0b38-4fe9-81dc-23ae8fe26c6d-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tzqx6\" (UID: \"404a2601-0b38-4fe9-81dc-23ae8fe26c6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6"
Jul 06 22:55:32.062891 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:32.062780    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/404a2601-0b38-4fe9-81dc-23ae8fe26c6d-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tzqx6\" (UID: \"404a2601-0b38-4fe9-81dc-23ae8fe26c6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6"
Jul 06 22:55:32.062891 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:32.062828    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/404a2601-0b38-4fe9-81dc-23ae8fe26c6d-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tzqx6\" (UID: \"404a2601-0b38-4fe9-81dc-23ae8fe26c6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6"
Jul 06 22:55:32.062891 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:32.062872    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/404a2601-0b38-4fe9-81dc-23ae8fe26c6d-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tzqx6\" (UID: \"404a2601-0b38-4fe9-81dc-23ae8fe26c6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6"
Jul 06 22:55:32.063413 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:32.063391    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/404a2601-0b38-4fe9-81dc-23ae8fe26c6d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tzqx6\" (UID: \"404a2601-0b38-4fe9-81dc-23ae8fe26c6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6"
Jul 06 22:55:32.064682 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:32.064662    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/404a2601-0b38-4fe9-81dc-23ae8fe26c6d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tzqx6\" (UID: \"404a2601-0b38-4fe9-81dc-23ae8fe26c6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6"
Jul 06 22:55:32.065016 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:32.064999    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/404a2601-0b38-4fe9-81dc-23ae8fe26c6d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tzqx6\" (UID: \"404a2601-0b38-4fe9-81dc-23ae8fe26c6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6"
Jul 06 22:55:32.071310 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:32.071287    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/404a2601-0b38-4fe9-81dc-23ae8fe26c6d-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tzqx6\" (UID: \"404a2601-0b38-4fe9-81dc-23ae8fe26c6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6"
Jul 06 22:55:32.071503 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:32.071482    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbdql\" (UniqueName: \"kubernetes.io/projected/404a2601-0b38-4fe9-81dc-23ae8fe26c6d-kube-api-access-dbdql\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tzqx6\" (UID: \"404a2601-0b38-4fe9-81dc-23ae8fe26c6d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6"
Jul 06 22:55:32.187652 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:32.187625    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6"
Jul 06 22:55:32.330058 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:32.330036    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6"]
Jul 06 22:55:32.332086 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:55:32.332059    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod404a2601_0b38_4fe9_81dc_23ae8fe26c6d.slice/crio-c7a517a48e54af59f5bf33ada6a5d8257abdc3d7e6b2c76b25e1af075a4380e9 WatchSource:0}: Error finding container c7a517a48e54af59f5bf33ada6a5d8257abdc3d7e6b2c76b25e1af075a4380e9: Status 404 returned error can't find the container with id c7a517a48e54af59f5bf33ada6a5d8257abdc3d7e6b2c76b25e1af075a4380e9
Jul 06 22:55:32.462756 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:32.462724    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6" event={"ID":"404a2601-0b38-4fe9-81dc-23ae8fe26c6d","Type":"ContainerStarted","Data":"c7a517a48e54af59f5bf33ada6a5d8257abdc3d7e6b2c76b25e1af075a4380e9"}
Jul 06 22:55:35.389191 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:35.389146    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 22:55:35.389438 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:35.389234    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 22:55:35.389438 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:35.389265    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 22:55:35.481687 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:35.481654    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6" event={"ID":"404a2601-0b38-4fe9-81dc-23ae8fe26c6d","Type":"ContainerStarted","Data":"7d06c23e61c346c80eb5e4da277c9096798837f32c3d2912dc70d2548c383a87"}
Jul 06 22:55:35.502617 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:35.502570    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6" podStartSLOduration=1.447909358 podStartE2EDuration="4.502555264s" totalImagesPullingTime="3.054645906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:55:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:55:35.4999499 +0000 UTC m=+547.904199804" watchObservedRunningTime="2026-07-06 22:55:35.502555264 +0000 UTC m=+547.906805167"
Jul 06 22:55:36.188124 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:36.188089    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6"
Jul 06 22:55:36.192542 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:36.192520    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6"
Jul 06 22:55:36.485617 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:36.485545    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6"
Jul 06 22:55:36.486338 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:36.486318    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6"
Jul 06 22:55:45.978446 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:45.978410    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-nhnvr"]
Jul 06 22:55:45.997468 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:45.997432    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-nhnvr"]
Jul 06 22:55:45.997607 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:45.997554    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-nhnvr"
Jul 06 22:55:46.000260 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:46.000236    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 06 22:55:46.071345 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:46.071309    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-nhnvr"]
Jul 06 22:55:46.172133 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:46.172105    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cdf05707-88d8-4063-87de-2ffa82fbe5e4-config-file\") pod \"limitador-limitador-64c94fd64f-nhnvr\" (UID: \"cdf05707-88d8-4063-87de-2ffa82fbe5e4\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-nhnvr"
Jul 06 22:55:46.172307 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:46.172196    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fw88\" (UniqueName: \"kubernetes.io/projected/cdf05707-88d8-4063-87de-2ffa82fbe5e4-kube-api-access-8fw88\") pod \"limitador-limitador-64c94fd64f-nhnvr\" (UID: \"cdf05707-88d8-4063-87de-2ffa82fbe5e4\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-nhnvr"
Jul 06 22:55:46.272732 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:46.272660    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fw88\" (UniqueName: \"kubernetes.io/projected/cdf05707-88d8-4063-87de-2ffa82fbe5e4-kube-api-access-8fw88\") pod \"limitador-limitador-64c94fd64f-nhnvr\" (UID: \"cdf05707-88d8-4063-87de-2ffa82fbe5e4\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-nhnvr"
Jul 06 22:55:46.272732 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:46.272720    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cdf05707-88d8-4063-87de-2ffa82fbe5e4-config-file\") pod \"limitador-limitador-64c94fd64f-nhnvr\" (UID: \"cdf05707-88d8-4063-87de-2ffa82fbe5e4\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-nhnvr"
Jul 06 22:55:46.273452 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:46.273431    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cdf05707-88d8-4063-87de-2ffa82fbe5e4-config-file\") pod \"limitador-limitador-64c94fd64f-nhnvr\" (UID: \"cdf05707-88d8-4063-87de-2ffa82fbe5e4\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-nhnvr"
Jul 06 22:55:46.281470 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:46.281448    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fw88\" (UniqueName: \"kubernetes.io/projected/cdf05707-88d8-4063-87de-2ffa82fbe5e4-kube-api-access-8fw88\") pod \"limitador-limitador-64c94fd64f-nhnvr\" (UID: \"cdf05707-88d8-4063-87de-2ffa82fbe5e4\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-nhnvr"
Jul 06 22:55:46.308693 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:46.308670    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-nhnvr"
Jul 06 22:55:46.440444 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:46.440417    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-nhnvr"]
Jul 06 22:55:46.442724 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:55:46.442686    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdf05707_88d8_4063_87de_2ffa82fbe5e4.slice/crio-2ebb2f28b7b83acc44b86a54750976e093a37ec3f6aadb9b8040ce8e0fafaf95 WatchSource:0}: Error finding container 2ebb2f28b7b83acc44b86a54750976e093a37ec3f6aadb9b8040ce8e0fafaf95: Status 404 returned error can't find the container with id 2ebb2f28b7b83acc44b86a54750976e093a37ec3f6aadb9b8040ce8e0fafaf95
Jul 06 22:55:46.528272 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:46.528202    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-nhnvr" event={"ID":"cdf05707-88d8-4063-87de-2ffa82fbe5e4","Type":"ContainerStarted","Data":"2ebb2f28b7b83acc44b86a54750976e093a37ec3f6aadb9b8040ce8e0fafaf95"}
Jul 06 22:55:46.769721 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:46.769694    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-jsbsc"]
Jul 06 22:55:46.774127 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:46.774108    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-jsbsc"
Jul 06 22:55:46.777077 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:46.777055    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-xwbh5\""
Jul 06 22:55:46.779439 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:46.779388    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-jsbsc"]
Jul 06 22:55:46.877956 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:46.877795    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp2vr\" (UniqueName: \"kubernetes.io/projected/767f67bb-5204-44d9-82b8-5013369fc274-kube-api-access-xp2vr\") pod \"authorino-58cd955675-jsbsc\" (UID: \"767f67bb-5204-44d9-82b8-5013369fc274\") " pod="kuadrant-system/authorino-58cd955675-jsbsc"
Jul 06 22:55:46.979486 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:46.979092    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp2vr\" (UniqueName: \"kubernetes.io/projected/767f67bb-5204-44d9-82b8-5013369fc274-kube-api-access-xp2vr\") pod \"authorino-58cd955675-jsbsc\" (UID: \"767f67bb-5204-44d9-82b8-5013369fc274\") " pod="kuadrant-system/authorino-58cd955675-jsbsc"
Jul 06 22:55:46.988626 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:46.988564    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp2vr\" (UniqueName: \"kubernetes.io/projected/767f67bb-5204-44d9-82b8-5013369fc274-kube-api-access-xp2vr\") pod \"authorino-58cd955675-jsbsc\" (UID: \"767f67bb-5204-44d9-82b8-5013369fc274\") " pod="kuadrant-system/authorino-58cd955675-jsbsc"
Jul 06 22:55:47.084747 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:47.084672    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-jsbsc"
Jul 06 22:55:47.255501 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:47.255440    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-jsbsc"]
Jul 06 22:55:47.259987 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:55:47.259953    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod767f67bb_5204_44d9_82b8_5013369fc274.slice/crio-56ae984ae0cd901ec4095a7d67ad26fedbcb0d2e2a24652279d3712b043b9008 WatchSource:0}: Error finding container 56ae984ae0cd901ec4095a7d67ad26fedbcb0d2e2a24652279d3712b043b9008: Status 404 returned error can't find the container with id 56ae984ae0cd901ec4095a7d67ad26fedbcb0d2e2a24652279d3712b043b9008
Jul 06 22:55:47.533916 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:47.533887    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-jsbsc" event={"ID":"767f67bb-5204-44d9-82b8-5013369fc274","Type":"ContainerStarted","Data":"56ae984ae0cd901ec4095a7d67ad26fedbcb0d2e2a24652279d3712b043b9008"}
Jul 06 22:55:49.547715 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:49.547643    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-nhnvr" event={"ID":"cdf05707-88d8-4063-87de-2ffa82fbe5e4","Type":"ContainerStarted","Data":"5cc60b3893019d0759ce9407a51889444609906c2015cd1c167d323c021e9d76"}
Jul 06 22:55:49.548267 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:49.547786    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-nhnvr"
Jul 06 22:55:49.565484 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:49.565428    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-nhnvr" podStartSLOduration=1.9556879230000002 podStartE2EDuration="4.565409868s" totalImagesPullingTime="2.609721945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:55:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:55:49.564780331 +0000 UTC m=+561.969030236" watchObservedRunningTime="2026-07-06 22:55:49.565409868 +0000 UTC m=+561.969659773"
Jul 06 22:55:50.667234 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:50.667196    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-jsbsc"]
Jul 06 22:55:51.560054 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:51.560017    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-jsbsc" event={"ID":"767f67bb-5204-44d9-82b8-5013369fc274","Type":"ContainerStarted","Data":"8432a13325c1507685d5ba20703931406b013d0acb9e20eb1ad4e54249ceaf8f"}
Jul 06 22:55:51.560272 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:51.560109    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-jsbsc" podUID="767f67bb-5204-44d9-82b8-5013369fc274" containerName="authorino" containerID="cri-o://8432a13325c1507685d5ba20703931406b013d0acb9e20eb1ad4e54249ceaf8f" gracePeriod=30
Jul 06 22:55:51.577844 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:51.577803    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-jsbsc" podStartSLOduration=1.911919854 podStartE2EDuration="5.577787951s" totalImagesPullingTime="3.665868097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:55:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:55:51.576006615 +0000 UTC m=+563.980256521" watchObservedRunningTime="2026-07-06 22:55:51.577787951 +0000 UTC m=+563.982037890"
Jul 06 22:55:51.805885 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:51.805862    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-jsbsc"
Jul 06 22:55:51.921913 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:51.921847    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xp2vr\" (UniqueName: \"kubernetes.io/projected/767f67bb-5204-44d9-82b8-5013369fc274-kube-api-access-xp2vr\") pod \"767f67bb-5204-44d9-82b8-5013369fc274\" (UID: \"767f67bb-5204-44d9-82b8-5013369fc274\") "
Jul 06 22:55:51.923848 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:51.923823    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/767f67bb-5204-44d9-82b8-5013369fc274-kube-api-access-xp2vr" (OuterVolumeSpecName: "kube-api-access-xp2vr") pod "767f67bb-5204-44d9-82b8-5013369fc274" (UID: "767f67bb-5204-44d9-82b8-5013369fc274"). InnerVolumeSpecName "kube-api-access-xp2vr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 22:55:52.023187 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:52.023163    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp2vr\" (UniqueName: \"kubernetes.io/projected/767f67bb-5204-44d9-82b8-5013369fc274-kube-api-access-xp2vr\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 22:55:52.565930 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:52.565897    2569 generic.go:356] "Generic (PLEG): container finished" podID="767f67bb-5204-44d9-82b8-5013369fc274" containerID="8432a13325c1507685d5ba20703931406b013d0acb9e20eb1ad4e54249ceaf8f" exitCode=0
Jul 06 22:55:52.566100 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:52.565933    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-jsbsc"
Jul 06 22:55:52.566100 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:52.565981    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-jsbsc" event={"ID":"767f67bb-5204-44d9-82b8-5013369fc274","Type":"ContainerDied","Data":"8432a13325c1507685d5ba20703931406b013d0acb9e20eb1ad4e54249ceaf8f"}
Jul 06 22:55:52.566100 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:52.566017    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-jsbsc" event={"ID":"767f67bb-5204-44d9-82b8-5013369fc274","Type":"ContainerDied","Data":"56ae984ae0cd901ec4095a7d67ad26fedbcb0d2e2a24652279d3712b043b9008"}
Jul 06 22:55:52.566100 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:52.566033    2569 scope.go:117] "RemoveContainer" containerID="8432a13325c1507685d5ba20703931406b013d0acb9e20eb1ad4e54249ceaf8f"
Jul 06 22:55:52.574980 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:52.574965    2569 scope.go:117] "RemoveContainer" containerID="8432a13325c1507685d5ba20703931406b013d0acb9e20eb1ad4e54249ceaf8f"
Jul 06 22:55:52.575335 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:55:52.575314    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8432a13325c1507685d5ba20703931406b013d0acb9e20eb1ad4e54249ceaf8f\": container with ID starting with 8432a13325c1507685d5ba20703931406b013d0acb9e20eb1ad4e54249ceaf8f not found: ID does not exist" containerID="8432a13325c1507685d5ba20703931406b013d0acb9e20eb1ad4e54249ceaf8f"
Jul 06 22:55:52.575399 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:52.575342    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8432a13325c1507685d5ba20703931406b013d0acb9e20eb1ad4e54249ceaf8f"} err="failed to get container status \"8432a13325c1507685d5ba20703931406b013d0acb9e20eb1ad4e54249ceaf8f\": rpc error: code = NotFound desc = could not find container \"8432a13325c1507685d5ba20703931406b013d0acb9e20eb1ad4e54249ceaf8f\": container with ID starting with 8432a13325c1507685d5ba20703931406b013d0acb9e20eb1ad4e54249ceaf8f not found: ID does not exist"
Jul 06 22:55:52.582513 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:52.582492    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-jsbsc"]
Jul 06 22:55:52.585853 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:52.585833    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-jsbsc"]
Jul 06 22:55:54.169583 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:55:54.169550    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="767f67bb-5204-44d9-82b8-5013369fc274" path="/var/lib/kubelet/pods/767f67bb-5204-44d9-82b8-5013369fc274/volumes"
Jul 06 22:56:00.555190 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:00.555157    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-nhnvr"
Jul 06 22:56:01.378510 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:01.378480    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-nhnvr"]
Jul 06 22:56:01.378726 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:01.378697    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-nhnvr" podUID="cdf05707-88d8-4063-87de-2ffa82fbe5e4" containerName="limitador" containerID="cri-o://5cc60b3893019d0759ce9407a51889444609906c2015cd1c167d323c021e9d76" gracePeriod=30
Jul 06 22:56:01.929662 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:01.929634    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-nhnvr"
Jul 06 22:56:02.001327 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:02.001271    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fw88\" (UniqueName: \"kubernetes.io/projected/cdf05707-88d8-4063-87de-2ffa82fbe5e4-kube-api-access-8fw88\") pod \"cdf05707-88d8-4063-87de-2ffa82fbe5e4\" (UID: \"cdf05707-88d8-4063-87de-2ffa82fbe5e4\") "
Jul 06 22:56:02.001438 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:02.001354    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cdf05707-88d8-4063-87de-2ffa82fbe5e4-config-file\") pod \"cdf05707-88d8-4063-87de-2ffa82fbe5e4\" (UID: \"cdf05707-88d8-4063-87de-2ffa82fbe5e4\") "
Jul 06 22:56:02.001672 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:02.001650    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdf05707-88d8-4063-87de-2ffa82fbe5e4-config-file" (OuterVolumeSpecName: "config-file") pod "cdf05707-88d8-4063-87de-2ffa82fbe5e4" (UID: "cdf05707-88d8-4063-87de-2ffa82fbe5e4"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 22:56:02.003407 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:02.003387    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdf05707-88d8-4063-87de-2ffa82fbe5e4-kube-api-access-8fw88" (OuterVolumeSpecName: "kube-api-access-8fw88") pod "cdf05707-88d8-4063-87de-2ffa82fbe5e4" (UID: "cdf05707-88d8-4063-87de-2ffa82fbe5e4"). InnerVolumeSpecName "kube-api-access-8fw88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 22:56:02.101821 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:02.101795    2569 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cdf05707-88d8-4063-87de-2ffa82fbe5e4-config-file\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 22:56:02.101821 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:02.101820    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fw88\" (UniqueName: \"kubernetes.io/projected/cdf05707-88d8-4063-87de-2ffa82fbe5e4-kube-api-access-8fw88\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 22:56:02.613146 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:02.613113    2569 generic.go:356] "Generic (PLEG): container finished" podID="cdf05707-88d8-4063-87de-2ffa82fbe5e4" containerID="5cc60b3893019d0759ce9407a51889444609906c2015cd1c167d323c021e9d76" exitCode=0
Jul 06 22:56:02.613355 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:02.613159    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-nhnvr"
Jul 06 22:56:02.613355 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:02.613201    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-nhnvr" event={"ID":"cdf05707-88d8-4063-87de-2ffa82fbe5e4","Type":"ContainerDied","Data":"5cc60b3893019d0759ce9407a51889444609906c2015cd1c167d323c021e9d76"}
Jul 06 22:56:02.613355 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:02.613284    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-nhnvr" event={"ID":"cdf05707-88d8-4063-87de-2ffa82fbe5e4","Type":"ContainerDied","Data":"2ebb2f28b7b83acc44b86a54750976e093a37ec3f6aadb9b8040ce8e0fafaf95"}
Jul 06 22:56:02.613355 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:02.613314    2569 scope.go:117] "RemoveContainer" containerID="5cc60b3893019d0759ce9407a51889444609906c2015cd1c167d323c021e9d76"
Jul 06 22:56:02.621959 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:02.621943    2569 scope.go:117] "RemoveContainer" containerID="5cc60b3893019d0759ce9407a51889444609906c2015cd1c167d323c021e9d76"
Jul 06 22:56:02.622224 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:56:02.622206    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cc60b3893019d0759ce9407a51889444609906c2015cd1c167d323c021e9d76\": container with ID starting with 5cc60b3893019d0759ce9407a51889444609906c2015cd1c167d323c021e9d76 not found: ID does not exist" containerID="5cc60b3893019d0759ce9407a51889444609906c2015cd1c167d323c021e9d76"
Jul 06 22:56:02.622290 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:02.622231    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cc60b3893019d0759ce9407a51889444609906c2015cd1c167d323c021e9d76"} err="failed to get container status \"5cc60b3893019d0759ce9407a51889444609906c2015cd1c167d323c021e9d76\": rpc error: code = NotFound desc = could not find container \"5cc60b3893019d0759ce9407a51889444609906c2015cd1c167d323c021e9d76\": container with ID starting with 5cc60b3893019d0759ce9407a51889444609906c2015cd1c167d323c021e9d76 not found: ID does not exist"
Jul 06 22:56:02.629367 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:02.629348    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-nhnvr"]
Jul 06 22:56:02.633413 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:02.633389    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-nhnvr"]
Jul 06 22:56:03.154556 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:03.154519    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/postgres-69fb4594cf-l29j5"]
Jul 06 22:56:03.155008 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:03.154839    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdf05707-88d8-4063-87de-2ffa82fbe5e4" containerName="limitador"
Jul 06 22:56:03.155008 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:03.154850    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdf05707-88d8-4063-87de-2ffa82fbe5e4" containerName="limitador"
Jul 06 22:56:03.155008 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:03.154863    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="767f67bb-5204-44d9-82b8-5013369fc274" containerName="authorino"
Jul 06 22:56:03.155008 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:03.154869    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="767f67bb-5204-44d9-82b8-5013369fc274" containerName="authorino"
Jul 06 22:56:03.155008 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:03.154924    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="767f67bb-5204-44d9-82b8-5013369fc274" containerName="authorino"
Jul 06 22:56:03.155008 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:03.154931    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdf05707-88d8-4063-87de-2ffa82fbe5e4" containerName="limitador"
Jul 06 22:56:03.158733 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:03.158713    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-l29j5"
Jul 06 22:56:03.161279 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:03.161248    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"postgres-creds\""
Jul 06 22:56:03.161380 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:03.161297    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-jw58c\""
Jul 06 22:56:03.165299 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:03.165277    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-l29j5"]
Jul 06 22:56:03.209955 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:03.209932    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/67be9ca0-f203-4586-968f-3b06828f4899-data\") pod \"postgres-69fb4594cf-l29j5\" (UID: \"67be9ca0-f203-4586-968f-3b06828f4899\") " pod="opendatahub/postgres-69fb4594cf-l29j5"
Jul 06 22:56:03.210047 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:03.209998    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgcbx\" (UniqueName: \"kubernetes.io/projected/67be9ca0-f203-4586-968f-3b06828f4899-kube-api-access-vgcbx\") pod \"postgres-69fb4594cf-l29j5\" (UID: \"67be9ca0-f203-4586-968f-3b06828f4899\") " pod="opendatahub/postgres-69fb4594cf-l29j5"
Jul 06 22:56:03.311044 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:03.311012    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgcbx\" (UniqueName: \"kubernetes.io/projected/67be9ca0-f203-4586-968f-3b06828f4899-kube-api-access-vgcbx\") pod \"postgres-69fb4594cf-l29j5\" (UID: \"67be9ca0-f203-4586-968f-3b06828f4899\") " pod="opendatahub/postgres-69fb4594cf-l29j5"
Jul 06 22:56:03.311168 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:03.311075    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/67be9ca0-f203-4586-968f-3b06828f4899-data\") pod \"postgres-69fb4594cf-l29j5\" (UID: \"67be9ca0-f203-4586-968f-3b06828f4899\") " pod="opendatahub/postgres-69fb4594cf-l29j5"
Jul 06 22:56:03.311462 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:03.311442    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/67be9ca0-f203-4586-968f-3b06828f4899-data\") pod \"postgres-69fb4594cf-l29j5\" (UID: \"67be9ca0-f203-4586-968f-3b06828f4899\") " pod="opendatahub/postgres-69fb4594cf-l29j5"
Jul 06 22:56:03.319615 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:03.319598    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgcbx\" (UniqueName: \"kubernetes.io/projected/67be9ca0-f203-4586-968f-3b06828f4899-kube-api-access-vgcbx\") pod \"postgres-69fb4594cf-l29j5\" (UID: \"67be9ca0-f203-4586-968f-3b06828f4899\") " pod="opendatahub/postgres-69fb4594cf-l29j5"
Jul 06 22:56:03.469719 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:03.469698    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-l29j5"
Jul 06 22:56:03.591924 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:03.591896    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-l29j5"]
Jul 06 22:56:03.593848 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:56:03.593816    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67be9ca0_f203_4586_968f_3b06828f4899.slice/crio-41864464f9ec0973a812e003f06e05fa0d225b43657b70a52379c7ec03774421 WatchSource:0}: Error finding container 41864464f9ec0973a812e003f06e05fa0d225b43657b70a52379c7ec03774421: Status 404 returned error can't find the container with id 41864464f9ec0973a812e003f06e05fa0d225b43657b70a52379c7ec03774421
Jul 06 22:56:03.618755 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:03.618728    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-l29j5" event={"ID":"67be9ca0-f203-4586-968f-3b06828f4899","Type":"ContainerStarted","Data":"41864464f9ec0973a812e003f06e05fa0d225b43657b70a52379c7ec03774421"}
Jul 06 22:56:04.170042 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:04.170017    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdf05707-88d8-4063-87de-2ffa82fbe5e4" path="/var/lib/kubelet/pods/cdf05707-88d8-4063-87de-2ffa82fbe5e4/volumes"
Jul 06 22:56:08.648753 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:08.648720    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-l29j5" event={"ID":"67be9ca0-f203-4586-968f-3b06828f4899","Type":"ContainerStarted","Data":"11f4908c3487b89e661872ef8c94e398e752207e139885e8abc6a601b1bbad91"}
Jul 06 22:56:08.649119 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:08.648876    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/postgres-69fb4594cf-l29j5"
Jul 06 22:56:08.664120 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:08.664075    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/postgres-69fb4594cf-l29j5" podStartSLOduration=0.766706938 podStartE2EDuration="5.66406149s" totalImagesPullingTime="4.897354552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:56:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:56:08.662902677 +0000 UTC m=+581.067152580" watchObservedRunningTime="2026-07-06 22:56:08.66406149 +0000 UTC m=+581.068311424"
Jul 06 22:56:14.682998 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:14.682968    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/postgres-69fb4594cf-l29j5"
Jul 06 22:56:15.232623 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:15.232594    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-c5k45"]
Jul 06 22:56:15.235901 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:15.235885    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-c5k45"
Jul 06 22:56:15.240431 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:15.240412    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-xwbh5\""
Jul 06 22:56:15.253936 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:15.253913    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-c5k45"]
Jul 06 22:56:15.300418 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:15.300397    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6blw\" (UniqueName: \"kubernetes.io/projected/697b3597-3695-4dfc-9484-d4ecc3349fda-kube-api-access-f6blw\") pod \"authorino-5d4b67c446-c5k45\" (UID: \"697b3597-3695-4dfc-9484-d4ecc3349fda\") " pod="kuadrant-system/authorino-5d4b67c446-c5k45"
Jul 06 22:56:15.401056 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:15.401035    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6blw\" (UniqueName: \"kubernetes.io/projected/697b3597-3695-4dfc-9484-d4ecc3349fda-kube-api-access-f6blw\") pod \"authorino-5d4b67c446-c5k45\" (UID: \"697b3597-3695-4dfc-9484-d4ecc3349fda\") " pod="kuadrant-system/authorino-5d4b67c446-c5k45"
Jul 06 22:56:15.409130 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:15.409108    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6blw\" (UniqueName: \"kubernetes.io/projected/697b3597-3695-4dfc-9484-d4ecc3349fda-kube-api-access-f6blw\") pod \"authorino-5d4b67c446-c5k45\" (UID: \"697b3597-3695-4dfc-9484-d4ecc3349fda\") " pod="kuadrant-system/authorino-5d4b67c446-c5k45"
Jul 06 22:56:15.465291 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:15.465273    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-c5k45"]
Jul 06 22:56:15.465450 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:15.465439    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-c5k45"
Jul 06 22:56:15.497270 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:15.497215    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7948b75cc7-t5qbv"]
Jul 06 22:56:15.501515 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:15.501498    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7948b75cc7-t5qbv"
Jul 06 22:56:15.516565 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:15.516536    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7948b75cc7-t5qbv"]
Jul 06 22:56:15.596217 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:15.596163    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-c5k45"]
Jul 06 22:56:15.599236 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:56:15.599211    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod697b3597_3695_4dfc_9484_d4ecc3349fda.slice/crio-d5e50fff026b329c43d7c261b0a42a88d12dddb700e6eb064127d711fcce11c0 WatchSource:0}: Error finding container d5e50fff026b329c43d7c261b0a42a88d12dddb700e6eb064127d711fcce11c0: Status 404 returned error can't find the container with id d5e50fff026b329c43d7c261b0a42a88d12dddb700e6eb064127d711fcce11c0
Jul 06 22:56:15.602503 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:15.602480    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c6zf\" (UniqueName: \"kubernetes.io/projected/0aa81ebe-ce8c-485e-85e6-6d4c4de82b45-kube-api-access-5c6zf\") pod \"authorino-7948b75cc7-t5qbv\" (UID: \"0aa81ebe-ce8c-485e-85e6-6d4c4de82b45\") " pod="kuadrant-system/authorino-7948b75cc7-t5qbv"
Jul 06 22:56:15.681475 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:15.681448    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-c5k45" event={"ID":"697b3597-3695-4dfc-9484-d4ecc3349fda","Type":"ContainerStarted","Data":"d5e50fff026b329c43d7c261b0a42a88d12dddb700e6eb064127d711fcce11c0"}
Jul 06 22:56:15.684527 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:15.684506    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7948b75cc7-t5qbv"]
Jul 06 22:56:15.684803 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:56:15.684701    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-5c6zf], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-7948b75cc7-t5qbv" podUID="0aa81ebe-ce8c-485e-85e6-6d4c4de82b45"
Jul 06 22:56:15.703358 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:15.703337    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c6zf\" (UniqueName: \"kubernetes.io/projected/0aa81ebe-ce8c-485e-85e6-6d4c4de82b45-kube-api-access-5c6zf\") pod \"authorino-7948b75cc7-t5qbv\" (UID: \"0aa81ebe-ce8c-485e-85e6-6d4c4de82b45\") " pod="kuadrant-system/authorino-7948b75cc7-t5qbv"
Jul 06 22:56:15.716046 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:15.716019    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c6zf\" (UniqueName: \"kubernetes.io/projected/0aa81ebe-ce8c-485e-85e6-6d4c4de82b45-kube-api-access-5c6zf\") pod \"authorino-7948b75cc7-t5qbv\" (UID: \"0aa81ebe-ce8c-485e-85e6-6d4c4de82b45\") " pod="kuadrant-system/authorino-7948b75cc7-t5qbv"
Jul 06 22:56:16.687099 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:16.687075    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7948b75cc7-t5qbv"
Jul 06 22:56:16.687099 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:16.687069    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-c5k45" podUID="697b3597-3695-4dfc-9484-d4ecc3349fda" containerName="authorino" containerID="cri-o://98ba7193bf538fd835960531b25cef4742df4d02181897f4cc30a69603b3d004" gracePeriod=30
Jul 06 22:56:16.687552 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:16.687065    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-c5k45" event={"ID":"697b3597-3695-4dfc-9484-d4ecc3349fda","Type":"ContainerStarted","Data":"98ba7193bf538fd835960531b25cef4742df4d02181897f4cc30a69603b3d004"}
Jul 06 22:56:16.694557 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:16.694533    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7948b75cc7-t5qbv"
Jul 06 22:56:16.702426 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:16.702385    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-c5k45" podStartSLOduration=1.434645156 podStartE2EDuration="1.702369986s" totalImagesPullingTime="267.72483ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:56:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:56:16.70109699 +0000 UTC m=+589.105346894" watchObservedRunningTime="2026-07-06 22:56:16.702369986 +0000 UTC m=+589.106619892"
Jul 06 22:56:16.811870 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:16.811843    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5c6zf\" (UniqueName: \"kubernetes.io/projected/0aa81ebe-ce8c-485e-85e6-6d4c4de82b45-kube-api-access-5c6zf\") pod \"0aa81ebe-ce8c-485e-85e6-6d4c4de82b45\" (UID: \"0aa81ebe-ce8c-485e-85e6-6d4c4de82b45\") "
Jul 06 22:56:16.813803 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:16.813776    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aa81ebe-ce8c-485e-85e6-6d4c4de82b45-kube-api-access-5c6zf" (OuterVolumeSpecName: "kube-api-access-5c6zf") pod "0aa81ebe-ce8c-485e-85e6-6d4c4de82b45" (UID: "0aa81ebe-ce8c-485e-85e6-6d4c4de82b45"). InnerVolumeSpecName "kube-api-access-5c6zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 22:56:16.912897 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:16.912874    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c6zf\" (UniqueName: \"kubernetes.io/projected/0aa81ebe-ce8c-485e-85e6-6d4c4de82b45-kube-api-access-5c6zf\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 22:56:16.929587 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:16.929565    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-c5k45"
Jul 06 22:56:17.013379 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:17.013316    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6blw\" (UniqueName: \"kubernetes.io/projected/697b3597-3695-4dfc-9484-d4ecc3349fda-kube-api-access-f6blw\") pod \"697b3597-3695-4dfc-9484-d4ecc3349fda\" (UID: \"697b3597-3695-4dfc-9484-d4ecc3349fda\") "
Jul 06 22:56:17.015189 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:17.015161    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/697b3597-3695-4dfc-9484-d4ecc3349fda-kube-api-access-f6blw" (OuterVolumeSpecName: "kube-api-access-f6blw") pod "697b3597-3695-4dfc-9484-d4ecc3349fda" (UID: "697b3597-3695-4dfc-9484-d4ecc3349fda"). InnerVolumeSpecName "kube-api-access-f6blw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 22:56:17.114715 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:17.114680    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6blw\" (UniqueName: \"kubernetes.io/projected/697b3597-3695-4dfc-9484-d4ecc3349fda-kube-api-access-f6blw\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 22:56:17.692899 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:17.692866    2569 generic.go:356] "Generic (PLEG): container finished" podID="697b3597-3695-4dfc-9484-d4ecc3349fda" containerID="98ba7193bf538fd835960531b25cef4742df4d02181897f4cc30a69603b3d004" exitCode=0
Jul 06 22:56:17.693297 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:17.692906    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-c5k45"
Jul 06 22:56:17.693297 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:17.692950    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-c5k45" event={"ID":"697b3597-3695-4dfc-9484-d4ecc3349fda","Type":"ContainerDied","Data":"98ba7193bf538fd835960531b25cef4742df4d02181897f4cc30a69603b3d004"}
Jul 06 22:56:17.693297 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:17.692990    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-c5k45" event={"ID":"697b3597-3695-4dfc-9484-d4ecc3349fda","Type":"ContainerDied","Data":"d5e50fff026b329c43d7c261b0a42a88d12dddb700e6eb064127d711fcce11c0"}
Jul 06 22:56:17.693297 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:17.693010    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7948b75cc7-t5qbv"
Jul 06 22:56:17.693297 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:17.693011    2569 scope.go:117] "RemoveContainer" containerID="98ba7193bf538fd835960531b25cef4742df4d02181897f4cc30a69603b3d004"
Jul 06 22:56:17.701953 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:17.701830    2569 scope.go:117] "RemoveContainer" containerID="98ba7193bf538fd835960531b25cef4742df4d02181897f4cc30a69603b3d004"
Jul 06 22:56:17.702158 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:56:17.702132    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98ba7193bf538fd835960531b25cef4742df4d02181897f4cc30a69603b3d004\": container with ID starting with 98ba7193bf538fd835960531b25cef4742df4d02181897f4cc30a69603b3d004 not found: ID does not exist" containerID="98ba7193bf538fd835960531b25cef4742df4d02181897f4cc30a69603b3d004"
Jul 06 22:56:17.702289 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:17.702168    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98ba7193bf538fd835960531b25cef4742df4d02181897f4cc30a69603b3d004"} err="failed to get container status \"98ba7193bf538fd835960531b25cef4742df4d02181897f4cc30a69603b3d004\": rpc error: code = NotFound desc = could not find container \"98ba7193bf538fd835960531b25cef4742df4d02181897f4cc30a69603b3d004\": container with ID starting with 98ba7193bf538fd835960531b25cef4742df4d02181897f4cc30a69603b3d004 not found: ID does not exist"
Jul 06 22:56:17.725433 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:17.725413    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7948b75cc7-t5qbv"]
Jul 06 22:56:17.729581 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:17.729560    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7948b75cc7-t5qbv"]
Jul 06 22:56:17.739156 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:17.739133    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-c5k45"]
Jul 06 22:56:17.741078 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:17.741060    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-c5k45"]
Jul 06 22:56:18.169830 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:18.169740    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0aa81ebe-ce8c-485e-85e6-6d4c4de82b45" path="/var/lib/kubelet/pods/0aa81ebe-ce8c-485e-85e6-6d4c4de82b45/volumes"
Jul 06 22:56:18.170047 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:18.170026    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="697b3597-3695-4dfc-9484-d4ecc3349fda" path="/var/lib/kubelet/pods/697b3597-3695-4dfc-9484-d4ecc3349fda/volumes"
Jul 06 22:56:21.001981 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:21.001950    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7cd9bffc57-2rvf9"]
Jul 06 22:56:21.002368 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:21.002289    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="697b3597-3695-4dfc-9484-d4ecc3349fda" containerName="authorino"
Jul 06 22:56:21.002368 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:21.002301    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="697b3597-3695-4dfc-9484-d4ecc3349fda" containerName="authorino"
Jul 06 22:56:21.002368 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:21.002356    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="697b3597-3695-4dfc-9484-d4ecc3349fda" containerName="authorino"
Jul 06 22:56:21.006736 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:21.006717    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7cd9bffc57-2rvf9"
Jul 06 22:56:21.009309 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:21.009290    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 06 22:56:21.009309 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:21.009300    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8zknt\""
Jul 06 22:56:21.010244 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:21.010232    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 06 22:56:21.014555 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:21.014529    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7cd9bffc57-2rvf9"]
Jul 06 22:56:21.040452 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:21.040430    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcdvz\" (UniqueName: \"kubernetes.io/projected/56fced8f-0770-4d9f-9043-fe17088b0a68-kube-api-access-zcdvz\") pod \"maas-controller-7cd9bffc57-2rvf9\" (UID: \"56fced8f-0770-4d9f-9043-fe17088b0a68\") " pod="opendatahub/maas-controller-7cd9bffc57-2rvf9"
Jul 06 22:56:21.040579 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:21.040473    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/56fced8f-0770-4d9f-9043-fe17088b0a68-webhook-cert\") pod \"maas-controller-7cd9bffc57-2rvf9\" (UID: \"56fced8f-0770-4d9f-9043-fe17088b0a68\") " pod="opendatahub/maas-controller-7cd9bffc57-2rvf9"
Jul 06 22:56:21.141210 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:21.141166    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcdvz\" (UniqueName: \"kubernetes.io/projected/56fced8f-0770-4d9f-9043-fe17088b0a68-kube-api-access-zcdvz\") pod \"maas-controller-7cd9bffc57-2rvf9\" (UID: \"56fced8f-0770-4d9f-9043-fe17088b0a68\") " pod="opendatahub/maas-controller-7cd9bffc57-2rvf9"
Jul 06 22:56:21.141335 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:21.141219    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/56fced8f-0770-4d9f-9043-fe17088b0a68-webhook-cert\") pod \"maas-controller-7cd9bffc57-2rvf9\" (UID: \"56fced8f-0770-4d9f-9043-fe17088b0a68\") " pod="opendatahub/maas-controller-7cd9bffc57-2rvf9"
Jul 06 22:56:21.141335 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:56:21.141319    2569 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 06 22:56:21.141453 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:56:21.141368    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56fced8f-0770-4d9f-9043-fe17088b0a68-webhook-cert podName:56fced8f-0770-4d9f-9043-fe17088b0a68 nodeName:}" failed. No retries permitted until 2026-07-06 22:56:21.641353444 +0000 UTC m=+594.045603326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/56fced8f-0770-4d9f-9043-fe17088b0a68-webhook-cert") pod "maas-controller-7cd9bffc57-2rvf9" (UID: "56fced8f-0770-4d9f-9043-fe17088b0a68") : secret "maas-controller-webhook-cert" not found
Jul 06 22:56:21.149859 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:21.149839    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcdvz\" (UniqueName: \"kubernetes.io/projected/56fced8f-0770-4d9f-9043-fe17088b0a68-kube-api-access-zcdvz\") pod \"maas-controller-7cd9bffc57-2rvf9\" (UID: \"56fced8f-0770-4d9f-9043-fe17088b0a68\") " pod="opendatahub/maas-controller-7cd9bffc57-2rvf9"
Jul 06 22:56:21.188432 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:21.188406    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5dfdd7fcb5-684w6"]
Jul 06 22:56:21.190861 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:21.190843    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5dfdd7fcb5-684w6"
Jul 06 22:56:21.201332 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:21.201308    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5dfdd7fcb5-684w6"]
Jul 06 22:56:21.241986 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:21.241952    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fa04db79-f0f1-44b5-92a0-5ed5a52bba0b-webhook-cert\") pod \"maas-controller-5dfdd7fcb5-684w6\" (UID: \"fa04db79-f0f1-44b5-92a0-5ed5a52bba0b\") " pod="opendatahub/maas-controller-5dfdd7fcb5-684w6"
Jul 06 22:56:21.242101 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:21.242057    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29qf5\" (UniqueName: \"kubernetes.io/projected/fa04db79-f0f1-44b5-92a0-5ed5a52bba0b-kube-api-access-29qf5\") pod \"maas-controller-5dfdd7fcb5-684w6\" (UID: \"fa04db79-f0f1-44b5-92a0-5ed5a52bba0b\") " pod="opendatahub/maas-controller-5dfdd7fcb5-684w6"
Jul 06 22:56:21.343154 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:21.343095    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fa04db79-f0f1-44b5-92a0-5ed5a52bba0b-webhook-cert\") pod \"maas-controller-5dfdd7fcb5-684w6\" (UID: \"fa04db79-f0f1-44b5-92a0-5ed5a52bba0b\") " pod="opendatahub/maas-controller-5dfdd7fcb5-684w6"
Jul 06 22:56:21.343154 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:21.343139    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29qf5\" (UniqueName: \"kubernetes.io/projected/fa04db79-f0f1-44b5-92a0-5ed5a52bba0b-kube-api-access-29qf5\") pod \"maas-controller-5dfdd7fcb5-684w6\" (UID: \"fa04db79-f0f1-44b5-92a0-5ed5a52bba0b\") " pod="opendatahub/maas-controller-5dfdd7fcb5-684w6"
Jul 06 22:56:21.345462 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:21.345440    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fa04db79-f0f1-44b5-92a0-5ed5a52bba0b-webhook-cert\") pod \"maas-controller-5dfdd7fcb5-684w6\" (UID: \"fa04db79-f0f1-44b5-92a0-5ed5a52bba0b\") " pod="opendatahub/maas-controller-5dfdd7fcb5-684w6"
Jul 06 22:56:21.351562 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:21.351541    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29qf5\" (UniqueName: \"kubernetes.io/projected/fa04db79-f0f1-44b5-92a0-5ed5a52bba0b-kube-api-access-29qf5\") pod \"maas-controller-5dfdd7fcb5-684w6\" (UID: \"fa04db79-f0f1-44b5-92a0-5ed5a52bba0b\") " pod="opendatahub/maas-controller-5dfdd7fcb5-684w6"
Jul 06 22:56:21.500283 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:21.500248    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5dfdd7fcb5-684w6"
Jul 06 22:56:21.626065 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:21.626036    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5dfdd7fcb5-684w6"]
Jul 06 22:56:21.628510 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:56:21.628489    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa04db79_f0f1_44b5_92a0_5ed5a52bba0b.slice/crio-266e2db3fc56542d87fed7dd536df9364b4628e5ea6e2fc3d47d707b4efd61fc WatchSource:0}: Error finding container 266e2db3fc56542d87fed7dd536df9364b4628e5ea6e2fc3d47d707b4efd61fc: Status 404 returned error can't find the container with id 266e2db3fc56542d87fed7dd536df9364b4628e5ea6e2fc3d47d707b4efd61fc
Jul 06 22:56:21.647311 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:21.647290    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/56fced8f-0770-4d9f-9043-fe17088b0a68-webhook-cert\") pod \"maas-controller-7cd9bffc57-2rvf9\" (UID: \"56fced8f-0770-4d9f-9043-fe17088b0a68\") " pod="opendatahub/maas-controller-7cd9bffc57-2rvf9"
Jul 06 22:56:21.649452 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:21.649436    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/56fced8f-0770-4d9f-9043-fe17088b0a68-webhook-cert\") pod \"maas-controller-7cd9bffc57-2rvf9\" (UID: \"56fced8f-0770-4d9f-9043-fe17088b0a68\") " pod="opendatahub/maas-controller-7cd9bffc57-2rvf9"
Jul 06 22:56:21.712678 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:21.712654    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5dfdd7fcb5-684w6" event={"ID":"fa04db79-f0f1-44b5-92a0-5ed5a52bba0b","Type":"ContainerStarted","Data":"266e2db3fc56542d87fed7dd536df9364b4628e5ea6e2fc3d47d707b4efd61fc"}
Jul 06 22:56:21.918014 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:21.917948    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7cd9bffc57-2rvf9"
Jul 06 22:56:22.041546 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:22.041513    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7cd9bffc57-2rvf9"]
Jul 06 22:56:22.044485 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:56:22.044456    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56fced8f_0770_4d9f_9043_fe17088b0a68.slice/crio-395fc1bce9869075a588bf31b788792b9ea206fa181f8e0dd6f20e6f04ba0b1d WatchSource:0}: Error finding container 395fc1bce9869075a588bf31b788792b9ea206fa181f8e0dd6f20e6f04ba0b1d: Status 404 returned error can't find the container with id 395fc1bce9869075a588bf31b788792b9ea206fa181f8e0dd6f20e6f04ba0b1d
Jul 06 22:56:22.721449 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:22.721410    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7cd9bffc57-2rvf9" event={"ID":"56fced8f-0770-4d9f-9043-fe17088b0a68","Type":"ContainerStarted","Data":"395fc1bce9869075a588bf31b788792b9ea206fa181f8e0dd6f20e6f04ba0b1d"}
Jul 06 22:56:24.732338 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:24.732298    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5dfdd7fcb5-684w6" event={"ID":"fa04db79-f0f1-44b5-92a0-5ed5a52bba0b","Type":"ContainerStarted","Data":"5179d00eda97ded7d1ab6fbed9595008b706e659a13b9e6a101904349efc8423"}
Jul 06 22:56:24.732762 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:24.732433    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5dfdd7fcb5-684w6"
Jul 06 22:56:24.733762 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:24.733739    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7cd9bffc57-2rvf9" event={"ID":"56fced8f-0770-4d9f-9043-fe17088b0a68","Type":"ContainerStarted","Data":"245efa3823feea4a29d85fc1730358db1ff998bd27f78862941ebc07503cc44d"}
Jul 06 22:56:24.733840 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:24.733795    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7cd9bffc57-2rvf9"
Jul 06 22:56:24.751964 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:24.751924    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5dfdd7fcb5-684w6" podStartSLOduration=1.228304586 podStartE2EDuration="3.751911413s" totalImagesPullingTime="2.523606827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:56:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:56:24.751153607 +0000 UTC m=+597.155403511" watchObservedRunningTime="2026-07-06 22:56:24.751911413 +0000 UTC m=+597.156161320"
Jul 06 22:56:24.767649 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:24.767610    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7cd9bffc57-2rvf9" podStartSLOduration=2.648560531 podStartE2EDuration="4.767598168s" totalImagesPullingTime="2.119037637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:56:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:56:24.767003809 +0000 UTC m=+597.171253741" watchObservedRunningTime="2026-07-06 22:56:24.767598168 +0000 UTC m=+597.171848073"
Jul 06 22:56:28.026803 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:28.026772    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bvsb_571f407a-249a-4fd9-8c15-53a6541db635/ovn-acl-logging/0.log"
Jul 06 22:56:28.031197 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:28.031153    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bvsb_571f407a-249a-4fd9-8c15-53a6541db635/ovn-acl-logging/0.log"
Jul 06 22:56:35.742831 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:35.742795    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7cd9bffc57-2rvf9"
Jul 06 22:56:35.743310 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:35.742865    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5dfdd7fcb5-684w6"
Jul 06 22:56:35.797666 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:35.797635    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7cd9bffc57-2rvf9"]
Jul 06 22:56:35.797836 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:35.797814    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-7cd9bffc57-2rvf9" podUID="56fced8f-0770-4d9f-9043-fe17088b0a68" containerName="manager" containerID="cri-o://245efa3823feea4a29d85fc1730358db1ff998bd27f78862941ebc07503cc44d" gracePeriod=30
Jul 06 22:56:36.038005 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:36.037984    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7cd9bffc57-2rvf9"
Jul 06 22:56:36.154059 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:36.154017    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/56fced8f-0770-4d9f-9043-fe17088b0a68-webhook-cert\") pod \"56fced8f-0770-4d9f-9043-fe17088b0a68\" (UID: \"56fced8f-0770-4d9f-9043-fe17088b0a68\") "
Jul 06 22:56:36.154254 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:36.154106    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcdvz\" (UniqueName: \"kubernetes.io/projected/56fced8f-0770-4d9f-9043-fe17088b0a68-kube-api-access-zcdvz\") pod \"56fced8f-0770-4d9f-9043-fe17088b0a68\" (UID: \"56fced8f-0770-4d9f-9043-fe17088b0a68\") "
Jul 06 22:56:36.156167 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:36.156123    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56fced8f-0770-4d9f-9043-fe17088b0a68-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "56fced8f-0770-4d9f-9043-fe17088b0a68" (UID: "56fced8f-0770-4d9f-9043-fe17088b0a68"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 22:56:36.156286 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:36.156255    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56fced8f-0770-4d9f-9043-fe17088b0a68-kube-api-access-zcdvz" (OuterVolumeSpecName: "kube-api-access-zcdvz") pod "56fced8f-0770-4d9f-9043-fe17088b0a68" (UID: "56fced8f-0770-4d9f-9043-fe17088b0a68"). InnerVolumeSpecName "kube-api-access-zcdvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 22:56:36.254907 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:36.254881    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcdvz\" (UniqueName: \"kubernetes.io/projected/56fced8f-0770-4d9f-9043-fe17088b0a68-kube-api-access-zcdvz\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 22:56:36.254907 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:36.254907    2569 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/56fced8f-0770-4d9f-9043-fe17088b0a68-webhook-cert\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 22:56:36.788107 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:36.788073    2569 generic.go:356] "Generic (PLEG): container finished" podID="56fced8f-0770-4d9f-9043-fe17088b0a68" containerID="245efa3823feea4a29d85fc1730358db1ff998bd27f78862941ebc07503cc44d" exitCode=0
Jul 06 22:56:36.788609 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:36.788119    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7cd9bffc57-2rvf9" event={"ID":"56fced8f-0770-4d9f-9043-fe17088b0a68","Type":"ContainerDied","Data":"245efa3823feea4a29d85fc1730358db1ff998bd27f78862941ebc07503cc44d"}
Jul 06 22:56:36.788609 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:36.788123    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7cd9bffc57-2rvf9"
Jul 06 22:56:36.788609 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:36.788169    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7cd9bffc57-2rvf9" event={"ID":"56fced8f-0770-4d9f-9043-fe17088b0a68","Type":"ContainerDied","Data":"395fc1bce9869075a588bf31b788792b9ea206fa181f8e0dd6f20e6f04ba0b1d"}
Jul 06 22:56:36.788609 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:36.788214    2569 scope.go:117] "RemoveContainer" containerID="245efa3823feea4a29d85fc1730358db1ff998bd27f78862941ebc07503cc44d"
Jul 06 22:56:36.797641 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:36.797622    2569 scope.go:117] "RemoveContainer" containerID="245efa3823feea4a29d85fc1730358db1ff998bd27f78862941ebc07503cc44d"
Jul 06 22:56:36.797905 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:56:36.797887    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"245efa3823feea4a29d85fc1730358db1ff998bd27f78862941ebc07503cc44d\": container with ID starting with 245efa3823feea4a29d85fc1730358db1ff998bd27f78862941ebc07503cc44d not found: ID does not exist" containerID="245efa3823feea4a29d85fc1730358db1ff998bd27f78862941ebc07503cc44d"
Jul 06 22:56:36.797968 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:36.797919    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"245efa3823feea4a29d85fc1730358db1ff998bd27f78862941ebc07503cc44d"} err="failed to get container status \"245efa3823feea4a29d85fc1730358db1ff998bd27f78862941ebc07503cc44d\": rpc error: code = NotFound desc = could not find container \"245efa3823feea4a29d85fc1730358db1ff998bd27f78862941ebc07503cc44d\": container with ID starting with 245efa3823feea4a29d85fc1730358db1ff998bd27f78862941ebc07503cc44d not found: ID does not exist"
Jul 06 22:56:36.804622 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:36.804603    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7cd9bffc57-2rvf9"]
Jul 06 22:56:36.807969 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:36.807948    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7cd9bffc57-2rvf9"]
Jul 06 22:56:38.169976 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:56:38.169942    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56fced8f-0770-4d9f-9043-fe17088b0a68" path="/var/lib/kubelet/pods/56fced8f-0770-4d9f-9043-fe17088b0a68/volumes"
Jul 06 22:57:00.113324 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:00.113251    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-7cc74d4bd-wz876"]
Jul 06 22:57:00.113724 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:00.113601    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56fced8f-0770-4d9f-9043-fe17088b0a68" containerName="manager"
Jul 06 22:57:00.113724 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:00.113615    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="56fced8f-0770-4d9f-9043-fe17088b0a68" containerName="manager"
Jul 06 22:57:00.113724 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:00.113673    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="56fced8f-0770-4d9f-9043-fe17088b0a68" containerName="manager"
Jul 06 22:57:00.118041 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:00.118025    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-7cc74d4bd-wz876"
Jul 06 22:57:00.120670 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:00.120650    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\""
Jul 06 22:57:00.121227 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:00.121211    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-tjzvc\""
Jul 06 22:57:00.126583 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:00.126564    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-7cc74d4bd-wz876"]
Jul 06 22:57:00.230170 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:00.230141    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f382ed-5d3a-4006-bac8-c39fd893e6a2-openshift-service-ca\") pod \"maas-api-7cc74d4bd-wz876\" (UID: \"11f382ed-5d3a-4006-bac8-c39fd893e6a2\") " pod="opendatahub/maas-api-7cc74d4bd-wz876"
Jul 06 22:57:00.230334 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:00.230204    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f382ed-5d3a-4006-bac8-c39fd893e6a2-maas-api-tls\") pod \"maas-api-7cc74d4bd-wz876\" (UID: \"11f382ed-5d3a-4006-bac8-c39fd893e6a2\") " pod="opendatahub/maas-api-7cc74d4bd-wz876"
Jul 06 22:57:00.230334 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:00.230251    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqzrk\" (UniqueName: \"kubernetes.io/projected/11f382ed-5d3a-4006-bac8-c39fd893e6a2-kube-api-access-xqzrk\") pod \"maas-api-7cc74d4bd-wz876\" (UID: \"11f382ed-5d3a-4006-bac8-c39fd893e6a2\") " pod="opendatahub/maas-api-7cc74d4bd-wz876"
Jul 06 22:57:00.331013 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:00.330958    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f382ed-5d3a-4006-bac8-c39fd893e6a2-openshift-service-ca\") pod \"maas-api-7cc74d4bd-wz876\" (UID: \"11f382ed-5d3a-4006-bac8-c39fd893e6a2\") " pod="opendatahub/maas-api-7cc74d4bd-wz876"
Jul 06 22:57:00.331195 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:00.331024    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f382ed-5d3a-4006-bac8-c39fd893e6a2-maas-api-tls\") pod \"maas-api-7cc74d4bd-wz876\" (UID: \"11f382ed-5d3a-4006-bac8-c39fd893e6a2\") " pod="opendatahub/maas-api-7cc74d4bd-wz876"
Jul 06 22:57:00.331195 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:00.331125    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqzrk\" (UniqueName: \"kubernetes.io/projected/11f382ed-5d3a-4006-bac8-c39fd893e6a2-kube-api-access-xqzrk\") pod \"maas-api-7cc74d4bd-wz876\" (UID: \"11f382ed-5d3a-4006-bac8-c39fd893e6a2\") " pod="opendatahub/maas-api-7cc74d4bd-wz876"
Jul 06 22:57:00.331329 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:57:00.331139    2569 secret.go:189] Couldn't get secret opendatahub/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Jul 06 22:57:00.331329 ip-10-0-134-116 kubenswrapper[2569]: E0706 22:57:00.331287    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11f382ed-5d3a-4006-bac8-c39fd893e6a2-maas-api-tls podName:11f382ed-5d3a-4006-bac8-c39fd893e6a2 nodeName:}" failed. No retries permitted until 2026-07-06 22:57:00.831262138 +0000 UTC m=+633.235512032 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/11f382ed-5d3a-4006-bac8-c39fd893e6a2-maas-api-tls") pod "maas-api-7cc74d4bd-wz876" (UID: "11f382ed-5d3a-4006-bac8-c39fd893e6a2") : secret "maas-api-serving-cert" not found
Jul 06 22:57:00.331660 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:00.331643    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f382ed-5d3a-4006-bac8-c39fd893e6a2-openshift-service-ca\") pod \"maas-api-7cc74d4bd-wz876\" (UID: \"11f382ed-5d3a-4006-bac8-c39fd893e6a2\") " pod="opendatahub/maas-api-7cc74d4bd-wz876"
Jul 06 22:57:00.340493 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:00.340471    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqzrk\" (UniqueName: \"kubernetes.io/projected/11f382ed-5d3a-4006-bac8-c39fd893e6a2-kube-api-access-xqzrk\") pod \"maas-api-7cc74d4bd-wz876\" (UID: \"11f382ed-5d3a-4006-bac8-c39fd893e6a2\") " pod="opendatahub/maas-api-7cc74d4bd-wz876"
Jul 06 22:57:00.581965 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:00.581936    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-qzkbl"]
Jul 06 22:57:00.584389 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:00.584368    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-qzkbl"
Jul 06 22:57:00.586864 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:00.586843    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-hjvmv\""
Jul 06 22:57:00.586943 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:00.586847    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 06 22:57:00.600218 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:00.600197    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-qzkbl"]
Jul 06 22:57:00.625370 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:00.625341    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-jd8zt"]
Jul 06 22:57:00.627510 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:00.627490    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-jd8zt"
Jul 06 22:57:00.638963 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:00.638944    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-jd8zt"]
Jul 06 22:57:00.734589 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:00.734546    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe012d97-8ca1-4fd7-a0d1-afdccd427929-plugins-config-volume\") pod \"payload-pre-processing-7964457696-qzkbl\" (UID: \"fe012d97-8ca1-4fd7-a0d1-afdccd427929\") " pod="openshift-ingress/payload-pre-processing-7964457696-qzkbl"
Jul 06 22:57:00.734760 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:00.734608    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjvxq\" (UniqueName: \"kubernetes.io/projected/02af0679-599a-44a8-b470-a7e480ddf47d-kube-api-access-tjvxq\") pod \"payload-processing-58f86c7798-jd8zt\" (UID: \"02af0679-599a-44a8-b470-a7e480ddf47d\") " pod="openshift-ingress/payload-processing-58f86c7798-jd8zt"
Jul 06 22:57:00.734760 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:00.734682    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg6lp\" (UniqueName: \"kubernetes.io/projected/fe012d97-8ca1-4fd7-a0d1-afdccd427929-kube-api-access-jg6lp\") pod \"payload-pre-processing-7964457696-qzkbl\" (UID: \"fe012d97-8ca1-4fd7-a0d1-afdccd427929\") " pod="openshift-ingress/payload-pre-processing-7964457696-qzkbl"
Jul 06 22:57:00.734840 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:00.734804    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02af0679-599a-44a8-b470-a7e480ddf47d-plugins-config-volume\") pod \"payload-processing-58f86c7798-jd8zt\" (UID: \"02af0679-599a-44a8-b470-a7e480ddf47d\") " pod="openshift-ingress/payload-processing-58f86c7798-jd8zt"
Jul 06 22:57:00.835309 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:00.835230    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02af0679-599a-44a8-b470-a7e480ddf47d-plugins-config-volume\") pod \"payload-processing-58f86c7798-jd8zt\" (UID: \"02af0679-599a-44a8-b470-a7e480ddf47d\") " pod="openshift-ingress/payload-processing-58f86c7798-jd8zt"
Jul 06 22:57:00.835309 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:00.835295    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe012d97-8ca1-4fd7-a0d1-afdccd427929-plugins-config-volume\") pod \"payload-pre-processing-7964457696-qzkbl\" (UID: \"fe012d97-8ca1-4fd7-a0d1-afdccd427929\") " pod="openshift-ingress/payload-pre-processing-7964457696-qzkbl"
Jul 06 22:57:00.835494 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:00.835455    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjvxq\" (UniqueName: \"kubernetes.io/projected/02af0679-599a-44a8-b470-a7e480ddf47d-kube-api-access-tjvxq\") pod \"payload-processing-58f86c7798-jd8zt\" (UID: \"02af0679-599a-44a8-b470-a7e480ddf47d\") " pod="openshift-ingress/payload-processing-58f86c7798-jd8zt"
Jul 06 22:57:00.835535 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:00.835503    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg6lp\" (UniqueName: \"kubernetes.io/projected/fe012d97-8ca1-4fd7-a0d1-afdccd427929-kube-api-access-jg6lp\") pod \"payload-pre-processing-7964457696-qzkbl\" (UID: \"fe012d97-8ca1-4fd7-a0d1-afdccd427929\") " pod="openshift-ingress/payload-pre-processing-7964457696-qzkbl"
Jul 06 22:57:00.835582 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:00.835566    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f382ed-5d3a-4006-bac8-c39fd893e6a2-maas-api-tls\") pod \"maas-api-7cc74d4bd-wz876\" (UID: \"11f382ed-5d3a-4006-bac8-c39fd893e6a2\") " pod="opendatahub/maas-api-7cc74d4bd-wz876"
Jul 06 22:57:00.836172 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:00.836146    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe012d97-8ca1-4fd7-a0d1-afdccd427929-plugins-config-volume\") pod \"payload-pre-processing-7964457696-qzkbl\" (UID: \"fe012d97-8ca1-4fd7-a0d1-afdccd427929\") " pod="openshift-ingress/payload-pre-processing-7964457696-qzkbl"
Jul 06 22:57:00.836310 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:00.836163    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02af0679-599a-44a8-b470-a7e480ddf47d-plugins-config-volume\") pod \"payload-processing-58f86c7798-jd8zt\" (UID: \"02af0679-599a-44a8-b470-a7e480ddf47d\") " pod="openshift-ingress/payload-processing-58f86c7798-jd8zt"
Jul 06 22:57:00.838126 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:00.838102    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11f382ed-5d3a-4006-bac8-c39fd893e6a2-maas-api-tls\") pod \"maas-api-7cc74d4bd-wz876\" (UID: \"11f382ed-5d3a-4006-bac8-c39fd893e6a2\") " pod="opendatahub/maas-api-7cc74d4bd-wz876"
Jul 06 22:57:00.843493 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:00.843466    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg6lp\" (UniqueName: \"kubernetes.io/projected/fe012d97-8ca1-4fd7-a0d1-afdccd427929-kube-api-access-jg6lp\") pod \"payload-pre-processing-7964457696-qzkbl\" (UID: \"fe012d97-8ca1-4fd7-a0d1-afdccd427929\") " pod="openshift-ingress/payload-pre-processing-7964457696-qzkbl"
Jul 06 22:57:00.843613 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:00.843593    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjvxq\" (UniqueName: \"kubernetes.io/projected/02af0679-599a-44a8-b470-a7e480ddf47d-kube-api-access-tjvxq\") pod \"payload-processing-58f86c7798-jd8zt\" (UID: \"02af0679-599a-44a8-b470-a7e480ddf47d\") " pod="openshift-ingress/payload-processing-58f86c7798-jd8zt"
Jul 06 22:57:00.894491 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:00.894470    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-qzkbl"
Jul 06 22:57:00.936170 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:00.936145    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-jd8zt"
Jul 06 22:57:01.028587 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:01.028547    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-7cc74d4bd-wz876"
Jul 06 22:57:01.032492 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:01.032457    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-qzkbl"]
Jul 06 22:57:01.034086 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:57:01.034061    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe012d97_8ca1_4fd7_a0d1_afdccd427929.slice/crio-d9f4cac5f28c1199189ac2802b84b1b2935f1b4fce8de306c7e0bc23d6fa79dd WatchSource:0}: Error finding container d9f4cac5f28c1199189ac2802b84b1b2935f1b4fce8de306c7e0bc23d6fa79dd: Status 404 returned error can't find the container with id d9f4cac5f28c1199189ac2802b84b1b2935f1b4fce8de306c7e0bc23d6fa79dd
Jul 06 22:57:01.087249 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:01.086873    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-jd8zt"]
Jul 06 22:57:01.096488 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:57:01.096408    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02af0679_599a_44a8_b470_a7e480ddf47d.slice/crio-bba29505616755a33170a342aae6fdacdfc38d11ee302ca133e02892e5038a3e WatchSource:0}: Error finding container bba29505616755a33170a342aae6fdacdfc38d11ee302ca133e02892e5038a3e: Status 404 returned error can't find the container with id bba29505616755a33170a342aae6fdacdfc38d11ee302ca133e02892e5038a3e
Jul 06 22:57:01.419703 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:01.419681    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-7cc74d4bd-wz876"]
Jul 06 22:57:01.421576 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:57:01.421552    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11f382ed_5d3a_4006_bac8_c39fd893e6a2.slice/crio-0f417dba15614596591f251e6696fc61631ba17b53be9fb53460baf6af71e072 WatchSource:0}: Error finding container 0f417dba15614596591f251e6696fc61631ba17b53be9fb53460baf6af71e072: Status 404 returned error can't find the container with id 0f417dba15614596591f251e6696fc61631ba17b53be9fb53460baf6af71e072
Jul 06 22:57:01.905817 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:01.905744    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-jd8zt" event={"ID":"02af0679-599a-44a8-b470-a7e480ddf47d","Type":"ContainerStarted","Data":"bba29505616755a33170a342aae6fdacdfc38d11ee302ca133e02892e5038a3e"}
Jul 06 22:57:01.907207 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:01.907126    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-7cc74d4bd-wz876" event={"ID":"11f382ed-5d3a-4006-bac8-c39fd893e6a2","Type":"ContainerStarted","Data":"0f417dba15614596591f251e6696fc61631ba17b53be9fb53460baf6af71e072"}
Jul 06 22:57:01.908563 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:01.908503    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-qzkbl" event={"ID":"fe012d97-8ca1-4fd7-a0d1-afdccd427929","Type":"ContainerStarted","Data":"d9f4cac5f28c1199189ac2802b84b1b2935f1b4fce8de306c7e0bc23d6fa79dd"}
Jul 06 22:57:02.919024 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:02.918981    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-jd8zt" event={"ID":"02af0679-599a-44a8-b470-a7e480ddf47d","Type":"ContainerStarted","Data":"cf5ff919700602e7ded60e4504dec6ff94fcc2d0862964b87ebd7309411a4144"}
Jul 06 22:57:02.919495 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:02.919076    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-jd8zt"
Jul 06 22:57:02.922023 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:02.921996    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-qzkbl" event={"ID":"fe012d97-8ca1-4fd7-a0d1-afdccd427929","Type":"ContainerStarted","Data":"842b589eeb50eb290db5aa4ee462150c4a98e425e17f3e90cf073b0ad1d1b236"}
Jul 06 22:57:02.922432 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:02.922412    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-qzkbl"
Jul 06 22:57:02.934589 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:02.934532    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-jd8zt" podStartSLOduration=1.615582194 podStartE2EDuration="2.934514555s" totalImagesPullingTime="1.318932361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:57:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:57:02.934432813 +0000 UTC m=+635.338682717" watchObservedRunningTime="2026-07-06 22:57:02.934514555 +0000 UTC m=+635.338764441"
Jul 06 22:57:02.949149 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:02.949113    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-qzkbl" podStartSLOduration=1.5646828830000001 podStartE2EDuration="2.949101436s" totalImagesPullingTime="1.384418553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:57:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:57:02.947857759 +0000 UTC m=+635.352107664" watchObservedRunningTime="2026-07-06 22:57:02.949101436 +0000 UTC m=+635.353351342"
Jul 06 22:57:03.929741 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:03.929702    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-7cc74d4bd-wz876" event={"ID":"11f382ed-5d3a-4006-bac8-c39fd893e6a2","Type":"ContainerStarted","Data":"7b06b95233f6ea71a9b3c730dcc14acacc3eb97df2e1e99b16fe011b3a133417"}
Jul 06 22:57:03.930137 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:03.929779    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-7cc74d4bd-wz876"
Jul 06 22:57:03.946963 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:03.946918    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-7cc74d4bd-wz876" podStartSLOduration=1.6036691159999998 podStartE2EDuration="3.946902423s" totalImagesPullingTime="2.343233307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:57:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:57:03.944514725 +0000 UTC m=+636.348764630" watchObservedRunningTime="2026-07-06 22:57:03.946902423 +0000 UTC m=+636.351152326"
Jul 06 22:57:07.526338 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:07.526295    2569 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-tzqx6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 06 22:57:07.526752 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:07.526361    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6" podUID="404a2601-0b38-4fe9-81dc-23ae8fe26c6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 06 22:57:08.955305 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:08.955275    2569 generic.go:356] "Generic (PLEG): container finished" podID="404a2601-0b38-4fe9-81dc-23ae8fe26c6d" containerID="7d06c23e61c346c80eb5e4da277c9096798837f32c3d2912dc70d2548c383a87" exitCode=0
Jul 06 22:57:08.955758 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:08.955362    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6" event={"ID":"404a2601-0b38-4fe9-81dc-23ae8fe26c6d","Type":"ContainerDied","Data":"7d06c23e61c346c80eb5e4da277c9096798837f32c3d2912dc70d2548c383a87"}
Jul 06 22:57:08.955842 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:08.955823    2569 scope.go:117] "RemoveContainer" containerID="7d06c23e61c346c80eb5e4da277c9096798837f32c3d2912dc70d2548c383a87"
Jul 06 22:57:08.956884 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:08.956853    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 22:57:08.956946 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:08.956932    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 22:57:08.956986 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:08.956966    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 22:57:09.942045 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:09.942016    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-7cc74d4bd-wz876"
Jul 06 22:57:09.961958 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:09.961929    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6" event={"ID":"404a2601-0b38-4fe9-81dc-23ae8fe26c6d","Type":"ContainerStarted","Data":"789a1e3264d082af234f3f58a075323ab2787e9a294b5f00bb410f7c946e722e"}
Jul 06 22:57:10.188853 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:10.188129    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6"
Jul 06 22:57:10.188853 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:10.188535    2569 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-tzqx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Jul 06 22:57:10.188853 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:10.188598    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6" podUID="404a2601-0b38-4fe9-81dc-23ae8fe26c6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Jul 06 22:57:11.188457 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:11.188413    2569 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-tzqx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Jul 06 22:57:11.189080 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:11.188492    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6" podUID="404a2601-0b38-4fe9-81dc-23ae8fe26c6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Jul 06 22:57:12.188579 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:12.188540    2569 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-tzqx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Jul 06 22:57:12.189070 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:12.188604    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6" podUID="404a2601-0b38-4fe9-81dc-23ae8fe26c6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Jul 06 22:57:13.188286 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:13.188248    2569 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-tzqx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Jul 06 22:57:13.188559 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:13.188529    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6" podUID="404a2601-0b38-4fe9-81dc-23ae8fe26c6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Jul 06 22:57:13.931635 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:13.931606    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-qzkbl"
Jul 06 22:57:13.932123 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:13.931680    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-jd8zt"
Jul 06 22:57:14.192234 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:14.192142    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6"
Jul 06 22:57:14.192518 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:14.192498    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6"
Jul 06 22:57:14.193167 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:14.193147    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tzqx6"
Jul 06 22:57:16.142219 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:16.142166    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv"]
Jul 06 22:57:16.144703 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:16.144687    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv"
Jul 06 22:57:16.147803 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:16.147783    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xcwzz\""
Jul 06 22:57:16.147803 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:16.147794    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 06 22:57:16.147991 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:16.147833    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 06 22:57:16.148875 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:16.148854    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 06 22:57:16.155887 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:16.155866    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv"]
Jul 06 22:57:16.266078 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:16.266044    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9578aaef-e242-4fe1-a9ca-6f00b96bc360-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv\" (UID: \"9578aaef-e242-4fe1-a9ca-6f00b96bc360\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv"
Jul 06 22:57:16.266259 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:16.266087    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9578aaef-e242-4fe1-a9ca-6f00b96bc360-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv\" (UID: \"9578aaef-e242-4fe1-a9ca-6f00b96bc360\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv"
Jul 06 22:57:16.266259 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:16.266128    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9578aaef-e242-4fe1-a9ca-6f00b96bc360-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv\" (UID: \"9578aaef-e242-4fe1-a9ca-6f00b96bc360\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv"
Jul 06 22:57:16.266358 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:16.266264    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9578aaef-e242-4fe1-a9ca-6f00b96bc360-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv\" (UID: \"9578aaef-e242-4fe1-a9ca-6f00b96bc360\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv"
Jul 06 22:57:16.266358 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:16.266323    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9578aaef-e242-4fe1-a9ca-6f00b96bc360-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv\" (UID: \"9578aaef-e242-4fe1-a9ca-6f00b96bc360\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv"
Jul 06 22:57:16.266424 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:16.266360    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdzrt\" (UniqueName: \"kubernetes.io/projected/9578aaef-e242-4fe1-a9ca-6f00b96bc360-kube-api-access-jdzrt\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv\" (UID: \"9578aaef-e242-4fe1-a9ca-6f00b96bc360\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv"
Jul 06 22:57:16.367039 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:16.367006    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9578aaef-e242-4fe1-a9ca-6f00b96bc360-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv\" (UID: \"9578aaef-e242-4fe1-a9ca-6f00b96bc360\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv"
Jul 06 22:57:16.367217 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:16.367047    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdzrt\" (UniqueName: \"kubernetes.io/projected/9578aaef-e242-4fe1-a9ca-6f00b96bc360-kube-api-access-jdzrt\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv\" (UID: \"9578aaef-e242-4fe1-a9ca-6f00b96bc360\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv"
Jul 06 22:57:16.367217 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:16.367087    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9578aaef-e242-4fe1-a9ca-6f00b96bc360-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv\" (UID: \"9578aaef-e242-4fe1-a9ca-6f00b96bc360\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv"
Jul 06 22:57:16.367217 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:16.367123    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9578aaef-e242-4fe1-a9ca-6f00b96bc360-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv\" (UID: \"9578aaef-e242-4fe1-a9ca-6f00b96bc360\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv"
Jul 06 22:57:16.367217 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:16.367152    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9578aaef-e242-4fe1-a9ca-6f00b96bc360-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv\" (UID: \"9578aaef-e242-4fe1-a9ca-6f00b96bc360\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv"
Jul 06 22:57:16.367425 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:16.367221    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9578aaef-e242-4fe1-a9ca-6f00b96bc360-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv\" (UID: \"9578aaef-e242-4fe1-a9ca-6f00b96bc360\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv"
Jul 06 22:57:16.367596 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:16.367579    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9578aaef-e242-4fe1-a9ca-6f00b96bc360-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv\" (UID: \"9578aaef-e242-4fe1-a9ca-6f00b96bc360\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv"
Jul 06 22:57:16.367662 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:16.367636    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9578aaef-e242-4fe1-a9ca-6f00b96bc360-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv\" (UID: \"9578aaef-e242-4fe1-a9ca-6f00b96bc360\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv"
Jul 06 22:57:16.367710 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:16.367656    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9578aaef-e242-4fe1-a9ca-6f00b96bc360-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv\" (UID: \"9578aaef-e242-4fe1-a9ca-6f00b96bc360\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv"
Jul 06 22:57:16.369491 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:16.369461    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9578aaef-e242-4fe1-a9ca-6f00b96bc360-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv\" (UID: \"9578aaef-e242-4fe1-a9ca-6f00b96bc360\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv"
Jul 06 22:57:16.369677 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:16.369661    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9578aaef-e242-4fe1-a9ca-6f00b96bc360-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv\" (UID: \"9578aaef-e242-4fe1-a9ca-6f00b96bc360\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv"
Jul 06 22:57:16.375147 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:16.375119    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdzrt\" (UniqueName: \"kubernetes.io/projected/9578aaef-e242-4fe1-a9ca-6f00b96bc360-kube-api-access-jdzrt\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv\" (UID: \"9578aaef-e242-4fe1-a9ca-6f00b96bc360\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv"
Jul 06 22:57:16.455717 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:16.455680    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv"
Jul 06 22:57:16.585782 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:16.585759    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv"]
Jul 06 22:57:16.587551 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:57:16.587525    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9578aaef_e242_4fe1_a9ca_6f00b96bc360.slice/crio-83665463f0a03d075195cae7e0923a5835ec80c28185c21c055f0236b0ac3f71 WatchSource:0}: Error finding container 83665463f0a03d075195cae7e0923a5835ec80c28185c21c055f0236b0ac3f71: Status 404 returned error can't find the container with id 83665463f0a03d075195cae7e0923a5835ec80c28185c21c055f0236b0ac3f71
Jul 06 22:57:16.588906 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:16.588881    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 22:57:17.002385 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:17.002340    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv" event={"ID":"9578aaef-e242-4fe1-a9ca-6f00b96bc360","Type":"ContainerStarted","Data":"83665463f0a03d075195cae7e0923a5835ec80c28185c21c055f0236b0ac3f71"}
Jul 06 22:57:18.008447 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:18.008409    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv" event={"ID":"9578aaef-e242-4fe1-a9ca-6f00b96bc360","Type":"ContainerStarted","Data":"04763bda52543230cc1ab1d8b3ecca79e80114bc8958b2d4531525e3adb7377a"}
Jul 06 22:57:18.028316 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:18.028268    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv" podStartSLOduration=0.960788075 podStartE2EDuration="2.028253112s" totalImagesPullingTime="1.067465037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:57:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:57:18.026754968 +0000 UTC m=+650.431004884" watchObservedRunningTime="2026-07-06 22:57:18.028253112 +0000 UTC m=+650.432503031"
Jul 06 22:57:26.456774 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:26.456746    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv"
Jul 06 22:57:26.457259 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:26.456807    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv"
Jul 06 22:57:26.469837 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:26.469812    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv"
Jul 06 22:57:27.061325 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:27.061295    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv"
Jul 06 22:57:37.728055 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:37.728024    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-7lc2v"]
Jul 06 22:57:37.740461 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:37.740431    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lc2v"
Jul 06 22:57:37.742336 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:37.742282    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-7lc2v"]
Jul 06 22:57:37.743023 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:37.743001    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 06 22:57:37.926139 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:37.926106    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1f8b9a7-86fa-4557-a90e-f3c2cf5a879b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7lc2v\" (UID: \"b1f8b9a7-86fa-4557-a90e-f3c2cf5a879b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lc2v"
Jul 06 22:57:37.926313 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:37.926140    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1f8b9a7-86fa-4557-a90e-f3c2cf5a879b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7lc2v\" (UID: \"b1f8b9a7-86fa-4557-a90e-f3c2cf5a879b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lc2v"
Jul 06 22:57:37.926313 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:37.926210    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1f8b9a7-86fa-4557-a90e-f3c2cf5a879b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7lc2v\" (UID: \"b1f8b9a7-86fa-4557-a90e-f3c2cf5a879b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lc2v"
Jul 06 22:57:37.926313 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:37.926274    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1f8b9a7-86fa-4557-a90e-f3c2cf5a879b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7lc2v\" (UID: \"b1f8b9a7-86fa-4557-a90e-f3c2cf5a879b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lc2v"
Jul 06 22:57:37.926313 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:37.926295    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1f8b9a7-86fa-4557-a90e-f3c2cf5a879b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7lc2v\" (UID: \"b1f8b9a7-86fa-4557-a90e-f3c2cf5a879b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lc2v"
Jul 06 22:57:37.926541 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:37.926325    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfnsg\" (UniqueName: \"kubernetes.io/projected/b1f8b9a7-86fa-4557-a90e-f3c2cf5a879b-kube-api-access-tfnsg\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7lc2v\" (UID: \"b1f8b9a7-86fa-4557-a90e-f3c2cf5a879b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lc2v"
Jul 06 22:57:38.027631 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:38.027561    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfnsg\" (UniqueName: \"kubernetes.io/projected/b1f8b9a7-86fa-4557-a90e-f3c2cf5a879b-kube-api-access-tfnsg\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7lc2v\" (UID: \"b1f8b9a7-86fa-4557-a90e-f3c2cf5a879b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lc2v"
Jul 06 22:57:38.027631 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:38.027620    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1f8b9a7-86fa-4557-a90e-f3c2cf5a879b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7lc2v\" (UID: \"b1f8b9a7-86fa-4557-a90e-f3c2cf5a879b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lc2v"
Jul 06 22:57:38.027800 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:38.027641    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1f8b9a7-86fa-4557-a90e-f3c2cf5a879b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7lc2v\" (UID: \"b1f8b9a7-86fa-4557-a90e-f3c2cf5a879b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lc2v"
Jul 06 22:57:38.027800 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:38.027671    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1f8b9a7-86fa-4557-a90e-f3c2cf5a879b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7lc2v\" (UID: \"b1f8b9a7-86fa-4557-a90e-f3c2cf5a879b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lc2v"
Jul 06 22:57:38.027800 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:38.027695    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1f8b9a7-86fa-4557-a90e-f3c2cf5a879b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7lc2v\" (UID: \"b1f8b9a7-86fa-4557-a90e-f3c2cf5a879b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lc2v"
Jul 06 22:57:38.027800 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:38.027711    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1f8b9a7-86fa-4557-a90e-f3c2cf5a879b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7lc2v\" (UID: \"b1f8b9a7-86fa-4557-a90e-f3c2cf5a879b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lc2v"
Jul 06 22:57:38.028105 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:38.028082    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1f8b9a7-86fa-4557-a90e-f3c2cf5a879b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7lc2v\" (UID: \"b1f8b9a7-86fa-4557-a90e-f3c2cf5a879b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lc2v"
Jul 06 22:57:38.028170 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:38.028095    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1f8b9a7-86fa-4557-a90e-f3c2cf5a879b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7lc2v\" (UID: \"b1f8b9a7-86fa-4557-a90e-f3c2cf5a879b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lc2v"
Jul 06 22:57:38.028170 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:38.028159    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1f8b9a7-86fa-4557-a90e-f3c2cf5a879b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7lc2v\" (UID: \"b1f8b9a7-86fa-4557-a90e-f3c2cf5a879b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lc2v"
Jul 06 22:57:38.030034 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:38.030009    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1f8b9a7-86fa-4557-a90e-f3c2cf5a879b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7lc2v\" (UID: \"b1f8b9a7-86fa-4557-a90e-f3c2cf5a879b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lc2v"
Jul 06 22:57:38.030286 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:38.030266    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1f8b9a7-86fa-4557-a90e-f3c2cf5a879b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7lc2v\" (UID: \"b1f8b9a7-86fa-4557-a90e-f3c2cf5a879b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lc2v"
Jul 06 22:57:38.035781 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:38.035762    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfnsg\" (UniqueName: \"kubernetes.io/projected/b1f8b9a7-86fa-4557-a90e-f3c2cf5a879b-kube-api-access-tfnsg\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7lc2v\" (UID: \"b1f8b9a7-86fa-4557-a90e-f3c2cf5a879b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lc2v"
Jul 06 22:57:38.052641 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:38.052617    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lc2v"
Jul 06 22:57:38.186332 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:38.186309    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-7lc2v"]
Jul 06 22:57:39.106998 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:39.106963    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lc2v" event={"ID":"b1f8b9a7-86fa-4557-a90e-f3c2cf5a879b","Type":"ContainerStarted","Data":"b1fcf2fb7a303d13035d94fd120caf5e25798c091096f22da81faabf369b1757"}
Jul 06 22:57:39.107376 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:39.107014    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lc2v" event={"ID":"b1f8b9a7-86fa-4557-a90e-f3c2cf5a879b","Type":"ContainerStarted","Data":"688824d72b55eafc44ffd825d90518dc19d835447ffd5573bfc8ab5b52d0011b"}
Jul 06 22:57:39.128684 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:39.128633    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lc2v" podStartSLOduration=1.975112669 podStartE2EDuration="2.12861952s" totalImagesPullingTime="153.506851ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:57:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:57:39.127627793 +0000 UTC m=+671.531877716" watchObservedRunningTime="2026-07-06 22:57:39.12861952 +0000 UTC m=+671.532869423"
Jul 06 22:57:48.053490 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:48.053455    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lc2v"
Jul 06 22:57:48.053879 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:48.053515    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lc2v"
Jul 06 22:57:48.069114 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:48.069085    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lc2v"
Jul 06 22:57:48.158985 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:57:48.158952    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7lc2v"
Jul 06 22:59:56.816081 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:59:56.816007    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d66d76884-trqqr"]
Jul 06 22:59:56.818368 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:59:56.818351    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d66d76884-trqqr"
Jul 06 22:59:56.822157 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:59:56.822137    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-xwbh5\""
Jul 06 22:59:56.822288 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:59:56.822210    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 06 22:59:56.827841 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:59:56.827819    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d66d76884-trqqr"]
Jul 06 22:59:56.899095 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:59:56.899071    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqk8q\" (UniqueName: \"kubernetes.io/projected/3c96c19d-0c87-49e1-a550-361c2ce200d9-kube-api-access-xqk8q\") pod \"authorino-6d66d76884-trqqr\" (UID: \"3c96c19d-0c87-49e1-a550-361c2ce200d9\") " pod="kuadrant-system/authorino-6d66d76884-trqqr"
Jul 06 22:59:56.899238 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:59:56.899117    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3c96c19d-0c87-49e1-a550-361c2ce200d9-tls-cert\") pod \"authorino-6d66d76884-trqqr\" (UID: \"3c96c19d-0c87-49e1-a550-361c2ce200d9\") " pod="kuadrant-system/authorino-6d66d76884-trqqr"
Jul 06 22:59:57.000149 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:59:57.000126    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3c96c19d-0c87-49e1-a550-361c2ce200d9-tls-cert\") pod \"authorino-6d66d76884-trqqr\" (UID: \"3c96c19d-0c87-49e1-a550-361c2ce200d9\") " pod="kuadrant-system/authorino-6d66d76884-trqqr"
Jul 06 22:59:57.000276 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:59:57.000231    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqk8q\" (UniqueName: \"kubernetes.io/projected/3c96c19d-0c87-49e1-a550-361c2ce200d9-kube-api-access-xqk8q\") pod \"authorino-6d66d76884-trqqr\" (UID: \"3c96c19d-0c87-49e1-a550-361c2ce200d9\") " pod="kuadrant-system/authorino-6d66d76884-trqqr"
Jul 06 22:59:57.002417 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:59:57.002392    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3c96c19d-0c87-49e1-a550-361c2ce200d9-tls-cert\") pod \"authorino-6d66d76884-trqqr\" (UID: \"3c96c19d-0c87-49e1-a550-361c2ce200d9\") " pod="kuadrant-system/authorino-6d66d76884-trqqr"
Jul 06 22:59:57.008059 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:59:57.008035    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqk8q\" (UniqueName: \"kubernetes.io/projected/3c96c19d-0c87-49e1-a550-361c2ce200d9-kube-api-access-xqk8q\") pod \"authorino-6d66d76884-trqqr\" (UID: \"3c96c19d-0c87-49e1-a550-361c2ce200d9\") " pod="kuadrant-system/authorino-6d66d76884-trqqr"
Jul 06 22:59:57.127669 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:59:57.127618    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d66d76884-trqqr"
Jul 06 22:59:57.253744 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:59:57.253720    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d66d76884-trqqr"]
Jul 06 22:59:57.255989 ip-10-0-134-116 kubenswrapper[2569]: W0706 22:59:57.255966    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c96c19d_0c87_49e1_a550_361c2ce200d9.slice/crio-435697110fb30d9bdd279bbb12df73841fb567de74f7f08c7668c40770d3deba WatchSource:0}: Error finding container 435697110fb30d9bdd279bbb12df73841fb567de74f7f08c7668c40770d3deba: Status 404 returned error can't find the container with id 435697110fb30d9bdd279bbb12df73841fb567de74f7f08c7668c40770d3deba
Jul 06 22:59:57.735326 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:59:57.735295    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d66d76884-trqqr" event={"ID":"3c96c19d-0c87-49e1-a550-361c2ce200d9","Type":"ContainerStarted","Data":"4425298f908c8af1b42538016c6e8aaff5ca2ebc168c29eca1e1a49eb5d3aa04"}
Jul 06 22:59:57.735326 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:59:57.735336    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d66d76884-trqqr" event={"ID":"3c96c19d-0c87-49e1-a550-361c2ce200d9","Type":"ContainerStarted","Data":"435697110fb30d9bdd279bbb12df73841fb567de74f7f08c7668c40770d3deba"}
Jul 06 22:59:57.749690 ip-10-0-134-116 kubenswrapper[2569]: I0706 22:59:57.749645    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d66d76884-trqqr" podStartSLOduration=1.436321397 podStartE2EDuration="1.749630849s" totalImagesPullingTime="313.309452ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:59:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:59:57.749033829 +0000 UTC m=+810.153283732" watchObservedRunningTime="2026-07-06 22:59:57.749630849 +0000 UTC m=+810.153880753"
Jul 06 23:01:09.226432 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:01:09.226401    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5dfdd7fcb5-684w6"]
Jul 06 23:01:09.227030 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:01:09.226686    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-5dfdd7fcb5-684w6" podUID="fa04db79-f0f1-44b5-92a0-5ed5a52bba0b" containerName="manager" containerID="cri-o://5179d00eda97ded7d1ab6fbed9595008b706e659a13b9e6a101904349efc8423" gracePeriod=30
Jul 06 23:01:09.470594 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:01:09.470571    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5dfdd7fcb5-684w6"
Jul 06 23:01:09.592650 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:01:09.592573    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29qf5\" (UniqueName: \"kubernetes.io/projected/fa04db79-f0f1-44b5-92a0-5ed5a52bba0b-kube-api-access-29qf5\") pod \"fa04db79-f0f1-44b5-92a0-5ed5a52bba0b\" (UID: \"fa04db79-f0f1-44b5-92a0-5ed5a52bba0b\") "
Jul 06 23:01:09.592778 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:01:09.592673    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fa04db79-f0f1-44b5-92a0-5ed5a52bba0b-webhook-cert\") pod \"fa04db79-f0f1-44b5-92a0-5ed5a52bba0b\" (UID: \"fa04db79-f0f1-44b5-92a0-5ed5a52bba0b\") "
Jul 06 23:01:09.594668 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:01:09.594635    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa04db79-f0f1-44b5-92a0-5ed5a52bba0b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "fa04db79-f0f1-44b5-92a0-5ed5a52bba0b" (UID: "fa04db79-f0f1-44b5-92a0-5ed5a52bba0b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 23:01:09.594788 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:01:09.594724    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa04db79-f0f1-44b5-92a0-5ed5a52bba0b-kube-api-access-29qf5" (OuterVolumeSpecName: "kube-api-access-29qf5") pod "fa04db79-f0f1-44b5-92a0-5ed5a52bba0b" (UID: "fa04db79-f0f1-44b5-92a0-5ed5a52bba0b"). InnerVolumeSpecName "kube-api-access-29qf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 23:01:09.693549 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:01:09.693525    2569 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fa04db79-f0f1-44b5-92a0-5ed5a52bba0b-webhook-cert\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:01:09.693549 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:01:09.693546    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29qf5\" (UniqueName: \"kubernetes.io/projected/fa04db79-f0f1-44b5-92a0-5ed5a52bba0b-kube-api-access-29qf5\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:01:10.096272 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:01:10.096220    2569 generic.go:356] "Generic (PLEG): container finished" podID="fa04db79-f0f1-44b5-92a0-5ed5a52bba0b" containerID="5179d00eda97ded7d1ab6fbed9595008b706e659a13b9e6a101904349efc8423" exitCode=0
Jul 06 23:01:10.096420 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:01:10.096282    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5dfdd7fcb5-684w6"
Jul 06 23:01:10.096420 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:01:10.096295    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5dfdd7fcb5-684w6" event={"ID":"fa04db79-f0f1-44b5-92a0-5ed5a52bba0b","Type":"ContainerDied","Data":"5179d00eda97ded7d1ab6fbed9595008b706e659a13b9e6a101904349efc8423"}
Jul 06 23:01:10.096420 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:01:10.096347    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5dfdd7fcb5-684w6" event={"ID":"fa04db79-f0f1-44b5-92a0-5ed5a52bba0b","Type":"ContainerDied","Data":"266e2db3fc56542d87fed7dd536df9364b4628e5ea6e2fc3d47d707b4efd61fc"}
Jul 06 23:01:10.096420 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:01:10.096370    2569 scope.go:117] "RemoveContainer" containerID="5179d00eda97ded7d1ab6fbed9595008b706e659a13b9e6a101904349efc8423"
Jul 06 23:01:10.104947 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:01:10.104934    2569 scope.go:117] "RemoveContainer" containerID="5179d00eda97ded7d1ab6fbed9595008b706e659a13b9e6a101904349efc8423"
Jul 06 23:01:10.105258 ip-10-0-134-116 kubenswrapper[2569]: E0706 23:01:10.105240    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5179d00eda97ded7d1ab6fbed9595008b706e659a13b9e6a101904349efc8423\": container with ID starting with 5179d00eda97ded7d1ab6fbed9595008b706e659a13b9e6a101904349efc8423 not found: ID does not exist" containerID="5179d00eda97ded7d1ab6fbed9595008b706e659a13b9e6a101904349efc8423"
Jul 06 23:01:10.105310 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:01:10.105267    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5179d00eda97ded7d1ab6fbed9595008b706e659a13b9e6a101904349efc8423"} err="failed to get container status \"5179d00eda97ded7d1ab6fbed9595008b706e659a13b9e6a101904349efc8423\": rpc error: code = NotFound desc = could not find container \"5179d00eda97ded7d1ab6fbed9595008b706e659a13b9e6a101904349efc8423\": container with ID starting with 5179d00eda97ded7d1ab6fbed9595008b706e659a13b9e6a101904349efc8423 not found: ID does not exist"
Jul 06 23:01:10.116879 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:01:10.116860    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5dfdd7fcb5-684w6"]
Jul 06 23:01:10.120668 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:01:10.120649    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5dfdd7fcb5-684w6"]
Jul 06 23:01:10.170266 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:01:10.170247    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa04db79-f0f1-44b5-92a0-5ed5a52bba0b" path="/var/lib/kubelet/pods/fa04db79-f0f1-44b5-92a0-5ed5a52bba0b/volumes"
Jul 06 23:01:10.863591 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:01:10.863561    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5dfdd7fcb5-m6mtz"]
Jul 06 23:01:10.863959 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:01:10.863927    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa04db79-f0f1-44b5-92a0-5ed5a52bba0b" containerName="manager"
Jul 06 23:01:10.863959 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:01:10.863938    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa04db79-f0f1-44b5-92a0-5ed5a52bba0b" containerName="manager"
Jul 06 23:01:10.864029 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:01:10.863998    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa04db79-f0f1-44b5-92a0-5ed5a52bba0b" containerName="manager"
Jul 06 23:01:10.865991 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:01:10.865971    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5dfdd7fcb5-m6mtz"
Jul 06 23:01:10.869884 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:01:10.869862    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 06 23:01:10.869991 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:01:10.869868    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 06 23:01:10.870046 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:01:10.869868    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8zknt\""
Jul 06 23:01:10.876976 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:01:10.876945    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5dfdd7fcb5-m6mtz"]
Jul 06 23:01:11.004867 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:01:11.004842    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2xd5\" (UniqueName: \"kubernetes.io/projected/02009b0e-a948-4de4-9eef-9a963f017c5a-kube-api-access-n2xd5\") pod \"maas-controller-5dfdd7fcb5-m6mtz\" (UID: \"02009b0e-a948-4de4-9eef-9a963f017c5a\") " pod="opendatahub/maas-controller-5dfdd7fcb5-m6mtz"
Jul 06 23:01:11.004972 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:01:11.004876    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/02009b0e-a948-4de4-9eef-9a963f017c5a-webhook-cert\") pod \"maas-controller-5dfdd7fcb5-m6mtz\" (UID: \"02009b0e-a948-4de4-9eef-9a963f017c5a\") " pod="opendatahub/maas-controller-5dfdd7fcb5-m6mtz"
Jul 06 23:01:11.105514 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:01:11.105486    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2xd5\" (UniqueName: \"kubernetes.io/projected/02009b0e-a948-4de4-9eef-9a963f017c5a-kube-api-access-n2xd5\") pod \"maas-controller-5dfdd7fcb5-m6mtz\" (UID: \"02009b0e-a948-4de4-9eef-9a963f017c5a\") " pod="opendatahub/maas-controller-5dfdd7fcb5-m6mtz"
Jul 06 23:01:11.105627 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:01:11.105521    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/02009b0e-a948-4de4-9eef-9a963f017c5a-webhook-cert\") pod \"maas-controller-5dfdd7fcb5-m6mtz\" (UID: \"02009b0e-a948-4de4-9eef-9a963f017c5a\") " pod="opendatahub/maas-controller-5dfdd7fcb5-m6mtz"
Jul 06 23:01:11.107806 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:01:11.107785    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/02009b0e-a948-4de4-9eef-9a963f017c5a-webhook-cert\") pod \"maas-controller-5dfdd7fcb5-m6mtz\" (UID: \"02009b0e-a948-4de4-9eef-9a963f017c5a\") " pod="opendatahub/maas-controller-5dfdd7fcb5-m6mtz"
Jul 06 23:01:11.113447 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:01:11.113419    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2xd5\" (UniqueName: \"kubernetes.io/projected/02009b0e-a948-4de4-9eef-9a963f017c5a-kube-api-access-n2xd5\") pod \"maas-controller-5dfdd7fcb5-m6mtz\" (UID: \"02009b0e-a948-4de4-9eef-9a963f017c5a\") " pod="opendatahub/maas-controller-5dfdd7fcb5-m6mtz"
Jul 06 23:01:11.176462 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:01:11.176409    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5dfdd7fcb5-m6mtz"
Jul 06 23:01:11.305736 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:01:11.305709    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5dfdd7fcb5-m6mtz"]
Jul 06 23:01:11.307825 ip-10-0-134-116 kubenswrapper[2569]: W0706 23:01:11.307803    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02009b0e_a948_4de4_9eef_9a963f017c5a.slice/crio-c8ebf9310a71fb0bcbfffd547050e82a044b8be7e72880bfcbfa4d6e79948296 WatchSource:0}: Error finding container c8ebf9310a71fb0bcbfffd547050e82a044b8be7e72880bfcbfa4d6e79948296: Status 404 returned error can't find the container with id c8ebf9310a71fb0bcbfffd547050e82a044b8be7e72880bfcbfa4d6e79948296
Jul 06 23:01:12.108481 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:01:12.108396    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5dfdd7fcb5-m6mtz" event={"ID":"02009b0e-a948-4de4-9eef-9a963f017c5a","Type":"ContainerStarted","Data":"5a51d7bba175bb3043984bbe11def0c0a99ab80c19d26190a98cdecfb3a7c3b3"}
Jul 06 23:01:12.108843 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:01:12.108496    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5dfdd7fcb5-m6mtz" event={"ID":"02009b0e-a948-4de4-9eef-9a963f017c5a","Type":"ContainerStarted","Data":"c8ebf9310a71fb0bcbfffd547050e82a044b8be7e72880bfcbfa4d6e79948296"}
Jul 06 23:01:12.108843 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:01:12.108529    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5dfdd7fcb5-m6mtz"
Jul 06 23:01:12.128336 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:01:12.128291    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5dfdd7fcb5-m6mtz" podStartSLOduration=1.7141267980000001 podStartE2EDuration="2.128276254s" totalImagesPullingTime="414.149456ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:01:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:01:12.126668178 +0000 UTC m=+884.530918107" watchObservedRunningTime="2026-07-06 23:01:12.128276254 +0000 UTC m=+884.532526158"
Jul 06 23:01:23.121443 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:01:23.121358    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5dfdd7fcb5-m6mtz"
Jul 06 23:01:28.059530 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:01:28.059497    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bvsb_571f407a-249a-4fd9-8c15-53a6541db635/ovn-acl-logging/0.log"
Jul 06 23:01:28.061719 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:01:28.061699    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bvsb_571f407a-249a-4fd9-8c15-53a6541db635/ovn-acl-logging/0.log"
Jul 06 23:06:28.089677 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:06:28.089649    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bvsb_571f407a-249a-4fd9-8c15-53a6541db635/ovn-acl-logging/0.log"
Jul 06 23:06:28.090213 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:06:28.089653    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bvsb_571f407a-249a-4fd9-8c15-53a6541db635/ovn-acl-logging/0.log"
Jul 06 23:11:28.116789 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:11:28.116763    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bvsb_571f407a-249a-4fd9-8c15-53a6541db635/ovn-acl-logging/0.log"
Jul 06 23:11:28.119271 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:11:28.118871    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bvsb_571f407a-249a-4fd9-8c15-53a6541db635/ovn-acl-logging/0.log"
Jul 06 23:13:05.477736 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:05.477699    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g"]
Jul 06 23:13:05.481453 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:05.481431    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g"
Jul 06 23:13:05.484289 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:05.484269    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0c98d6bc-openshift-default-dockercfg-5rg4q\""
Jul 06 23:13:05.505312 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:05.505287    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g"]
Jul 06 23:13:05.557267 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:05.557242    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pndv8\" (UniqueName: \"kubernetes.io/projected/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-kube-api-access-pndv8\") pod \"e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g\" (UID: \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\") " pod="openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g"
Jul 06 23:13:05.557389 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:05.557293    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-workload-socket\") pod \"e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g\" (UID: \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\") " pod="openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g"
Jul 06 23:13:05.557389 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:05.557338    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-credential-socket\") pod \"e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g\" (UID: \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\") " pod="openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g"
Jul 06 23:13:05.557389 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:05.557357    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-istio-token\") pod \"e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g\" (UID: \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\") " pod="openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g"
Jul 06 23:13:05.557389 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:05.557373    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-istiod-ca-cert\") pod \"e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g\" (UID: \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\") " pod="openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g"
Jul 06 23:13:05.557525 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:05.557459    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-workload-certs\") pod \"e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g\" (UID: \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\") " pod="openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g"
Jul 06 23:13:05.557525 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:05.557488    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-istio-podinfo\") pod \"e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g\" (UID: \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\") " pod="openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g"
Jul 06 23:13:05.557525 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:05.557505    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-istio-envoy\") pod \"e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g\" (UID: \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\") " pod="openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g"
Jul 06 23:13:05.557618 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:05.557587    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-istio-data\") pod \"e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g\" (UID: \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\") " pod="openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g"
Jul 06 23:13:05.658094 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:05.658066    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-workload-certs\") pod \"e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g\" (UID: \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\") " pod="openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g"
Jul 06 23:13:05.658267 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:05.658097    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-istio-podinfo\") pod \"e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g\" (UID: \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\") " pod="openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g"
Jul 06 23:13:05.658267 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:05.658119    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-istio-envoy\") pod \"e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g\" (UID: \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\") " pod="openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g"
Jul 06 23:13:05.658267 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:05.658144    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-istio-data\") pod \"e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g\" (UID: \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\") " pod="openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g"
Jul 06 23:13:05.658267 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:05.658215    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pndv8\" (UniqueName: \"kubernetes.io/projected/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-kube-api-access-pndv8\") pod \"e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g\" (UID: \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\") " pod="openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g"
Jul 06 23:13:05.658267 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:05.658261    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-workload-socket\") pod \"e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g\" (UID: \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\") " pod="openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g"
Jul 06 23:13:05.658523 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:05.658292    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-credential-socket\") pod \"e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g\" (UID: \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\") " pod="openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g"
Jul 06 23:13:05.658523 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:05.658319    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-istio-token\") pod \"e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g\" (UID: \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\") " pod="openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g"
Jul 06 23:13:05.658523 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:05.658341    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-istiod-ca-cert\") pod \"e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g\" (UID: \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\") " pod="openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g"
Jul 06 23:13:05.658676 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:05.658653    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-istio-data\") pod \"e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g\" (UID: \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\") " pod="openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g"
Jul 06 23:13:05.658735 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:05.658694    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-credential-socket\") pod \"e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g\" (UID: \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\") " pod="openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g"
Jul 06 23:13:05.658814 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:05.658791    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-workload-socket\") pod \"e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g\" (UID: \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\") " pod="openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g"
Jul 06 23:13:05.658875 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:05.658858    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-workload-certs\") pod \"e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g\" (UID: \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\") " pod="openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g"
Jul 06 23:13:05.659078 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:05.659062    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-istiod-ca-cert\") pod \"e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g\" (UID: \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\") " pod="openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g"
Jul 06 23:13:05.660459 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:05.660436    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-istio-envoy\") pod \"e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g\" (UID: \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\") " pod="openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g"
Jul 06 23:13:05.661235 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:05.661209    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-istio-podinfo\") pod \"e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g\" (UID: \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\") " pod="openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g"
Jul 06 23:13:05.668001 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:05.667116    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-istio-token\") pod \"e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g\" (UID: \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\") " pod="openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g"
Jul 06 23:13:05.669029 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:05.669008    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pndv8\" (UniqueName: \"kubernetes.io/projected/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-kube-api-access-pndv8\") pod \"e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g\" (UID: \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\") " pod="openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g"
Jul 06 23:13:05.792227 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:05.792117    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g"
Jul 06 23:13:05.925373 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:05.925141    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g"]
Jul 06 23:13:05.927716 ip-10-0-134-116 kubenswrapper[2569]: W0706 23:13:05.927691    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod669166c8_88ee_4c33_8bf4_f1e6fb04ce2a.slice/crio-5559209a4aa730dd0b38d755fb7b41b015de0cfa0e6f4aa424156f867bddc6c3 WatchSource:0}: Error finding container 5559209a4aa730dd0b38d755fb7b41b015de0cfa0e6f4aa424156f867bddc6c3: Status 404 returned error can't find the container with id 5559209a4aa730dd0b38d755fb7b41b015de0cfa0e6f4aa424156f867bddc6c3
Jul 06 23:13:05.929721 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:05.929685    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 23:13:05.929790 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:05.929750    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 23:13:05.929790 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:05.929777    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 23:13:06.520234 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:06.520173    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g" event={"ID":"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a","Type":"ContainerStarted","Data":"bd9506819ef921d1417ba3069559aeb1cb4f8e0d2afd179932c227392647cca7"}
Jul 06 23:13:06.520656 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:06.520253    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g" event={"ID":"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a","Type":"ContainerStarted","Data":"5559209a4aa730dd0b38d755fb7b41b015de0cfa0e6f4aa424156f867bddc6c3"}
Jul 06 23:13:06.537565 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:06.537515    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g" podStartSLOduration=1.537497241 podStartE2EDuration="1.537497241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:13:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:13:06.53748205 +0000 UTC m=+1598.941731955" watchObservedRunningTime="2026-07-06 23:13:06.537497241 +0000 UTC m=+1598.941747146"
Jul 06 23:13:06.792829 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:06.792745    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g"
Jul 06 23:13:06.797689 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:06.797665    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g"
Jul 06 23:13:06.819488 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:06.819459    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-0c98d6bc-6cc98695d-grcmr"]
Jul 06 23:13:06.822912 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:06.822894    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-0c98d6bc-6cc98695d-grcmr"
Jul 06 23:13:06.825859 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:06.825836    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-0c98d6bc-serving-cert\""
Jul 06 23:13:06.847991 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:06.847971    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-0c98d6bc-6cc98695d-grcmr"]
Jul 06 23:13:06.869073 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:06.869046    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/414ebf2f-daef-48dd-ac68-e1f53fc3e7cb-maas-api-tls\") pod \"maas-api-e2e-ait-0c98d6bc-6cc98695d-grcmr\" (UID: \"414ebf2f-daef-48dd-ac68-e1f53fc3e7cb\") " pod="opendatahub/maas-api-e2e-ait-0c98d6bc-6cc98695d-grcmr"
Jul 06 23:13:06.869227 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:06.869116    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbnr5\" (UniqueName: \"kubernetes.io/projected/414ebf2f-daef-48dd-ac68-e1f53fc3e7cb-kube-api-access-pbnr5\") pod \"maas-api-e2e-ait-0c98d6bc-6cc98695d-grcmr\" (UID: \"414ebf2f-daef-48dd-ac68-e1f53fc3e7cb\") " pod="opendatahub/maas-api-e2e-ait-0c98d6bc-6cc98695d-grcmr"
Jul 06 23:13:06.869227 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:06.869151    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/414ebf2f-daef-48dd-ac68-e1f53fc3e7cb-openshift-service-ca\") pod \"maas-api-e2e-ait-0c98d6bc-6cc98695d-grcmr\" (UID: \"414ebf2f-daef-48dd-ac68-e1f53fc3e7cb\") " pod="opendatahub/maas-api-e2e-ait-0c98d6bc-6cc98695d-grcmr"
Jul 06 23:13:06.970591 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:06.970556    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/414ebf2f-daef-48dd-ac68-e1f53fc3e7cb-maas-api-tls\") pod \"maas-api-e2e-ait-0c98d6bc-6cc98695d-grcmr\" (UID: \"414ebf2f-daef-48dd-ac68-e1f53fc3e7cb\") " pod="opendatahub/maas-api-e2e-ait-0c98d6bc-6cc98695d-grcmr"
Jul 06 23:13:06.970741 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:06.970606    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbnr5\" (UniqueName: \"kubernetes.io/projected/414ebf2f-daef-48dd-ac68-e1f53fc3e7cb-kube-api-access-pbnr5\") pod \"maas-api-e2e-ait-0c98d6bc-6cc98695d-grcmr\" (UID: \"414ebf2f-daef-48dd-ac68-e1f53fc3e7cb\") " pod="opendatahub/maas-api-e2e-ait-0c98d6bc-6cc98695d-grcmr"
Jul 06 23:13:06.970741 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:06.970627    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/414ebf2f-daef-48dd-ac68-e1f53fc3e7cb-openshift-service-ca\") pod \"maas-api-e2e-ait-0c98d6bc-6cc98695d-grcmr\" (UID: \"414ebf2f-daef-48dd-ac68-e1f53fc3e7cb\") " pod="opendatahub/maas-api-e2e-ait-0c98d6bc-6cc98695d-grcmr"
Jul 06 23:13:06.970741 ip-10-0-134-116 kubenswrapper[2569]: E0706 23:13:06.970704    2569 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-0c98d6bc-serving-cert: secret "maas-api-e2e-ait-0c98d6bc-serving-cert" not found
Jul 06 23:13:06.970885 ip-10-0-134-116 kubenswrapper[2569]: E0706 23:13:06.970770    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/414ebf2f-daef-48dd-ac68-e1f53fc3e7cb-maas-api-tls podName:414ebf2f-daef-48dd-ac68-e1f53fc3e7cb nodeName:}" failed. No retries permitted until 2026-07-06 23:13:07.470754698 +0000 UTC m=+1599.875004579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/414ebf2f-daef-48dd-ac68-e1f53fc3e7cb-maas-api-tls") pod "maas-api-e2e-ait-0c98d6bc-6cc98695d-grcmr" (UID: "414ebf2f-daef-48dd-ac68-e1f53fc3e7cb") : secret "maas-api-e2e-ait-0c98d6bc-serving-cert" not found
Jul 06 23:13:06.971270 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:06.971254    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/414ebf2f-daef-48dd-ac68-e1f53fc3e7cb-openshift-service-ca\") pod \"maas-api-e2e-ait-0c98d6bc-6cc98695d-grcmr\" (UID: \"414ebf2f-daef-48dd-ac68-e1f53fc3e7cb\") " pod="opendatahub/maas-api-e2e-ait-0c98d6bc-6cc98695d-grcmr"
Jul 06 23:13:06.982016 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:06.981983    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbnr5\" (UniqueName: \"kubernetes.io/projected/414ebf2f-daef-48dd-ac68-e1f53fc3e7cb-kube-api-access-pbnr5\") pod \"maas-api-e2e-ait-0c98d6bc-6cc98695d-grcmr\" (UID: \"414ebf2f-daef-48dd-ac68-e1f53fc3e7cb\") " pod="opendatahub/maas-api-e2e-ait-0c98d6bc-6cc98695d-grcmr"
Jul 06 23:13:07.474983 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:07.474952    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/414ebf2f-daef-48dd-ac68-e1f53fc3e7cb-maas-api-tls\") pod \"maas-api-e2e-ait-0c98d6bc-6cc98695d-grcmr\" (UID: \"414ebf2f-daef-48dd-ac68-e1f53fc3e7cb\") " pod="opendatahub/maas-api-e2e-ait-0c98d6bc-6cc98695d-grcmr"
Jul 06 23:13:07.477259 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:07.477236    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/414ebf2f-daef-48dd-ac68-e1f53fc3e7cb-maas-api-tls\") pod \"maas-api-e2e-ait-0c98d6bc-6cc98695d-grcmr\" (UID: \"414ebf2f-daef-48dd-ac68-e1f53fc3e7cb\") " pod="opendatahub/maas-api-e2e-ait-0c98d6bc-6cc98695d-grcmr"
Jul 06 23:13:07.525560 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:07.525533    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g"
Jul 06 23:13:07.526361 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:07.526343    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g"
Jul 06 23:13:07.733934 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:07.733856    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-0c98d6bc-6cc98695d-grcmr"
Jul 06 23:13:07.873488 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:07.873461    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-0c98d6bc-6cc98695d-grcmr"]
Jul 06 23:13:07.875825 ip-10-0-134-116 kubenswrapper[2569]: W0706 23:13:07.875796    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod414ebf2f_daef_48dd_ac68_e1f53fc3e7cb.slice/crio-4feefc29aa3651a8e902c3486c56a1be76db8e30fcf204bc690895d661de3ddc WatchSource:0}: Error finding container 4feefc29aa3651a8e902c3486c56a1be76db8e30fcf204bc690895d661de3ddc: Status 404 returned error can't find the container with id 4feefc29aa3651a8e902c3486c56a1be76db8e30fcf204bc690895d661de3ddc
Jul 06 23:13:07.877706 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:07.877685    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 23:13:08.531489 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:08.531456    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-0c98d6bc-6cc98695d-grcmr" event={"ID":"414ebf2f-daef-48dd-ac68-e1f53fc3e7cb","Type":"ContainerStarted","Data":"f0e2f4e935ee4f65055cc31316cb6f9477acf11cae7125af2ff963d12efa73db"}
Jul 06 23:13:08.531944 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:08.531506    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-0c98d6bc-6cc98695d-grcmr" event={"ID":"414ebf2f-daef-48dd-ac68-e1f53fc3e7cb","Type":"ContainerStarted","Data":"4feefc29aa3651a8e902c3486c56a1be76db8e30fcf204bc690895d661de3ddc"}
Jul 06 23:13:08.531944 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:08.531547    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-0c98d6bc-6cc98695d-grcmr"
Jul 06 23:13:08.552584 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:08.552539    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-0c98d6bc-6cc98695d-grcmr" podStartSLOduration=2.060623701 podStartE2EDuration="2.552524028s" totalImagesPullingTime="491.900327ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:13:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:13:08.550439852 +0000 UTC m=+1600.954689804" watchObservedRunningTime="2026-07-06 23:13:08.552524028 +0000 UTC m=+1600.956773932"
Jul 06 23:13:11.843137 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:11.843103    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g"]
Jul 06 23:13:11.843534 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:11.843316    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g" podUID="669166c8-88ee-4c33-8bf4-f1e6fb04ce2a" containerName="istio-proxy" containerID="cri-o://bd9506819ef921d1417ba3069559aeb1cb4f8e0d2afd179932c227392647cca7" gracePeriod=30
Jul 06 23:13:12.058272 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:12.058241    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-0c98d6bc-6cc98695d-grcmr"]
Jul 06 23:13:12.058435 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:12.058413    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-0c98d6bc-6cc98695d-grcmr" podUID="414ebf2f-daef-48dd-ac68-e1f53fc3e7cb" containerName="maas-api" containerID="cri-o://f0e2f4e935ee4f65055cc31316cb6f9477acf11cae7125af2ff963d12efa73db" gracePeriod=30
Jul 06 23:13:12.308581 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:12.308552    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-0c98d6bc-6cc98695d-grcmr"
Jul 06 23:13:12.418639 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:12.418607    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbnr5\" (UniqueName: \"kubernetes.io/projected/414ebf2f-daef-48dd-ac68-e1f53fc3e7cb-kube-api-access-pbnr5\") pod \"414ebf2f-daef-48dd-ac68-e1f53fc3e7cb\" (UID: \"414ebf2f-daef-48dd-ac68-e1f53fc3e7cb\") "
Jul 06 23:13:12.418854 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:12.418659    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/414ebf2f-daef-48dd-ac68-e1f53fc3e7cb-maas-api-tls\") pod \"414ebf2f-daef-48dd-ac68-e1f53fc3e7cb\" (UID: \"414ebf2f-daef-48dd-ac68-e1f53fc3e7cb\") "
Jul 06 23:13:12.418854 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:12.418735    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/414ebf2f-daef-48dd-ac68-e1f53fc3e7cb-openshift-service-ca\") pod \"414ebf2f-daef-48dd-ac68-e1f53fc3e7cb\" (UID: \"414ebf2f-daef-48dd-ac68-e1f53fc3e7cb\") "
Jul 06 23:13:12.419144 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:12.419118    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/414ebf2f-daef-48dd-ac68-e1f53fc3e7cb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "414ebf2f-daef-48dd-ac68-e1f53fc3e7cb" (UID: "414ebf2f-daef-48dd-ac68-e1f53fc3e7cb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 23:13:12.420874 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:12.420845    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/414ebf2f-daef-48dd-ac68-e1f53fc3e7cb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "414ebf2f-daef-48dd-ac68-e1f53fc3e7cb" (UID: "414ebf2f-daef-48dd-ac68-e1f53fc3e7cb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 23:13:12.420967 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:12.420894    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/414ebf2f-daef-48dd-ac68-e1f53fc3e7cb-kube-api-access-pbnr5" (OuterVolumeSpecName: "kube-api-access-pbnr5") pod "414ebf2f-daef-48dd-ac68-e1f53fc3e7cb" (UID: "414ebf2f-daef-48dd-ac68-e1f53fc3e7cb"). InnerVolumeSpecName "kube-api-access-pbnr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 23:13:12.520128 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:12.520101    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/414ebf2f-daef-48dd-ac68-e1f53fc3e7cb-maas-api-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:13:12.520128 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:12.520124    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/414ebf2f-daef-48dd-ac68-e1f53fc3e7cb-openshift-service-ca\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:13:12.520128 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:12.520134    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbnr5\" (UniqueName: \"kubernetes.io/projected/414ebf2f-daef-48dd-ac68-e1f53fc3e7cb-kube-api-access-pbnr5\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:13:12.551484 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:12.551453    2569 generic.go:356] "Generic (PLEG): container finished" podID="414ebf2f-daef-48dd-ac68-e1f53fc3e7cb" containerID="f0e2f4e935ee4f65055cc31316cb6f9477acf11cae7125af2ff963d12efa73db" exitCode=0
Jul 06 23:13:12.551639 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:12.551496    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-0c98d6bc-6cc98695d-grcmr" event={"ID":"414ebf2f-daef-48dd-ac68-e1f53fc3e7cb","Type":"ContainerDied","Data":"f0e2f4e935ee4f65055cc31316cb6f9477acf11cae7125af2ff963d12efa73db"}
Jul 06 23:13:12.551639 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:12.551544    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-0c98d6bc-6cc98695d-grcmr" event={"ID":"414ebf2f-daef-48dd-ac68-e1f53fc3e7cb","Type":"ContainerDied","Data":"4feefc29aa3651a8e902c3486c56a1be76db8e30fcf204bc690895d661de3ddc"}
Jul 06 23:13:12.551639 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:12.551576    2569 scope.go:117] "RemoveContainer" containerID="f0e2f4e935ee4f65055cc31316cb6f9477acf11cae7125af2ff963d12efa73db"
Jul 06 23:13:12.551639 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:12.551500    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-0c98d6bc-6cc98695d-grcmr"
Jul 06 23:13:12.561040 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:12.561019    2569 scope.go:117] "RemoveContainer" containerID="f0e2f4e935ee4f65055cc31316cb6f9477acf11cae7125af2ff963d12efa73db"
Jul 06 23:13:12.561329 ip-10-0-134-116 kubenswrapper[2569]: E0706 23:13:12.561309    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0e2f4e935ee4f65055cc31316cb6f9477acf11cae7125af2ff963d12efa73db\": container with ID starting with f0e2f4e935ee4f65055cc31316cb6f9477acf11cae7125af2ff963d12efa73db not found: ID does not exist" containerID="f0e2f4e935ee4f65055cc31316cb6f9477acf11cae7125af2ff963d12efa73db"
Jul 06 23:13:12.561423 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:12.561336    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0e2f4e935ee4f65055cc31316cb6f9477acf11cae7125af2ff963d12efa73db"} err="failed to get container status \"f0e2f4e935ee4f65055cc31316cb6f9477acf11cae7125af2ff963d12efa73db\": rpc error: code = NotFound desc = could not find container \"f0e2f4e935ee4f65055cc31316cb6f9477acf11cae7125af2ff963d12efa73db\": container with ID starting with f0e2f4e935ee4f65055cc31316cb6f9477acf11cae7125af2ff963d12efa73db not found: ID does not exist"
Jul 06 23:13:12.574579 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:12.574555    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-0c98d6bc-6cc98695d-grcmr"]
Jul 06 23:13:12.576270 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:12.576252    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-0c98d6bc-6cc98695d-grcmr"]
Jul 06 23:13:14.176038 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:14.176006    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="414ebf2f-daef-48dd-ac68-e1f53fc3e7cb" path="/var/lib/kubelet/pods/414ebf2f-daef-48dd-ac68-e1f53fc3e7cb/volumes"
Jul 06 23:13:17.098331 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.098310    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g"
Jul 06 23:13:17.256397 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.256370    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-istiod-ca-cert\") pod \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\" (UID: \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\") "
Jul 06 23:13:17.256558 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.256422    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-workload-certs\") pod \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\" (UID: \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\") "
Jul 06 23:13:17.256558 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.256451    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-istio-token\") pod \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\" (UID: \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\") "
Jul 06 23:13:17.256558 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.256477    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-credential-socket\") pod \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\" (UID: \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\") "
Jul 06 23:13:17.256558 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.256505    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-workload-socket\") pod \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\" (UID: \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\") "
Jul 06 23:13:17.256558 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.256547    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pndv8\" (UniqueName: \"kubernetes.io/projected/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-kube-api-access-pndv8\") pod \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\" (UID: \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\") "
Jul 06 23:13:17.256803 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.256569    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-istio-envoy\") pod \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\" (UID: \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\") "
Jul 06 23:13:17.256803 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.256603    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-istio-data\") pod \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\" (UID: \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\") "
Jul 06 23:13:17.256803 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.256633    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-istio-podinfo\") pod \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\" (UID: \"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a\") "
Jul 06 23:13:17.256803 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.256662    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "669166c8-88ee-4c33-8bf4-f1e6fb04ce2a" (UID: "669166c8-88ee-4c33-8bf4-f1e6fb04ce2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:13:17.257008 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.256846    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "669166c8-88ee-4c33-8bf4-f1e6fb04ce2a" (UID: "669166c8-88ee-4c33-8bf4-f1e6fb04ce2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:13:17.257008 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.256902    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "669166c8-88ee-4c33-8bf4-f1e6fb04ce2a" (UID: "669166c8-88ee-4c33-8bf4-f1e6fb04ce2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 23:13:17.257008 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.256956    2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-workload-certs\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:13:17.257008 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.256963    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "669166c8-88ee-4c33-8bf4-f1e6fb04ce2a" (UID: "669166c8-88ee-4c33-8bf4-f1e6fb04ce2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:13:17.257008 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.256975    2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-credential-socket\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:13:17.257297 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.257062    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-istio-data" (OuterVolumeSpecName: "istio-data") pod "669166c8-88ee-4c33-8bf4-f1e6fb04ce2a" (UID: "669166c8-88ee-4c33-8bf4-f1e6fb04ce2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:13:17.258741 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.258692    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "669166c8-88ee-4c33-8bf4-f1e6fb04ce2a" (UID: "669166c8-88ee-4c33-8bf4-f1e6fb04ce2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:13:17.258863 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.258772    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "669166c8-88ee-4c33-8bf4-f1e6fb04ce2a" (UID: "669166c8-88ee-4c33-8bf4-f1e6fb04ce2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 06 23:13:17.258863 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.258797    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-istio-token" (OuterVolumeSpecName: "istio-token") pod "669166c8-88ee-4c33-8bf4-f1e6fb04ce2a" (UID: "669166c8-88ee-4c33-8bf4-f1e6fb04ce2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 23:13:17.258863 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.258823    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-kube-api-access-pndv8" (OuterVolumeSpecName: "kube-api-access-pndv8") pod "669166c8-88ee-4c33-8bf4-f1e6fb04ce2a" (UID: "669166c8-88ee-4c33-8bf4-f1e6fb04ce2a"). InnerVolumeSpecName "kube-api-access-pndv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 23:13:17.357903 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.357867    2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-istio-podinfo\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:13:17.357903 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.357900    2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-istiod-ca-cert\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:13:17.357903 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.357909    2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-istio-token\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:13:17.357903 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.357917    2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-workload-socket\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:13:17.358112 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.357925    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pndv8\" (UniqueName: \"kubernetes.io/projected/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-kube-api-access-pndv8\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:13:17.358112 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.357946    2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-istio-envoy\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:13:17.358112 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.357954    2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a-istio-data\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:13:17.578500 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.578419    2569 generic.go:356] "Generic (PLEG): container finished" podID="669166c8-88ee-4c33-8bf4-f1e6fb04ce2a" containerID="bd9506819ef921d1417ba3069559aeb1cb4f8e0d2afd179932c227392647cca7" exitCode=0
Jul 06 23:13:17.578500 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.578469    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g"
Jul 06 23:13:17.578719 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.578497    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g" event={"ID":"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a","Type":"ContainerDied","Data":"bd9506819ef921d1417ba3069559aeb1cb4f8e0d2afd179932c227392647cca7"}
Jul 06 23:13:17.578719 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.578553    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g" event={"ID":"669166c8-88ee-4c33-8bf4-f1e6fb04ce2a","Type":"ContainerDied","Data":"5559209a4aa730dd0b38d755fb7b41b015de0cfa0e6f4aa424156f867bddc6c3"}
Jul 06 23:13:17.578719 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.578575    2569 scope.go:117] "RemoveContainer" containerID="bd9506819ef921d1417ba3069559aeb1cb4f8e0d2afd179932c227392647cca7"
Jul 06 23:13:17.588369 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.588353    2569 scope.go:117] "RemoveContainer" containerID="bd9506819ef921d1417ba3069559aeb1cb4f8e0d2afd179932c227392647cca7"
Jul 06 23:13:17.588642 ip-10-0-134-116 kubenswrapper[2569]: E0706 23:13:17.588624    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd9506819ef921d1417ba3069559aeb1cb4f8e0d2afd179932c227392647cca7\": container with ID starting with bd9506819ef921d1417ba3069559aeb1cb4f8e0d2afd179932c227392647cca7 not found: ID does not exist" containerID="bd9506819ef921d1417ba3069559aeb1cb4f8e0d2afd179932c227392647cca7"
Jul 06 23:13:17.588694 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.588652    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd9506819ef921d1417ba3069559aeb1cb4f8e0d2afd179932c227392647cca7"} err="failed to get container status \"bd9506819ef921d1417ba3069559aeb1cb4f8e0d2afd179932c227392647cca7\": rpc error: code = NotFound desc = could not find container \"bd9506819ef921d1417ba3069559aeb1cb4f8e0d2afd179932c227392647cca7\": container with ID starting with bd9506819ef921d1417ba3069559aeb1cb4f8e0d2afd179932c227392647cca7 not found: ID does not exist"
Jul 06 23:13:17.601350 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.601329    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g"]
Jul 06 23:13:17.604813 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.604795    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0c98d6bc-openshift-default-9dd97cf97-4k26g"]
Jul 06 23:13:17.973293 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.973258    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k"]
Jul 06 23:13:17.973694 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.973675    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="414ebf2f-daef-48dd-ac68-e1f53fc3e7cb" containerName="maas-api"
Jul 06 23:13:17.973786 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.973698    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="414ebf2f-daef-48dd-ac68-e1f53fc3e7cb" containerName="maas-api"
Jul 06 23:13:17.973786 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.973716    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="669166c8-88ee-4c33-8bf4-f1e6fb04ce2a" containerName="istio-proxy"
Jul 06 23:13:17.973786 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.973724    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="669166c8-88ee-4c33-8bf4-f1e6fb04ce2a" containerName="istio-proxy"
Jul 06 23:13:17.973933 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.973848    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="414ebf2f-daef-48dd-ac68-e1f53fc3e7cb" containerName="maas-api"
Jul 06 23:13:17.973933 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.973861    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="669166c8-88ee-4c33-8bf4-f1e6fb04ce2a" containerName="istio-proxy"
Jul 06 23:13:17.978130 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.978107    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k"
Jul 06 23:13:17.980776 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.980759    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b3a9ee34-openshift-default-dockercfg-vk6wg\""
Jul 06 23:13:17.986482 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:17.986405    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k"]
Jul 06 23:13:18.064271 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:18.064244    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-workload-socket\") pod \"e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k\" (UID: \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\") " pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k"
Jul 06 23:13:18.064271 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:18.064272    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-credential-socket\") pod \"e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k\" (UID: \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\") " pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k"
Jul 06 23:13:18.064439 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:18.064335    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-istio-envoy\") pod \"e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k\" (UID: \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\") " pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k"
Jul 06 23:13:18.064439 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:18.064393    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-istiod-ca-cert\") pod \"e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k\" (UID: \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\") " pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k"
Jul 06 23:13:18.064504 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:18.064438    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrnwr\" (UniqueName: \"kubernetes.io/projected/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-kube-api-access-hrnwr\") pod \"e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k\" (UID: \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\") " pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k"
Jul 06 23:13:18.064504 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:18.064490    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-istio-data\") pod \"e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k\" (UID: \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\") " pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k"
Jul 06 23:13:18.064569 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:18.064519    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-istio-podinfo\") pod \"e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k\" (UID: \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\") " pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k"
Jul 06 23:13:18.064569 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:18.064548    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-workload-certs\") pod \"e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k\" (UID: \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\") " pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k"
Jul 06 23:13:18.064628 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:18.064570    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-istio-token\") pod \"e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k\" (UID: \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\") " pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k"
Jul 06 23:13:18.165959 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:18.165918    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-workload-certs\") pod \"e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k\" (UID: \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\") " pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k"
Jul 06 23:13:18.166425 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:18.165971    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-istio-token\") pod \"e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k\" (UID: \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\") " pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k"
Jul 06 23:13:18.166425 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:18.166010    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-workload-socket\") pod \"e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k\" (UID: \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\") " pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k"
Jul 06 23:13:18.166425 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:18.166033    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-credential-socket\") pod \"e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k\" (UID: \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\") " pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k"
Jul 06 23:13:18.166425 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:18.166057    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-istio-envoy\") pod \"e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k\" (UID: \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\") " pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k"
Jul 06 23:13:18.166425 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:18.166111    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-istiod-ca-cert\") pod \"e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k\" (UID: \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\") " pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k"
Jul 06 23:13:18.166425 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:18.166193    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrnwr\" (UniqueName: \"kubernetes.io/projected/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-kube-api-access-hrnwr\") pod \"e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k\" (UID: \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\") " pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k"
Jul 06 23:13:18.166425 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:18.166238    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-istio-data\") pod \"e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k\" (UID: \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\") " pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k"
Jul 06 23:13:18.166425 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:18.166276    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-istio-podinfo\") pod \"e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k\" (UID: \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\") " pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k"
Jul 06 23:13:18.166425 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:18.166420    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-credential-socket\") pod \"e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k\" (UID: \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\") " pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k"
Jul 06 23:13:18.166842 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:18.166461    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-workload-socket\") pod \"e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k\" (UID: \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\") " pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k"
Jul 06 23:13:18.166842 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:18.166694    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-istio-data\") pod \"e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k\" (UID: \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\") " pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k"
Jul 06 23:13:18.166842 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:18.166760    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-workload-certs\") pod \"e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k\" (UID: \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\") " pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k"
Jul 06 23:13:18.167167 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:18.167143    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-istiod-ca-cert\") pod \"e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k\" (UID: \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\") " pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k"
Jul 06 23:13:18.168778 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:18.168751    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-istio-envoy\") pod \"e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k\" (UID: \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\") " pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k"
Jul 06 23:13:18.168873 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:18.168834    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-istio-podinfo\") pod \"e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k\" (UID: \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\") " pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k"
Jul 06 23:13:18.171959 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:18.171938    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="669166c8-88ee-4c33-8bf4-f1e6fb04ce2a" path="/var/lib/kubelet/pods/669166c8-88ee-4c33-8bf4-f1e6fb04ce2a/volumes"
Jul 06 23:13:18.176038 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:18.176008    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-istio-token\") pod \"e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k\" (UID: \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\") " pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k"
Jul 06 23:13:18.176240 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:18.176223    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrnwr\" (UniqueName: \"kubernetes.io/projected/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-kube-api-access-hrnwr\") pod \"e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k\" (UID: \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\") " pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k"
Jul 06 23:13:18.290293 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:18.290223    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k"
Jul 06 23:13:18.442096 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:18.442072    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k"]
Jul 06 23:13:18.445061 ip-10-0-134-116 kubenswrapper[2569]: W0706 23:13:18.445033    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e929c1_b9f3_409d_bdb4_b7717a5e6991.slice/crio-c0772a2a0385c9af0f0e97a756c55990e91e73754c2b78fd8fda3a7dd24350b0 WatchSource:0}: Error finding container c0772a2a0385c9af0f0e97a756c55990e91e73754c2b78fd8fda3a7dd24350b0: Status 404 returned error can't find the container with id c0772a2a0385c9af0f0e97a756c55990e91e73754c2b78fd8fda3a7dd24350b0
Jul 06 23:13:18.447279 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:18.447241    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 23:13:18.447376 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:18.447323    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 23:13:18.447376 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:18.447353    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 23:13:18.586938 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:18.586840    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k" event={"ID":"a2e929c1-b9f3-409d-bdb4-b7717a5e6991","Type":"ContainerStarted","Data":"d6f1c976da649bae2c8a3e71002885c125087be20d548b45d59ffc4bfde08b80"}
Jul 06 23:13:18.586938 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:18.586905    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k" event={"ID":"a2e929c1-b9f3-409d-bdb4-b7717a5e6991","Type":"ContainerStarted","Data":"c0772a2a0385c9af0f0e97a756c55990e91e73754c2b78fd8fda3a7dd24350b0"}
Jul 06 23:13:18.609123 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:18.609077    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k" podStartSLOduration=1.609062416 podStartE2EDuration="1.609062416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:13:18.606892745 +0000 UTC m=+1611.011142652" watchObservedRunningTime="2026-07-06 23:13:18.609062416 +0000 UTC m=+1611.013312360"
Jul 06 23:13:19.243501 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:19.243471    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-b3a9ee34-6fd469549c-k6g6c"]
Jul 06 23:13:19.246883 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:19.246860    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-b3a9ee34-6fd469549c-k6g6c"
Jul 06 23:13:19.249742 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:19.249719    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-b3a9ee34-serving-cert\""
Jul 06 23:13:19.260388 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:19.260362    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-b3a9ee34-6fd469549c-k6g6c"]
Jul 06 23:13:19.291028 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:19.290999    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k"
Jul 06 23:13:19.292076 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:19.292057    2569 patch_prober.go:28] interesting pod/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Jul 06 23:13:19.292168 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:19.292101    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k" podUID="a2e929c1-b9f3-409d-bdb4-b7717a5e6991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Jul 06 23:13:19.377733 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:19.377699    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c7f9a78-6199-497f-8868-fa90ae30a371-openshift-service-ca\") pod \"maas-api-e2e-ait-b3a9ee34-6fd469549c-k6g6c\" (UID: \"9c7f9a78-6199-497f-8868-fa90ae30a371\") " pod="opendatahub/maas-api-e2e-ait-b3a9ee34-6fd469549c-k6g6c"
Jul 06 23:13:19.377916 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:19.377743    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxxvh\" (UniqueName: \"kubernetes.io/projected/9c7f9a78-6199-497f-8868-fa90ae30a371-kube-api-access-rxxvh\") pod \"maas-api-e2e-ait-b3a9ee34-6fd469549c-k6g6c\" (UID: \"9c7f9a78-6199-497f-8868-fa90ae30a371\") " pod="opendatahub/maas-api-e2e-ait-b3a9ee34-6fd469549c-k6g6c"
Jul 06 23:13:19.377916 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:19.377867    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c7f9a78-6199-497f-8868-fa90ae30a371-maas-api-tls\") pod \"maas-api-e2e-ait-b3a9ee34-6fd469549c-k6g6c\" (UID: \"9c7f9a78-6199-497f-8868-fa90ae30a371\") " pod="opendatahub/maas-api-e2e-ait-b3a9ee34-6fd469549c-k6g6c"
Jul 06 23:13:19.479524 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:19.479481    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c7f9a78-6199-497f-8868-fa90ae30a371-openshift-service-ca\") pod \"maas-api-e2e-ait-b3a9ee34-6fd469549c-k6g6c\" (UID: \"9c7f9a78-6199-497f-8868-fa90ae30a371\") " pod="opendatahub/maas-api-e2e-ait-b3a9ee34-6fd469549c-k6g6c"
Jul 06 23:13:19.479524 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:19.479530    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxxvh\" (UniqueName: \"kubernetes.io/projected/9c7f9a78-6199-497f-8868-fa90ae30a371-kube-api-access-rxxvh\") pod \"maas-api-e2e-ait-b3a9ee34-6fd469549c-k6g6c\" (UID: \"9c7f9a78-6199-497f-8868-fa90ae30a371\") " pod="opendatahub/maas-api-e2e-ait-b3a9ee34-6fd469549c-k6g6c"
Jul 06 23:13:19.479748 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:19.479599    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c7f9a78-6199-497f-8868-fa90ae30a371-maas-api-tls\") pod \"maas-api-e2e-ait-b3a9ee34-6fd469549c-k6g6c\" (UID: \"9c7f9a78-6199-497f-8868-fa90ae30a371\") " pod="opendatahub/maas-api-e2e-ait-b3a9ee34-6fd469549c-k6g6c"
Jul 06 23:13:19.479748 ip-10-0-134-116 kubenswrapper[2569]: E0706 23:13:19.479736    2569 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-b3a9ee34-serving-cert: secret "maas-api-e2e-ait-b3a9ee34-serving-cert" not found
Jul 06 23:13:19.479880 ip-10-0-134-116 kubenswrapper[2569]: E0706 23:13:19.479804    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c7f9a78-6199-497f-8868-fa90ae30a371-maas-api-tls podName:9c7f9a78-6199-497f-8868-fa90ae30a371 nodeName:}" failed. No retries permitted until 2026-07-06 23:13:19.979783117 +0000 UTC m=+1612.384033001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9c7f9a78-6199-497f-8868-fa90ae30a371-maas-api-tls") pod "maas-api-e2e-ait-b3a9ee34-6fd469549c-k6g6c" (UID: "9c7f9a78-6199-497f-8868-fa90ae30a371") : secret "maas-api-e2e-ait-b3a9ee34-serving-cert" not found
Jul 06 23:13:19.480258 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:19.480234    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c7f9a78-6199-497f-8868-fa90ae30a371-openshift-service-ca\") pod \"maas-api-e2e-ait-b3a9ee34-6fd469549c-k6g6c\" (UID: \"9c7f9a78-6199-497f-8868-fa90ae30a371\") " pod="opendatahub/maas-api-e2e-ait-b3a9ee34-6fd469549c-k6g6c"
Jul 06 23:13:19.489289 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:19.489267    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxxvh\" (UniqueName: \"kubernetes.io/projected/9c7f9a78-6199-497f-8868-fa90ae30a371-kube-api-access-rxxvh\") pod \"maas-api-e2e-ait-b3a9ee34-6fd469549c-k6g6c\" (UID: \"9c7f9a78-6199-497f-8868-fa90ae30a371\") " pod="opendatahub/maas-api-e2e-ait-b3a9ee34-6fd469549c-k6g6c"
Jul 06 23:13:19.984379 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:19.984342    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c7f9a78-6199-497f-8868-fa90ae30a371-maas-api-tls\") pod \"maas-api-e2e-ait-b3a9ee34-6fd469549c-k6g6c\" (UID: \"9c7f9a78-6199-497f-8868-fa90ae30a371\") " pod="opendatahub/maas-api-e2e-ait-b3a9ee34-6fd469549c-k6g6c"
Jul 06 23:13:19.987163 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:19.987128    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c7f9a78-6199-497f-8868-fa90ae30a371-maas-api-tls\") pod \"maas-api-e2e-ait-b3a9ee34-6fd469549c-k6g6c\" (UID: \"9c7f9a78-6199-497f-8868-fa90ae30a371\") " pod="opendatahub/maas-api-e2e-ait-b3a9ee34-6fd469549c-k6g6c"
Jul 06 23:13:20.158392 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:20.158314    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-b3a9ee34-6fd469549c-k6g6c"
Jul 06 23:13:20.294212 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:20.294170    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-b3a9ee34-6fd469549c-k6g6c"]
Jul 06 23:13:20.295137 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:20.295116    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k"
Jul 06 23:13:20.296329 ip-10-0-134-116 kubenswrapper[2569]: W0706 23:13:20.296307    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c7f9a78_6199_497f_8868_fa90ae30a371.slice/crio-9acb4821a4f0299617bce072990c918842a9e7dc921e5c83c38df871e0254493 WatchSource:0}: Error finding container 9acb4821a4f0299617bce072990c918842a9e7dc921e5c83c38df871e0254493: Status 404 returned error can't find the container with id 9acb4821a4f0299617bce072990c918842a9e7dc921e5c83c38df871e0254493
Jul 06 23:13:20.598401 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:20.598325    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-b3a9ee34-6fd469549c-k6g6c" event={"ID":"9c7f9a78-6199-497f-8868-fa90ae30a371","Type":"ContainerStarted","Data":"9acb4821a4f0299617bce072990c918842a9e7dc921e5c83c38df871e0254493"}
Jul 06 23:13:20.598613 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:20.598598    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k"
Jul 06 23:13:20.599626 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:20.599606    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k"
Jul 06 23:13:21.604595 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:21.604562    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-b3a9ee34-6fd469549c-k6g6c" event={"ID":"9c7f9a78-6199-497f-8868-fa90ae30a371","Type":"ContainerStarted","Data":"010367f70346d63ff5386811c613429b677af8fbfb09d669128eb740ca98373e"}
Jul 06 23:13:21.604984 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:21.604644    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-b3a9ee34-6fd469549c-k6g6c"
Jul 06 23:13:21.621732 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:21.621688    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-b3a9ee34-6fd469549c-k6g6c" podStartSLOduration=2.2688855869999998 podStartE2EDuration="2.6216702s" totalImagesPullingTime="352.784613ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:13:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:13:21.620722884 +0000 UTC m=+1614.024972790" watchObservedRunningTime="2026-07-06 23:13:21.6216702 +0000 UTC m=+1614.025920107"
Jul 06 23:13:24.538284 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:24.538248    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-b3a9ee34-6fd469549c-k6g6c"]
Jul 06 23:13:24.539065 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:24.538945    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-b3a9ee34-6fd469549c-k6g6c" podUID="9c7f9a78-6199-497f-8868-fa90ae30a371" containerName="maas-api" containerID="cri-o://010367f70346d63ff5386811c613429b677af8fbfb09d669128eb740ca98373e" gracePeriod=30
Jul 06 23:13:24.849978 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:24.849947    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-b3a9ee34-6fd469549c-k6g6c"
Jul 06 23:13:24.929478 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:24.929443    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c7f9a78-6199-497f-8868-fa90ae30a371-openshift-service-ca\") pod \"9c7f9a78-6199-497f-8868-fa90ae30a371\" (UID: \"9c7f9a78-6199-497f-8868-fa90ae30a371\") "
Jul 06 23:13:24.929634 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:24.929541    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxxvh\" (UniqueName: \"kubernetes.io/projected/9c7f9a78-6199-497f-8868-fa90ae30a371-kube-api-access-rxxvh\") pod \"9c7f9a78-6199-497f-8868-fa90ae30a371\" (UID: \"9c7f9a78-6199-497f-8868-fa90ae30a371\") "
Jul 06 23:13:24.929634 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:24.929579    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c7f9a78-6199-497f-8868-fa90ae30a371-maas-api-tls\") pod \"9c7f9a78-6199-497f-8868-fa90ae30a371\" (UID: \"9c7f9a78-6199-497f-8868-fa90ae30a371\") "
Jul 06 23:13:24.929825 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:24.929790    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c7f9a78-6199-497f-8868-fa90ae30a371-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9c7f9a78-6199-497f-8868-fa90ae30a371" (UID: "9c7f9a78-6199-497f-8868-fa90ae30a371"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 23:13:24.931554 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:24.931522    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c7f9a78-6199-497f-8868-fa90ae30a371-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9c7f9a78-6199-497f-8868-fa90ae30a371" (UID: "9c7f9a78-6199-497f-8868-fa90ae30a371"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 23:13:24.931624 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:24.931570    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c7f9a78-6199-497f-8868-fa90ae30a371-kube-api-access-rxxvh" (OuterVolumeSpecName: "kube-api-access-rxxvh") pod "9c7f9a78-6199-497f-8868-fa90ae30a371" (UID: "9c7f9a78-6199-497f-8868-fa90ae30a371"). InnerVolumeSpecName "kube-api-access-rxxvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 23:13:25.030165 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:25.030128    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c7f9a78-6199-497f-8868-fa90ae30a371-maas-api-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:13:25.030165 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:25.030160    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c7f9a78-6199-497f-8868-fa90ae30a371-openshift-service-ca\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:13:25.030377 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:25.030173    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxxvh\" (UniqueName: \"kubernetes.io/projected/9c7f9a78-6199-497f-8868-fa90ae30a371-kube-api-access-rxxvh\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:13:25.624774 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:25.624739    2569 generic.go:356] "Generic (PLEG): container finished" podID="9c7f9a78-6199-497f-8868-fa90ae30a371" containerID="010367f70346d63ff5386811c613429b677af8fbfb09d669128eb740ca98373e" exitCode=0
Jul 06 23:13:25.625236 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:25.624780    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-b3a9ee34-6fd469549c-k6g6c"
Jul 06 23:13:25.625236 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:25.624791    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-b3a9ee34-6fd469549c-k6g6c" event={"ID":"9c7f9a78-6199-497f-8868-fa90ae30a371","Type":"ContainerDied","Data":"010367f70346d63ff5386811c613429b677af8fbfb09d669128eb740ca98373e"}
Jul 06 23:13:25.625236 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:25.624831    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-b3a9ee34-6fd469549c-k6g6c" event={"ID":"9c7f9a78-6199-497f-8868-fa90ae30a371","Type":"ContainerDied","Data":"9acb4821a4f0299617bce072990c918842a9e7dc921e5c83c38df871e0254493"}
Jul 06 23:13:25.625236 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:25.624853    2569 scope.go:117] "RemoveContainer" containerID="010367f70346d63ff5386811c613429b677af8fbfb09d669128eb740ca98373e"
Jul 06 23:13:25.633985 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:25.633967    2569 scope.go:117] "RemoveContainer" containerID="010367f70346d63ff5386811c613429b677af8fbfb09d669128eb740ca98373e"
Jul 06 23:13:25.634297 ip-10-0-134-116 kubenswrapper[2569]: E0706 23:13:25.634272    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"010367f70346d63ff5386811c613429b677af8fbfb09d669128eb740ca98373e\": container with ID starting with 010367f70346d63ff5386811c613429b677af8fbfb09d669128eb740ca98373e not found: ID does not exist" containerID="010367f70346d63ff5386811c613429b677af8fbfb09d669128eb740ca98373e"
Jul 06 23:13:25.634390 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:25.634307    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"010367f70346d63ff5386811c613429b677af8fbfb09d669128eb740ca98373e"} err="failed to get container status \"010367f70346d63ff5386811c613429b677af8fbfb09d669128eb740ca98373e\": rpc error: code = NotFound desc = could not find container \"010367f70346d63ff5386811c613429b677af8fbfb09d669128eb740ca98373e\": container with ID starting with 010367f70346d63ff5386811c613429b677af8fbfb09d669128eb740ca98373e not found: ID does not exist"
Jul 06 23:13:25.647683 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:25.647662    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-b3a9ee34-6fd469549c-k6g6c"]
Jul 06 23:13:25.652584 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:25.652566    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-b3a9ee34-6fd469549c-k6g6c"]
Jul 06 23:13:26.170137 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:26.170101    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c7f9a78-6199-497f-8868-fa90ae30a371" path="/var/lib/kubelet/pods/9c7f9a78-6199-497f-8868-fa90ae30a371/volumes"
Jul 06 23:13:29.648274 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:29.648242    2569 generic.go:356] "Generic (PLEG): container finished" podID="a2e929c1-b9f3-409d-bdb4-b7717a5e6991" containerID="d6f1c976da649bae2c8a3e71002885c125087be20d548b45d59ffc4bfde08b80" exitCode=0
Jul 06 23:13:29.648713 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:29.648307    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k" event={"ID":"a2e929c1-b9f3-409d-bdb4-b7717a5e6991","Type":"ContainerDied","Data":"d6f1c976da649bae2c8a3e71002885c125087be20d548b45d59ffc4bfde08b80"}
Jul 06 23:13:29.648713 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:29.648649    2569 scope.go:117] "RemoveContainer" containerID="d6f1c976da649bae2c8a3e71002885c125087be20d548b45d59ffc4bfde08b80"
Jul 06 23:13:29.649579 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:29.649552    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 23:13:29.649669 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:29.649619    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 23:13:29.649669 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:29.649653    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 23:13:30.295382 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:30.295347    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k"]
Jul 06 23:13:30.655109 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:30.655021    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k" event={"ID":"a2e929c1-b9f3-409d-bdb4-b7717a5e6991","Type":"ContainerStarted","Data":"b7c834da037d94c3f8c2fdf7b4728b8b1aa2f00536ff02173a14377cffe8b0f9"}
Jul 06 23:13:30.655109 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:30.655094    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k" podUID="a2e929c1-b9f3-409d-bdb4-b7717a5e6991" containerName="istio-proxy" containerID="cri-o://b7c834da037d94c3f8c2fdf7b4728b8b1aa2f00536ff02173a14377cffe8b0f9" gracePeriod=30
Jul 06 23:13:35.598819 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:35.598786    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k"
Jul 06 23:13:35.910109 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:35.910084    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k"
Jul 06 23:13:36.020426 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.020403    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-credential-socket\") pod \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\" (UID: \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\") "
Jul 06 23:13:36.020571 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.020431    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-istiod-ca-cert\") pod \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\" (UID: \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\") "
Jul 06 23:13:36.020571 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.020449    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrnwr\" (UniqueName: \"kubernetes.io/projected/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-kube-api-access-hrnwr\") pod \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\" (UID: \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\") "
Jul 06 23:13:36.020571 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.020484    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-istio-data\") pod \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\" (UID: \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\") "
Jul 06 23:13:36.020571 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.020550    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-workload-socket\") pod \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\" (UID: \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\") "
Jul 06 23:13:36.020783 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.020577    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-workload-certs\") pod \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\" (UID: \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\") "
Jul 06 23:13:36.020783 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.020649    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-istio-envoy\") pod \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\" (UID: \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\") "
Jul 06 23:13:36.020783 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.020708    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-istio-podinfo\") pod \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\" (UID: \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\") "
Jul 06 23:13:36.020783 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.020709    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a2e929c1-b9f3-409d-bdb4-b7717a5e6991" (UID: "a2e929c1-b9f3-409d-bdb4-b7717a5e6991"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:13:36.020783 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.020734    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-istio-token\") pod \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\" (UID: \"a2e929c1-b9f3-409d-bdb4-b7717a5e6991\") "
Jul 06 23:13:36.021003 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.020867    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a2e929c1-b9f3-409d-bdb4-b7717a5e6991" (UID: "a2e929c1-b9f3-409d-bdb4-b7717a5e6991"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:13:36.021003 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.020878    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a2e929c1-b9f3-409d-bdb4-b7717a5e6991" (UID: "a2e929c1-b9f3-409d-bdb4-b7717a5e6991"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:13:36.021003 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.020889    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a2e929c1-b9f3-409d-bdb4-b7717a5e6991" (UID: "a2e929c1-b9f3-409d-bdb4-b7717a5e6991"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 23:13:36.021145 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.021006    2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-credential-socket\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:13:36.021145 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.021022    2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-istiod-ca-cert\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:13:36.021145 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.021032    2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-workload-socket\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:13:36.021145 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.021043    2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-workload-certs\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:13:36.021492 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.021454    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-istio-data" (OuterVolumeSpecName: "istio-data") pod "a2e929c1-b9f3-409d-bdb4-b7717a5e6991" (UID: "a2e929c1-b9f3-409d-bdb4-b7717a5e6991"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:13:36.022736 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.022710    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-kube-api-access-hrnwr" (OuterVolumeSpecName: "kube-api-access-hrnwr") pod "a2e929c1-b9f3-409d-bdb4-b7717a5e6991" (UID: "a2e929c1-b9f3-409d-bdb4-b7717a5e6991"). InnerVolumeSpecName "kube-api-access-hrnwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 23:13:36.022870 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.022852    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-istio-token" (OuterVolumeSpecName: "istio-token") pod "a2e929c1-b9f3-409d-bdb4-b7717a5e6991" (UID: "a2e929c1-b9f3-409d-bdb4-b7717a5e6991"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 23:13:36.022920 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.022882    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a2e929c1-b9f3-409d-bdb4-b7717a5e6991" (UID: "a2e929c1-b9f3-409d-bdb4-b7717a5e6991"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 06 23:13:36.022920 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.022900    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a2e929c1-b9f3-409d-bdb4-b7717a5e6991" (UID: "a2e929c1-b9f3-409d-bdb4-b7717a5e6991"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:13:36.121741 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.121721    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrnwr\" (UniqueName: \"kubernetes.io/projected/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-kube-api-access-hrnwr\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:13:36.121741 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.121740    2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-istio-data\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:13:36.121882 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.121750    2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-istio-envoy\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:13:36.121882 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.121757    2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-istio-podinfo\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:13:36.121882 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.121765    2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2e929c1-b9f3-409d-bdb4-b7717a5e6991-istio-token\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:13:36.129385 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.129363    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj"]
Jul 06 23:13:36.130099 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.130075    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2e929c1-b9f3-409d-bdb4-b7717a5e6991" containerName="istio-proxy"
Jul 06 23:13:36.130099 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.130100    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2e929c1-b9f3-409d-bdb4-b7717a5e6991" containerName="istio-proxy"
Jul 06 23:13:36.130326 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.130124    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c7f9a78-6199-497f-8868-fa90ae30a371" containerName="maas-api"
Jul 06 23:13:36.130326 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.130132    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c7f9a78-6199-497f-8868-fa90ae30a371" containerName="maas-api"
Jul 06 23:13:36.130326 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.130239    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c7f9a78-6199-497f-8868-fa90ae30a371" containerName="maas-api"
Jul 06 23:13:36.130326 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.130252    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2e929c1-b9f3-409d-bdb4-b7717a5e6991" containerName="istio-proxy"
Jul 06 23:13:36.133165 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.133149    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj"
Jul 06 23:13:36.135916 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.135898    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-0f938eb0-openshift-default-dockercfg-ckzjm\""
Jul 06 23:13:36.145888 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.145865    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj"]
Jul 06 23:13:36.323535 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.323501    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e25e93-c184-4a91-a364-ec2e36127940-credential-socket\") pod \"e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj\" (UID: \"b9e25e93-c184-4a91-a364-ec2e36127940\") " pod="openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj"
Jul 06 23:13:36.323671 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.323540    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p2zd\" (UniqueName: \"kubernetes.io/projected/b9e25e93-c184-4a91-a364-ec2e36127940-kube-api-access-7p2zd\") pod \"e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj\" (UID: \"b9e25e93-c184-4a91-a364-ec2e36127940\") " pod="openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj"
Jul 06 23:13:36.323671 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.323563    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e25e93-c184-4a91-a364-ec2e36127940-workload-socket\") pod \"e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj\" (UID: \"b9e25e93-c184-4a91-a364-ec2e36127940\") " pod="openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj"
Jul 06 23:13:36.323671 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.323586    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9e25e93-c184-4a91-a364-ec2e36127940-istio-token\") pod \"e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj\" (UID: \"b9e25e93-c184-4a91-a364-ec2e36127940\") " pod="openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj"
Jul 06 23:13:36.323671 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.323606    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9e25e93-c184-4a91-a364-ec2e36127940-istio-podinfo\") pod \"e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj\" (UID: \"b9e25e93-c184-4a91-a364-ec2e36127940\") " pod="openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj"
Jul 06 23:13:36.323671 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.323653    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9e25e93-c184-4a91-a364-ec2e36127940-istio-envoy\") pod \"e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj\" (UID: \"b9e25e93-c184-4a91-a364-ec2e36127940\") " pod="openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj"
Jul 06 23:13:36.323861 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.323674    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9e25e93-c184-4a91-a364-ec2e36127940-istio-data\") pod \"e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj\" (UID: \"b9e25e93-c184-4a91-a364-ec2e36127940\") " pod="openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj"
Jul 06 23:13:36.323861 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.323701    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9e25e93-c184-4a91-a364-ec2e36127940-workload-certs\") pod \"e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj\" (UID: \"b9e25e93-c184-4a91-a364-ec2e36127940\") " pod="openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj"
Jul 06 23:13:36.323861 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.323755    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9e25e93-c184-4a91-a364-ec2e36127940-istiod-ca-cert\") pod \"e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj\" (UID: \"b9e25e93-c184-4a91-a364-ec2e36127940\") " pod="openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj"
Jul 06 23:13:36.424905 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.424842    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9e25e93-c184-4a91-a364-ec2e36127940-istiod-ca-cert\") pod \"e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj\" (UID: \"b9e25e93-c184-4a91-a364-ec2e36127940\") " pod="openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj"
Jul 06 23:13:36.424905 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.424890    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e25e93-c184-4a91-a364-ec2e36127940-credential-socket\") pod \"e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj\" (UID: \"b9e25e93-c184-4a91-a364-ec2e36127940\") " pod="openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj"
Jul 06 23:13:36.425077 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.424919    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p2zd\" (UniqueName: \"kubernetes.io/projected/b9e25e93-c184-4a91-a364-ec2e36127940-kube-api-access-7p2zd\") pod \"e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj\" (UID: \"b9e25e93-c184-4a91-a364-ec2e36127940\") " pod="openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj"
Jul 06 23:13:36.425077 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.424951    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e25e93-c184-4a91-a364-ec2e36127940-workload-socket\") pod \"e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj\" (UID: \"b9e25e93-c184-4a91-a364-ec2e36127940\") " pod="openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj"
Jul 06 23:13:36.425077 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.424983    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9e25e93-c184-4a91-a364-ec2e36127940-istio-token\") pod \"e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj\" (UID: \"b9e25e93-c184-4a91-a364-ec2e36127940\") " pod="openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj"
Jul 06 23:13:36.425077 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.425007    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9e25e93-c184-4a91-a364-ec2e36127940-istio-podinfo\") pod \"e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj\" (UID: \"b9e25e93-c184-4a91-a364-ec2e36127940\") " pod="openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj"
Jul 06 23:13:36.425077 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.425049    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9e25e93-c184-4a91-a364-ec2e36127940-istio-envoy\") pod \"e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj\" (UID: \"b9e25e93-c184-4a91-a364-ec2e36127940\") " pod="openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj"
Jul 06 23:13:36.425077 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.425071    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9e25e93-c184-4a91-a364-ec2e36127940-istio-data\") pod \"e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj\" (UID: \"b9e25e93-c184-4a91-a364-ec2e36127940\") " pod="openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj"
Jul 06 23:13:36.425408 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.425118    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9e25e93-c184-4a91-a364-ec2e36127940-workload-certs\") pod \"e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj\" (UID: \"b9e25e93-c184-4a91-a364-ec2e36127940\") " pod="openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj"
Jul 06 23:13:36.425408 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.425345    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e25e93-c184-4a91-a364-ec2e36127940-credential-socket\") pod \"e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj\" (UID: \"b9e25e93-c184-4a91-a364-ec2e36127940\") " pod="openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj"
Jul 06 23:13:36.425408 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.425360    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e25e93-c184-4a91-a364-ec2e36127940-workload-socket\") pod \"e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj\" (UID: \"b9e25e93-c184-4a91-a364-ec2e36127940\") " pod="openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj"
Jul 06 23:13:36.425536 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.425457    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9e25e93-c184-4a91-a364-ec2e36127940-workload-certs\") pod \"e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj\" (UID: \"b9e25e93-c184-4a91-a364-ec2e36127940\") " pod="openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj"
Jul 06 23:13:36.425606 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.425585    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9e25e93-c184-4a91-a364-ec2e36127940-istio-data\") pod \"e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj\" (UID: \"b9e25e93-c184-4a91-a364-ec2e36127940\") " pod="openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj"
Jul 06 23:13:36.425685 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.425666    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9e25e93-c184-4a91-a364-ec2e36127940-istiod-ca-cert\") pod \"e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj\" (UID: \"b9e25e93-c184-4a91-a364-ec2e36127940\") " pod="openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj"
Jul 06 23:13:36.427320 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.427299    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9e25e93-c184-4a91-a364-ec2e36127940-istio-envoy\") pod \"e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj\" (UID: \"b9e25e93-c184-4a91-a364-ec2e36127940\") " pod="openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj"
Jul 06 23:13:36.427568 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.427521    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9e25e93-c184-4a91-a364-ec2e36127940-istio-podinfo\") pod \"e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj\" (UID: \"b9e25e93-c184-4a91-a364-ec2e36127940\") " pod="openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj"
Jul 06 23:13:36.432789 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.432771    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p2zd\" (UniqueName: \"kubernetes.io/projected/b9e25e93-c184-4a91-a364-ec2e36127940-kube-api-access-7p2zd\") pod \"e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj\" (UID: \"b9e25e93-c184-4a91-a364-ec2e36127940\") " pod="openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj"
Jul 06 23:13:36.432935 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.432917    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9e25e93-c184-4a91-a364-ec2e36127940-istio-token\") pod \"e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj\" (UID: \"b9e25e93-c184-4a91-a364-ec2e36127940\") " pod="openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj"
Jul 06 23:13:36.445882 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.445866    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj"
Jul 06 23:13:36.582925 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.582900    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj"]
Jul 06 23:13:36.584572 ip-10-0-134-116 kubenswrapper[2569]: W0706 23:13:36.584547    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9e25e93_c184_4a91_a364_ec2e36127940.slice/crio-820d8c7532a64297cc4928b11079a850eebb40c5ef26ddbf5f6948b142006cf8 WatchSource:0}: Error finding container 820d8c7532a64297cc4928b11079a850eebb40c5ef26ddbf5f6948b142006cf8: Status 404 returned error can't find the container with id 820d8c7532a64297cc4928b11079a850eebb40c5ef26ddbf5f6948b142006cf8
Jul 06 23:13:36.586659 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.586620    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 23:13:36.586714 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.586684    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 23:13:36.586754 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.586714    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 23:13:36.686125 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.686098    2569 generic.go:356] "Generic (PLEG): container finished" podID="a2e929c1-b9f3-409d-bdb4-b7717a5e6991" containerID="b7c834da037d94c3f8c2fdf7b4728b8b1aa2f00536ff02173a14377cffe8b0f9" exitCode=0
Jul 06 23:13:36.686573 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.686136    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k"
Jul 06 23:13:36.686573 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.686195    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k" event={"ID":"a2e929c1-b9f3-409d-bdb4-b7717a5e6991","Type":"ContainerDied","Data":"b7c834da037d94c3f8c2fdf7b4728b8b1aa2f00536ff02173a14377cffe8b0f9"}
Jul 06 23:13:36.686573 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.686259    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k" event={"ID":"a2e929c1-b9f3-409d-bdb4-b7717a5e6991","Type":"ContainerDied","Data":"c0772a2a0385c9af0f0e97a756c55990e91e73754c2b78fd8fda3a7dd24350b0"}
Jul 06 23:13:36.686573 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.686291    2569 scope.go:117] "RemoveContainer" containerID="b7c834da037d94c3f8c2fdf7b4728b8b1aa2f00536ff02173a14377cffe8b0f9"
Jul 06 23:13:36.688047 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.688019    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj" event={"ID":"b9e25e93-c184-4a91-a364-ec2e36127940","Type":"ContainerStarted","Data":"0c313d158ee0925280ae26849aa53e1eeff98303bc681a8a353f5679054946e5"}
Jul 06 23:13:36.688137 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.688073    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj" event={"ID":"b9e25e93-c184-4a91-a364-ec2e36127940","Type":"ContainerStarted","Data":"820d8c7532a64297cc4928b11079a850eebb40c5ef26ddbf5f6948b142006cf8"}
Jul 06 23:13:36.695166 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.695150    2569 scope.go:117] "RemoveContainer" containerID="d6f1c976da649bae2c8a3e71002885c125087be20d548b45d59ffc4bfde08b80"
Jul 06 23:13:36.703296 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.703275    2569 scope.go:117] "RemoveContainer" containerID="b7c834da037d94c3f8c2fdf7b4728b8b1aa2f00536ff02173a14377cffe8b0f9"
Jul 06 23:13:36.703580 ip-10-0-134-116 kubenswrapper[2569]: E0706 23:13:36.703558    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7c834da037d94c3f8c2fdf7b4728b8b1aa2f00536ff02173a14377cffe8b0f9\": container with ID starting with b7c834da037d94c3f8c2fdf7b4728b8b1aa2f00536ff02173a14377cffe8b0f9 not found: ID does not exist" containerID="b7c834da037d94c3f8c2fdf7b4728b8b1aa2f00536ff02173a14377cffe8b0f9"
Jul 06 23:13:36.703654 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.703592    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7c834da037d94c3f8c2fdf7b4728b8b1aa2f00536ff02173a14377cffe8b0f9"} err="failed to get container status \"b7c834da037d94c3f8c2fdf7b4728b8b1aa2f00536ff02173a14377cffe8b0f9\": rpc error: code = NotFound desc = could not find container \"b7c834da037d94c3f8c2fdf7b4728b8b1aa2f00536ff02173a14377cffe8b0f9\": container with ID starting with b7c834da037d94c3f8c2fdf7b4728b8b1aa2f00536ff02173a14377cffe8b0f9 not found: ID does not exist"
Jul 06 23:13:36.703654 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.703617    2569 scope.go:117] "RemoveContainer" containerID="d6f1c976da649bae2c8a3e71002885c125087be20d548b45d59ffc4bfde08b80"
Jul 06 23:13:36.703851 ip-10-0-134-116 kubenswrapper[2569]: E0706 23:13:36.703830    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6f1c976da649bae2c8a3e71002885c125087be20d548b45d59ffc4bfde08b80\": container with ID starting with d6f1c976da649bae2c8a3e71002885c125087be20d548b45d59ffc4bfde08b80 not found: ID does not exist" containerID="d6f1c976da649bae2c8a3e71002885c125087be20d548b45d59ffc4bfde08b80"
Jul 06 23:13:36.703906 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.703859    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6f1c976da649bae2c8a3e71002885c125087be20d548b45d59ffc4bfde08b80"} err="failed to get container status \"d6f1c976da649bae2c8a3e71002885c125087be20d548b45d59ffc4bfde08b80\": rpc error: code = NotFound desc = could not find container \"d6f1c976da649bae2c8a3e71002885c125087be20d548b45d59ffc4bfde08b80\": container with ID starting with d6f1c976da649bae2c8a3e71002885c125087be20d548b45d59ffc4bfde08b80 not found: ID does not exist"
Jul 06 23:13:36.704843 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.704824    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k"]
Jul 06 23:13:36.708244 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.708224    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b3a9ee34-openshift-default-546b679887-wrr9k"]
Jul 06 23:13:36.725359 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:36.725309    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj" podStartSLOduration=0.725293042 podStartE2EDuration="725.293042ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:13:36.723307224 +0000 UTC m=+1629.127557159" watchObservedRunningTime="2026-07-06 23:13:36.725293042 +0000 UTC m=+1629.129542948"
Jul 06 23:13:37.431928 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:37.431893    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-0f938eb0-64d4f456bf-h67tq"]
Jul 06 23:13:37.432368 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:37.432347    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2e929c1-b9f3-409d-bdb4-b7717a5e6991" containerName="istio-proxy"
Jul 06 23:13:37.432368 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:37.432366    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2e929c1-b9f3-409d-bdb4-b7717a5e6991" containerName="istio-proxy"
Jul 06 23:13:37.432581 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:37.432466    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2e929c1-b9f3-409d-bdb4-b7717a5e6991" containerName="istio-proxy"
Jul 06 23:13:37.435761 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:37.435741    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-0f938eb0-64d4f456bf-h67tq"
Jul 06 23:13:37.438520 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:37.438501    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-0f938eb0-serving-cert\""
Jul 06 23:13:37.446342 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:37.446321    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj"
Jul 06 23:13:37.449475 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:37.449445    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-0f938eb0-64d4f456bf-h67tq"]
Jul 06 23:13:37.452004 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:37.451985    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj"
Jul 06 23:13:37.533541 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:37.533508    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afef2c1d-731c-4f27-be8f-eb5c86e52349-maas-api-tls\") pod \"maas-api-e2e-derive-0f938eb0-64d4f456bf-h67tq\" (UID: \"afef2c1d-731c-4f27-be8f-eb5c86e52349\") " pod="opendatahub/maas-api-e2e-derive-0f938eb0-64d4f456bf-h67tq"
Jul 06 23:13:37.533770 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:37.533562    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afef2c1d-731c-4f27-be8f-eb5c86e52349-openshift-service-ca\") pod \"maas-api-e2e-derive-0f938eb0-64d4f456bf-h67tq\" (UID: \"afef2c1d-731c-4f27-be8f-eb5c86e52349\") " pod="opendatahub/maas-api-e2e-derive-0f938eb0-64d4f456bf-h67tq"
Jul 06 23:13:37.533770 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:37.533600    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kxkm\" (UniqueName: \"kubernetes.io/projected/afef2c1d-731c-4f27-be8f-eb5c86e52349-kube-api-access-6kxkm\") pod \"maas-api-e2e-derive-0f938eb0-64d4f456bf-h67tq\" (UID: \"afef2c1d-731c-4f27-be8f-eb5c86e52349\") " pod="opendatahub/maas-api-e2e-derive-0f938eb0-64d4f456bf-h67tq"
Jul 06 23:13:37.634724 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:37.634693    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afef2c1d-731c-4f27-be8f-eb5c86e52349-maas-api-tls\") pod \"maas-api-e2e-derive-0f938eb0-64d4f456bf-h67tq\" (UID: \"afef2c1d-731c-4f27-be8f-eb5c86e52349\") " pod="opendatahub/maas-api-e2e-derive-0f938eb0-64d4f456bf-h67tq"
Jul 06 23:13:37.634846 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:37.634731    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afef2c1d-731c-4f27-be8f-eb5c86e52349-openshift-service-ca\") pod \"maas-api-e2e-derive-0f938eb0-64d4f456bf-h67tq\" (UID: \"afef2c1d-731c-4f27-be8f-eb5c86e52349\") " pod="opendatahub/maas-api-e2e-derive-0f938eb0-64d4f456bf-h67tq"
Jul 06 23:13:37.634846 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:37.634756    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kxkm\" (UniqueName: \"kubernetes.io/projected/afef2c1d-731c-4f27-be8f-eb5c86e52349-kube-api-access-6kxkm\") pod \"maas-api-e2e-derive-0f938eb0-64d4f456bf-h67tq\" (UID: \"afef2c1d-731c-4f27-be8f-eb5c86e52349\") " pod="opendatahub/maas-api-e2e-derive-0f938eb0-64d4f456bf-h67tq"
Jul 06 23:13:37.634961 ip-10-0-134-116 kubenswrapper[2569]: E0706 23:13:37.634847    2569 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-derive-0f938eb0-serving-cert: secret "maas-api-e2e-derive-0f938eb0-serving-cert" not found
Jul 06 23:13:37.634961 ip-10-0-134-116 kubenswrapper[2569]: E0706 23:13:37.634912    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afef2c1d-731c-4f27-be8f-eb5c86e52349-maas-api-tls podName:afef2c1d-731c-4f27-be8f-eb5c86e52349 nodeName:}" failed. No retries permitted until 2026-07-06 23:13:38.134894935 +0000 UTC m=+1630.539144829 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/afef2c1d-731c-4f27-be8f-eb5c86e52349-maas-api-tls") pod "maas-api-e2e-derive-0f938eb0-64d4f456bf-h67tq" (UID: "afef2c1d-731c-4f27-be8f-eb5c86e52349") : secret "maas-api-e2e-derive-0f938eb0-serving-cert" not found
Jul 06 23:13:37.635389 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:37.635371    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afef2c1d-731c-4f27-be8f-eb5c86e52349-openshift-service-ca\") pod \"maas-api-e2e-derive-0f938eb0-64d4f456bf-h67tq\" (UID: \"afef2c1d-731c-4f27-be8f-eb5c86e52349\") " pod="opendatahub/maas-api-e2e-derive-0f938eb0-64d4f456bf-h67tq"
Jul 06 23:13:37.643732 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:37.643711    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kxkm\" (UniqueName: \"kubernetes.io/projected/afef2c1d-731c-4f27-be8f-eb5c86e52349-kube-api-access-6kxkm\") pod \"maas-api-e2e-derive-0f938eb0-64d4f456bf-h67tq\" (UID: \"afef2c1d-731c-4f27-be8f-eb5c86e52349\") " pod="opendatahub/maas-api-e2e-derive-0f938eb0-64d4f456bf-h67tq"
Jul 06 23:13:37.693471 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:37.693406    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj"
Jul 06 23:13:37.694164 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:37.694147    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj"
Jul 06 23:13:38.138824 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:38.138747    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afef2c1d-731c-4f27-be8f-eb5c86e52349-maas-api-tls\") pod \"maas-api-e2e-derive-0f938eb0-64d4f456bf-h67tq\" (UID: \"afef2c1d-731c-4f27-be8f-eb5c86e52349\") " pod="opendatahub/maas-api-e2e-derive-0f938eb0-64d4f456bf-h67tq"
Jul 06 23:13:38.141134 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:38.141109    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afef2c1d-731c-4f27-be8f-eb5c86e52349-maas-api-tls\") pod \"maas-api-e2e-derive-0f938eb0-64d4f456bf-h67tq\" (UID: \"afef2c1d-731c-4f27-be8f-eb5c86e52349\") " pod="opendatahub/maas-api-e2e-derive-0f938eb0-64d4f456bf-h67tq"
Jul 06 23:13:38.171358 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:38.171334    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2e929c1-b9f3-409d-bdb4-b7717a5e6991" path="/var/lib/kubelet/pods/a2e929c1-b9f3-409d-bdb4-b7717a5e6991/volumes"
Jul 06 23:13:38.346834 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:38.346807    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-0f938eb0-64d4f456bf-h67tq"
Jul 06 23:13:38.482319 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:38.482295    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-0f938eb0-64d4f456bf-h67tq"]
Jul 06 23:13:38.484386 ip-10-0-134-116 kubenswrapper[2569]: W0706 23:13:38.484354    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafef2c1d_731c_4f27_be8f_eb5c86e52349.slice/crio-2572c97b421cf019661165a1f98113be27a4b8a2d1b93baabff97752b9e332d2 WatchSource:0}: Error finding container 2572c97b421cf019661165a1f98113be27a4b8a2d1b93baabff97752b9e332d2: Status 404 returned error can't find the container with id 2572c97b421cf019661165a1f98113be27a4b8a2d1b93baabff97752b9e332d2
Jul 06 23:13:38.698662 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:38.698632    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-0f938eb0-64d4f456bf-h67tq" event={"ID":"afef2c1d-731c-4f27-be8f-eb5c86e52349","Type":"ContainerStarted","Data":"2572c97b421cf019661165a1f98113be27a4b8a2d1b93baabff97752b9e332d2"}
Jul 06 23:13:39.705609 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:39.705574    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-0f938eb0-64d4f456bf-h67tq" event={"ID":"afef2c1d-731c-4f27-be8f-eb5c86e52349","Type":"ContainerStarted","Data":"2ee2f6bd5bc56e95df108deabfb67d3fccb1bf4e1bcea142bc82e5631b4716a9"}
Jul 06 23:13:39.721612 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:39.721570    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-0f938eb0-64d4f456bf-h67tq" podStartSLOduration=2.335592898 podStartE2EDuration="2.721555541s" totalImagesPullingTime="385.962643ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:13:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:13:39.720955457 +0000 UTC m=+1632.125205362" watchObservedRunningTime="2026-07-06 23:13:39.721555541 +0000 UTC m=+1632.125805445"
Jul 06 23:13:40.711335 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:40.711299    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-0f938eb0-64d4f456bf-h67tq"
Jul 06 23:13:41.885261 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:41.885225    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj"]
Jul 06 23:13:41.885764 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:41.885442    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj" podUID="b9e25e93-c184-4a91-a364-ec2e36127940" containerName="istio-proxy" containerID="cri-o://0c313d158ee0925280ae26849aa53e1eeff98303bc681a8a353f5679054946e5" gracePeriod=30
Jul 06 23:13:42.044121 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:42.044089    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-0f938eb0-64d4f456bf-h67tq"]
Jul 06 23:13:42.721464 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:42.721428    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-0f938eb0-64d4f456bf-h67tq" podUID="afef2c1d-731c-4f27-be8f-eb5c86e52349" containerName="maas-api" containerID="cri-o://2ee2f6bd5bc56e95df108deabfb67d3fccb1bf4e1bcea142bc82e5631b4716a9" gracePeriod=30
Jul 06 23:13:43.001883 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:43.001819    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-0f938eb0-64d4f456bf-h67tq"
Jul 06 23:13:43.097457 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:43.097428    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afef2c1d-731c-4f27-be8f-eb5c86e52349-openshift-service-ca\") pod \"afef2c1d-731c-4f27-be8f-eb5c86e52349\" (UID: \"afef2c1d-731c-4f27-be8f-eb5c86e52349\") "
Jul 06 23:13:43.097646 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:43.097494    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afef2c1d-731c-4f27-be8f-eb5c86e52349-maas-api-tls\") pod \"afef2c1d-731c-4f27-be8f-eb5c86e52349\" (UID: \"afef2c1d-731c-4f27-be8f-eb5c86e52349\") "
Jul 06 23:13:43.097646 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:43.097545    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6kxkm\" (UniqueName: \"kubernetes.io/projected/afef2c1d-731c-4f27-be8f-eb5c86e52349-kube-api-access-6kxkm\") pod \"afef2c1d-731c-4f27-be8f-eb5c86e52349\" (UID: \"afef2c1d-731c-4f27-be8f-eb5c86e52349\") "
Jul 06 23:13:43.097796 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:43.097768    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afef2c1d-731c-4f27-be8f-eb5c86e52349-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "afef2c1d-731c-4f27-be8f-eb5c86e52349" (UID: "afef2c1d-731c-4f27-be8f-eb5c86e52349"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 23:13:43.099596 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:43.099560    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afef2c1d-731c-4f27-be8f-eb5c86e52349-kube-api-access-6kxkm" (OuterVolumeSpecName: "kube-api-access-6kxkm") pod "afef2c1d-731c-4f27-be8f-eb5c86e52349" (UID: "afef2c1d-731c-4f27-be8f-eb5c86e52349"). InnerVolumeSpecName "kube-api-access-6kxkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 23:13:43.099596 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:43.099574    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afef2c1d-731c-4f27-be8f-eb5c86e52349-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "afef2c1d-731c-4f27-be8f-eb5c86e52349" (UID: "afef2c1d-731c-4f27-be8f-eb5c86e52349"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 23:13:43.198844 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:43.198814    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kxkm\" (UniqueName: \"kubernetes.io/projected/afef2c1d-731c-4f27-be8f-eb5c86e52349-kube-api-access-6kxkm\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:13:43.198844 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:43.198840    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afef2c1d-731c-4f27-be8f-eb5c86e52349-openshift-service-ca\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:13:43.198844 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:43.198849    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afef2c1d-731c-4f27-be8f-eb5c86e52349-maas-api-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:13:43.726933 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:43.726898    2569 generic.go:356] "Generic (PLEG): container finished" podID="afef2c1d-731c-4f27-be8f-eb5c86e52349" containerID="2ee2f6bd5bc56e95df108deabfb67d3fccb1bf4e1bcea142bc82e5631b4716a9" exitCode=0
Jul 06 23:13:43.727102 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:43.726947    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-0f938eb0-64d4f456bf-h67tq" event={"ID":"afef2c1d-731c-4f27-be8f-eb5c86e52349","Type":"ContainerDied","Data":"2ee2f6bd5bc56e95df108deabfb67d3fccb1bf4e1bcea142bc82e5631b4716a9"}
Jul 06 23:13:43.727102 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:43.726983    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-0f938eb0-64d4f456bf-h67tq" event={"ID":"afef2c1d-731c-4f27-be8f-eb5c86e52349","Type":"ContainerDied","Data":"2572c97b421cf019661165a1f98113be27a4b8a2d1b93baabff97752b9e332d2"}
Jul 06 23:13:43.727102 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:43.727005    2569 scope.go:117] "RemoveContainer" containerID="2ee2f6bd5bc56e95df108deabfb67d3fccb1bf4e1bcea142bc82e5631b4716a9"
Jul 06 23:13:43.727102 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:43.726947    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-0f938eb0-64d4f456bf-h67tq"
Jul 06 23:13:43.735948 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:43.735930    2569 scope.go:117] "RemoveContainer" containerID="2ee2f6bd5bc56e95df108deabfb67d3fccb1bf4e1bcea142bc82e5631b4716a9"
Jul 06 23:13:43.736239 ip-10-0-134-116 kubenswrapper[2569]: E0706 23:13:43.736219    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ee2f6bd5bc56e95df108deabfb67d3fccb1bf4e1bcea142bc82e5631b4716a9\": container with ID starting with 2ee2f6bd5bc56e95df108deabfb67d3fccb1bf4e1bcea142bc82e5631b4716a9 not found: ID does not exist" containerID="2ee2f6bd5bc56e95df108deabfb67d3fccb1bf4e1bcea142bc82e5631b4716a9"
Jul 06 23:13:43.736289 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:43.736248    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ee2f6bd5bc56e95df108deabfb67d3fccb1bf4e1bcea142bc82e5631b4716a9"} err="failed to get container status \"2ee2f6bd5bc56e95df108deabfb67d3fccb1bf4e1bcea142bc82e5631b4716a9\": rpc error: code = NotFound desc = could not find container \"2ee2f6bd5bc56e95df108deabfb67d3fccb1bf4e1bcea142bc82e5631b4716a9\": container with ID starting with 2ee2f6bd5bc56e95df108deabfb67d3fccb1bf4e1bcea142bc82e5631b4716a9 not found: ID does not exist"
Jul 06 23:13:43.747442 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:43.747406    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-0f938eb0-64d4f456bf-h67tq"]
Jul 06 23:13:43.749681 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:43.749662    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-0f938eb0-64d4f456bf-h67tq"]
Jul 06 23:13:44.170082 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:44.170051    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afef2c1d-731c-4f27-be8f-eb5c86e52349" path="/var/lib/kubelet/pods/afef2c1d-731c-4f27-be8f-eb5c86e52349/volumes"
Jul 06 23:13:48.247543 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:48.247486    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj"
Jul 06 23:13:48.344611 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:48.344574    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e25e93-c184-4a91-a364-ec2e36127940-credential-socket\") pod \"b9e25e93-c184-4a91-a364-ec2e36127940\" (UID: \"b9e25e93-c184-4a91-a364-ec2e36127940\") "
Jul 06 23:13:48.344611 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:48.344610    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9e25e93-c184-4a91-a364-ec2e36127940-istio-data\") pod \"b9e25e93-c184-4a91-a364-ec2e36127940\" (UID: \"b9e25e93-c184-4a91-a364-ec2e36127940\") "
Jul 06 23:13:48.344820 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:48.344628    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9e25e93-c184-4a91-a364-ec2e36127940-istio-token\") pod \"b9e25e93-c184-4a91-a364-ec2e36127940\" (UID: \"b9e25e93-c184-4a91-a364-ec2e36127940\") "
Jul 06 23:13:48.344820 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:48.344652    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7p2zd\" (UniqueName: \"kubernetes.io/projected/b9e25e93-c184-4a91-a364-ec2e36127940-kube-api-access-7p2zd\") pod \"b9e25e93-c184-4a91-a364-ec2e36127940\" (UID: \"b9e25e93-c184-4a91-a364-ec2e36127940\") "
Jul 06 23:13:48.344820 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:48.344681    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9e25e93-c184-4a91-a364-ec2e36127940-workload-certs\") pod \"b9e25e93-c184-4a91-a364-ec2e36127940\" (UID: \"b9e25e93-c184-4a91-a364-ec2e36127940\") "
Jul 06 23:13:48.344820 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:48.344715    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9e25e93-c184-4a91-a364-ec2e36127940-istiod-ca-cert\") pod \"b9e25e93-c184-4a91-a364-ec2e36127940\" (UID: \"b9e25e93-c184-4a91-a364-ec2e36127940\") "
Jul 06 23:13:48.344992 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:48.344952    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9e25e93-c184-4a91-a364-ec2e36127940-istio-podinfo\") pod \"b9e25e93-c184-4a91-a364-ec2e36127940\" (UID: \"b9e25e93-c184-4a91-a364-ec2e36127940\") "
Jul 06 23:13:48.345049 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:48.344958    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e25e93-c184-4a91-a364-ec2e36127940-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b9e25e93-c184-4a91-a364-ec2e36127940" (UID: "b9e25e93-c184-4a91-a364-ec2e36127940"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:13:48.345049 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:48.345032    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e25e93-c184-4a91-a364-ec2e36127940-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b9e25e93-c184-4a91-a364-ec2e36127940" (UID: "b9e25e93-c184-4a91-a364-ec2e36127940"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:13:48.345160 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:48.345038    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e25e93-c184-4a91-a364-ec2e36127940-workload-socket\") pod \"b9e25e93-c184-4a91-a364-ec2e36127940\" (UID: \"b9e25e93-c184-4a91-a364-ec2e36127940\") "
Jul 06 23:13:48.345160 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:48.345114    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9e25e93-c184-4a91-a364-ec2e36127940-istio-envoy\") pod \"b9e25e93-c184-4a91-a364-ec2e36127940\" (UID: \"b9e25e93-c184-4a91-a364-ec2e36127940\") "
Jul 06 23:13:48.345293 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:48.345255    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e25e93-c184-4a91-a364-ec2e36127940-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b9e25e93-c184-4a91-a364-ec2e36127940" (UID: "b9e25e93-c184-4a91-a364-ec2e36127940"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:13:48.345399 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:48.345376    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9e25e93-c184-4a91-a364-ec2e36127940-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b9e25e93-c184-4a91-a364-ec2e36127940" (UID: "b9e25e93-c184-4a91-a364-ec2e36127940"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 23:13:48.345504 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:48.345486    2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e25e93-c184-4a91-a364-ec2e36127940-credential-socket\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:13:48.345570 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:48.345509    2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9e25e93-c184-4a91-a364-ec2e36127940-workload-certs\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:13:48.345570 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:48.345523    2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9e25e93-c184-4a91-a364-ec2e36127940-istiod-ca-cert\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:13:48.345570 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:48.345535    2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e25e93-c184-4a91-a364-ec2e36127940-workload-socket\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:13:48.345785 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:48.345760    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e25e93-c184-4a91-a364-ec2e36127940-istio-data" (OuterVolumeSpecName: "istio-data") pod "b9e25e93-c184-4a91-a364-ec2e36127940" (UID: "b9e25e93-c184-4a91-a364-ec2e36127940"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:13:48.347029 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:48.347007    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9e25e93-c184-4a91-a364-ec2e36127940-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b9e25e93-c184-4a91-a364-ec2e36127940" (UID: "b9e25e93-c184-4a91-a364-ec2e36127940"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 06 23:13:48.347148 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:48.347032    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9e25e93-c184-4a91-a364-ec2e36127940-istio-token" (OuterVolumeSpecName: "istio-token") pod "b9e25e93-c184-4a91-a364-ec2e36127940" (UID: "b9e25e93-c184-4a91-a364-ec2e36127940"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 23:13:48.347148 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:48.347100    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9e25e93-c184-4a91-a364-ec2e36127940-kube-api-access-7p2zd" (OuterVolumeSpecName: "kube-api-access-7p2zd") pod "b9e25e93-c184-4a91-a364-ec2e36127940" (UID: "b9e25e93-c184-4a91-a364-ec2e36127940"). InnerVolumeSpecName "kube-api-access-7p2zd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 23:13:48.347277 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:48.347146    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e25e93-c184-4a91-a364-ec2e36127940-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b9e25e93-c184-4a91-a364-ec2e36127940" (UID: "b9e25e93-c184-4a91-a364-ec2e36127940"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:13:48.446490 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:48.446466    2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9e25e93-c184-4a91-a364-ec2e36127940-istio-podinfo\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:13:48.446490 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:48.446488    2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9e25e93-c184-4a91-a364-ec2e36127940-istio-envoy\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:13:48.446652 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:48.446497    2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9e25e93-c184-4a91-a364-ec2e36127940-istio-data\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:13:48.446652 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:48.446505    2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9e25e93-c184-4a91-a364-ec2e36127940-istio-token\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:13:48.446652 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:48.446513    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p2zd\" (UniqueName: \"kubernetes.io/projected/b9e25e93-c184-4a91-a364-ec2e36127940-kube-api-access-7p2zd\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:13:48.753740 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:48.753703    2569 generic.go:356] "Generic (PLEG): container finished" podID="b9e25e93-c184-4a91-a364-ec2e36127940" containerID="0c313d158ee0925280ae26849aa53e1eeff98303bc681a8a353f5679054946e5" exitCode=0
Jul 06 23:13:48.753927 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:48.753749    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj"
Jul 06 23:13:48.753927 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:48.753784    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj" event={"ID":"b9e25e93-c184-4a91-a364-ec2e36127940","Type":"ContainerDied","Data":"0c313d158ee0925280ae26849aa53e1eeff98303bc681a8a353f5679054946e5"}
Jul 06 23:13:48.753927 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:48.753841    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj" event={"ID":"b9e25e93-c184-4a91-a364-ec2e36127940","Type":"ContainerDied","Data":"820d8c7532a64297cc4928b11079a850eebb40c5ef26ddbf5f6948b142006cf8"}
Jul 06 23:13:48.753927 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:48.753863    2569 scope.go:117] "RemoveContainer" containerID="0c313d158ee0925280ae26849aa53e1eeff98303bc681a8a353f5679054946e5"
Jul 06 23:13:48.762982 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:48.762965    2569 scope.go:117] "RemoveContainer" containerID="0c313d158ee0925280ae26849aa53e1eeff98303bc681a8a353f5679054946e5"
Jul 06 23:13:48.763227 ip-10-0-134-116 kubenswrapper[2569]: E0706 23:13:48.763206    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c313d158ee0925280ae26849aa53e1eeff98303bc681a8a353f5679054946e5\": container with ID starting with 0c313d158ee0925280ae26849aa53e1eeff98303bc681a8a353f5679054946e5 not found: ID does not exist" containerID="0c313d158ee0925280ae26849aa53e1eeff98303bc681a8a353f5679054946e5"
Jul 06 23:13:48.763366 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:48.763232    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c313d158ee0925280ae26849aa53e1eeff98303bc681a8a353f5679054946e5"} err="failed to get container status \"0c313d158ee0925280ae26849aa53e1eeff98303bc681a8a353f5679054946e5\": rpc error: code = NotFound desc = could not find container \"0c313d158ee0925280ae26849aa53e1eeff98303bc681a8a353f5679054946e5\": container with ID starting with 0c313d158ee0925280ae26849aa53e1eeff98303bc681a8a353f5679054946e5 not found: ID does not exist"
Jul 06 23:13:48.775054 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:48.775030    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj"]
Jul 06 23:13:48.780727 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:48.780708    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-0f938eb0-openshift-default-5ddff74d67-55vvj"]
Jul 06 23:13:50.169901 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:13:50.169866    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9e25e93-c184-4a91-a364-ec2e36127940" path="/var/lib/kubelet/pods/b9e25e93-c184-4a91-a364-ec2e36127940/volumes"
Jul 06 23:15:22.442284 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:22.442252    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"]
Jul 06 23:15:22.442761 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:22.442733    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9e25e93-c184-4a91-a364-ec2e36127940" containerName="istio-proxy"
Jul 06 23:15:22.442761 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:22.442751    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9e25e93-c184-4a91-a364-ec2e36127940" containerName="istio-proxy"
Jul 06 23:15:22.442761 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:22.442759    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afef2c1d-731c-4f27-be8f-eb5c86e52349" containerName="maas-api"
Jul 06 23:15:22.442892 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:22.442764    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="afef2c1d-731c-4f27-be8f-eb5c86e52349" containerName="maas-api"
Jul 06 23:15:22.442892 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:22.442823    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9e25e93-c184-4a91-a364-ec2e36127940" containerName="istio-proxy"
Jul 06 23:15:22.442892 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:22.442834    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="afef2c1d-731c-4f27-be8f-eb5c86e52349" containerName="maas-api"
Jul 06 23:15:22.445417 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:22.445396    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"
Jul 06 23:15:22.447870 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:22.447848    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-32a20285-openshift-default-dockercfg-k86qf\""
Jul 06 23:15:22.459782 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:22.459761    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"]
Jul 06 23:15:22.593427 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:22.593388    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b7d6d16-e04b-492b-9c65-ddc86d024423-istiod-ca-cert\") pod \"e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp\" (UID: \"8b7d6d16-e04b-492b-9c65-ddc86d024423\") " pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"
Jul 06 23:15:22.593595 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:22.593450    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6d16-e04b-492b-9c65-ddc86d024423-credential-socket\") pod \"e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp\" (UID: \"8b7d6d16-e04b-492b-9c65-ddc86d024423\") " pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"
Jul 06 23:15:22.593595 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:22.593501    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8vzs\" (UniqueName: \"kubernetes.io/projected/8b7d6d16-e04b-492b-9c65-ddc86d024423-kube-api-access-v8vzs\") pod \"e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp\" (UID: \"8b7d6d16-e04b-492b-9c65-ddc86d024423\") " pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"
Jul 06 23:15:22.593595 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:22.593558    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6d16-e04b-492b-9c65-ddc86d024423-workload-socket\") pod \"e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp\" (UID: \"8b7d6d16-e04b-492b-9c65-ddc86d024423\") " pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"
Jul 06 23:15:22.593595 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:22.593575    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6d16-e04b-492b-9c65-ddc86d024423-istio-data\") pod \"e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp\" (UID: \"8b7d6d16-e04b-492b-9c65-ddc86d024423\") " pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"
Jul 06 23:15:22.593595 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:22.593589    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b7d6d16-e04b-492b-9c65-ddc86d024423-istio-podinfo\") pod \"e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp\" (UID: \"8b7d6d16-e04b-492b-9c65-ddc86d024423\") " pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"
Jul 06 23:15:22.593849 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:22.593613    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6d16-e04b-492b-9c65-ddc86d024423-workload-certs\") pod \"e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp\" (UID: \"8b7d6d16-e04b-492b-9c65-ddc86d024423\") " pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"
Jul 06 23:15:22.593849 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:22.593721    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6d16-e04b-492b-9c65-ddc86d024423-istio-envoy\") pod \"e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp\" (UID: \"8b7d6d16-e04b-492b-9c65-ddc86d024423\") " pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"
Jul 06 23:15:22.593849 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:22.593765    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b7d6d16-e04b-492b-9c65-ddc86d024423-istio-token\") pod \"e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp\" (UID: \"8b7d6d16-e04b-492b-9c65-ddc86d024423\") " pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"
Jul 06 23:15:22.695033 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:22.694970    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6d16-e04b-492b-9c65-ddc86d024423-credential-socket\") pod \"e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp\" (UID: \"8b7d6d16-e04b-492b-9c65-ddc86d024423\") " pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"
Jul 06 23:15:22.695033 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:22.695001    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8vzs\" (UniqueName: \"kubernetes.io/projected/8b7d6d16-e04b-492b-9c65-ddc86d024423-kube-api-access-v8vzs\") pod \"e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp\" (UID: \"8b7d6d16-e04b-492b-9c65-ddc86d024423\") " pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"
Jul 06 23:15:22.695033 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:22.695028    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6d16-e04b-492b-9c65-ddc86d024423-workload-socket\") pod \"e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp\" (UID: \"8b7d6d16-e04b-492b-9c65-ddc86d024423\") " pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"
Jul 06 23:15:22.695328 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:22.695051    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6d16-e04b-492b-9c65-ddc86d024423-istio-data\") pod \"e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp\" (UID: \"8b7d6d16-e04b-492b-9c65-ddc86d024423\") " pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"
Jul 06 23:15:22.695328 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:22.695076    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b7d6d16-e04b-492b-9c65-ddc86d024423-istio-podinfo\") pod \"e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp\" (UID: \"8b7d6d16-e04b-492b-9c65-ddc86d024423\") " pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"
Jul 06 23:15:22.695328 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:22.695110    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6d16-e04b-492b-9c65-ddc86d024423-workload-certs\") pod \"e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp\" (UID: \"8b7d6d16-e04b-492b-9c65-ddc86d024423\") " pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"
Jul 06 23:15:22.695328 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:22.695154    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6d16-e04b-492b-9c65-ddc86d024423-istio-envoy\") pod \"e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp\" (UID: \"8b7d6d16-e04b-492b-9c65-ddc86d024423\") " pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"
Jul 06 23:15:22.695328 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:22.695208    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b7d6d16-e04b-492b-9c65-ddc86d024423-istio-token\") pod \"e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp\" (UID: \"8b7d6d16-e04b-492b-9c65-ddc86d024423\") " pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"
Jul 06 23:15:22.695328 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:22.695251    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b7d6d16-e04b-492b-9c65-ddc86d024423-istiod-ca-cert\") pod \"e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp\" (UID: \"8b7d6d16-e04b-492b-9c65-ddc86d024423\") " pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"
Jul 06 23:15:22.695641 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:22.695458    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6d16-e04b-492b-9c65-ddc86d024423-credential-socket\") pod \"e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp\" (UID: \"8b7d6d16-e04b-492b-9c65-ddc86d024423\") " pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"
Jul 06 23:15:22.695641 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:22.695605    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6d16-e04b-492b-9c65-ddc86d024423-workload-certs\") pod \"e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp\" (UID: \"8b7d6d16-e04b-492b-9c65-ddc86d024423\") " pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"
Jul 06 23:15:22.695727 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:22.695675    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6d16-e04b-492b-9c65-ddc86d024423-workload-socket\") pod \"e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp\" (UID: \"8b7d6d16-e04b-492b-9c65-ddc86d024423\") " pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"
Jul 06 23:15:22.695727 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:22.695704    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6d16-e04b-492b-9c65-ddc86d024423-istio-data\") pod \"e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp\" (UID: \"8b7d6d16-e04b-492b-9c65-ddc86d024423\") " pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"
Jul 06 23:15:22.695959 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:22.695936    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b7d6d16-e04b-492b-9c65-ddc86d024423-istiod-ca-cert\") pod \"e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp\" (UID: \"8b7d6d16-e04b-492b-9c65-ddc86d024423\") " pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"
Jul 06 23:15:22.697624 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:22.697595    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6d16-e04b-492b-9c65-ddc86d024423-istio-envoy\") pod \"e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp\" (UID: \"8b7d6d16-e04b-492b-9c65-ddc86d024423\") " pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"
Jul 06 23:15:22.697711 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:22.697599    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b7d6d16-e04b-492b-9c65-ddc86d024423-istio-podinfo\") pod \"e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp\" (UID: \"8b7d6d16-e04b-492b-9c65-ddc86d024423\") " pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"
Jul 06 23:15:22.703171 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:22.703147    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8vzs\" (UniqueName: \"kubernetes.io/projected/8b7d6d16-e04b-492b-9c65-ddc86d024423-kube-api-access-v8vzs\") pod \"e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp\" (UID: \"8b7d6d16-e04b-492b-9c65-ddc86d024423\") " pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"
Jul 06 23:15:22.703881 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:22.703863    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b7d6d16-e04b-492b-9c65-ddc86d024423-istio-token\") pod \"e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp\" (UID: \"8b7d6d16-e04b-492b-9c65-ddc86d024423\") " pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"
Jul 06 23:15:22.757088 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:22.757062    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"
Jul 06 23:15:22.888563 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:22.888539    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"]
Jul 06 23:15:22.890390 ip-10-0-134-116 kubenswrapper[2569]: W0706 23:15:22.890359    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b7d6d16_e04b_492b_9c65_ddc86d024423.slice/crio-5326c03d33f4cd54e1b9863a456dfe01dc5531901aaf337285e049b10aa88fa1 WatchSource:0}: Error finding container 5326c03d33f4cd54e1b9863a456dfe01dc5531901aaf337285e049b10aa88fa1: Status 404 returned error can't find the container with id 5326c03d33f4cd54e1b9863a456dfe01dc5531901aaf337285e049b10aa88fa1
Jul 06 23:15:22.892747 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:22.892709    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 23:15:22.892881 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:22.892783    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 23:15:22.892881 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:22.892824    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 23:15:23.220933 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:23.220845    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp" event={"ID":"8b7d6d16-e04b-492b-9c65-ddc86d024423","Type":"ContainerStarted","Data":"ec6404df3c5f8e8b323ce77572fd68de28df0eb5e7ca07c0a79a9de511528789"}
Jul 06 23:15:23.220933 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:23.220899    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp" event={"ID":"8b7d6d16-e04b-492b-9c65-ddc86d024423","Type":"ContainerStarted","Data":"5326c03d33f4cd54e1b9863a456dfe01dc5531901aaf337285e049b10aa88fa1"}
Jul 06 23:15:23.241825 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:23.241773    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp" podStartSLOduration=1.241753205 podStartE2EDuration="1.241753205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:15:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:15:23.238574229 +0000 UTC m=+1735.642824138" watchObservedRunningTime="2026-07-06 23:15:23.241753205 +0000 UTC m=+1735.646003111"
Jul 06 23:15:23.757392 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:23.757359    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"
Jul 06 23:15:23.762094 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:23.762069    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"
Jul 06 23:15:24.225750 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:24.225718    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"
Jul 06 23:15:24.226672 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:24.226653    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"
Jul 06 23:15:25.203525 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:25.203461    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-32a20285-855c9b7bb7-6jsg7"]
Jul 06 23:15:25.206790 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:25.206758    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-32a20285-855c9b7bb7-6jsg7"
Jul 06 23:15:25.209354 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:25.209332    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-32a20285-serving-cert\""
Jul 06 23:15:25.217551 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:25.217525    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-32a20285-855c9b7bb7-6jsg7"]
Jul 06 23:15:25.314579 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:25.314550    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsbgn\" (UniqueName: \"kubernetes.io/projected/fde166c9-923d-4202-8454-90bc5b608526-kube-api-access-xsbgn\") pod \"maas-api-e2e-mt-32a20285-855c9b7bb7-6jsg7\" (UID: \"fde166c9-923d-4202-8454-90bc5b608526\") " pod="opendatahub/maas-api-e2e-mt-32a20285-855c9b7bb7-6jsg7"
Jul 06 23:15:25.314735 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:25.314614    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fde166c9-923d-4202-8454-90bc5b608526-maas-api-tls\") pod \"maas-api-e2e-mt-32a20285-855c9b7bb7-6jsg7\" (UID: \"fde166c9-923d-4202-8454-90bc5b608526\") " pod="opendatahub/maas-api-e2e-mt-32a20285-855c9b7bb7-6jsg7"
Jul 06 23:15:25.314735 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:25.314654    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fde166c9-923d-4202-8454-90bc5b608526-openshift-service-ca\") pod \"maas-api-e2e-mt-32a20285-855c9b7bb7-6jsg7\" (UID: \"fde166c9-923d-4202-8454-90bc5b608526\") " pod="opendatahub/maas-api-e2e-mt-32a20285-855c9b7bb7-6jsg7"
Jul 06 23:15:25.415427 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:25.415399    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsbgn\" (UniqueName: \"kubernetes.io/projected/fde166c9-923d-4202-8454-90bc5b608526-kube-api-access-xsbgn\") pod \"maas-api-e2e-mt-32a20285-855c9b7bb7-6jsg7\" (UID: \"fde166c9-923d-4202-8454-90bc5b608526\") " pod="opendatahub/maas-api-e2e-mt-32a20285-855c9b7bb7-6jsg7"
Jul 06 23:15:25.415560 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:25.415446    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fde166c9-923d-4202-8454-90bc5b608526-maas-api-tls\") pod \"maas-api-e2e-mt-32a20285-855c9b7bb7-6jsg7\" (UID: \"fde166c9-923d-4202-8454-90bc5b608526\") " pod="opendatahub/maas-api-e2e-mt-32a20285-855c9b7bb7-6jsg7"
Jul 06 23:15:25.415560 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:25.415481    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fde166c9-923d-4202-8454-90bc5b608526-openshift-service-ca\") pod \"maas-api-e2e-mt-32a20285-855c9b7bb7-6jsg7\" (UID: \"fde166c9-923d-4202-8454-90bc5b608526\") " pod="opendatahub/maas-api-e2e-mt-32a20285-855c9b7bb7-6jsg7"
Jul 06 23:15:25.416239 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:25.416222    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fde166c9-923d-4202-8454-90bc5b608526-openshift-service-ca\") pod \"maas-api-e2e-mt-32a20285-855c9b7bb7-6jsg7\" (UID: \"fde166c9-923d-4202-8454-90bc5b608526\") " pod="opendatahub/maas-api-e2e-mt-32a20285-855c9b7bb7-6jsg7"
Jul 06 23:15:25.417975 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:25.417955    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fde166c9-923d-4202-8454-90bc5b608526-maas-api-tls\") pod \"maas-api-e2e-mt-32a20285-855c9b7bb7-6jsg7\" (UID: \"fde166c9-923d-4202-8454-90bc5b608526\") " pod="opendatahub/maas-api-e2e-mt-32a20285-855c9b7bb7-6jsg7"
Jul 06 23:15:25.425612 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:25.425586    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsbgn\" (UniqueName: \"kubernetes.io/projected/fde166c9-923d-4202-8454-90bc5b608526-kube-api-access-xsbgn\") pod \"maas-api-e2e-mt-32a20285-855c9b7bb7-6jsg7\" (UID: \"fde166c9-923d-4202-8454-90bc5b608526\") " pod="opendatahub/maas-api-e2e-mt-32a20285-855c9b7bb7-6jsg7"
Jul 06 23:15:25.520318 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:25.520260    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-32a20285-855c9b7bb7-6jsg7"
Jul 06 23:15:25.652379 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:25.652356    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-32a20285-855c9b7bb7-6jsg7"]
Jul 06 23:15:25.654780 ip-10-0-134-116 kubenswrapper[2569]: W0706 23:15:25.654751    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfde166c9_923d_4202_8454_90bc5b608526.slice/crio-15921b80fc3aa939326db2dead757475d82d9a28372e25acc105aee3bcabefbc WatchSource:0}: Error finding container 15921b80fc3aa939326db2dead757475d82d9a28372e25acc105aee3bcabefbc: Status 404 returned error can't find the container with id 15921b80fc3aa939326db2dead757475d82d9a28372e25acc105aee3bcabefbc
Jul 06 23:15:26.238309 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:26.238270    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-32a20285-855c9b7bb7-6jsg7" event={"ID":"fde166c9-923d-4202-8454-90bc5b608526","Type":"ContainerStarted","Data":"15921b80fc3aa939326db2dead757475d82d9a28372e25acc105aee3bcabefbc"}
Jul 06 23:15:27.244918 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:27.244878    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-32a20285-855c9b7bb7-6jsg7" event={"ID":"fde166c9-923d-4202-8454-90bc5b608526","Type":"ContainerStarted","Data":"dcefe4e237667215eb75160e9d68daf4d3260479383950ea88229e47c79ddfda"}
Jul 06 23:15:27.245311 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:27.244971    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-32a20285-855c9b7bb7-6jsg7"
Jul 06 23:15:27.261679 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:27.261625    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-32a20285-855c9b7bb7-6jsg7" podStartSLOduration=1.557051361 podStartE2EDuration="2.261608872s" totalImagesPullingTime="704.557511ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:15:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:15:27.260565011 +0000 UTC m=+1739.664814918" watchObservedRunningTime="2026-07-06 23:15:27.261608872 +0000 UTC m=+1739.665858776"
Jul 06 23:15:33.255435 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:33.255409    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-32a20285-855c9b7bb7-6jsg7"
Jul 06 23:15:37.267706 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:37.267672    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-32a20285-855c9b7bb7-6jsg7"]
Jul 06 23:15:37.268152 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:37.267867    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-32a20285-855c9b7bb7-6jsg7" podUID="fde166c9-923d-4202-8454-90bc5b608526" containerName="maas-api" containerID="cri-o://dcefe4e237667215eb75160e9d68daf4d3260479383950ea88229e47c79ddfda" gracePeriod=30
Jul 06 23:15:37.526901 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:37.526877    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-32a20285-855c9b7bb7-6jsg7"
Jul 06 23:15:37.622697 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:37.622666    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fde166c9-923d-4202-8454-90bc5b608526-openshift-service-ca\") pod \"fde166c9-923d-4202-8454-90bc5b608526\" (UID: \"fde166c9-923d-4202-8454-90bc5b608526\") "
Jul 06 23:15:37.622885 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:37.622720    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsbgn\" (UniqueName: \"kubernetes.io/projected/fde166c9-923d-4202-8454-90bc5b608526-kube-api-access-xsbgn\") pod \"fde166c9-923d-4202-8454-90bc5b608526\" (UID: \"fde166c9-923d-4202-8454-90bc5b608526\") "
Jul 06 23:15:37.622885 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:37.622760    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fde166c9-923d-4202-8454-90bc5b608526-maas-api-tls\") pod \"fde166c9-923d-4202-8454-90bc5b608526\" (UID: \"fde166c9-923d-4202-8454-90bc5b608526\") "
Jul 06 23:15:37.623160 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:37.623123    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fde166c9-923d-4202-8454-90bc5b608526-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fde166c9-923d-4202-8454-90bc5b608526" (UID: "fde166c9-923d-4202-8454-90bc5b608526"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 23:15:37.624927 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:37.624891    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fde166c9-923d-4202-8454-90bc5b608526-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fde166c9-923d-4202-8454-90bc5b608526" (UID: "fde166c9-923d-4202-8454-90bc5b608526"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 23:15:37.625035 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:37.624941    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fde166c9-923d-4202-8454-90bc5b608526-kube-api-access-xsbgn" (OuterVolumeSpecName: "kube-api-access-xsbgn") pod "fde166c9-923d-4202-8454-90bc5b608526" (UID: "fde166c9-923d-4202-8454-90bc5b608526"). InnerVolumeSpecName "kube-api-access-xsbgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 23:15:37.723541 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:37.723511    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fde166c9-923d-4202-8454-90bc5b608526-openshift-service-ca\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:15:37.723541 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:37.723542    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsbgn\" (UniqueName: \"kubernetes.io/projected/fde166c9-923d-4202-8454-90bc5b608526-kube-api-access-xsbgn\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:15:37.723729 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:37.723556    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fde166c9-923d-4202-8454-90bc5b608526-maas-api-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:15:38.307313 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:38.307275    2569 generic.go:356] "Generic (PLEG): container finished" podID="fde166c9-923d-4202-8454-90bc5b608526" containerID="dcefe4e237667215eb75160e9d68daf4d3260479383950ea88229e47c79ddfda" exitCode=0
Jul 06 23:15:38.307755 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:38.307323    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-32a20285-855c9b7bb7-6jsg7"
Jul 06 23:15:38.307755 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:38.307359    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-32a20285-855c9b7bb7-6jsg7" event={"ID":"fde166c9-923d-4202-8454-90bc5b608526","Type":"ContainerDied","Data":"dcefe4e237667215eb75160e9d68daf4d3260479383950ea88229e47c79ddfda"}
Jul 06 23:15:38.307755 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:38.307417    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-32a20285-855c9b7bb7-6jsg7" event={"ID":"fde166c9-923d-4202-8454-90bc5b608526","Type":"ContainerDied","Data":"15921b80fc3aa939326db2dead757475d82d9a28372e25acc105aee3bcabefbc"}
Jul 06 23:15:38.307755 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:38.307444    2569 scope.go:117] "RemoveContainer" containerID="dcefe4e237667215eb75160e9d68daf4d3260479383950ea88229e47c79ddfda"
Jul 06 23:15:38.316531 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:38.316500    2569 scope.go:117] "RemoveContainer" containerID="dcefe4e237667215eb75160e9d68daf4d3260479383950ea88229e47c79ddfda"
Jul 06 23:15:38.316780 ip-10-0-134-116 kubenswrapper[2569]: E0706 23:15:38.316761    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcefe4e237667215eb75160e9d68daf4d3260479383950ea88229e47c79ddfda\": container with ID starting with dcefe4e237667215eb75160e9d68daf4d3260479383950ea88229e47c79ddfda not found: ID does not exist" containerID="dcefe4e237667215eb75160e9d68daf4d3260479383950ea88229e47c79ddfda"
Jul 06 23:15:38.316858 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:38.316788    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcefe4e237667215eb75160e9d68daf4d3260479383950ea88229e47c79ddfda"} err="failed to get container status \"dcefe4e237667215eb75160e9d68daf4d3260479383950ea88229e47c79ddfda\": rpc error: code = NotFound desc = could not find container \"dcefe4e237667215eb75160e9d68daf4d3260479383950ea88229e47c79ddfda\": container with ID starting with dcefe4e237667215eb75160e9d68daf4d3260479383950ea88229e47c79ddfda not found: ID does not exist"
Jul 06 23:15:38.325025 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:38.325005    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-32a20285-855c9b7bb7-6jsg7"]
Jul 06 23:15:38.328899 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:38.328876    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-32a20285-855c9b7bb7-6jsg7"]
Jul 06 23:15:39.918371 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:39.918340    2569 patch_prober.go:28] interesting pod/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": read tcp 10.134.0.2:37150->10.134.0.51:15021: read: connection reset by peer" start-of-body=
Jul 06 23:15:39.918785 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:39.918442    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp" podUID="8b7d6d16-e04b-492b-9c65-ddc86d024423" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": read tcp 10.134.0.2:37150->10.134.0.51:15021: read: connection reset by peer"
Jul 06 23:15:40.170147 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:40.170058    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fde166c9-923d-4202-8454-90bc5b608526" path="/var/lib/kubelet/pods/fde166c9-923d-4202-8454-90bc5b608526/volumes"
Jul 06 23:15:40.319438 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:40.319409    2569 generic.go:356] "Generic (PLEG): container finished" podID="8b7d6d16-e04b-492b-9c65-ddc86d024423" containerID="ec6404df3c5f8e8b323ce77572fd68de28df0eb5e7ca07c0a79a9de511528789" exitCode=0
Jul 06 23:15:40.319614 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:40.319485    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp" event={"ID":"8b7d6d16-e04b-492b-9c65-ddc86d024423","Type":"ContainerDied","Data":"ec6404df3c5f8e8b323ce77572fd68de28df0eb5e7ca07c0a79a9de511528789"}
Jul 06 23:15:40.319903 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:40.319887    2569 scope.go:117] "RemoveContainer" containerID="ec6404df3c5f8e8b323ce77572fd68de28df0eb5e7ca07c0a79a9de511528789"
Jul 06 23:15:40.320868 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:40.320840    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 23:15:40.320928 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:40.320906    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 23:15:40.320964 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:40.320941    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 23:15:41.326338 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:41.326303    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp" event={"ID":"8b7d6d16-e04b-492b-9c65-ddc86d024423","Type":"ContainerStarted","Data":"802f16310462db10e7b8dd33ad6811229ba11b9b186a8de0f807a56985f10ee4"}
Jul 06 23:15:41.758249 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:41.758216    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"
Jul 06 23:15:41.761582 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:41.761561    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"
Jul 06 23:15:42.331282 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:42.331252    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"
Jul 06 23:15:42.332231 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:42.332212    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"
Jul 06 23:15:43.084025 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:43.083990    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"]
Jul 06 23:15:43.335634 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:43.335549    2569 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp" secret="" err="secret \"e2e-mt-32a20285-openshift-default-dockercfg-k86qf\" not found"
Jul 06 23:15:44.340292 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:44.340234    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp" podUID="8b7d6d16-e04b-492b-9c65-ddc86d024423" containerName="istio-proxy" containerID="cri-o://802f16310462db10e7b8dd33ad6811229ba11b9b186a8de0f807a56985f10ee4" gracePeriod=30
Jul 06 23:15:49.593420 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:49.593396    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"
Jul 06 23:15:49.618076 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:49.618021    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6d16-e04b-492b-9c65-ddc86d024423-istio-data\") pod \"8b7d6d16-e04b-492b-9c65-ddc86d024423\" (UID: \"8b7d6d16-e04b-492b-9c65-ddc86d024423\") "
Jul 06 23:15:49.618076 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:49.618059    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b7d6d16-e04b-492b-9c65-ddc86d024423-istio-podinfo\") pod \"8b7d6d16-e04b-492b-9c65-ddc86d024423\" (UID: \"8b7d6d16-e04b-492b-9c65-ddc86d024423\") "
Jul 06 23:15:49.618273 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:49.618088    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8vzs\" (UniqueName: \"kubernetes.io/projected/8b7d6d16-e04b-492b-9c65-ddc86d024423-kube-api-access-v8vzs\") pod \"8b7d6d16-e04b-492b-9c65-ddc86d024423\" (UID: \"8b7d6d16-e04b-492b-9c65-ddc86d024423\") "
Jul 06 23:15:49.618273 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:49.618140    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6d16-e04b-492b-9c65-ddc86d024423-istio-envoy\") pod \"8b7d6d16-e04b-492b-9c65-ddc86d024423\" (UID: \"8b7d6d16-e04b-492b-9c65-ddc86d024423\") "
Jul 06 23:15:49.618273 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:49.618231    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6d16-e04b-492b-9c65-ddc86d024423-workload-socket\") pod \"8b7d6d16-e04b-492b-9c65-ddc86d024423\" (UID: \"8b7d6d16-e04b-492b-9c65-ddc86d024423\") "
Jul 06 23:15:49.618273 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:49.618268    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b7d6d16-e04b-492b-9c65-ddc86d024423-istiod-ca-cert\") pod \"8b7d6d16-e04b-492b-9c65-ddc86d024423\" (UID: \"8b7d6d16-e04b-492b-9c65-ddc86d024423\") "
Jul 06 23:15:49.618500 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:49.618296    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b7d6d16-e04b-492b-9c65-ddc86d024423-istio-token\") pod \"8b7d6d16-e04b-492b-9c65-ddc86d024423\" (UID: \"8b7d6d16-e04b-492b-9c65-ddc86d024423\") "
Jul 06 23:15:49.618500 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:49.618322    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6d16-e04b-492b-9c65-ddc86d024423-workload-certs\") pod \"8b7d6d16-e04b-492b-9c65-ddc86d024423\" (UID: \"8b7d6d16-e04b-492b-9c65-ddc86d024423\") "
Jul 06 23:15:49.618500 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:49.618348    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6d16-e04b-492b-9c65-ddc86d024423-credential-socket\") pod \"8b7d6d16-e04b-492b-9c65-ddc86d024423\" (UID: \"8b7d6d16-e04b-492b-9c65-ddc86d024423\") "
Jul 06 23:15:49.618850 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:49.618823    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b7d6d16-e04b-492b-9c65-ddc86d024423-istio-data" (OuterVolumeSpecName: "istio-data") pod "8b7d6d16-e04b-492b-9c65-ddc86d024423" (UID: "8b7d6d16-e04b-492b-9c65-ddc86d024423"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:15:49.618946 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:49.618806    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b7d6d16-e04b-492b-9c65-ddc86d024423-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8b7d6d16-e04b-492b-9c65-ddc86d024423" (UID: "8b7d6d16-e04b-492b-9c65-ddc86d024423"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:15:49.619088 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:49.619066    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b7d6d16-e04b-492b-9c65-ddc86d024423-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8b7d6d16-e04b-492b-9c65-ddc86d024423" (UID: "8b7d6d16-e04b-492b-9c65-ddc86d024423"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:15:49.620391 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:49.619312    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b7d6d16-e04b-492b-9c65-ddc86d024423-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8b7d6d16-e04b-492b-9c65-ddc86d024423" (UID: "8b7d6d16-e04b-492b-9c65-ddc86d024423"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 23:15:49.620582 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:49.620551    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b7d6d16-e04b-492b-9c65-ddc86d024423-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8b7d6d16-e04b-492b-9c65-ddc86d024423" (UID: "8b7d6d16-e04b-492b-9c65-ddc86d024423"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:15:49.620859 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:49.620765    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b7d6d16-e04b-492b-9c65-ddc86d024423-kube-api-access-v8vzs" (OuterVolumeSpecName: "kube-api-access-v8vzs") pod "8b7d6d16-e04b-492b-9c65-ddc86d024423" (UID: "8b7d6d16-e04b-492b-9c65-ddc86d024423"). InnerVolumeSpecName "kube-api-access-v8vzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 23:15:49.621517 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:49.621490    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b7d6d16-e04b-492b-9c65-ddc86d024423-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8b7d6d16-e04b-492b-9c65-ddc86d024423" (UID: "8b7d6d16-e04b-492b-9c65-ddc86d024423"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:15:49.621714 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:49.621669    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b7d6d16-e04b-492b-9c65-ddc86d024423-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8b7d6d16-e04b-492b-9c65-ddc86d024423" (UID: "8b7d6d16-e04b-492b-9c65-ddc86d024423"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 06 23:15:49.621821 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:49.621744    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b7d6d16-e04b-492b-9c65-ddc86d024423-istio-token" (OuterVolumeSpecName: "istio-token") pod "8b7d6d16-e04b-492b-9c65-ddc86d024423" (UID: "8b7d6d16-e04b-492b-9c65-ddc86d024423"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 23:15:49.719148 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:49.719108    2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6d16-e04b-492b-9c65-ddc86d024423-workload-socket\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:15:49.719148 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:49.719143    2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b7d6d16-e04b-492b-9c65-ddc86d024423-istiod-ca-cert\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:15:49.719366 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:49.719158    2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b7d6d16-e04b-492b-9c65-ddc86d024423-istio-token\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:15:49.719366 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:49.719171    2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6d16-e04b-492b-9c65-ddc86d024423-workload-certs\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:15:49.719366 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:49.719208    2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6d16-e04b-492b-9c65-ddc86d024423-credential-socket\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:15:49.719366 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:49.719222    2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6d16-e04b-492b-9c65-ddc86d024423-istio-data\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:15:49.719366 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:49.719235    2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b7d6d16-e04b-492b-9c65-ddc86d024423-istio-podinfo\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:15:49.719366 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:49.719248    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8vzs\" (UniqueName: \"kubernetes.io/projected/8b7d6d16-e04b-492b-9c65-ddc86d024423-kube-api-access-v8vzs\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:15:49.719366 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:49.719261    2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b7d6d16-e04b-492b-9c65-ddc86d024423-istio-envoy\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:15:50.369814 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:50.369783    2569 generic.go:356] "Generic (PLEG): container finished" podID="8b7d6d16-e04b-492b-9c65-ddc86d024423" containerID="802f16310462db10e7b8dd33ad6811229ba11b9b186a8de0f807a56985f10ee4" exitCode=0
Jul 06 23:15:50.369971 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:50.369843    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"
Jul 06 23:15:50.369971 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:50.369863    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp" event={"ID":"8b7d6d16-e04b-492b-9c65-ddc86d024423","Type":"ContainerDied","Data":"802f16310462db10e7b8dd33ad6811229ba11b9b186a8de0f807a56985f10ee4"}
Jul 06 23:15:50.369971 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:50.369915    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp" event={"ID":"8b7d6d16-e04b-492b-9c65-ddc86d024423","Type":"ContainerDied","Data":"5326c03d33f4cd54e1b9863a456dfe01dc5531901aaf337285e049b10aa88fa1"}
Jul 06 23:15:50.369971 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:50.369936    2569 scope.go:117] "RemoveContainer" containerID="802f16310462db10e7b8dd33ad6811229ba11b9b186a8de0f807a56985f10ee4"
Jul 06 23:15:50.378377 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:50.378363    2569 scope.go:117] "RemoveContainer" containerID="ec6404df3c5f8e8b323ce77572fd68de28df0eb5e7ca07c0a79a9de511528789"
Jul 06 23:15:50.386080 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:50.386065    2569 scope.go:117] "RemoveContainer" containerID="802f16310462db10e7b8dd33ad6811229ba11b9b186a8de0f807a56985f10ee4"
Jul 06 23:15:50.386404 ip-10-0-134-116 kubenswrapper[2569]: E0706 23:15:50.386383    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"802f16310462db10e7b8dd33ad6811229ba11b9b186a8de0f807a56985f10ee4\": container with ID starting with 802f16310462db10e7b8dd33ad6811229ba11b9b186a8de0f807a56985f10ee4 not found: ID does not exist" containerID="802f16310462db10e7b8dd33ad6811229ba11b9b186a8de0f807a56985f10ee4"
Jul 06 23:15:50.386535 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:50.386415    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"802f16310462db10e7b8dd33ad6811229ba11b9b186a8de0f807a56985f10ee4"} err="failed to get container status \"802f16310462db10e7b8dd33ad6811229ba11b9b186a8de0f807a56985f10ee4\": rpc error: code = NotFound desc = could not find container \"802f16310462db10e7b8dd33ad6811229ba11b9b186a8de0f807a56985f10ee4\": container with ID starting with 802f16310462db10e7b8dd33ad6811229ba11b9b186a8de0f807a56985f10ee4 not found: ID does not exist"
Jul 06 23:15:50.386535 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:50.386435    2569 scope.go:117] "RemoveContainer" containerID="ec6404df3c5f8e8b323ce77572fd68de28df0eb5e7ca07c0a79a9de511528789"
Jul 06 23:15:50.386698 ip-10-0-134-116 kubenswrapper[2569]: E0706 23:15:50.386670    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec6404df3c5f8e8b323ce77572fd68de28df0eb5e7ca07c0a79a9de511528789\": container with ID starting with ec6404df3c5f8e8b323ce77572fd68de28df0eb5e7ca07c0a79a9de511528789 not found: ID does not exist" containerID="ec6404df3c5f8e8b323ce77572fd68de28df0eb5e7ca07c0a79a9de511528789"
Jul 06 23:15:50.386753 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:50.386703    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec6404df3c5f8e8b323ce77572fd68de28df0eb5e7ca07c0a79a9de511528789"} err="failed to get container status \"ec6404df3c5f8e8b323ce77572fd68de28df0eb5e7ca07c0a79a9de511528789\": rpc error: code = NotFound desc = could not find container \"ec6404df3c5f8e8b323ce77572fd68de28df0eb5e7ca07c0a79a9de511528789\": container with ID starting with ec6404df3c5f8e8b323ce77572fd68de28df0eb5e7ca07c0a79a9de511528789 not found: ID does not exist"
Jul 06 23:15:50.388739 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:50.388721    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"]
Jul 06 23:15:50.392361 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:50.392343    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-32a20285-openshift-default-b6fbb6987-4hsjp"]
Jul 06 23:15:52.170218 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:15:52.170165    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b7d6d16-e04b-492b-9c65-ddc86d024423" path="/var/lib/kubelet/pods/8b7d6d16-e04b-492b-9c65-ddc86d024423/volumes"
Jul 06 23:16:02.289612 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.289578    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"]
Jul 06 23:16:02.290126 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.290109    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b7d6d16-e04b-492b-9c65-ddc86d024423" containerName="istio-proxy"
Jul 06 23:16:02.290190 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.290128    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b7d6d16-e04b-492b-9c65-ddc86d024423" containerName="istio-proxy"
Jul 06 23:16:02.290190 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.290148    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fde166c9-923d-4202-8454-90bc5b608526" containerName="maas-api"
Jul 06 23:16:02.290190 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.290156    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fde166c9-923d-4202-8454-90bc5b608526" containerName="maas-api"
Jul 06 23:16:02.290303 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.290174    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b7d6d16-e04b-492b-9c65-ddc86d024423" containerName="istio-proxy"
Jul 06 23:16:02.290303 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.290199    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b7d6d16-e04b-492b-9c65-ddc86d024423" containerName="istio-proxy"
Jul 06 23:16:02.290303 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.290293    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b7d6d16-e04b-492b-9c65-ddc86d024423" containerName="istio-proxy"
Jul 06 23:16:02.290413 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.290307    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b7d6d16-e04b-492b-9c65-ddc86d024423" containerName="istio-proxy"
Jul 06 23:16:02.290413 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.290318    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="fde166c9-923d-4202-8454-90bc5b608526" containerName="maas-api"
Jul 06 23:16:02.293051 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.293023    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"
Jul 06 23:16:02.295923 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.295900    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-93d8da3f-openshift-default-dockercfg-gtvg2\""
Jul 06 23:16:02.304623 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.304597    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"]
Jul 06 23:16:02.425656 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.425625    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e9d91df-a00e-4a48-a494-75050b179100-istio-envoy\") pod \"e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5\" (UID: \"5e9d91df-a00e-4a48-a494-75050b179100\") " pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"
Jul 06 23:16:02.425656 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.425661    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e9d91df-a00e-4a48-a494-75050b179100-istio-token\") pod \"e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5\" (UID: \"5e9d91df-a00e-4a48-a494-75050b179100\") " pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"
Jul 06 23:16:02.425858 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.425684    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e9d91df-a00e-4a48-a494-75050b179100-istio-data\") pod \"e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5\" (UID: \"5e9d91df-a00e-4a48-a494-75050b179100\") " pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"
Jul 06 23:16:02.425858 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.425805    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnz87\" (UniqueName: \"kubernetes.io/projected/5e9d91df-a00e-4a48-a494-75050b179100-kube-api-access-gnz87\") pod \"e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5\" (UID: \"5e9d91df-a00e-4a48-a494-75050b179100\") " pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"
Jul 06 23:16:02.425931 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.425873    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e9d91df-a00e-4a48-a494-75050b179100-istio-podinfo\") pod \"e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5\" (UID: \"5e9d91df-a00e-4a48-a494-75050b179100\") " pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"
Jul 06 23:16:02.425931 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.425911    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9d91df-a00e-4a48-a494-75050b179100-workload-socket\") pod \"e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5\" (UID: \"5e9d91df-a00e-4a48-a494-75050b179100\") " pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"
Jul 06 23:16:02.425995 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.425945    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9d91df-a00e-4a48-a494-75050b179100-credential-socket\") pod \"e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5\" (UID: \"5e9d91df-a00e-4a48-a494-75050b179100\") " pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"
Jul 06 23:16:02.425995 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.425983    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e9d91df-a00e-4a48-a494-75050b179100-istiod-ca-cert\") pod \"e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5\" (UID: \"5e9d91df-a00e-4a48-a494-75050b179100\") " pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"
Jul 06 23:16:02.426065 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.426036    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e9d91df-a00e-4a48-a494-75050b179100-workload-certs\") pod \"e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5\" (UID: \"5e9d91df-a00e-4a48-a494-75050b179100\") " pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"
Jul 06 23:16:02.526480 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.526446    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e9d91df-a00e-4a48-a494-75050b179100-istio-envoy\") pod \"e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5\" (UID: \"5e9d91df-a00e-4a48-a494-75050b179100\") " pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"
Jul 06 23:16:02.526480 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.526484    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e9d91df-a00e-4a48-a494-75050b179100-istio-token\") pod \"e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5\" (UID: \"5e9d91df-a00e-4a48-a494-75050b179100\") " pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"
Jul 06 23:16:02.526731 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.526504    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e9d91df-a00e-4a48-a494-75050b179100-istio-data\") pod \"e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5\" (UID: \"5e9d91df-a00e-4a48-a494-75050b179100\") " pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"
Jul 06 23:16:02.526731 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.526530    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnz87\" (UniqueName: \"kubernetes.io/projected/5e9d91df-a00e-4a48-a494-75050b179100-kube-api-access-gnz87\") pod \"e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5\" (UID: \"5e9d91df-a00e-4a48-a494-75050b179100\") " pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"
Jul 06 23:16:02.526731 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.526567    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e9d91df-a00e-4a48-a494-75050b179100-istio-podinfo\") pod \"e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5\" (UID: \"5e9d91df-a00e-4a48-a494-75050b179100\") " pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"
Jul 06 23:16:02.526731 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.526602    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9d91df-a00e-4a48-a494-75050b179100-workload-socket\") pod \"e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5\" (UID: \"5e9d91df-a00e-4a48-a494-75050b179100\") " pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"
Jul 06 23:16:02.526951 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.526826    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9d91df-a00e-4a48-a494-75050b179100-credential-socket\") pod \"e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5\" (UID: \"5e9d91df-a00e-4a48-a494-75050b179100\") " pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"
Jul 06 23:16:02.526951 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.526891    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e9d91df-a00e-4a48-a494-75050b179100-istiod-ca-cert\") pod \"e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5\" (UID: \"5e9d91df-a00e-4a48-a494-75050b179100\") " pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"
Jul 06 23:16:02.526951 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.526933    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e9d91df-a00e-4a48-a494-75050b179100-workload-certs\") pod \"e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5\" (UID: \"5e9d91df-a00e-4a48-a494-75050b179100\") " pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"
Jul 06 23:16:02.526951 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.526937    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e9d91df-a00e-4a48-a494-75050b179100-istio-data\") pod \"e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5\" (UID: \"5e9d91df-a00e-4a48-a494-75050b179100\") " pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"
Jul 06 23:16:02.526951 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.526950    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9d91df-a00e-4a48-a494-75050b179100-workload-socket\") pod \"e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5\" (UID: \"5e9d91df-a00e-4a48-a494-75050b179100\") " pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"
Jul 06 23:16:02.527297 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.527216    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9d91df-a00e-4a48-a494-75050b179100-credential-socket\") pod \"e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5\" (UID: \"5e9d91df-a00e-4a48-a494-75050b179100\") " pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"
Jul 06 23:16:02.527297 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.527283    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e9d91df-a00e-4a48-a494-75050b179100-workload-certs\") pod \"e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5\" (UID: \"5e9d91df-a00e-4a48-a494-75050b179100\") " pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"
Jul 06 23:16:02.527644 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.527627    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e9d91df-a00e-4a48-a494-75050b179100-istiod-ca-cert\") pod \"e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5\" (UID: \"5e9d91df-a00e-4a48-a494-75050b179100\") " pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"
Jul 06 23:16:02.528825 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.528804    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e9d91df-a00e-4a48-a494-75050b179100-istio-envoy\") pod \"e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5\" (UID: \"5e9d91df-a00e-4a48-a494-75050b179100\") " pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"
Jul 06 23:16:02.529720 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.529695    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e9d91df-a00e-4a48-a494-75050b179100-istio-podinfo\") pod \"e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5\" (UID: \"5e9d91df-a00e-4a48-a494-75050b179100\") " pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"
Jul 06 23:16:02.535456 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.535427    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e9d91df-a00e-4a48-a494-75050b179100-istio-token\") pod \"e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5\" (UID: \"5e9d91df-a00e-4a48-a494-75050b179100\") " pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"
Jul 06 23:16:02.535594 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.535568    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnz87\" (UniqueName: \"kubernetes.io/projected/5e9d91df-a00e-4a48-a494-75050b179100-kube-api-access-gnz87\") pod \"e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5\" (UID: \"5e9d91df-a00e-4a48-a494-75050b179100\") " pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"
Jul 06 23:16:02.605548 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.605460    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"
Jul 06 23:16:02.748463 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.748439    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"]
Jul 06 23:16:02.750277 ip-10-0-134-116 kubenswrapper[2569]: W0706 23:16:02.750242    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e9d91df_a00e_4a48_a494_75050b179100.slice/crio-c238cfc0af654a33bc48fb99527077960733874f034fe007d5fdf635ea05555e WatchSource:0}: Error finding container c238cfc0af654a33bc48fb99527077960733874f034fe007d5fdf635ea05555e: Status 404 returned error can't find the container with id c238cfc0af654a33bc48fb99527077960733874f034fe007d5fdf635ea05555e
Jul 06 23:16:02.752416 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.752381    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 23:16:02.752508 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.752445    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 23:16:02.752508 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:02.752476    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 23:16:03.446577 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:03.446542    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5" event={"ID":"5e9d91df-a00e-4a48-a494-75050b179100","Type":"ContainerStarted","Data":"4f3725985c9382edb149c1a04a7dd18fc99277ef63968f88548a91dff06ea5f2"}
Jul 06 23:16:03.446962 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:03.446597    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5" event={"ID":"5e9d91df-a00e-4a48-a494-75050b179100","Type":"ContainerStarted","Data":"c238cfc0af654a33bc48fb99527077960733874f034fe007d5fdf635ea05555e"}
Jul 06 23:16:03.468349 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:03.468303    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5" podStartSLOduration=1.468288326 podStartE2EDuration="1.468288326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:16:03.466064164 +0000 UTC m=+1775.870314086" watchObservedRunningTime="2026-07-06 23:16:03.468288326 +0000 UTC m=+1775.872538229"
Jul 06 23:16:03.606250 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:03.606217    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"
Jul 06 23:16:03.610750 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:03.610727    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"
Jul 06 23:16:04.452330 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:04.452298    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"
Jul 06 23:16:04.453355 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:04.453334    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"
Jul 06 23:16:05.530257 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:05.530212    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-93d8da3f-7c69bbf9d8-j2ztq"]
Jul 06 23:16:05.533984 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:05.533961    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-93d8da3f-7c69bbf9d8-j2ztq"
Jul 06 23:16:05.536451 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:05.536431    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-93d8da3f-serving-cert\""
Jul 06 23:16:05.542812 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:05.542792    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-93d8da3f-7c69bbf9d8-j2ztq"]
Jul 06 23:16:05.654526 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:05.654494    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8dxb\" (UniqueName: \"kubernetes.io/projected/6f4f6c1c-fad1-476a-b752-015adcfba59d-kube-api-access-v8dxb\") pod \"maas-api-e2e-mt-93d8da3f-7c69bbf9d8-j2ztq\" (UID: \"6f4f6c1c-fad1-476a-b752-015adcfba59d\") " pod="opendatahub/maas-api-e2e-mt-93d8da3f-7c69bbf9d8-j2ztq"
Jul 06 23:16:05.654684 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:05.654558    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f4f6c1c-fad1-476a-b752-015adcfba59d-maas-api-tls\") pod \"maas-api-e2e-mt-93d8da3f-7c69bbf9d8-j2ztq\" (UID: \"6f4f6c1c-fad1-476a-b752-015adcfba59d\") " pod="opendatahub/maas-api-e2e-mt-93d8da3f-7c69bbf9d8-j2ztq"
Jul 06 23:16:05.654684 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:05.654588    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f4f6c1c-fad1-476a-b752-015adcfba59d-openshift-service-ca\") pod \"maas-api-e2e-mt-93d8da3f-7c69bbf9d8-j2ztq\" (UID: \"6f4f6c1c-fad1-476a-b752-015adcfba59d\") " pod="opendatahub/maas-api-e2e-mt-93d8da3f-7c69bbf9d8-j2ztq"
Jul 06 23:16:05.755341 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:05.755311    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f4f6c1c-fad1-476a-b752-015adcfba59d-maas-api-tls\") pod \"maas-api-e2e-mt-93d8da3f-7c69bbf9d8-j2ztq\" (UID: \"6f4f6c1c-fad1-476a-b752-015adcfba59d\") " pod="opendatahub/maas-api-e2e-mt-93d8da3f-7c69bbf9d8-j2ztq"
Jul 06 23:16:05.755463 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:05.755347    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f4f6c1c-fad1-476a-b752-015adcfba59d-openshift-service-ca\") pod \"maas-api-e2e-mt-93d8da3f-7c69bbf9d8-j2ztq\" (UID: \"6f4f6c1c-fad1-476a-b752-015adcfba59d\") " pod="opendatahub/maas-api-e2e-mt-93d8da3f-7c69bbf9d8-j2ztq"
Jul 06 23:16:05.755463 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:05.755395    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8dxb\" (UniqueName: \"kubernetes.io/projected/6f4f6c1c-fad1-476a-b752-015adcfba59d-kube-api-access-v8dxb\") pod \"maas-api-e2e-mt-93d8da3f-7c69bbf9d8-j2ztq\" (UID: \"6f4f6c1c-fad1-476a-b752-015adcfba59d\") " pod="opendatahub/maas-api-e2e-mt-93d8da3f-7c69bbf9d8-j2ztq"
Jul 06 23:16:05.755463 ip-10-0-134-116 kubenswrapper[2569]: E0706 23:16:05.755448    2569 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-93d8da3f-serving-cert: secret "maas-api-e2e-mt-93d8da3f-serving-cert" not found
Jul 06 23:16:05.755570 ip-10-0-134-116 kubenswrapper[2569]: E0706 23:16:05.755526    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f4f6c1c-fad1-476a-b752-015adcfba59d-maas-api-tls podName:6f4f6c1c-fad1-476a-b752-015adcfba59d nodeName:}" failed. No retries permitted until 2026-07-06 23:16:06.2555066 +0000 UTC m=+1778.659756482 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6f4f6c1c-fad1-476a-b752-015adcfba59d-maas-api-tls") pod "maas-api-e2e-mt-93d8da3f-7c69bbf9d8-j2ztq" (UID: "6f4f6c1c-fad1-476a-b752-015adcfba59d") : secret "maas-api-e2e-mt-93d8da3f-serving-cert" not found
Jul 06 23:16:05.756061 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:05.756042    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f4f6c1c-fad1-476a-b752-015adcfba59d-openshift-service-ca\") pod \"maas-api-e2e-mt-93d8da3f-7c69bbf9d8-j2ztq\" (UID: \"6f4f6c1c-fad1-476a-b752-015adcfba59d\") " pod="opendatahub/maas-api-e2e-mt-93d8da3f-7c69bbf9d8-j2ztq"
Jul 06 23:16:05.764248 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:05.764225    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8dxb\" (UniqueName: \"kubernetes.io/projected/6f4f6c1c-fad1-476a-b752-015adcfba59d-kube-api-access-v8dxb\") pod \"maas-api-e2e-mt-93d8da3f-7c69bbf9d8-j2ztq\" (UID: \"6f4f6c1c-fad1-476a-b752-015adcfba59d\") " pod="opendatahub/maas-api-e2e-mt-93d8da3f-7c69bbf9d8-j2ztq"
Jul 06 23:16:06.259419 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:06.259388    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f4f6c1c-fad1-476a-b752-015adcfba59d-maas-api-tls\") pod \"maas-api-e2e-mt-93d8da3f-7c69bbf9d8-j2ztq\" (UID: \"6f4f6c1c-fad1-476a-b752-015adcfba59d\") " pod="opendatahub/maas-api-e2e-mt-93d8da3f-7c69bbf9d8-j2ztq"
Jul 06 23:16:06.261914 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:06.261891    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f4f6c1c-fad1-476a-b752-015adcfba59d-maas-api-tls\") pod \"maas-api-e2e-mt-93d8da3f-7c69bbf9d8-j2ztq\" (UID: \"6f4f6c1c-fad1-476a-b752-015adcfba59d\") " pod="opendatahub/maas-api-e2e-mt-93d8da3f-7c69bbf9d8-j2ztq"
Jul 06 23:16:06.446727 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:06.446694    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-93d8da3f-7c69bbf9d8-j2ztq"
Jul 06 23:16:06.589243 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:06.589210    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-93d8da3f-7c69bbf9d8-j2ztq"]
Jul 06 23:16:06.590597 ip-10-0-134-116 kubenswrapper[2569]: W0706 23:16:06.590574    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f4f6c1c_fad1_476a_b752_015adcfba59d.slice/crio-b7fcc543b92577fadbff3c9b2b48e4d10cada4327f51b07338ba0cae68be6c87 WatchSource:0}: Error finding container b7fcc543b92577fadbff3c9b2b48e4d10cada4327f51b07338ba0cae68be6c87: Status 404 returned error can't find the container with id b7fcc543b92577fadbff3c9b2b48e4d10cada4327f51b07338ba0cae68be6c87
Jul 06 23:16:07.469036 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:07.469003    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-93d8da3f-7c69bbf9d8-j2ztq" event={"ID":"6f4f6c1c-fad1-476a-b752-015adcfba59d","Type":"ContainerStarted","Data":"668cb6e3d7cc5387366e77a0e9b34c231dd945d52c3a0842287dbc08aaed53e6"}
Jul 06 23:16:07.469198 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:07.469059    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-93d8da3f-7c69bbf9d8-j2ztq" event={"ID":"6f4f6c1c-fad1-476a-b752-015adcfba59d","Type":"ContainerStarted","Data":"b7fcc543b92577fadbff3c9b2b48e4d10cada4327f51b07338ba0cae68be6c87"}
Jul 06 23:16:07.469198 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:07.469084    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-93d8da3f-7c69bbf9d8-j2ztq"
Jul 06 23:16:07.486031 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:07.485994    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-93d8da3f-7c69bbf9d8-j2ztq" podStartSLOduration=2.10048069 podStartE2EDuration="2.485980078s" totalImagesPullingTime="385.499388ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:16:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:16:07.485896809 +0000 UTC m=+1779.890146713" watchObservedRunningTime="2026-07-06 23:16:07.485980078 +0000 UTC m=+1779.890229982"
Jul 06 23:16:13.489513 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:13.489482    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-93d8da3f-7c69bbf9d8-j2ztq"
Jul 06 23:16:19.453569 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:19.453533    2569 patch_prober.go:28] interesting pod/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Jul 06 23:16:19.453954 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:19.453591    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5" podUID="5e9d91df-a00e-4a48-a494-75050b179100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Jul 06 23:16:19.534405 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:19.534342    2569 generic.go:356] "Generic (PLEG): container finished" podID="5e9d91df-a00e-4a48-a494-75050b179100" containerID="4f3725985c9382edb149c1a04a7dd18fc99277ef63968f88548a91dff06ea5f2" exitCode=0
Jul 06 23:16:19.534504 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:19.534416    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5" event={"ID":"5e9d91df-a00e-4a48-a494-75050b179100","Type":"ContainerDied","Data":"4f3725985c9382edb149c1a04a7dd18fc99277ef63968f88548a91dff06ea5f2"}
Jul 06 23:16:19.534792 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:19.534778    2569 scope.go:117] "RemoveContainer" containerID="4f3725985c9382edb149c1a04a7dd18fc99277ef63968f88548a91dff06ea5f2"
Jul 06 23:16:19.535776 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:19.535744    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 23:16:19.535845 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:19.535832    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 23:16:19.535885 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:19.535873    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 23:16:20.541743 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:20.541662    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5" event={"ID":"5e9d91df-a00e-4a48-a494-75050b179100","Type":"ContainerStarted","Data":"3817b6cfbb7e7b888da8931588af2b83490912af38eb6bde68b8e8da3957389e"}
Jul 06 23:16:20.605695 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:20.605651    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"
Jul 06 23:16:20.605907 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:20.605886    2569 patch_prober.go:28] interesting pod/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Jul 06 23:16:20.605961 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:20.605944    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5" podUID="5e9d91df-a00e-4a48-a494-75050b179100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Jul 06 23:16:21.606470 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:21.606433    2569 patch_prober.go:28] interesting pod/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Jul 06 23:16:21.606865 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:21.606489    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5" podUID="5e9d91df-a00e-4a48-a494-75050b179100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Jul 06 23:16:22.605923 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:22.605887    2569 patch_prober.go:28] interesting pod/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Jul 06 23:16:22.606105 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:22.605944    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5" podUID="5e9d91df-a00e-4a48-a494-75050b179100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Jul 06 23:16:23.606201 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:23.606150    2569 patch_prober.go:28] interesting pod/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Jul 06 23:16:23.606583 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:23.606217    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5" podUID="5e9d91df-a00e-4a48-a494-75050b179100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Jul 06 23:16:24.609722 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:24.609689    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"
Jul 06 23:16:24.610102 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:24.610010    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"
Jul 06 23:16:24.610593 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:24.610577    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"
Jul 06 23:16:28.144593 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:28.144561    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bvsb_571f407a-249a-4fd9-8c15-53a6541db635/ovn-acl-logging/0.log"
Jul 06 23:16:28.148172 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:28.148147    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bvsb_571f407a-249a-4fd9-8c15-53a6541db635/ovn-acl-logging/0.log"
Jul 06 23:16:47.985080 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:47.985049    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-93d8da3f-7c69bbf9d8-j2ztq"]
Jul 06 23:16:47.985663 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:47.985291    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-93d8da3f-7c69bbf9d8-j2ztq" podUID="6f4f6c1c-fad1-476a-b752-015adcfba59d" containerName="maas-api" containerID="cri-o://668cb6e3d7cc5387366e77a0e9b34c231dd945d52c3a0842287dbc08aaed53e6" gracePeriod=30
Jul 06 23:16:48.234780 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:48.234756    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-93d8da3f-7c69bbf9d8-j2ztq"
Jul 06 23:16:48.313584 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:48.313504    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8dxb\" (UniqueName: \"kubernetes.io/projected/6f4f6c1c-fad1-476a-b752-015adcfba59d-kube-api-access-v8dxb\") pod \"6f4f6c1c-fad1-476a-b752-015adcfba59d\" (UID: \"6f4f6c1c-fad1-476a-b752-015adcfba59d\") "
Jul 06 23:16:48.313584 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:48.313561    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f4f6c1c-fad1-476a-b752-015adcfba59d-maas-api-tls\") pod \"6f4f6c1c-fad1-476a-b752-015adcfba59d\" (UID: \"6f4f6c1c-fad1-476a-b752-015adcfba59d\") "
Jul 06 23:16:48.313792 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:48.313601    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f4f6c1c-fad1-476a-b752-015adcfba59d-openshift-service-ca\") pod \"6f4f6c1c-fad1-476a-b752-015adcfba59d\" (UID: \"6f4f6c1c-fad1-476a-b752-015adcfba59d\") "
Jul 06 23:16:48.314072 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:48.314041    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f4f6c1c-fad1-476a-b752-015adcfba59d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6f4f6c1c-fad1-476a-b752-015adcfba59d" (UID: "6f4f6c1c-fad1-476a-b752-015adcfba59d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 23:16:48.315609 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:48.315587    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f4f6c1c-fad1-476a-b752-015adcfba59d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6f4f6c1c-fad1-476a-b752-015adcfba59d" (UID: "6f4f6c1c-fad1-476a-b752-015adcfba59d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 23:16:48.315758 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:48.315736    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f4f6c1c-fad1-476a-b752-015adcfba59d-kube-api-access-v8dxb" (OuterVolumeSpecName: "kube-api-access-v8dxb") pod "6f4f6c1c-fad1-476a-b752-015adcfba59d" (UID: "6f4f6c1c-fad1-476a-b752-015adcfba59d"). InnerVolumeSpecName "kube-api-access-v8dxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 23:16:48.414341 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:48.414314    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8dxb\" (UniqueName: \"kubernetes.io/projected/6f4f6c1c-fad1-476a-b752-015adcfba59d-kube-api-access-v8dxb\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:16:48.414341 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:48.414342    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f4f6c1c-fad1-476a-b752-015adcfba59d-maas-api-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:16:48.414483 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:48.414353    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f4f6c1c-fad1-476a-b752-015adcfba59d-openshift-service-ca\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:16:48.677688 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:48.677604    2569 generic.go:356] "Generic (PLEG): container finished" podID="6f4f6c1c-fad1-476a-b752-015adcfba59d" containerID="668cb6e3d7cc5387366e77a0e9b34c231dd945d52c3a0842287dbc08aaed53e6" exitCode=0
Jul 06 23:16:48.677688 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:48.677656    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-93d8da3f-7c69bbf9d8-j2ztq"
Jul 06 23:16:48.677905 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:48.677660    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-93d8da3f-7c69bbf9d8-j2ztq" event={"ID":"6f4f6c1c-fad1-476a-b752-015adcfba59d","Type":"ContainerDied","Data":"668cb6e3d7cc5387366e77a0e9b34c231dd945d52c3a0842287dbc08aaed53e6"}
Jul 06 23:16:48.677905 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:48.677771    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-93d8da3f-7c69bbf9d8-j2ztq" event={"ID":"6f4f6c1c-fad1-476a-b752-015adcfba59d","Type":"ContainerDied","Data":"b7fcc543b92577fadbff3c9b2b48e4d10cada4327f51b07338ba0cae68be6c87"}
Jul 06 23:16:48.677905 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:48.677793    2569 scope.go:117] "RemoveContainer" containerID="668cb6e3d7cc5387366e77a0e9b34c231dd945d52c3a0842287dbc08aaed53e6"
Jul 06 23:16:48.686977 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:48.686960    2569 scope.go:117] "RemoveContainer" containerID="668cb6e3d7cc5387366e77a0e9b34c231dd945d52c3a0842287dbc08aaed53e6"
Jul 06 23:16:48.687247 ip-10-0-134-116 kubenswrapper[2569]: E0706 23:16:48.687229    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"668cb6e3d7cc5387366e77a0e9b34c231dd945d52c3a0842287dbc08aaed53e6\": container with ID starting with 668cb6e3d7cc5387366e77a0e9b34c231dd945d52c3a0842287dbc08aaed53e6 not found: ID does not exist" containerID="668cb6e3d7cc5387366e77a0e9b34c231dd945d52c3a0842287dbc08aaed53e6"
Jul 06 23:16:48.687328 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:48.687254    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"668cb6e3d7cc5387366e77a0e9b34c231dd945d52c3a0842287dbc08aaed53e6"} err="failed to get container status \"668cb6e3d7cc5387366e77a0e9b34c231dd945d52c3a0842287dbc08aaed53e6\": rpc error: code = NotFound desc = could not find container \"668cb6e3d7cc5387366e77a0e9b34c231dd945d52c3a0842287dbc08aaed53e6\": container with ID starting with 668cb6e3d7cc5387366e77a0e9b34c231dd945d52c3a0842287dbc08aaed53e6 not found: ID does not exist"
Jul 06 23:16:48.698144 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:48.698123    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-93d8da3f-7c69bbf9d8-j2ztq"]
Jul 06 23:16:48.701646 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:48.701628    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-93d8da3f-7c69bbf9d8-j2ztq"]
Jul 06 23:16:50.170413 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:16:50.170379    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f4f6c1c-fad1-476a-b752-015adcfba59d" path="/var/lib/kubelet/pods/6f4f6c1c-fad1-476a-b752-015adcfba59d/volumes"
Jul 06 23:17:40.914293 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:17:40.914265    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"]
Jul 06 23:17:40.914789 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:17:40.914474    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5" podUID="5e9d91df-a00e-4a48-a494-75050b179100" containerName="istio-proxy" containerID="cri-o://3817b6cfbb7e7b888da8931588af2b83490912af38eb6bde68b8e8da3957389e" gracePeriod=30
Jul 06 23:17:46.176668 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:17:46.176610    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"
Jul 06 23:17:46.371592 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:17:46.371555    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e9d91df-a00e-4a48-a494-75050b179100-istio-envoy\") pod \"5e9d91df-a00e-4a48-a494-75050b179100\" (UID: \"5e9d91df-a00e-4a48-a494-75050b179100\") "
Jul 06 23:17:46.371787 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:17:46.371601    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e9d91df-a00e-4a48-a494-75050b179100-istio-token\") pod \"5e9d91df-a00e-4a48-a494-75050b179100\" (UID: \"5e9d91df-a00e-4a48-a494-75050b179100\") "
Jul 06 23:17:46.371787 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:17:46.371635    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e9d91df-a00e-4a48-a494-75050b179100-istio-data\") pod \"5e9d91df-a00e-4a48-a494-75050b179100\" (UID: \"5e9d91df-a00e-4a48-a494-75050b179100\") "
Jul 06 23:17:46.371787 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:17:46.371697    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9d91df-a00e-4a48-a494-75050b179100-workload-socket\") pod \"5e9d91df-a00e-4a48-a494-75050b179100\" (UID: \"5e9d91df-a00e-4a48-a494-75050b179100\") "
Jul 06 23:17:46.371787 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:17:46.371733    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e9d91df-a00e-4a48-a494-75050b179100-istio-podinfo\") pod \"5e9d91df-a00e-4a48-a494-75050b179100\" (UID: \"5e9d91df-a00e-4a48-a494-75050b179100\") "
Jul 06 23:17:46.371787 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:17:46.371768    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e9d91df-a00e-4a48-a494-75050b179100-istiod-ca-cert\") pod \"5e9d91df-a00e-4a48-a494-75050b179100\" (UID: \"5e9d91df-a00e-4a48-a494-75050b179100\") "
Jul 06 23:17:46.372068 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:17:46.371793    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9d91df-a00e-4a48-a494-75050b179100-credential-socket\") pod \"5e9d91df-a00e-4a48-a494-75050b179100\" (UID: \"5e9d91df-a00e-4a48-a494-75050b179100\") "
Jul 06 23:17:46.372068 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:17:46.371831    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e9d91df-a00e-4a48-a494-75050b179100-workload-certs\") pod \"5e9d91df-a00e-4a48-a494-75050b179100\" (UID: \"5e9d91df-a00e-4a48-a494-75050b179100\") "
Jul 06 23:17:46.372068 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:17:46.371887    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnz87\" (UniqueName: \"kubernetes.io/projected/5e9d91df-a00e-4a48-a494-75050b179100-kube-api-access-gnz87\") pod \"5e9d91df-a00e-4a48-a494-75050b179100\" (UID: \"5e9d91df-a00e-4a48-a494-75050b179100\") "
Jul 06 23:17:46.372281 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:17:46.372073    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e9d91df-a00e-4a48-a494-75050b179100-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5e9d91df-a00e-4a48-a494-75050b179100" (UID: "5e9d91df-a00e-4a48-a494-75050b179100"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:17:46.372281 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:17:46.372159    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e9d91df-a00e-4a48-a494-75050b179100-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5e9d91df-a00e-4a48-a494-75050b179100" (UID: "5e9d91df-a00e-4a48-a494-75050b179100"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 23:17:46.372402 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:17:46.372287    2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9d91df-a00e-4a48-a494-75050b179100-credential-socket\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:17:46.372402 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:17:46.372299    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e9d91df-a00e-4a48-a494-75050b179100-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5e9d91df-a00e-4a48-a494-75050b179100" (UID: "5e9d91df-a00e-4a48-a494-75050b179100"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:17:46.372402 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:17:46.372357    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e9d91df-a00e-4a48-a494-75050b179100-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5e9d91df-a00e-4a48-a494-75050b179100" (UID: "5e9d91df-a00e-4a48-a494-75050b179100"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:17:46.372560 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:17:46.372500    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e9d91df-a00e-4a48-a494-75050b179100-istio-data" (OuterVolumeSpecName: "istio-data") pod "5e9d91df-a00e-4a48-a494-75050b179100" (UID: "5e9d91df-a00e-4a48-a494-75050b179100"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:17:46.374139 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:17:46.374112    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e9d91df-a00e-4a48-a494-75050b179100-istio-token" (OuterVolumeSpecName: "istio-token") pod "5e9d91df-a00e-4a48-a494-75050b179100" (UID: "5e9d91df-a00e-4a48-a494-75050b179100"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 23:17:46.374372 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:17:46.374349    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e9d91df-a00e-4a48-a494-75050b179100-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5e9d91df-a00e-4a48-a494-75050b179100" (UID: "5e9d91df-a00e-4a48-a494-75050b179100"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:17:46.374465 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:17:46.374428    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e9d91df-a00e-4a48-a494-75050b179100-kube-api-access-gnz87" (OuterVolumeSpecName: "kube-api-access-gnz87") pod "5e9d91df-a00e-4a48-a494-75050b179100" (UID: "5e9d91df-a00e-4a48-a494-75050b179100"). InnerVolumeSpecName "kube-api-access-gnz87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 23:17:46.374569 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:17:46.374549    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e9d91df-a00e-4a48-a494-75050b179100-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5e9d91df-a00e-4a48-a494-75050b179100" (UID: "5e9d91df-a00e-4a48-a494-75050b179100"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 06 23:17:46.473133 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:17:46.473096    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnz87\" (UniqueName: \"kubernetes.io/projected/5e9d91df-a00e-4a48-a494-75050b179100-kube-api-access-gnz87\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:17:46.473377 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:17:46.473139    2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e9d91df-a00e-4a48-a494-75050b179100-istio-envoy\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:17:46.473377 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:17:46.473155    2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e9d91df-a00e-4a48-a494-75050b179100-istio-token\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:17:46.473377 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:17:46.473165    2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e9d91df-a00e-4a48-a494-75050b179100-istio-data\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:17:46.473377 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:17:46.473196    2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9d91df-a00e-4a48-a494-75050b179100-workload-socket\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:17:46.473377 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:17:46.473211    2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e9d91df-a00e-4a48-a494-75050b179100-istio-podinfo\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:17:46.473377 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:17:46.473225    2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e9d91df-a00e-4a48-a494-75050b179100-istiod-ca-cert\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:17:46.473377 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:17:46.473239    2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e9d91df-a00e-4a48-a494-75050b179100-workload-certs\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:17:46.969716 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:17:46.969682    2569 generic.go:356] "Generic (PLEG): container finished" podID="5e9d91df-a00e-4a48-a494-75050b179100" containerID="3817b6cfbb7e7b888da8931588af2b83490912af38eb6bde68b8e8da3957389e" exitCode=0
Jul 06 23:17:46.969900 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:17:46.969726    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"
Jul 06 23:17:46.969900 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:17:46.969754    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5" event={"ID":"5e9d91df-a00e-4a48-a494-75050b179100","Type":"ContainerDied","Data":"3817b6cfbb7e7b888da8931588af2b83490912af38eb6bde68b8e8da3957389e"}
Jul 06 23:17:46.969900 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:17:46.969806    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5" event={"ID":"5e9d91df-a00e-4a48-a494-75050b179100","Type":"ContainerDied","Data":"c238cfc0af654a33bc48fb99527077960733874f034fe007d5fdf635ea05555e"}
Jul 06 23:17:46.969900 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:17:46.969835    2569 scope.go:117] "RemoveContainer" containerID="3817b6cfbb7e7b888da8931588af2b83490912af38eb6bde68b8e8da3957389e"
Jul 06 23:17:46.978862 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:17:46.978843    2569 scope.go:117] "RemoveContainer" containerID="4f3725985c9382edb149c1a04a7dd18fc99277ef63968f88548a91dff06ea5f2"
Jul 06 23:17:46.986540 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:17:46.986524    2569 scope.go:117] "RemoveContainer" containerID="3817b6cfbb7e7b888da8931588af2b83490912af38eb6bde68b8e8da3957389e"
Jul 06 23:17:46.986793 ip-10-0-134-116 kubenswrapper[2569]: E0706 23:17:46.986775    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3817b6cfbb7e7b888da8931588af2b83490912af38eb6bde68b8e8da3957389e\": container with ID starting with 3817b6cfbb7e7b888da8931588af2b83490912af38eb6bde68b8e8da3957389e not found: ID does not exist" containerID="3817b6cfbb7e7b888da8931588af2b83490912af38eb6bde68b8e8da3957389e"
Jul 06 23:17:46.986868 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:17:46.986806    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3817b6cfbb7e7b888da8931588af2b83490912af38eb6bde68b8e8da3957389e"} err="failed to get container status \"3817b6cfbb7e7b888da8931588af2b83490912af38eb6bde68b8e8da3957389e\": rpc error: code = NotFound desc = could not find container \"3817b6cfbb7e7b888da8931588af2b83490912af38eb6bde68b8e8da3957389e\": container with ID starting with 3817b6cfbb7e7b888da8931588af2b83490912af38eb6bde68b8e8da3957389e not found: ID does not exist"
Jul 06 23:17:46.986868 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:17:46.986828    2569 scope.go:117] "RemoveContainer" containerID="4f3725985c9382edb149c1a04a7dd18fc99277ef63968f88548a91dff06ea5f2"
Jul 06 23:17:46.987041 ip-10-0-134-116 kubenswrapper[2569]: E0706 23:17:46.987026    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f3725985c9382edb149c1a04a7dd18fc99277ef63968f88548a91dff06ea5f2\": container with ID starting with 4f3725985c9382edb149c1a04a7dd18fc99277ef63968f88548a91dff06ea5f2 not found: ID does not exist" containerID="4f3725985c9382edb149c1a04a7dd18fc99277ef63968f88548a91dff06ea5f2"
Jul 06 23:17:46.987080 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:17:46.987046    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f3725985c9382edb149c1a04a7dd18fc99277ef63968f88548a91dff06ea5f2"} err="failed to get container status \"4f3725985c9382edb149c1a04a7dd18fc99277ef63968f88548a91dff06ea5f2\": rpc error: code = NotFound desc = could not find container \"4f3725985c9382edb149c1a04a7dd18fc99277ef63968f88548a91dff06ea5f2\": container with ID starting with 4f3725985c9382edb149c1a04a7dd18fc99277ef63968f88548a91dff06ea5f2 not found: ID does not exist"
Jul 06 23:17:46.991275 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:17:46.991256    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"]
Jul 06 23:17:46.997802 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:17:46.997783    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-93d8da3f-openshift-default-59ff844f95-bq5r5"]
Jul 06 23:17:48.169702 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:17:48.169674    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e9d91df-a00e-4a48-a494-75050b179100" path="/var/lib/kubelet/pods/5e9d91df-a00e-4a48-a494-75050b179100/volumes"
Jul 06 23:18:35.176093 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.176062    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"]
Jul 06 23:18:35.176632 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.176516    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f4f6c1c-fad1-476a-b752-015adcfba59d" containerName="maas-api"
Jul 06 23:18:35.176632 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.176530    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f4f6c1c-fad1-476a-b752-015adcfba59d" containerName="maas-api"
Jul 06 23:18:35.176632 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.176540    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e9d91df-a00e-4a48-a494-75050b179100" containerName="istio-proxy"
Jul 06 23:18:35.176632 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.176547    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e9d91df-a00e-4a48-a494-75050b179100" containerName="istio-proxy"
Jul 06 23:18:35.176632 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.176561    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e9d91df-a00e-4a48-a494-75050b179100" containerName="istio-proxy"
Jul 06 23:18:35.176632 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.176567    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e9d91df-a00e-4a48-a494-75050b179100" containerName="istio-proxy"
Jul 06 23:18:35.176632 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.176625    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e9d91df-a00e-4a48-a494-75050b179100" containerName="istio-proxy"
Jul 06 23:18:35.176632 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.176633    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e9d91df-a00e-4a48-a494-75050b179100" containerName="istio-proxy"
Jul 06 23:18:35.176889 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.176641    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f4f6c1c-fad1-476a-b752-015adcfba59d" containerName="maas-api"
Jul 06 23:18:35.179634 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.179615    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"
Jul 06 23:18:35.182317 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.182298    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-056fae-openshift-default-dockercfg-xz67s\""
Jul 06 23:18:35.191987 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.191965    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"]
Jul 06 23:18:35.225929 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.225902    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40525690-0960-4e67-a6b1-24f3245d6d96-istio-token\") pod \"e2e-inf-a-056fae-openshift-default-5d4b575947-w956v\" (UID: \"40525690-0960-4e67-a6b1-24f3245d6d96\") " pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"
Jul 06 23:18:35.226020 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.225941    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40525690-0960-4e67-a6b1-24f3245d6d96-workload-certs\") pod \"e2e-inf-a-056fae-openshift-default-5d4b575947-w956v\" (UID: \"40525690-0960-4e67-a6b1-24f3245d6d96\") " pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"
Jul 06 23:18:35.226020 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.225977    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40525690-0960-4e67-a6b1-24f3245d6d96-istio-envoy\") pod \"e2e-inf-a-056fae-openshift-default-5d4b575947-w956v\" (UID: \"40525690-0960-4e67-a6b1-24f3245d6d96\") " pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"
Jul 06 23:18:35.226097 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.226058    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40525690-0960-4e67-a6b1-24f3245d6d96-workload-socket\") pod \"e2e-inf-a-056fae-openshift-default-5d4b575947-w956v\" (UID: \"40525690-0960-4e67-a6b1-24f3245d6d96\") " pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"
Jul 06 23:18:35.226136 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.226100    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40525690-0960-4e67-a6b1-24f3245d6d96-istiod-ca-cert\") pod \"e2e-inf-a-056fae-openshift-default-5d4b575947-w956v\" (UID: \"40525690-0960-4e67-a6b1-24f3245d6d96\") " pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"
Jul 06 23:18:35.226136 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.226119    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tcmm\" (UniqueName: \"kubernetes.io/projected/40525690-0960-4e67-a6b1-24f3245d6d96-kube-api-access-8tcmm\") pod \"e2e-inf-a-056fae-openshift-default-5d4b575947-w956v\" (UID: \"40525690-0960-4e67-a6b1-24f3245d6d96\") " pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"
Jul 06 23:18:35.226245 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.226138    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40525690-0960-4e67-a6b1-24f3245d6d96-istio-podinfo\") pod \"e2e-inf-a-056fae-openshift-default-5d4b575947-w956v\" (UID: \"40525690-0960-4e67-a6b1-24f3245d6d96\") " pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"
Jul 06 23:18:35.226245 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.226211    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40525690-0960-4e67-a6b1-24f3245d6d96-istio-data\") pod \"e2e-inf-a-056fae-openshift-default-5d4b575947-w956v\" (UID: \"40525690-0960-4e67-a6b1-24f3245d6d96\") " pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"
Jul 06 23:18:35.226245 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.226242    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40525690-0960-4e67-a6b1-24f3245d6d96-credential-socket\") pod \"e2e-inf-a-056fae-openshift-default-5d4b575947-w956v\" (UID: \"40525690-0960-4e67-a6b1-24f3245d6d96\") " pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"
Jul 06 23:18:35.327093 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.327062    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40525690-0960-4e67-a6b1-24f3245d6d96-workload-certs\") pod \"e2e-inf-a-056fae-openshift-default-5d4b575947-w956v\" (UID: \"40525690-0960-4e67-a6b1-24f3245d6d96\") " pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"
Jul 06 23:18:35.327265 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.327108    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40525690-0960-4e67-a6b1-24f3245d6d96-istio-envoy\") pod \"e2e-inf-a-056fae-openshift-default-5d4b575947-w956v\" (UID: \"40525690-0960-4e67-a6b1-24f3245d6d96\") " pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"
Jul 06 23:18:35.327265 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.327152    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40525690-0960-4e67-a6b1-24f3245d6d96-workload-socket\") pod \"e2e-inf-a-056fae-openshift-default-5d4b575947-w956v\" (UID: \"40525690-0960-4e67-a6b1-24f3245d6d96\") " pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"
Jul 06 23:18:35.327265 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.327205    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40525690-0960-4e67-a6b1-24f3245d6d96-istiod-ca-cert\") pod \"e2e-inf-a-056fae-openshift-default-5d4b575947-w956v\" (UID: \"40525690-0960-4e67-a6b1-24f3245d6d96\") " pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"
Jul 06 23:18:35.327265 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.327230    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tcmm\" (UniqueName: \"kubernetes.io/projected/40525690-0960-4e67-a6b1-24f3245d6d96-kube-api-access-8tcmm\") pod \"e2e-inf-a-056fae-openshift-default-5d4b575947-w956v\" (UID: \"40525690-0960-4e67-a6b1-24f3245d6d96\") " pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"
Jul 06 23:18:35.327265 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.327256    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40525690-0960-4e67-a6b1-24f3245d6d96-istio-podinfo\") pod \"e2e-inf-a-056fae-openshift-default-5d4b575947-w956v\" (UID: \"40525690-0960-4e67-a6b1-24f3245d6d96\") " pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"
Jul 06 23:18:35.327552 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.327307    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40525690-0960-4e67-a6b1-24f3245d6d96-istio-data\") pod \"e2e-inf-a-056fae-openshift-default-5d4b575947-w956v\" (UID: \"40525690-0960-4e67-a6b1-24f3245d6d96\") " pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"
Jul 06 23:18:35.327552 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.327346    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40525690-0960-4e67-a6b1-24f3245d6d96-credential-socket\") pod \"e2e-inf-a-056fae-openshift-default-5d4b575947-w956v\" (UID: \"40525690-0960-4e67-a6b1-24f3245d6d96\") " pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"
Jul 06 23:18:35.327552 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.327400    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40525690-0960-4e67-a6b1-24f3245d6d96-istio-token\") pod \"e2e-inf-a-056fae-openshift-default-5d4b575947-w956v\" (UID: \"40525690-0960-4e67-a6b1-24f3245d6d96\") " pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"
Jul 06 23:18:35.327552 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.327513    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40525690-0960-4e67-a6b1-24f3245d6d96-workload-certs\") pod \"e2e-inf-a-056fae-openshift-default-5d4b575947-w956v\" (UID: \"40525690-0960-4e67-a6b1-24f3245d6d96\") " pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"
Jul 06 23:18:35.327552 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.327543    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40525690-0960-4e67-a6b1-24f3245d6d96-workload-socket\") pod \"e2e-inf-a-056fae-openshift-default-5d4b575947-w956v\" (UID: \"40525690-0960-4e67-a6b1-24f3245d6d96\") " pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"
Jul 06 23:18:35.327781 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.327761    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40525690-0960-4e67-a6b1-24f3245d6d96-credential-socket\") pod \"e2e-inf-a-056fae-openshift-default-5d4b575947-w956v\" (UID: \"40525690-0960-4e67-a6b1-24f3245d6d96\") " pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"
Jul 06 23:18:35.327927 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.327910    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40525690-0960-4e67-a6b1-24f3245d6d96-istiod-ca-cert\") pod \"e2e-inf-a-056fae-openshift-default-5d4b575947-w956v\" (UID: \"40525690-0960-4e67-a6b1-24f3245d6d96\") " pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"
Jul 06 23:18:35.328051 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.328029    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40525690-0960-4e67-a6b1-24f3245d6d96-istio-data\") pod \"e2e-inf-a-056fae-openshift-default-5d4b575947-w956v\" (UID: \"40525690-0960-4e67-a6b1-24f3245d6d96\") " pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"
Jul 06 23:18:35.329630 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.329609    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40525690-0960-4e67-a6b1-24f3245d6d96-istio-envoy\") pod \"e2e-inf-a-056fae-openshift-default-5d4b575947-w956v\" (UID: \"40525690-0960-4e67-a6b1-24f3245d6d96\") " pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"
Jul 06 23:18:35.329752 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.329735    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40525690-0960-4e67-a6b1-24f3245d6d96-istio-podinfo\") pod \"e2e-inf-a-056fae-openshift-default-5d4b575947-w956v\" (UID: \"40525690-0960-4e67-a6b1-24f3245d6d96\") " pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"
Jul 06 23:18:35.335669 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.335642    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40525690-0960-4e67-a6b1-24f3245d6d96-istio-token\") pod \"e2e-inf-a-056fae-openshift-default-5d4b575947-w956v\" (UID: \"40525690-0960-4e67-a6b1-24f3245d6d96\") " pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"
Jul 06 23:18:35.335898 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.335878    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tcmm\" (UniqueName: \"kubernetes.io/projected/40525690-0960-4e67-a6b1-24f3245d6d96-kube-api-access-8tcmm\") pod \"e2e-inf-a-056fae-openshift-default-5d4b575947-w956v\" (UID: \"40525690-0960-4e67-a6b1-24f3245d6d96\") " pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"
Jul 06 23:18:35.489954 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.489933    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"
Jul 06 23:18:35.628189 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.628143    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"]
Jul 06 23:18:35.630015 ip-10-0-134-116 kubenswrapper[2569]: W0706 23:18:35.629989    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40525690_0960_4e67_a6b1_24f3245d6d96.slice/crio-da0a1d17baf0769a6535b64b6aa87b073f335e144361d48c7bdbcb8cd6d5705c WatchSource:0}: Error finding container da0a1d17baf0769a6535b64b6aa87b073f335e144361d48c7bdbcb8cd6d5705c: Status 404 returned error can't find the container with id da0a1d17baf0769a6535b64b6aa87b073f335e144361d48c7bdbcb8cd6d5705c
Jul 06 23:18:35.632173 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.632138    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 23:18:35.632245 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.632231    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 23:18:35.632284 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:35.632261    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 23:18:36.202045 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:36.202005    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v" event={"ID":"40525690-0960-4e67-a6b1-24f3245d6d96","Type":"ContainerStarted","Data":"12c646ec6397d2deaccb854c11ce6df45ba3a87afb96c5e2440bf104cf5e57dd"}
Jul 06 23:18:36.202476 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:36.202072    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v" event={"ID":"40525690-0960-4e67-a6b1-24f3245d6d96","Type":"ContainerStarted","Data":"da0a1d17baf0769a6535b64b6aa87b073f335e144361d48c7bdbcb8cd6d5705c"}
Jul 06 23:18:36.222969 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:36.222921    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v" podStartSLOduration=1.222905623 podStartE2EDuration="1.222905623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:18:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:18:36.221865698 +0000 UTC m=+1928.626115643" watchObservedRunningTime="2026-07-06 23:18:36.222905623 +0000 UTC m=+1928.627155528"
Jul 06 23:18:36.490297 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:36.490218    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"
Jul 06 23:18:36.494937 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:36.494916    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"
Jul 06 23:18:37.207232 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:37.207197    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"
Jul 06 23:18:37.208069 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:37.208050    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"
Jul 06 23:18:38.334764 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:38.334731    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-056fae-57ccd859cc-5276p"]
Jul 06 23:18:38.340995 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:38.340966    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-056fae-57ccd859cc-5276p"
Jul 06 23:18:38.344653 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:38.344626    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-056fae-serving-cert\""
Jul 06 23:18:38.350253 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:38.350224    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-056fae-57ccd859cc-5276p"]
Jul 06 23:18:38.455345 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:38.455315    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d0fa088-1469-4035-b408-5120c67d035a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-056fae-57ccd859cc-5276p\" (UID: \"3d0fa088-1469-4035-b408-5120c67d035a\") " pod="opendatahub/maas-api-e2e-inf-a-056fae-57ccd859cc-5276p"
Jul 06 23:18:38.455345 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:38.455348    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8w9d\" (UniqueName: \"kubernetes.io/projected/3d0fa088-1469-4035-b408-5120c67d035a-kube-api-access-t8w9d\") pod \"maas-api-e2e-inf-a-056fae-57ccd859cc-5276p\" (UID: \"3d0fa088-1469-4035-b408-5120c67d035a\") " pod="opendatahub/maas-api-e2e-inf-a-056fae-57ccd859cc-5276p"
Jul 06 23:18:38.455545 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:38.455377    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d0fa088-1469-4035-b408-5120c67d035a-maas-api-tls\") pod \"maas-api-e2e-inf-a-056fae-57ccd859cc-5276p\" (UID: \"3d0fa088-1469-4035-b408-5120c67d035a\") " pod="opendatahub/maas-api-e2e-inf-a-056fae-57ccd859cc-5276p"
Jul 06 23:18:38.556069 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:38.556034    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d0fa088-1469-4035-b408-5120c67d035a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-056fae-57ccd859cc-5276p\" (UID: \"3d0fa088-1469-4035-b408-5120c67d035a\") " pod="opendatahub/maas-api-e2e-inf-a-056fae-57ccd859cc-5276p"
Jul 06 23:18:38.556069 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:38.556066    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8w9d\" (UniqueName: \"kubernetes.io/projected/3d0fa088-1469-4035-b408-5120c67d035a-kube-api-access-t8w9d\") pod \"maas-api-e2e-inf-a-056fae-57ccd859cc-5276p\" (UID: \"3d0fa088-1469-4035-b408-5120c67d035a\") " pod="opendatahub/maas-api-e2e-inf-a-056fae-57ccd859cc-5276p"
Jul 06 23:18:38.556298 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:38.556094    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d0fa088-1469-4035-b408-5120c67d035a-maas-api-tls\") pod \"maas-api-e2e-inf-a-056fae-57ccd859cc-5276p\" (UID: \"3d0fa088-1469-4035-b408-5120c67d035a\") " pod="opendatahub/maas-api-e2e-inf-a-056fae-57ccd859cc-5276p"
Jul 06 23:18:38.556298 ip-10-0-134-116 kubenswrapper[2569]: E0706 23:18:38.556213    2569 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-a-056fae-serving-cert: secret "maas-api-e2e-inf-a-056fae-serving-cert" not found
Jul 06 23:18:38.556298 ip-10-0-134-116 kubenswrapper[2569]: E0706 23:18:38.556286    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d0fa088-1469-4035-b408-5120c67d035a-maas-api-tls podName:3d0fa088-1469-4035-b408-5120c67d035a nodeName:}" failed. No retries permitted until 2026-07-06 23:18:39.056266577 +0000 UTC m=+1931.460516464 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3d0fa088-1469-4035-b408-5120c67d035a-maas-api-tls") pod "maas-api-e2e-inf-a-056fae-57ccd859cc-5276p" (UID: "3d0fa088-1469-4035-b408-5120c67d035a") : secret "maas-api-e2e-inf-a-056fae-serving-cert" not found
Jul 06 23:18:38.556663 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:38.556642    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d0fa088-1469-4035-b408-5120c67d035a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-056fae-57ccd859cc-5276p\" (UID: \"3d0fa088-1469-4035-b408-5120c67d035a\") " pod="opendatahub/maas-api-e2e-inf-a-056fae-57ccd859cc-5276p"
Jul 06 23:18:38.569028 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:38.569011    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8w9d\" (UniqueName: \"kubernetes.io/projected/3d0fa088-1469-4035-b408-5120c67d035a-kube-api-access-t8w9d\") pod \"maas-api-e2e-inf-a-056fae-57ccd859cc-5276p\" (UID: \"3d0fa088-1469-4035-b408-5120c67d035a\") " pod="opendatahub/maas-api-e2e-inf-a-056fae-57ccd859cc-5276p"
Jul 06 23:18:39.061048 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:39.061009    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d0fa088-1469-4035-b408-5120c67d035a-maas-api-tls\") pod \"maas-api-e2e-inf-a-056fae-57ccd859cc-5276p\" (UID: \"3d0fa088-1469-4035-b408-5120c67d035a\") " pod="opendatahub/maas-api-e2e-inf-a-056fae-57ccd859cc-5276p"
Jul 06 23:18:39.063638 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:39.063618    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d0fa088-1469-4035-b408-5120c67d035a-maas-api-tls\") pod \"maas-api-e2e-inf-a-056fae-57ccd859cc-5276p\" (UID: \"3d0fa088-1469-4035-b408-5120c67d035a\") " pod="opendatahub/maas-api-e2e-inf-a-056fae-57ccd859cc-5276p"
Jul 06 23:18:39.256240 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:39.256211    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-056fae-57ccd859cc-5276p"
Jul 06 23:18:39.595840 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:39.595812    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-056fae-57ccd859cc-5276p"]
Jul 06 23:18:39.597323 ip-10-0-134-116 kubenswrapper[2569]: W0706 23:18:39.597287    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d0fa088_1469_4035_b408_5120c67d035a.slice/crio-5d4deafe078e6f63d24eabb8b45c0d8a9c55a4d021f3bda2148aff85309af9e3 WatchSource:0}: Error finding container 5d4deafe078e6f63d24eabb8b45c0d8a9c55a4d021f3bda2148aff85309af9e3: Status 404 returned error can't find the container with id 5d4deafe078e6f63d24eabb8b45c0d8a9c55a4d021f3bda2148aff85309af9e3
Jul 06 23:18:39.598783 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:39.598756    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 23:18:40.223099 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:40.223064    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-056fae-57ccd859cc-5276p" event={"ID":"3d0fa088-1469-4035-b408-5120c67d035a","Type":"ContainerStarted","Data":"94719727b6a0fb29715a89fa6b48f2497a7b758f49917b7024253002a9aa3361"}
Jul 06 23:18:40.223309 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:40.223118    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-056fae-57ccd859cc-5276p" event={"ID":"3d0fa088-1469-4035-b408-5120c67d035a","Type":"ContainerStarted","Data":"5d4deafe078e6f63d24eabb8b45c0d8a9c55a4d021f3bda2148aff85309af9e3"}
Jul 06 23:18:40.223309 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:40.223138    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-056fae-57ccd859cc-5276p"
Jul 06 23:18:40.241722 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:40.241679    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-056fae-57ccd859cc-5276p" podStartSLOduration=1.7860630180000001 podStartE2EDuration="2.241664501s" totalImagesPullingTime="455.601483ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:18:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:18:40.240378645 +0000 UTC m=+1932.644628559" watchObservedRunningTime="2026-07-06 23:18:40.241664501 +0000 UTC m=+1932.645914404"
Jul 06 23:18:46.233671 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:46.233642    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-056fae-57ccd859cc-5276p"
Jul 06 23:18:53.248352 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:53.248318    2569 patch_prober.go:28] interesting pod/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 06 23:18:53.248737 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:53.248377    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v" podUID="40525690-0960-4e67-a6b1-24f3245d6d96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 06 23:18:54.295447 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:54.295387    2569 generic.go:356] "Generic (PLEG): container finished" podID="40525690-0960-4e67-a6b1-24f3245d6d96" containerID="12c646ec6397d2deaccb854c11ce6df45ba3a87afb96c5e2440bf104cf5e57dd" exitCode=0
Jul 06 23:18:54.295821 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:54.295453    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v" event={"ID":"40525690-0960-4e67-a6b1-24f3245d6d96","Type":"ContainerDied","Data":"12c646ec6397d2deaccb854c11ce6df45ba3a87afb96c5e2440bf104cf5e57dd"}
Jul 06 23:18:54.295821 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:54.295755    2569 scope.go:117] "RemoveContainer" containerID="12c646ec6397d2deaccb854c11ce6df45ba3a87afb96c5e2440bf104cf5e57dd"
Jul 06 23:18:54.296769 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:54.296740    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 23:18:54.296887 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:54.296805    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 23:18:54.296887 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:54.296837    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 23:18:55.303573 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:55.303540    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v" event={"ID":"40525690-0960-4e67-a6b1-24f3245d6d96","Type":"ContainerStarted","Data":"ca15319ba96f4ecb38919480e51766ecdc67ed1796e992dcdf40f0866a3f24fe"}
Jul 06 23:18:55.490698 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:55.490663    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"
Jul 06 23:18:55.490908 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:55.490782    2569 patch_prober.go:28] interesting pod/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Jul 06 23:18:55.490908 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:55.490824    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v" podUID="40525690-0960-4e67-a6b1-24f3245d6d96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Jul 06 23:18:56.491299 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:56.491261    2569 patch_prober.go:28] interesting pod/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Jul 06 23:18:56.491798 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:56.491322    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v" podUID="40525690-0960-4e67-a6b1-24f3245d6d96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Jul 06 23:18:57.492667 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:57.492227    2569 patch_prober.go:28] interesting pod/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Jul 06 23:18:57.492667 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:57.492298    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v" podUID="40525690-0960-4e67-a6b1-24f3245d6d96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Jul 06 23:18:58.490822 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:58.490786    2569 patch_prober.go:28] interesting pod/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Jul 06 23:18:58.491099 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:58.491069    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v" podUID="40525690-0960-4e67-a6b1-24f3245d6d96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Jul 06 23:18:59.494296 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:59.494264    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"
Jul 06 23:18:59.494681 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:59.494611    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"
Jul 06 23:18:59.495426 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:18:59.495403    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"
Jul 06 23:19:22.378999 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:22.378909    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-639686/test-model-639686-kserve-57c5dcfbbf-w8qz4"]
Jul 06 23:19:22.381445 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:22.381427    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-639686/test-model-639686-kserve-57c5dcfbbf-w8qz4"
Jul 06 23:19:22.384223 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:22.384195    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-639686\"/\"openshift-service-ca.crt\""
Jul 06 23:19:22.384350 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:22.384235    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-639686\"/\"kube-root-ca.crt\""
Jul 06 23:19:22.385597 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:22.385582    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-639686\"/\"default-dockercfg-xcsvz\""
Jul 06 23:19:22.385694 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:22.385602    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-639686\"/\"test-model-639686-kserve-self-signed-certs\""
Jul 06 23:19:22.392736 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:22.392717    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-639686/test-model-639686-kserve-57c5dcfbbf-w8qz4"]
Jul 06 23:19:22.420372 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:22.420349    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca5ade6c-cbe6-4873-8bf3-da7845732f93-dshm\") pod \"test-model-639686-kserve-57c5dcfbbf-w8qz4\" (UID: \"ca5ade6c-cbe6-4873-8bf3-da7845732f93\") " pod="ai-tenant-e2e-inf-b-639686/test-model-639686-kserve-57c5dcfbbf-w8qz4"
Jul 06 23:19:22.420477 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:22.420379    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22b7n\" (UniqueName: \"kubernetes.io/projected/ca5ade6c-cbe6-4873-8bf3-da7845732f93-kube-api-access-22b7n\") pod \"test-model-639686-kserve-57c5dcfbbf-w8qz4\" (UID: \"ca5ade6c-cbe6-4873-8bf3-da7845732f93\") " pod="ai-tenant-e2e-inf-b-639686/test-model-639686-kserve-57c5dcfbbf-w8qz4"
Jul 06 23:19:22.420477 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:22.420399    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca5ade6c-cbe6-4873-8bf3-da7845732f93-tls-certs\") pod \"test-model-639686-kserve-57c5dcfbbf-w8qz4\" (UID: \"ca5ade6c-cbe6-4873-8bf3-da7845732f93\") " pod="ai-tenant-e2e-inf-b-639686/test-model-639686-kserve-57c5dcfbbf-w8qz4"
Jul 06 23:19:22.420552 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:22.420536    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca5ade6c-cbe6-4873-8bf3-da7845732f93-home\") pod \"test-model-639686-kserve-57c5dcfbbf-w8qz4\" (UID: \"ca5ade6c-cbe6-4873-8bf3-da7845732f93\") " pod="ai-tenant-e2e-inf-b-639686/test-model-639686-kserve-57c5dcfbbf-w8qz4"
Jul 06 23:19:22.420592 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:22.420574    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca5ade6c-cbe6-4873-8bf3-da7845732f93-model-cache\") pod \"test-model-639686-kserve-57c5dcfbbf-w8qz4\" (UID: \"ca5ade6c-cbe6-4873-8bf3-da7845732f93\") " pod="ai-tenant-e2e-inf-b-639686/test-model-639686-kserve-57c5dcfbbf-w8qz4"
Jul 06 23:19:22.420627 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:22.420595    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca5ade6c-cbe6-4873-8bf3-da7845732f93-tmp-dir\") pod \"test-model-639686-kserve-57c5dcfbbf-w8qz4\" (UID: \"ca5ade6c-cbe6-4873-8bf3-da7845732f93\") " pod="ai-tenant-e2e-inf-b-639686/test-model-639686-kserve-57c5dcfbbf-w8qz4"
Jul 06 23:19:22.521056 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:22.521032    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca5ade6c-cbe6-4873-8bf3-da7845732f93-home\") pod \"test-model-639686-kserve-57c5dcfbbf-w8qz4\" (UID: \"ca5ade6c-cbe6-4873-8bf3-da7845732f93\") " pod="ai-tenant-e2e-inf-b-639686/test-model-639686-kserve-57c5dcfbbf-w8qz4"
Jul 06 23:19:22.521216 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:22.521063    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca5ade6c-cbe6-4873-8bf3-da7845732f93-model-cache\") pod \"test-model-639686-kserve-57c5dcfbbf-w8qz4\" (UID: \"ca5ade6c-cbe6-4873-8bf3-da7845732f93\") " pod="ai-tenant-e2e-inf-b-639686/test-model-639686-kserve-57c5dcfbbf-w8qz4"
Jul 06 23:19:22.521216 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:22.521080    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca5ade6c-cbe6-4873-8bf3-da7845732f93-tmp-dir\") pod \"test-model-639686-kserve-57c5dcfbbf-w8qz4\" (UID: \"ca5ade6c-cbe6-4873-8bf3-da7845732f93\") " pod="ai-tenant-e2e-inf-b-639686/test-model-639686-kserve-57c5dcfbbf-w8qz4"
Jul 06 23:19:22.521216 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:22.521105    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca5ade6c-cbe6-4873-8bf3-da7845732f93-dshm\") pod \"test-model-639686-kserve-57c5dcfbbf-w8qz4\" (UID: \"ca5ade6c-cbe6-4873-8bf3-da7845732f93\") " pod="ai-tenant-e2e-inf-b-639686/test-model-639686-kserve-57c5dcfbbf-w8qz4"
Jul 06 23:19:22.521216 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:22.521133    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22b7n\" (UniqueName: \"kubernetes.io/projected/ca5ade6c-cbe6-4873-8bf3-da7845732f93-kube-api-access-22b7n\") pod \"test-model-639686-kserve-57c5dcfbbf-w8qz4\" (UID: \"ca5ade6c-cbe6-4873-8bf3-da7845732f93\") " pod="ai-tenant-e2e-inf-b-639686/test-model-639686-kserve-57c5dcfbbf-w8qz4"
Jul 06 23:19:22.521216 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:22.521159    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca5ade6c-cbe6-4873-8bf3-da7845732f93-tls-certs\") pod \"test-model-639686-kserve-57c5dcfbbf-w8qz4\" (UID: \"ca5ade6c-cbe6-4873-8bf3-da7845732f93\") " pod="ai-tenant-e2e-inf-b-639686/test-model-639686-kserve-57c5dcfbbf-w8qz4"
Jul 06 23:19:22.521518 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:22.521495    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca5ade6c-cbe6-4873-8bf3-da7845732f93-home\") pod \"test-model-639686-kserve-57c5dcfbbf-w8qz4\" (UID: \"ca5ade6c-cbe6-4873-8bf3-da7845732f93\") " pod="ai-tenant-e2e-inf-b-639686/test-model-639686-kserve-57c5dcfbbf-w8qz4"
Jul 06 23:19:22.521584 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:22.521528    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca5ade6c-cbe6-4873-8bf3-da7845732f93-tmp-dir\") pod \"test-model-639686-kserve-57c5dcfbbf-w8qz4\" (UID: \"ca5ade6c-cbe6-4873-8bf3-da7845732f93\") " pod="ai-tenant-e2e-inf-b-639686/test-model-639686-kserve-57c5dcfbbf-w8qz4"
Jul 06 23:19:22.521584 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:22.521553    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca5ade6c-cbe6-4873-8bf3-da7845732f93-model-cache\") pod \"test-model-639686-kserve-57c5dcfbbf-w8qz4\" (UID: \"ca5ade6c-cbe6-4873-8bf3-da7845732f93\") " pod="ai-tenant-e2e-inf-b-639686/test-model-639686-kserve-57c5dcfbbf-w8qz4"
Jul 06 23:19:22.523461 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:22.523439    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca5ade6c-cbe6-4873-8bf3-da7845732f93-dshm\") pod \"test-model-639686-kserve-57c5dcfbbf-w8qz4\" (UID: \"ca5ade6c-cbe6-4873-8bf3-da7845732f93\") " pod="ai-tenant-e2e-inf-b-639686/test-model-639686-kserve-57c5dcfbbf-w8qz4"
Jul 06 23:19:22.523739 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:22.523718    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca5ade6c-cbe6-4873-8bf3-da7845732f93-tls-certs\") pod \"test-model-639686-kserve-57c5dcfbbf-w8qz4\" (UID: \"ca5ade6c-cbe6-4873-8bf3-da7845732f93\") " pod="ai-tenant-e2e-inf-b-639686/test-model-639686-kserve-57c5dcfbbf-w8qz4"
Jul 06 23:19:22.538557 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:22.538534    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22b7n\" (UniqueName: \"kubernetes.io/projected/ca5ade6c-cbe6-4873-8bf3-da7845732f93-kube-api-access-22b7n\") pod \"test-model-639686-kserve-57c5dcfbbf-w8qz4\" (UID: \"ca5ade6c-cbe6-4873-8bf3-da7845732f93\") " pod="ai-tenant-e2e-inf-b-639686/test-model-639686-kserve-57c5dcfbbf-w8qz4"
Jul 06 23:19:22.690844 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:22.690811    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-639686/test-model-639686-kserve-57c5dcfbbf-w8qz4"
Jul 06 23:19:22.818249 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:22.818197    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-639686/test-model-639686-kserve-57c5dcfbbf-w8qz4"]
Jul 06 23:19:22.820391 ip-10-0-134-116 kubenswrapper[2569]: W0706 23:19:22.820364    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca5ade6c_cbe6_4873_8bf3_da7845732f93.slice/crio-8e01b8b4df962efeeab4e560f41174b899e25c68d168c00260976320d52536dd WatchSource:0}: Error finding container 8e01b8b4df962efeeab4e560f41174b899e25c68d168c00260976320d52536dd: Status 404 returned error can't find the container with id 8e01b8b4df962efeeab4e560f41174b899e25c68d168c00260976320d52536dd
Jul 06 23:19:23.436550 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:23.436508    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-639686/test-model-639686-kserve-57c5dcfbbf-w8qz4" event={"ID":"ca5ade6c-cbe6-4873-8bf3-da7845732f93","Type":"ContainerStarted","Data":"11d3aff090d5b593d4e9a0c9c4e3edbceb52946bf7cff974e573e2abb7bece1b"}
Jul 06 23:19:23.436550 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:23.436553    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-639686/test-model-639686-kserve-57c5dcfbbf-w8qz4" event={"ID":"ca5ade6c-cbe6-4873-8bf3-da7845732f93","Type":"ContainerStarted","Data":"8e01b8b4df962efeeab4e560f41174b899e25c68d168c00260976320d52536dd"}
Jul 06 23:19:23.453534 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:23.453487    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-639686/test-model-639686-kserve-57c5dcfbbf-w8qz4" podStartSLOduration=1.4534720810000001 podStartE2EDuration="1.453472081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:19:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:19:23.451445471 +0000 UTC m=+1975.855695376" watchObservedRunningTime="2026-07-06 23:19:23.453472081 +0000 UTC m=+1975.857721984"
Jul 06 23:19:32.691111 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:32.691077    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-639686/test-model-639686-kserve-57c5dcfbbf-w8qz4"
Jul 06 23:19:32.691537 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:32.691125    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-639686/test-model-639686-kserve-57c5dcfbbf-w8qz4"
Jul 06 23:19:32.704084 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:32.704060    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-639686/test-model-639686-kserve-57c5dcfbbf-w8qz4"
Jul 06 23:19:33.493845 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:33.493819    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-639686/test-model-639686-kserve-57c5dcfbbf-w8qz4"
Jul 06 23:19:49.717562 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:49.717529    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-639686/test-model-639686-kserve-57c5dcfbbf-w8qz4"]
Jul 06 23:19:50.654366 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:50.654334    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-056fae-57ccd859cc-5276p"]
Jul 06 23:19:50.654678 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:50.654645    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-056fae-57ccd859cc-5276p" podUID="3d0fa088-1469-4035-b408-5120c67d035a" containerName="maas-api" containerID="cri-o://94719727b6a0fb29715a89fa6b48f2497a7b758f49917b7024253002a9aa3361" gracePeriod=30
Jul 06 23:19:51.108839 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:51.108815    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-056fae-57ccd859cc-5276p"
Jul 06 23:19:51.250991 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:51.250966    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d0fa088-1469-4035-b408-5120c67d035a-maas-api-tls\") pod \"3d0fa088-1469-4035-b408-5120c67d035a\" (UID: \"3d0fa088-1469-4035-b408-5120c67d035a\") "
Jul 06 23:19:51.251148 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:51.251012    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8w9d\" (UniqueName: \"kubernetes.io/projected/3d0fa088-1469-4035-b408-5120c67d035a-kube-api-access-t8w9d\") pod \"3d0fa088-1469-4035-b408-5120c67d035a\" (UID: \"3d0fa088-1469-4035-b408-5120c67d035a\") "
Jul 06 23:19:51.251148 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:51.251046    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d0fa088-1469-4035-b408-5120c67d035a-openshift-service-ca\") pod \"3d0fa088-1469-4035-b408-5120c67d035a\" (UID: \"3d0fa088-1469-4035-b408-5120c67d035a\") "
Jul 06 23:19:51.251478 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:51.251455    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d0fa088-1469-4035-b408-5120c67d035a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3d0fa088-1469-4035-b408-5120c67d035a" (UID: "3d0fa088-1469-4035-b408-5120c67d035a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 23:19:51.253096 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:51.253071    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d0fa088-1469-4035-b408-5120c67d035a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3d0fa088-1469-4035-b408-5120c67d035a" (UID: "3d0fa088-1469-4035-b408-5120c67d035a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 23:19:51.253167 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:51.253120    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d0fa088-1469-4035-b408-5120c67d035a-kube-api-access-t8w9d" (OuterVolumeSpecName: "kube-api-access-t8w9d") pod "3d0fa088-1469-4035-b408-5120c67d035a" (UID: "3d0fa088-1469-4035-b408-5120c67d035a"). InnerVolumeSpecName "kube-api-access-t8w9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 23:19:51.352068 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:51.352046    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d0fa088-1469-4035-b408-5120c67d035a-maas-api-tls\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:19:51.352068 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:51.352067    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8w9d\" (UniqueName: \"kubernetes.io/projected/3d0fa088-1469-4035-b408-5120c67d035a-kube-api-access-t8w9d\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:19:51.352305 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:51.352077    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d0fa088-1469-4035-b408-5120c67d035a-openshift-service-ca\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:19:51.571717 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:51.571622    2569 generic.go:356] "Generic (PLEG): container finished" podID="3d0fa088-1469-4035-b408-5120c67d035a" containerID="94719727b6a0fb29715a89fa6b48f2497a7b758f49917b7024253002a9aa3361" exitCode=0
Jul 06 23:19:51.571717 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:51.571673    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-056fae-57ccd859cc-5276p"
Jul 06 23:19:51.571896 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:51.571708    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-056fae-57ccd859cc-5276p" event={"ID":"3d0fa088-1469-4035-b408-5120c67d035a","Type":"ContainerDied","Data":"94719727b6a0fb29715a89fa6b48f2497a7b758f49917b7024253002a9aa3361"}
Jul 06 23:19:51.571896 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:51.571764    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-056fae-57ccd859cc-5276p" event={"ID":"3d0fa088-1469-4035-b408-5120c67d035a","Type":"ContainerDied","Data":"5d4deafe078e6f63d24eabb8b45c0d8a9c55a4d021f3bda2148aff85309af9e3"}
Jul 06 23:19:51.571896 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:51.571795    2569 scope.go:117] "RemoveContainer" containerID="94719727b6a0fb29715a89fa6b48f2497a7b758f49917b7024253002a9aa3361"
Jul 06 23:19:51.581503 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:51.581487    2569 scope.go:117] "RemoveContainer" containerID="94719727b6a0fb29715a89fa6b48f2497a7b758f49917b7024253002a9aa3361"
Jul 06 23:19:51.581731 ip-10-0-134-116 kubenswrapper[2569]: E0706 23:19:51.581714    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94719727b6a0fb29715a89fa6b48f2497a7b758f49917b7024253002a9aa3361\": container with ID starting with 94719727b6a0fb29715a89fa6b48f2497a7b758f49917b7024253002a9aa3361 not found: ID does not exist" containerID="94719727b6a0fb29715a89fa6b48f2497a7b758f49917b7024253002a9aa3361"
Jul 06 23:19:51.581782 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:51.581739    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94719727b6a0fb29715a89fa6b48f2497a7b758f49917b7024253002a9aa3361"} err="failed to get container status \"94719727b6a0fb29715a89fa6b48f2497a7b758f49917b7024253002a9aa3361\": rpc error: code = NotFound desc = could not find container \"94719727b6a0fb29715a89fa6b48f2497a7b758f49917b7024253002a9aa3361\": container with ID starting with 94719727b6a0fb29715a89fa6b48f2497a7b758f49917b7024253002a9aa3361 not found: ID does not exist"
Jul 06 23:19:51.595310 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:51.595280    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-056fae-57ccd859cc-5276p"]
Jul 06 23:19:51.599486 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:51.599465    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-056fae-57ccd859cc-5276p"]
Jul 06 23:19:52.170919 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:19:52.170890    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d0fa088-1469-4035-b408-5120c67d035a" path="/var/lib/kubelet/pods/3d0fa088-1469-4035-b408-5120c67d035a/volumes"
Jul 06 23:20:04.738005 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:04.737949    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-639686/test-model-639686-kserve-57c5dcfbbf-w8qz4" podUID="ca5ade6c-cbe6-4873-8bf3-da7845732f93" containerName="main" containerID="cri-o://11d3aff090d5b593d4e9a0c9c4e3edbceb52946bf7cff974e573e2abb7bece1b" gracePeriod=45
Jul 06 23:20:04.987824 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:04.987800    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-639686/test-model-639686-kserve-57c5dcfbbf-w8qz4"
Jul 06 23:20:05.167139 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:05.167075    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca5ade6c-cbe6-4873-8bf3-da7845732f93-tls-certs\") pod \"ca5ade6c-cbe6-4873-8bf3-da7845732f93\" (UID: \"ca5ade6c-cbe6-4873-8bf3-da7845732f93\") "
Jul 06 23:20:05.167139 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:05.167107    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca5ade6c-cbe6-4873-8bf3-da7845732f93-home\") pod \"ca5ade6c-cbe6-4873-8bf3-da7845732f93\" (UID: \"ca5ade6c-cbe6-4873-8bf3-da7845732f93\") "
Jul 06 23:20:05.167139 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:05.167123    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca5ade6c-cbe6-4873-8bf3-da7845732f93-tmp-dir\") pod \"ca5ade6c-cbe6-4873-8bf3-da7845732f93\" (UID: \"ca5ade6c-cbe6-4873-8bf3-da7845732f93\") "
Jul 06 23:20:05.167420 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:05.167147    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca5ade6c-cbe6-4873-8bf3-da7845732f93-dshm\") pod \"ca5ade6c-cbe6-4873-8bf3-da7845732f93\" (UID: \"ca5ade6c-cbe6-4873-8bf3-da7845732f93\") "
Jul 06 23:20:05.167420 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:05.167196    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22b7n\" (UniqueName: \"kubernetes.io/projected/ca5ade6c-cbe6-4873-8bf3-da7845732f93-kube-api-access-22b7n\") pod \"ca5ade6c-cbe6-4873-8bf3-da7845732f93\" (UID: \"ca5ade6c-cbe6-4873-8bf3-da7845732f93\") "
Jul 06 23:20:05.167420 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:05.167237    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca5ade6c-cbe6-4873-8bf3-da7845732f93-model-cache\") pod \"ca5ade6c-cbe6-4873-8bf3-da7845732f93\" (UID: \"ca5ade6c-cbe6-4873-8bf3-da7845732f93\") "
Jul 06 23:20:05.167420 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:05.167396    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca5ade6c-cbe6-4873-8bf3-da7845732f93-home" (OuterVolumeSpecName: "home") pod "ca5ade6c-cbe6-4873-8bf3-da7845732f93" (UID: "ca5ade6c-cbe6-4873-8bf3-da7845732f93"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:20:05.167621 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:05.167443    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca5ade6c-cbe6-4873-8bf3-da7845732f93-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ca5ade6c-cbe6-4873-8bf3-da7845732f93" (UID: "ca5ade6c-cbe6-4873-8bf3-da7845732f93"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:20:05.167621 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:05.167555    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca5ade6c-cbe6-4873-8bf3-da7845732f93-home\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:20:05.167621 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:05.167558    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca5ade6c-cbe6-4873-8bf3-da7845732f93-model-cache" (OuterVolumeSpecName: "model-cache") pod "ca5ade6c-cbe6-4873-8bf3-da7845732f93" (UID: "ca5ade6c-cbe6-4873-8bf3-da7845732f93"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:20:05.167621 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:05.167575    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca5ade6c-cbe6-4873-8bf3-da7845732f93-tmp-dir\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:20:05.169447 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:05.169421    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca5ade6c-cbe6-4873-8bf3-da7845732f93-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ca5ade6c-cbe6-4873-8bf3-da7845732f93" (UID: "ca5ade6c-cbe6-4873-8bf3-da7845732f93"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 23:20:05.169548 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:05.169451    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca5ade6c-cbe6-4873-8bf3-da7845732f93-kube-api-access-22b7n" (OuterVolumeSpecName: "kube-api-access-22b7n") pod "ca5ade6c-cbe6-4873-8bf3-da7845732f93" (UID: "ca5ade6c-cbe6-4873-8bf3-da7845732f93"). InnerVolumeSpecName "kube-api-access-22b7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 23:20:05.169548 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:05.169495    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca5ade6c-cbe6-4873-8bf3-da7845732f93-dshm" (OuterVolumeSpecName: "dshm") pod "ca5ade6c-cbe6-4873-8bf3-da7845732f93" (UID: "ca5ade6c-cbe6-4873-8bf3-da7845732f93"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:20:05.268802 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:05.268782    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca5ade6c-cbe6-4873-8bf3-da7845732f93-dshm\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:20:05.268802 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:05.268802    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22b7n\" (UniqueName: \"kubernetes.io/projected/ca5ade6c-cbe6-4873-8bf3-da7845732f93-kube-api-access-22b7n\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:20:05.268934 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:05.268814    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca5ade6c-cbe6-4873-8bf3-da7845732f93-model-cache\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:20:05.268934 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:05.268823    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca5ade6c-cbe6-4873-8bf3-da7845732f93-tls-certs\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:20:05.647936 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:05.647898    2569 generic.go:356] "Generic (PLEG): container finished" podID="ca5ade6c-cbe6-4873-8bf3-da7845732f93" containerID="11d3aff090d5b593d4e9a0c9c4e3edbceb52946bf7cff974e573e2abb7bece1b" exitCode=0
Jul 06 23:20:05.648096 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:05.647952    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-639686/test-model-639686-kserve-57c5dcfbbf-w8qz4" event={"ID":"ca5ade6c-cbe6-4873-8bf3-da7845732f93","Type":"ContainerDied","Data":"11d3aff090d5b593d4e9a0c9c4e3edbceb52946bf7cff974e573e2abb7bece1b"}
Jul 06 23:20:05.648096 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:05.647956    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-639686/test-model-639686-kserve-57c5dcfbbf-w8qz4"
Jul 06 23:20:05.648096 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:05.647989    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-639686/test-model-639686-kserve-57c5dcfbbf-w8qz4" event={"ID":"ca5ade6c-cbe6-4873-8bf3-da7845732f93","Type":"ContainerDied","Data":"8e01b8b4df962efeeab4e560f41174b899e25c68d168c00260976320d52536dd"}
Jul 06 23:20:05.648096 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:05.648007    2569 scope.go:117] "RemoveContainer" containerID="11d3aff090d5b593d4e9a0c9c4e3edbceb52946bf7cff974e573e2abb7bece1b"
Jul 06 23:20:05.657530 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:05.657511    2569 scope.go:117] "RemoveContainer" containerID="11d3aff090d5b593d4e9a0c9c4e3edbceb52946bf7cff974e573e2abb7bece1b"
Jul 06 23:20:05.657770 ip-10-0-134-116 kubenswrapper[2569]: E0706 23:20:05.657751    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11d3aff090d5b593d4e9a0c9c4e3edbceb52946bf7cff974e573e2abb7bece1b\": container with ID starting with 11d3aff090d5b593d4e9a0c9c4e3edbceb52946bf7cff974e573e2abb7bece1b not found: ID does not exist" containerID="11d3aff090d5b593d4e9a0c9c4e3edbceb52946bf7cff974e573e2abb7bece1b"
Jul 06 23:20:05.657822 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:05.657778    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11d3aff090d5b593d4e9a0c9c4e3edbceb52946bf7cff974e573e2abb7bece1b"} err="failed to get container status \"11d3aff090d5b593d4e9a0c9c4e3edbceb52946bf7cff974e573e2abb7bece1b\": rpc error: code = NotFound desc = could not find container \"11d3aff090d5b593d4e9a0c9c4e3edbceb52946bf7cff974e573e2abb7bece1b\": container with ID starting with 11d3aff090d5b593d4e9a0c9c4e3edbceb52946bf7cff974e573e2abb7bece1b not found: ID does not exist"
Jul 06 23:20:05.668758 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:05.668737    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-639686/test-model-639686-kserve-57c5dcfbbf-w8qz4"]
Jul 06 23:20:05.672697 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:05.672678    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-639686/test-model-639686-kserve-57c5dcfbbf-w8qz4"]
Jul 06 23:20:06.171077 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:06.171046    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca5ade6c-cbe6-4873-8bf3-da7845732f93" path="/var/lib/kubelet/pods/ca5ade6c-cbe6-4873-8bf3-da7845732f93/volumes"
Jul 06 23:20:32.466198 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:32.466144    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"]
Jul 06 23:20:32.466616 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:32.466394    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v" podUID="40525690-0960-4e67-a6b1-24f3245d6d96" containerName="istio-proxy" containerID="cri-o://ca15319ba96f4ecb38919480e51766ecdc67ed1796e992dcdf40f0866a3f24fe" gracePeriod=30
Jul 06 23:20:37.723191 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:37.723149    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"
Jul 06 23:20:37.808714 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:37.808642    2569 generic.go:356] "Generic (PLEG): container finished" podID="40525690-0960-4e67-a6b1-24f3245d6d96" containerID="ca15319ba96f4ecb38919480e51766ecdc67ed1796e992dcdf40f0866a3f24fe" exitCode=0
Jul 06 23:20:37.808714 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:37.808694    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"
Jul 06 23:20:37.808714 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:37.808706    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v" event={"ID":"40525690-0960-4e67-a6b1-24f3245d6d96","Type":"ContainerDied","Data":"ca15319ba96f4ecb38919480e51766ecdc67ed1796e992dcdf40f0866a3f24fe"}
Jul 06 23:20:37.808950 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:37.808772    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v" event={"ID":"40525690-0960-4e67-a6b1-24f3245d6d96","Type":"ContainerDied","Data":"da0a1d17baf0769a6535b64b6aa87b073f335e144361d48c7bdbcb8cd6d5705c"}
Jul 06 23:20:37.808950 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:37.808793    2569 scope.go:117] "RemoveContainer" containerID="ca15319ba96f4ecb38919480e51766ecdc67ed1796e992dcdf40f0866a3f24fe"
Jul 06 23:20:37.819897 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:37.819879    2569 scope.go:117] "RemoveContainer" containerID="12c646ec6397d2deaccb854c11ce6df45ba3a87afb96c5e2440bf104cf5e57dd"
Jul 06 23:20:37.827140 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:37.827119    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40525690-0960-4e67-a6b1-24f3245d6d96-istio-token\") pod \"40525690-0960-4e67-a6b1-24f3245d6d96\" (UID: \"40525690-0960-4e67-a6b1-24f3245d6d96\") "
Jul 06 23:20:37.827253 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:37.827190    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40525690-0960-4e67-a6b1-24f3245d6d96-credential-socket\") pod \"40525690-0960-4e67-a6b1-24f3245d6d96\" (UID: \"40525690-0960-4e67-a6b1-24f3245d6d96\") "
Jul 06 23:20:37.827253 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:37.827225    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40525690-0960-4e67-a6b1-24f3245d6d96-workload-socket\") pod \"40525690-0960-4e67-a6b1-24f3245d6d96\" (UID: \"40525690-0960-4e67-a6b1-24f3245d6d96\") "
Jul 06 23:20:37.827253 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:37.827252    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40525690-0960-4e67-a6b1-24f3245d6d96-istiod-ca-cert\") pod \"40525690-0960-4e67-a6b1-24f3245d6d96\" (UID: \"40525690-0960-4e67-a6b1-24f3245d6d96\") "
Jul 06 23:20:37.827421 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:37.827273    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40525690-0960-4e67-a6b1-24f3245d6d96-istio-data\") pod \"40525690-0960-4e67-a6b1-24f3245d6d96\" (UID: \"40525690-0960-4e67-a6b1-24f3245d6d96\") "
Jul 06 23:20:37.827421 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:37.827332    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40525690-0960-4e67-a6b1-24f3245d6d96-workload-certs\") pod \"40525690-0960-4e67-a6b1-24f3245d6d96\" (UID: \"40525690-0960-4e67-a6b1-24f3245d6d96\") "
Jul 06 23:20:37.827421 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:37.827354    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40525690-0960-4e67-a6b1-24f3245d6d96-istio-podinfo\") pod \"40525690-0960-4e67-a6b1-24f3245d6d96\" (UID: \"40525690-0960-4e67-a6b1-24f3245d6d96\") "
Jul 06 23:20:37.827421 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:37.827405    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40525690-0960-4e67-a6b1-24f3245d6d96-istio-envoy\") pod \"40525690-0960-4e67-a6b1-24f3245d6d96\" (UID: \"40525690-0960-4e67-a6b1-24f3245d6d96\") "
Jul 06 23:20:37.827642 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:37.827430    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tcmm\" (UniqueName: \"kubernetes.io/projected/40525690-0960-4e67-a6b1-24f3245d6d96-kube-api-access-8tcmm\") pod \"40525690-0960-4e67-a6b1-24f3245d6d96\" (UID: \"40525690-0960-4e67-a6b1-24f3245d6d96\") "
Jul 06 23:20:37.827642 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:37.827547    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40525690-0960-4e67-a6b1-24f3245d6d96-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "40525690-0960-4e67-a6b1-24f3245d6d96" (UID: "40525690-0960-4e67-a6b1-24f3245d6d96"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:20:37.827642 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:37.827561    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40525690-0960-4e67-a6b1-24f3245d6d96-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "40525690-0960-4e67-a6b1-24f3245d6d96" (UID: "40525690-0960-4e67-a6b1-24f3245d6d96"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:20:37.827642 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:37.827628    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40525690-0960-4e67-a6b1-24f3245d6d96-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "40525690-0960-4e67-a6b1-24f3245d6d96" (UID: "40525690-0960-4e67-a6b1-24f3245d6d96"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:20:37.827843 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:37.827648    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40525690-0960-4e67-a6b1-24f3245d6d96-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "40525690-0960-4e67-a6b1-24f3245d6d96" (UID: "40525690-0960-4e67-a6b1-24f3245d6d96"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 23:20:37.827843 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:37.827776    2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40525690-0960-4e67-a6b1-24f3245d6d96-istiod-ca-cert\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:20:37.827843 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:37.827795    2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40525690-0960-4e67-a6b1-24f3245d6d96-workload-certs\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:20:37.827843 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:37.827809    2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40525690-0960-4e67-a6b1-24f3245d6d96-credential-socket\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:20:37.827843 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:37.827823    2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40525690-0960-4e67-a6b1-24f3245d6d96-workload-socket\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:20:37.828089 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:37.828061    2569 scope.go:117] "RemoveContainer" containerID="ca15319ba96f4ecb38919480e51766ecdc67ed1796e992dcdf40f0866a3f24fe"
Jul 06 23:20:37.828369 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:37.828332    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40525690-0960-4e67-a6b1-24f3245d6d96-istio-data" (OuterVolumeSpecName: "istio-data") pod "40525690-0960-4e67-a6b1-24f3245d6d96" (UID: "40525690-0960-4e67-a6b1-24f3245d6d96"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:20:37.828493 ip-10-0-134-116 kubenswrapper[2569]: E0706 23:20:37.828411    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca15319ba96f4ecb38919480e51766ecdc67ed1796e992dcdf40f0866a3f24fe\": container with ID starting with ca15319ba96f4ecb38919480e51766ecdc67ed1796e992dcdf40f0866a3f24fe not found: ID does not exist" containerID="ca15319ba96f4ecb38919480e51766ecdc67ed1796e992dcdf40f0866a3f24fe"
Jul 06 23:20:37.828493 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:37.828439    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca15319ba96f4ecb38919480e51766ecdc67ed1796e992dcdf40f0866a3f24fe"} err="failed to get container status \"ca15319ba96f4ecb38919480e51766ecdc67ed1796e992dcdf40f0866a3f24fe\": rpc error: code = NotFound desc = could not find container \"ca15319ba96f4ecb38919480e51766ecdc67ed1796e992dcdf40f0866a3f24fe\": container with ID starting with ca15319ba96f4ecb38919480e51766ecdc67ed1796e992dcdf40f0866a3f24fe not found: ID does not exist"
Jul 06 23:20:37.828493 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:37.828466    2569 scope.go:117] "RemoveContainer" containerID="12c646ec6397d2deaccb854c11ce6df45ba3a87afb96c5e2440bf104cf5e57dd"
Jul 06 23:20:37.828721 ip-10-0-134-116 kubenswrapper[2569]: E0706 23:20:37.828698    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12c646ec6397d2deaccb854c11ce6df45ba3a87afb96c5e2440bf104cf5e57dd\": container with ID starting with 12c646ec6397d2deaccb854c11ce6df45ba3a87afb96c5e2440bf104cf5e57dd not found: ID does not exist" containerID="12c646ec6397d2deaccb854c11ce6df45ba3a87afb96c5e2440bf104cf5e57dd"
Jul 06 23:20:37.828805 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:37.828725    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12c646ec6397d2deaccb854c11ce6df45ba3a87afb96c5e2440bf104cf5e57dd"} err="failed to get container status \"12c646ec6397d2deaccb854c11ce6df45ba3a87afb96c5e2440bf104cf5e57dd\": rpc error: code = NotFound desc = could not find container \"12c646ec6397d2deaccb854c11ce6df45ba3a87afb96c5e2440bf104cf5e57dd\": container with ID starting with 12c646ec6397d2deaccb854c11ce6df45ba3a87afb96c5e2440bf104cf5e57dd not found: ID does not exist"
Jul 06 23:20:37.829742 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:37.829716    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40525690-0960-4e67-a6b1-24f3245d6d96-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "40525690-0960-4e67-a6b1-24f3245d6d96" (UID: "40525690-0960-4e67-a6b1-24f3245d6d96"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 06 23:20:37.829873 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:37.829855    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40525690-0960-4e67-a6b1-24f3245d6d96-istio-token" (OuterVolumeSpecName: "istio-token") pod "40525690-0960-4e67-a6b1-24f3245d6d96" (UID: "40525690-0960-4e67-a6b1-24f3245d6d96"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 23:20:37.829941 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:37.829893    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40525690-0960-4e67-a6b1-24f3245d6d96-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "40525690-0960-4e67-a6b1-24f3245d6d96" (UID: "40525690-0960-4e67-a6b1-24f3245d6d96"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:20:37.829941 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:37.829926    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40525690-0960-4e67-a6b1-24f3245d6d96-kube-api-access-8tcmm" (OuterVolumeSpecName: "kube-api-access-8tcmm") pod "40525690-0960-4e67-a6b1-24f3245d6d96" (UID: "40525690-0960-4e67-a6b1-24f3245d6d96"). InnerVolumeSpecName "kube-api-access-8tcmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 23:20:37.928618 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:37.928597    2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40525690-0960-4e67-a6b1-24f3245d6d96-istio-envoy\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:20:37.928709 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:37.928620    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tcmm\" (UniqueName: \"kubernetes.io/projected/40525690-0960-4e67-a6b1-24f3245d6d96-kube-api-access-8tcmm\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:20:37.928709 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:37.928629    2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40525690-0960-4e67-a6b1-24f3245d6d96-istio-token\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:20:37.928709 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:37.928638    2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40525690-0960-4e67-a6b1-24f3245d6d96-istio-data\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:20:37.928709 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:37.928646    2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40525690-0960-4e67-a6b1-24f3245d6d96-istio-podinfo\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:20:38.130478 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:38.130448    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"]
Jul 06 23:20:38.137722 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:38.137705    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-056fae-openshift-default-5d4b575947-w956v"]
Jul 06 23:20:38.170964 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:38.170943    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40525690-0960-4e67-a6b1-24f3245d6d96" path="/var/lib/kubelet/pods/40525690-0960-4e67-a6b1-24f3245d6d96/volumes"
Jul 06 23:20:40.867195 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:40.867157    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"]
Jul 06 23:20:40.867697 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:40.867678    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca5ade6c-cbe6-4873-8bf3-da7845732f93" containerName="main"
Jul 06 23:20:40.867782 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:40.867699    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca5ade6c-cbe6-4873-8bf3-da7845732f93" containerName="main"
Jul 06 23:20:40.867782 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:40.867713    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40525690-0960-4e67-a6b1-24f3245d6d96" containerName="istio-proxy"
Jul 06 23:20:40.867782 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:40.867721    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="40525690-0960-4e67-a6b1-24f3245d6d96" containerName="istio-proxy"
Jul 06 23:20:40.867782 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:40.867737    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d0fa088-1469-4035-b408-5120c67d035a" containerName="maas-api"
Jul 06 23:20:40.867782 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:40.867746    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d0fa088-1469-4035-b408-5120c67d035a" containerName="maas-api"
Jul 06 23:20:40.867782 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:40.867758    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40525690-0960-4e67-a6b1-24f3245d6d96" containerName="istio-proxy"
Jul 06 23:20:40.867782 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:40.867767    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="40525690-0960-4e67-a6b1-24f3245d6d96" containerName="istio-proxy"
Jul 06 23:20:40.868119 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:40.867880    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d0fa088-1469-4035-b408-5120c67d035a" containerName="maas-api"
Jul 06 23:20:40.868119 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:40.867893    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca5ade6c-cbe6-4873-8bf3-da7845732f93" containerName="main"
Jul 06 23:20:40.868119 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:40.867903    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="40525690-0960-4e67-a6b1-24f3245d6d96" containerName="istio-proxy"
Jul 06 23:20:40.872466 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:40.872442    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"
Jul 06 23:20:40.875591 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:40.875526    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-e0bef8-openshift-default-dockercfg-bgl8s\""
Jul 06 23:20:40.884993 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:40.884969    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"]
Jul 06 23:20:40.952988 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:40.952968    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e72aff77-68de-4d16-83ee-6db12d1625be-istio-envoy\") pod \"e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44\" (UID: \"e72aff77-68de-4d16-83ee-6db12d1625be\") " pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"
Jul 06 23:20:40.953088 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:40.953005    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxpc2\" (UniqueName: \"kubernetes.io/projected/e72aff77-68de-4d16-83ee-6db12d1625be-kube-api-access-kxpc2\") pod \"e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44\" (UID: \"e72aff77-68de-4d16-83ee-6db12d1625be\") " pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"
Jul 06 23:20:40.953088 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:40.953028    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e72aff77-68de-4d16-83ee-6db12d1625be-workload-socket\") pod \"e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44\" (UID: \"e72aff77-68de-4d16-83ee-6db12d1625be\") " pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"
Jul 06 23:20:40.953088 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:40.953045    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e72aff77-68de-4d16-83ee-6db12d1625be-istio-podinfo\") pod \"e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44\" (UID: \"e72aff77-68de-4d16-83ee-6db12d1625be\") " pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"
Jul 06 23:20:40.953212 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:40.953114    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e72aff77-68de-4d16-83ee-6db12d1625be-istio-data\") pod \"e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44\" (UID: \"e72aff77-68de-4d16-83ee-6db12d1625be\") " pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"
Jul 06 23:20:40.953212 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:40.953149    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e72aff77-68de-4d16-83ee-6db12d1625be-workload-certs\") pod \"e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44\" (UID: \"e72aff77-68de-4d16-83ee-6db12d1625be\") " pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"
Jul 06 23:20:40.953212 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:40.953168    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e72aff77-68de-4d16-83ee-6db12d1625be-istiod-ca-cert\") pod \"e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44\" (UID: \"e72aff77-68de-4d16-83ee-6db12d1625be\") " pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"
Jul 06 23:20:40.953313 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:40.953240    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e72aff77-68de-4d16-83ee-6db12d1625be-credential-socket\") pod \"e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44\" (UID: \"e72aff77-68de-4d16-83ee-6db12d1625be\") " pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"
Jul 06 23:20:40.953313 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:40.953266    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e72aff77-68de-4d16-83ee-6db12d1625be-istio-token\") pod \"e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44\" (UID: \"e72aff77-68de-4d16-83ee-6db12d1625be\") " pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"
Jul 06 23:20:41.053974 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:41.053946    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e72aff77-68de-4d16-83ee-6db12d1625be-workload-certs\") pod \"e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44\" (UID: \"e72aff77-68de-4d16-83ee-6db12d1625be\") " pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"
Jul 06 23:20:41.053974 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:41.053975    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e72aff77-68de-4d16-83ee-6db12d1625be-istiod-ca-cert\") pod \"e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44\" (UID: \"e72aff77-68de-4d16-83ee-6db12d1625be\") " pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"
Jul 06 23:20:41.054241 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:41.054006    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e72aff77-68de-4d16-83ee-6db12d1625be-credential-socket\") pod \"e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44\" (UID: \"e72aff77-68de-4d16-83ee-6db12d1625be\") " pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"
Jul 06 23:20:41.054241 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:41.054025    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e72aff77-68de-4d16-83ee-6db12d1625be-istio-token\") pod \"e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44\" (UID: \"e72aff77-68de-4d16-83ee-6db12d1625be\") " pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"
Jul 06 23:20:41.054241 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:41.054056    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e72aff77-68de-4d16-83ee-6db12d1625be-istio-envoy\") pod \"e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44\" (UID: \"e72aff77-68de-4d16-83ee-6db12d1625be\") " pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"
Jul 06 23:20:41.054241 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:41.054083    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxpc2\" (UniqueName: \"kubernetes.io/projected/e72aff77-68de-4d16-83ee-6db12d1625be-kube-api-access-kxpc2\") pod \"e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44\" (UID: \"e72aff77-68de-4d16-83ee-6db12d1625be\") " pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"
Jul 06 23:20:41.054241 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:41.054105    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e72aff77-68de-4d16-83ee-6db12d1625be-workload-socket\") pod \"e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44\" (UID: \"e72aff77-68de-4d16-83ee-6db12d1625be\") " pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"
Jul 06 23:20:41.054241 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:41.054129    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e72aff77-68de-4d16-83ee-6db12d1625be-istio-podinfo\") pod \"e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44\" (UID: \"e72aff77-68de-4d16-83ee-6db12d1625be\") " pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"
Jul 06 23:20:41.054241 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:41.054219    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e72aff77-68de-4d16-83ee-6db12d1625be-istio-data\") pod \"e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44\" (UID: \"e72aff77-68de-4d16-83ee-6db12d1625be\") " pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"
Jul 06 23:20:41.054617 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:41.054381    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e72aff77-68de-4d16-83ee-6db12d1625be-credential-socket\") pod \"e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44\" (UID: \"e72aff77-68de-4d16-83ee-6db12d1625be\") " pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"
Jul 06 23:20:41.054617 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:41.054416    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e72aff77-68de-4d16-83ee-6db12d1625be-workload-certs\") pod \"e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44\" (UID: \"e72aff77-68de-4d16-83ee-6db12d1625be\") " pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"
Jul 06 23:20:41.054617 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:41.054521    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e72aff77-68de-4d16-83ee-6db12d1625be-workload-socket\") pod \"e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44\" (UID: \"e72aff77-68de-4d16-83ee-6db12d1625be\") " pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"
Jul 06 23:20:41.054617 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:41.054593    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e72aff77-68de-4d16-83ee-6db12d1625be-istio-data\") pod \"e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44\" (UID: \"e72aff77-68de-4d16-83ee-6db12d1625be\") " pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"
Jul 06 23:20:41.054767 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:41.054699    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e72aff77-68de-4d16-83ee-6db12d1625be-istiod-ca-cert\") pod \"e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44\" (UID: \"e72aff77-68de-4d16-83ee-6db12d1625be\") " pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"
Jul 06 23:20:41.056409 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:41.056388    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e72aff77-68de-4d16-83ee-6db12d1625be-istio-envoy\") pod \"e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44\" (UID: \"e72aff77-68de-4d16-83ee-6db12d1625be\") " pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"
Jul 06 23:20:41.056681 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:41.056665    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e72aff77-68de-4d16-83ee-6db12d1625be-istio-podinfo\") pod \"e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44\" (UID: \"e72aff77-68de-4d16-83ee-6db12d1625be\") " pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"
Jul 06 23:20:41.063729 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:41.063702    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e72aff77-68de-4d16-83ee-6db12d1625be-istio-token\") pod \"e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44\" (UID: \"e72aff77-68de-4d16-83ee-6db12d1625be\") " pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"
Jul 06 23:20:41.064333 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:41.064302    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxpc2\" (UniqueName: \"kubernetes.io/projected/e72aff77-68de-4d16-83ee-6db12d1625be-kube-api-access-kxpc2\") pod \"e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44\" (UID: \"e72aff77-68de-4d16-83ee-6db12d1625be\") " pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"
Jul 06 23:20:41.183860 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:41.183838    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"
Jul 06 23:20:41.318717 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:41.318692    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"]
Jul 06 23:20:41.320998 ip-10-0-134-116 kubenswrapper[2569]: W0706 23:20:41.320968    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode72aff77_68de_4d16_83ee_6db12d1625be.slice/crio-b712dff95cab6f81959a998b236e354f26295b756f7c5694b309f86b7bb9a4cf WatchSource:0}: Error finding container b712dff95cab6f81959a998b236e354f26295b756f7c5694b309f86b7bb9a4cf: Status 404 returned error can't find the container with id b712dff95cab6f81959a998b236e354f26295b756f7c5694b309f86b7bb9a4cf
Jul 06 23:20:41.323401 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:41.323371    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 23:20:41.323465 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:41.323434    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 23:20:41.323506 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:41.323464    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 23:20:41.831695 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:41.831662    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44" event={"ID":"e72aff77-68de-4d16-83ee-6db12d1625be","Type":"ContainerStarted","Data":"4e91dd8cdceb971b96b2104cc58870e3dd8cab68dfd4dfa1fee6dcf7406d21eb"}
Jul 06 23:20:41.831871 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:41.831739    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44" event={"ID":"e72aff77-68de-4d16-83ee-6db12d1625be","Type":"ContainerStarted","Data":"b712dff95cab6f81959a998b236e354f26295b756f7c5694b309f86b7bb9a4cf"}
Jul 06 23:20:41.853051 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:41.853001    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44" podStartSLOduration=1.852986502 podStartE2EDuration="1.852986502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:20:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:20:41.8526589 +0000 UTC m=+2054.256908826" watchObservedRunningTime="2026-07-06 23:20:41.852986502 +0000 UTC m=+2054.257236406"
Jul 06 23:20:42.184688 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:42.184659    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"
Jul 06 23:20:42.189535 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:42.189510    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"
Jul 06 23:20:42.837718 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:42.837687    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"
Jul 06 23:20:42.838586 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:42.838564    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"
Jul 06 23:20:57.839070 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:57.839039    2569 patch_prober.go:28] interesting pod/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Jul 06 23:20:57.839498 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:57.839096    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44" podUID="e72aff77-68de-4d16-83ee-6db12d1625be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Jul 06 23:20:57.921705 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:57.921676    2569 generic.go:356] "Generic (PLEG): container finished" podID="e72aff77-68de-4d16-83ee-6db12d1625be" containerID="4e91dd8cdceb971b96b2104cc58870e3dd8cab68dfd4dfa1fee6dcf7406d21eb" exitCode=0
Jul 06 23:20:57.921850 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:57.921719    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44" event={"ID":"e72aff77-68de-4d16-83ee-6db12d1625be","Type":"ContainerDied","Data":"4e91dd8cdceb971b96b2104cc58870e3dd8cab68dfd4dfa1fee6dcf7406d21eb"}
Jul 06 23:20:57.922059 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:57.922045    2569 scope.go:117] "RemoveContainer" containerID="4e91dd8cdceb971b96b2104cc58870e3dd8cab68dfd4dfa1fee6dcf7406d21eb"
Jul 06 23:20:57.923079 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:57.923017    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 23:20:57.923158 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:57.923088    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 23:20:57.923158 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:57.923121    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 23:20:58.928822 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:58.928789    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44" event={"ID":"e72aff77-68de-4d16-83ee-6db12d1625be","Type":"ContainerStarted","Data":"3c6ded4f3d75ccef28cb35c26429023ba6f1a5323003dc6557bab30ff036b7c0"}
Jul 06 23:20:59.184434 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:59.184403    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"
Jul 06 23:20:59.184828 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:59.184736    2569 patch_prober.go:28] interesting pod/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Jul 06 23:20:59.184992 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:20:59.184971    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44" podUID="e72aff77-68de-4d16-83ee-6db12d1625be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Jul 06 23:21:00.184859 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:00.184825    2569 patch_prober.go:28] interesting pod/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Jul 06 23:21:00.185567 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:00.185529    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44" podUID="e72aff77-68de-4d16-83ee-6db12d1625be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Jul 06 23:21:01.184677 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:01.184637    2569 patch_prober.go:28] interesting pod/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Jul 06 23:21:01.184956 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:01.184926    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44" podUID="e72aff77-68de-4d16-83ee-6db12d1625be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Jul 06 23:21:02.184755 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:02.184717    2569 patch_prober.go:28] interesting pod/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Jul 06 23:21:02.184962 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:02.184797    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44" podUID="e72aff77-68de-4d16-83ee-6db12d1625be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Jul 06 23:21:03.188665 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:03.188636    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"
Jul 06 23:21:03.189051 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:03.188939    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"
Jul 06 23:21:03.189689 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:03.189673    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"
Jul 06 23:21:11.484334 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:11.484300    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"]
Jul 06 23:21:11.484774 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:11.484571    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44" podUID="e72aff77-68de-4d16-83ee-6db12d1625be" containerName="istio-proxy" containerID="cri-o://3c6ded4f3d75ccef28cb35c26429023ba6f1a5323003dc6557bab30ff036b7c0" gracePeriod=30
Jul 06 23:21:16.752230 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:16.752156    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"
Jul 06 23:21:16.857608 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:16.857573    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e72aff77-68de-4d16-83ee-6db12d1625be-istio-podinfo\") pod \"e72aff77-68de-4d16-83ee-6db12d1625be\" (UID: \"e72aff77-68de-4d16-83ee-6db12d1625be\") "
Jul 06 23:21:16.857740 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:16.857622    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e72aff77-68de-4d16-83ee-6db12d1625be-istio-envoy\") pod \"e72aff77-68de-4d16-83ee-6db12d1625be\" (UID: \"e72aff77-68de-4d16-83ee-6db12d1625be\") "
Jul 06 23:21:16.857740 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:16.857731    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e72aff77-68de-4d16-83ee-6db12d1625be-istiod-ca-cert\") pod \"e72aff77-68de-4d16-83ee-6db12d1625be\" (UID: \"e72aff77-68de-4d16-83ee-6db12d1625be\") "
Jul 06 23:21:16.857841 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:16.857770    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e72aff77-68de-4d16-83ee-6db12d1625be-workload-certs\") pod \"e72aff77-68de-4d16-83ee-6db12d1625be\" (UID: \"e72aff77-68de-4d16-83ee-6db12d1625be\") "
Jul 06 23:21:16.857841 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:16.857803    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e72aff77-68de-4d16-83ee-6db12d1625be-istio-data\") pod \"e72aff77-68de-4d16-83ee-6db12d1625be\" (UID: \"e72aff77-68de-4d16-83ee-6db12d1625be\") "
Jul 06 23:21:16.857841 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:16.857837    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e72aff77-68de-4d16-83ee-6db12d1625be-istio-token\") pod \"e72aff77-68de-4d16-83ee-6db12d1625be\" (UID: \"e72aff77-68de-4d16-83ee-6db12d1625be\") "
Jul 06 23:21:16.858023 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:16.857868    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kxpc2\" (UniqueName: \"kubernetes.io/projected/e72aff77-68de-4d16-83ee-6db12d1625be-kube-api-access-kxpc2\") pod \"e72aff77-68de-4d16-83ee-6db12d1625be\" (UID: \"e72aff77-68de-4d16-83ee-6db12d1625be\") "
Jul 06 23:21:16.858023 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:16.857917    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e72aff77-68de-4d16-83ee-6db12d1625be-workload-socket\") pod \"e72aff77-68de-4d16-83ee-6db12d1625be\" (UID: \"e72aff77-68de-4d16-83ee-6db12d1625be\") "
Jul 06 23:21:16.858023 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:16.857951    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e72aff77-68de-4d16-83ee-6db12d1625be-credential-socket\") pod \"e72aff77-68de-4d16-83ee-6db12d1625be\" (UID: \"e72aff77-68de-4d16-83ee-6db12d1625be\") "
Jul 06 23:21:16.858214 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:16.858119    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e72aff77-68de-4d16-83ee-6db12d1625be-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e72aff77-68de-4d16-83ee-6db12d1625be" (UID: "e72aff77-68de-4d16-83ee-6db12d1625be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 23:21:16.858284 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:16.858260    2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e72aff77-68de-4d16-83ee-6db12d1625be-istiod-ca-cert\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:21:16.858420 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:16.858366    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e72aff77-68de-4d16-83ee-6db12d1625be-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e72aff77-68de-4d16-83ee-6db12d1625be" (UID: "e72aff77-68de-4d16-83ee-6db12d1625be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:21:16.858420 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:16.858387    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e72aff77-68de-4d16-83ee-6db12d1625be-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e72aff77-68de-4d16-83ee-6db12d1625be" (UID: "e72aff77-68de-4d16-83ee-6db12d1625be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:21:16.858420 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:16.858377    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e72aff77-68de-4d16-83ee-6db12d1625be-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e72aff77-68de-4d16-83ee-6db12d1625be" (UID: "e72aff77-68de-4d16-83ee-6db12d1625be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:21:16.859210 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:16.859161    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e72aff77-68de-4d16-83ee-6db12d1625be-istio-data" (OuterVolumeSpecName: "istio-data") pod "e72aff77-68de-4d16-83ee-6db12d1625be" (UID: "e72aff77-68de-4d16-83ee-6db12d1625be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:21:16.859885 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:16.859863    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e72aff77-68de-4d16-83ee-6db12d1625be-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e72aff77-68de-4d16-83ee-6db12d1625be" (UID: "e72aff77-68de-4d16-83ee-6db12d1625be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 06 23:21:16.860059 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:16.860032    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e72aff77-68de-4d16-83ee-6db12d1625be-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e72aff77-68de-4d16-83ee-6db12d1625be" (UID: "e72aff77-68de-4d16-83ee-6db12d1625be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:21:16.860166 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:16.860123    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e72aff77-68de-4d16-83ee-6db12d1625be-istio-token" (OuterVolumeSpecName: "istio-token") pod "e72aff77-68de-4d16-83ee-6db12d1625be" (UID: "e72aff77-68de-4d16-83ee-6db12d1625be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 23:21:16.860166 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:16.860154    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e72aff77-68de-4d16-83ee-6db12d1625be-kube-api-access-kxpc2" (OuterVolumeSpecName: "kube-api-access-kxpc2") pod "e72aff77-68de-4d16-83ee-6db12d1625be" (UID: "e72aff77-68de-4d16-83ee-6db12d1625be"). InnerVolumeSpecName "kube-api-access-kxpc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 23:21:16.958951 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:16.958927    2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e72aff77-68de-4d16-83ee-6db12d1625be-workload-certs\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:21:16.958951 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:16.958949    2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e72aff77-68de-4d16-83ee-6db12d1625be-istio-data\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:21:16.959079 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:16.958959    2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e72aff77-68de-4d16-83ee-6db12d1625be-istio-token\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:21:16.959079 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:16.958968    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxpc2\" (UniqueName: \"kubernetes.io/projected/e72aff77-68de-4d16-83ee-6db12d1625be-kube-api-access-kxpc2\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:21:16.959079 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:16.958979    2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e72aff77-68de-4d16-83ee-6db12d1625be-workload-socket\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:21:16.959079 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:16.958987    2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e72aff77-68de-4d16-83ee-6db12d1625be-credential-socket\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:21:16.959079 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:16.958995    2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e72aff77-68de-4d16-83ee-6db12d1625be-istio-podinfo\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:21:16.959079 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:16.959003    2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e72aff77-68de-4d16-83ee-6db12d1625be-istio-envoy\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:21:17.019962 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:17.019898    2569 generic.go:356] "Generic (PLEG): container finished" podID="e72aff77-68de-4d16-83ee-6db12d1625be" containerID="3c6ded4f3d75ccef28cb35c26429023ba6f1a5323003dc6557bab30ff036b7c0" exitCode=0
Jul 06 23:21:17.019962 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:17.019935    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44" event={"ID":"e72aff77-68de-4d16-83ee-6db12d1625be","Type":"ContainerDied","Data":"3c6ded4f3d75ccef28cb35c26429023ba6f1a5323003dc6557bab30ff036b7c0"}
Jul 06 23:21:17.019962 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:17.019948    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"
Jul 06 23:21:17.020139 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:17.019989    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44" event={"ID":"e72aff77-68de-4d16-83ee-6db12d1625be","Type":"ContainerDied","Data":"b712dff95cab6f81959a998b236e354f26295b756f7c5694b309f86b7bb9a4cf"}
Jul 06 23:21:17.020139 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:17.020016    2569 scope.go:117] "RemoveContainer" containerID="3c6ded4f3d75ccef28cb35c26429023ba6f1a5323003dc6557bab30ff036b7c0"
Jul 06 23:21:17.031135 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:17.031110    2569 scope.go:117] "RemoveContainer" containerID="4e91dd8cdceb971b96b2104cc58870e3dd8cab68dfd4dfa1fee6dcf7406d21eb"
Jul 06 23:21:17.038980 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:17.038961    2569 scope.go:117] "RemoveContainer" containerID="3c6ded4f3d75ccef28cb35c26429023ba6f1a5323003dc6557bab30ff036b7c0"
Jul 06 23:21:17.039257 ip-10-0-134-116 kubenswrapper[2569]: E0706 23:21:17.039236    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c6ded4f3d75ccef28cb35c26429023ba6f1a5323003dc6557bab30ff036b7c0\": container with ID starting with 3c6ded4f3d75ccef28cb35c26429023ba6f1a5323003dc6557bab30ff036b7c0 not found: ID does not exist" containerID="3c6ded4f3d75ccef28cb35c26429023ba6f1a5323003dc6557bab30ff036b7c0"
Jul 06 23:21:17.039389 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:17.039269    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c6ded4f3d75ccef28cb35c26429023ba6f1a5323003dc6557bab30ff036b7c0"} err="failed to get container status \"3c6ded4f3d75ccef28cb35c26429023ba6f1a5323003dc6557bab30ff036b7c0\": rpc error: code = NotFound desc = could not find container \"3c6ded4f3d75ccef28cb35c26429023ba6f1a5323003dc6557bab30ff036b7c0\": container with ID starting with 3c6ded4f3d75ccef28cb35c26429023ba6f1a5323003dc6557bab30ff036b7c0 not found: ID does not exist"
Jul 06 23:21:17.039389 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:17.039294    2569 scope.go:117] "RemoveContainer" containerID="4e91dd8cdceb971b96b2104cc58870e3dd8cab68dfd4dfa1fee6dcf7406d21eb"
Jul 06 23:21:17.039560 ip-10-0-134-116 kubenswrapper[2569]: E0706 23:21:17.039543    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e91dd8cdceb971b96b2104cc58870e3dd8cab68dfd4dfa1fee6dcf7406d21eb\": container with ID starting with 4e91dd8cdceb971b96b2104cc58870e3dd8cab68dfd4dfa1fee6dcf7406d21eb not found: ID does not exist" containerID="4e91dd8cdceb971b96b2104cc58870e3dd8cab68dfd4dfa1fee6dcf7406d21eb"
Jul 06 23:21:17.039610 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:17.039564    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e91dd8cdceb971b96b2104cc58870e3dd8cab68dfd4dfa1fee6dcf7406d21eb"} err="failed to get container status \"4e91dd8cdceb971b96b2104cc58870e3dd8cab68dfd4dfa1fee6dcf7406d21eb\": rpc error: code = NotFound desc = could not find container \"4e91dd8cdceb971b96b2104cc58870e3dd8cab68dfd4dfa1fee6dcf7406d21eb\": container with ID starting with 4e91dd8cdceb971b96b2104cc58870e3dd8cab68dfd4dfa1fee6dcf7406d21eb not found: ID does not exist"
Jul 06 23:21:17.046743 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:17.046721    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"]
Jul 06 23:21:17.049535 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:17.049517    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e0bef8-openshift-default-8cd9bf97f-m4r44"]
Jul 06 23:21:18.176470 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:18.176440    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e72aff77-68de-4d16-83ee-6db12d1625be" path="/var/lib/kubelet/pods/e72aff77-68de-4d16-83ee-6db12d1625be/volumes"
Jul 06 23:21:19.675565 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:19.675533    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"]
Jul 06 23:21:19.675923 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:19.675887    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e72aff77-68de-4d16-83ee-6db12d1625be" containerName="istio-proxy"
Jul 06 23:21:19.675923 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:19.675897    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e72aff77-68de-4d16-83ee-6db12d1625be" containerName="istio-proxy"
Jul 06 23:21:19.675923 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:19.675916    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e72aff77-68de-4d16-83ee-6db12d1625be" containerName="istio-proxy"
Jul 06 23:21:19.675923 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:19.675921    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e72aff77-68de-4d16-83ee-6db12d1625be" containerName="istio-proxy"
Jul 06 23:21:19.676076 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:19.675973    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e72aff77-68de-4d16-83ee-6db12d1625be" containerName="istio-proxy"
Jul 06 23:21:19.676076 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:19.675983    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="40525690-0960-4e67-a6b1-24f3245d6d96" containerName="istio-proxy"
Jul 06 23:21:19.679768 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:19.679744    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"
Jul 06 23:21:19.682638 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:19.682617    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-669fee-openshift-default-dockercfg-s4mwk\""
Jul 06 23:21:19.694889 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:19.694865    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"]
Jul 06 23:21:19.783291 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:19.783261    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e57fe7-6888-4ff4-bd85-a541c58164ad-istio-podinfo\") pod \"e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47\" (UID: \"80e57fe7-6888-4ff4-bd85-a541c58164ad\") " pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"
Jul 06 23:21:19.783453 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:19.783301    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e57fe7-6888-4ff4-bd85-a541c58164ad-istio-data\") pod \"e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47\" (UID: \"80e57fe7-6888-4ff4-bd85-a541c58164ad\") " pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"
Jul 06 23:21:19.783453 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:19.783356    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e57fe7-6888-4ff4-bd85-a541c58164ad-istiod-ca-cert\") pod \"e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47\" (UID: \"80e57fe7-6888-4ff4-bd85-a541c58164ad\") " pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"
Jul 06 23:21:19.783453 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:19.783403    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkkjw\" (UniqueName: \"kubernetes.io/projected/80e57fe7-6888-4ff4-bd85-a541c58164ad-kube-api-access-nkkjw\") pod \"e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47\" (UID: \"80e57fe7-6888-4ff4-bd85-a541c58164ad\") " pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"
Jul 06 23:21:19.783453 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:19.783439    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e57fe7-6888-4ff4-bd85-a541c58164ad-workload-certs\") pod \"e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47\" (UID: \"80e57fe7-6888-4ff4-bd85-a541c58164ad\") " pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"
Jul 06 23:21:19.783453 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:19.783453    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e57fe7-6888-4ff4-bd85-a541c58164ad-istio-envoy\") pod \"e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47\" (UID: \"80e57fe7-6888-4ff4-bd85-a541c58164ad\") " pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"
Jul 06 23:21:19.783651 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:19.783497    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e57fe7-6888-4ff4-bd85-a541c58164ad-credential-socket\") pod \"e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47\" (UID: \"80e57fe7-6888-4ff4-bd85-a541c58164ad\") " pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"
Jul 06 23:21:19.783651 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:19.783540    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e57fe7-6888-4ff4-bd85-a541c58164ad-workload-socket\") pod \"e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47\" (UID: \"80e57fe7-6888-4ff4-bd85-a541c58164ad\") " pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"
Jul 06 23:21:19.783651 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:19.783565    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80e57fe7-6888-4ff4-bd85-a541c58164ad-istio-token\") pod \"e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47\" (UID: \"80e57fe7-6888-4ff4-bd85-a541c58164ad\") " pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"
Jul 06 23:21:19.884473 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:19.884446    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e57fe7-6888-4ff4-bd85-a541c58164ad-workload-socket\") pod \"e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47\" (UID: \"80e57fe7-6888-4ff4-bd85-a541c58164ad\") " pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"
Jul 06 23:21:19.884605 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:19.884474    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80e57fe7-6888-4ff4-bd85-a541c58164ad-istio-token\") pod \"e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47\" (UID: \"80e57fe7-6888-4ff4-bd85-a541c58164ad\") " pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"
Jul 06 23:21:19.884605 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:19.884497    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e57fe7-6888-4ff4-bd85-a541c58164ad-istio-podinfo\") pod \"e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47\" (UID: \"80e57fe7-6888-4ff4-bd85-a541c58164ad\") " pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"
Jul 06 23:21:19.884605 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:19.884523    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e57fe7-6888-4ff4-bd85-a541c58164ad-istio-data\") pod \"e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47\" (UID: \"80e57fe7-6888-4ff4-bd85-a541c58164ad\") " pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"
Jul 06 23:21:19.884605 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:19.884559    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e57fe7-6888-4ff4-bd85-a541c58164ad-istiod-ca-cert\") pod \"e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47\" (UID: \"80e57fe7-6888-4ff4-bd85-a541c58164ad\") " pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"
Jul 06 23:21:19.884605 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:19.884588    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkkjw\" (UniqueName: \"kubernetes.io/projected/80e57fe7-6888-4ff4-bd85-a541c58164ad-kube-api-access-nkkjw\") pod \"e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47\" (UID: \"80e57fe7-6888-4ff4-bd85-a541c58164ad\") " pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"
Jul 06 23:21:19.884836 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:19.884655    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e57fe7-6888-4ff4-bd85-a541c58164ad-workload-certs\") pod \"e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47\" (UID: \"80e57fe7-6888-4ff4-bd85-a541c58164ad\") " pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"
Jul 06 23:21:19.884836 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:19.884679    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e57fe7-6888-4ff4-bd85-a541c58164ad-istio-envoy\") pod \"e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47\" (UID: \"80e57fe7-6888-4ff4-bd85-a541c58164ad\") " pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"
Jul 06 23:21:19.884836 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:19.884708    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e57fe7-6888-4ff4-bd85-a541c58164ad-credential-socket\") pod \"e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47\" (UID: \"80e57fe7-6888-4ff4-bd85-a541c58164ad\") " pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"
Jul 06 23:21:19.884993 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:19.884835    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e57fe7-6888-4ff4-bd85-a541c58164ad-workload-socket\") pod \"e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47\" (UID: \"80e57fe7-6888-4ff4-bd85-a541c58164ad\") " pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"
Jul 06 23:21:19.884993 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:19.884886    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e57fe7-6888-4ff4-bd85-a541c58164ad-istio-data\") pod \"e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47\" (UID: \"80e57fe7-6888-4ff4-bd85-a541c58164ad\") " pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"
Jul 06 23:21:19.885074 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:19.884990    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e57fe7-6888-4ff4-bd85-a541c58164ad-credential-socket\") pod \"e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47\" (UID: \"80e57fe7-6888-4ff4-bd85-a541c58164ad\") " pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"
Jul 06 23:21:19.885109 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:19.885073    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e57fe7-6888-4ff4-bd85-a541c58164ad-workload-certs\") pod \"e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47\" (UID: \"80e57fe7-6888-4ff4-bd85-a541c58164ad\") " pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"
Jul 06 23:21:19.885278 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:19.885259    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e57fe7-6888-4ff4-bd85-a541c58164ad-istiod-ca-cert\") pod \"e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47\" (UID: \"80e57fe7-6888-4ff4-bd85-a541c58164ad\") " pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"
Jul 06 23:21:19.886919 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:19.886902    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e57fe7-6888-4ff4-bd85-a541c58164ad-istio-envoy\") pod \"e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47\" (UID: \"80e57fe7-6888-4ff4-bd85-a541c58164ad\") " pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"
Jul 06 23:21:19.886991 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:19.886975    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e57fe7-6888-4ff4-bd85-a541c58164ad-istio-podinfo\") pod \"e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47\" (UID: \"80e57fe7-6888-4ff4-bd85-a541c58164ad\") " pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"
Jul 06 23:21:19.895398 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:19.895378    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkkjw\" (UniqueName: \"kubernetes.io/projected/80e57fe7-6888-4ff4-bd85-a541c58164ad-kube-api-access-nkkjw\") pod \"e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47\" (UID: \"80e57fe7-6888-4ff4-bd85-a541c58164ad\") " pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"
Jul 06 23:21:19.895531 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:19.895394    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80e57fe7-6888-4ff4-bd85-a541c58164ad-istio-token\") pod \"e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47\" (UID: \"80e57fe7-6888-4ff4-bd85-a541c58164ad\") " pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"
Jul 06 23:21:19.990267 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:19.990248    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"
Jul 06 23:21:20.126043 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:20.126021    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"]
Jul 06 23:21:20.128145 ip-10-0-134-116 kubenswrapper[2569]: W0706 23:21:20.128122    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80e57fe7_6888_4ff4_bd85_a541c58164ad.slice/crio-948ec83327a8ec25dd2a224e0d1dc1ea94dbcded8efa08b533ebb458d39fb904 WatchSource:0}: Error finding container 948ec83327a8ec25dd2a224e0d1dc1ea94dbcded8efa08b533ebb458d39fb904: Status 404 returned error can't find the container with id 948ec83327a8ec25dd2a224e0d1dc1ea94dbcded8efa08b533ebb458d39fb904
Jul 06 23:21:20.130607 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:20.130565    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 23:21:20.130685 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:20.130658    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 23:21:20.130727 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:20.130703    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 23:21:21.041798 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:21.041763    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47" event={"ID":"80e57fe7-6888-4ff4-bd85-a541c58164ad","Type":"ContainerStarted","Data":"8a235296c09b3bae1525211802187255296d81f1bf39999ab80a00a246a34314"}
Jul 06 23:21:21.042243 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:21.041819    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47" event={"ID":"80e57fe7-6888-4ff4-bd85-a541c58164ad","Type":"ContainerStarted","Data":"948ec83327a8ec25dd2a224e0d1dc1ea94dbcded8efa08b533ebb458d39fb904"}
Jul 06 23:21:21.062773 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:21.062588    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47" podStartSLOduration=2.062568808 podStartE2EDuration="2.062568808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:21:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:21:21.061882096 +0000 UTC m=+2093.466132010" watchObservedRunningTime="2026-07-06 23:21:21.062568808 +0000 UTC m=+2093.466818711"
Jul 06 23:21:21.991194 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:21.991148    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"
Jul 06 23:21:21.995696 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:21.995670    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"
Jul 06 23:21:22.047200 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:22.047156    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"
Jul 06 23:21:22.048099 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:22.048077    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"
Jul 06 23:21:28.175921 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:28.175890    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bvsb_571f407a-249a-4fd9-8c15-53a6541db635/ovn-acl-logging/0.log"
Jul 06 23:21:28.181868 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:28.181844    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bvsb_571f407a-249a-4fd9-8c15-53a6541db635/ovn-acl-logging/0.log"
Jul 06 23:21:36.130729 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:36.130654    2569 generic.go:356] "Generic (PLEG): container finished" podID="80e57fe7-6888-4ff4-bd85-a541c58164ad" containerID="8a235296c09b3bae1525211802187255296d81f1bf39999ab80a00a246a34314" exitCode=0
Jul 06 23:21:36.131116 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:36.130735    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47" event={"ID":"80e57fe7-6888-4ff4-bd85-a541c58164ad","Type":"ContainerDied","Data":"8a235296c09b3bae1525211802187255296d81f1bf39999ab80a00a246a34314"}
Jul 06 23:21:36.131212 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:36.131135    2569 scope.go:117] "RemoveContainer" containerID="8a235296c09b3bae1525211802187255296d81f1bf39999ab80a00a246a34314"
Jul 06 23:21:36.132113 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:36.132084    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 23:21:36.132233 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:36.132149    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 23:21:36.132233 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:36.132195    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 23:21:37.047832 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:37.047803    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"
Jul 06 23:21:37.137714 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:37.137676    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47" event={"ID":"80e57fe7-6888-4ff4-bd85-a541c58164ad","Type":"ContainerStarted","Data":"1c2c45e7139560fa01f88aebe8fca0e47663e582a5a4b5abaa276faa86905e26"}
Jul 06 23:21:37.992323 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:37.992288    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"
Jul 06 23:21:37.992529 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:37.992401    2569 patch_prober.go:28] interesting pod/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Jul 06 23:21:37.992529 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:37.992445    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47" podUID="80e57fe7-6888-4ff4-bd85-a541c58164ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Jul 06 23:21:38.991310 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:38.991270    2569 patch_prober.go:28] interesting pod/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Jul 06 23:21:38.991789 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:38.991334    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47" podUID="80e57fe7-6888-4ff4-bd85-a541c58164ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Jul 06 23:21:39.991102 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:39.991059    2569 patch_prober.go:28] interesting pod/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Jul 06 23:21:39.991315 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:39.991136    2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47" podUID="80e57fe7-6888-4ff4-bd85-a541c58164ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Jul 06 23:21:40.994594 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:40.994567    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"
Jul 06 23:21:40.994964 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:40.994832    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"
Jul 06 23:21:40.995530 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:40.995509    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"
Jul 06 23:21:46.392482 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:46.392446    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-669fee/auth-test-model-669fee-kserve-796fc59cdc-kq9g7"]
Jul 06 23:21:46.393038 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:46.392972    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e72aff77-68de-4d16-83ee-6db12d1625be" containerName="istio-proxy"
Jul 06 23:21:46.395132 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:46.395117    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-669fee/auth-test-model-669fee-kserve-796fc59cdc-kq9g7"
Jul 06 23:21:46.403450 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:46.403433    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-669fee\"/\"default-dockercfg-6ncrl\""
Jul 06 23:21:46.404695 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:46.404678    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-669fee\"/\"openshift-service-ca.crt\""
Jul 06 23:21:46.404813 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:46.404785    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-669fee\"/\"auth-test-model-669fee-kserve-self-signed-certs\""
Jul 06 23:21:46.404889 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:46.404837    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-669fee\"/\"kube-root-ca.crt\""
Jul 06 23:21:46.525886 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:46.525850    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/358fd68e-d95f-49eb-af9e-2fbb084a1027-dshm\") pod \"auth-test-model-669fee-kserve-796fc59cdc-kq9g7\" (UID: \"358fd68e-d95f-49eb-af9e-2fbb084a1027\") " pod="ai-tenant-e2e-shared-a-669fee/auth-test-model-669fee-kserve-796fc59cdc-kq9g7"
Jul 06 23:21:46.526075 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:46.525905    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gmnf\" (UniqueName: \"kubernetes.io/projected/358fd68e-d95f-49eb-af9e-2fbb084a1027-kube-api-access-9gmnf\") pod \"auth-test-model-669fee-kserve-796fc59cdc-kq9g7\" (UID: \"358fd68e-d95f-49eb-af9e-2fbb084a1027\") " pod="ai-tenant-e2e-shared-a-669fee/auth-test-model-669fee-kserve-796fc59cdc-kq9g7"
Jul 06 23:21:46.526075 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:46.525997    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/358fd68e-d95f-49eb-af9e-2fbb084a1027-model-cache\") pod \"auth-test-model-669fee-kserve-796fc59cdc-kq9g7\" (UID: \"358fd68e-d95f-49eb-af9e-2fbb084a1027\") " pod="ai-tenant-e2e-shared-a-669fee/auth-test-model-669fee-kserve-796fc59cdc-kq9g7"
Jul 06 23:21:46.526075 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:46.526038    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/358fd68e-d95f-49eb-af9e-2fbb084a1027-tls-certs\") pod \"auth-test-model-669fee-kserve-796fc59cdc-kq9g7\" (UID: \"358fd68e-d95f-49eb-af9e-2fbb084a1027\") " pod="ai-tenant-e2e-shared-a-669fee/auth-test-model-669fee-kserve-796fc59cdc-kq9g7"
Jul 06 23:21:46.526075 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:46.526069    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/358fd68e-d95f-49eb-af9e-2fbb084a1027-tmp-dir\") pod \"auth-test-model-669fee-kserve-796fc59cdc-kq9g7\" (UID: \"358fd68e-d95f-49eb-af9e-2fbb084a1027\") " pod="ai-tenant-e2e-shared-a-669fee/auth-test-model-669fee-kserve-796fc59cdc-kq9g7"
Jul 06 23:21:46.526374 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:46.526114    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/358fd68e-d95f-49eb-af9e-2fbb084a1027-home\") pod \"auth-test-model-669fee-kserve-796fc59cdc-kq9g7\" (UID: \"358fd68e-d95f-49eb-af9e-2fbb084a1027\") " pod="ai-tenant-e2e-shared-a-669fee/auth-test-model-669fee-kserve-796fc59cdc-kq9g7"
Jul 06 23:21:46.627498 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:46.627465    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/358fd68e-d95f-49eb-af9e-2fbb084a1027-dshm\") pod \"auth-test-model-669fee-kserve-796fc59cdc-kq9g7\" (UID: \"358fd68e-d95f-49eb-af9e-2fbb084a1027\") " pod="ai-tenant-e2e-shared-a-669fee/auth-test-model-669fee-kserve-796fc59cdc-kq9g7"
Jul 06 23:21:46.627682 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:46.627513    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gmnf\" (UniqueName: \"kubernetes.io/projected/358fd68e-d95f-49eb-af9e-2fbb084a1027-kube-api-access-9gmnf\") pod \"auth-test-model-669fee-kserve-796fc59cdc-kq9g7\" (UID: \"358fd68e-d95f-49eb-af9e-2fbb084a1027\") " pod="ai-tenant-e2e-shared-a-669fee/auth-test-model-669fee-kserve-796fc59cdc-kq9g7"
Jul 06 23:21:46.627682 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:46.627552    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/358fd68e-d95f-49eb-af9e-2fbb084a1027-model-cache\") pod \"auth-test-model-669fee-kserve-796fc59cdc-kq9g7\" (UID: \"358fd68e-d95f-49eb-af9e-2fbb084a1027\") " pod="ai-tenant-e2e-shared-a-669fee/auth-test-model-669fee-kserve-796fc59cdc-kq9g7"
Jul 06 23:21:46.627682 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:46.627570    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/358fd68e-d95f-49eb-af9e-2fbb084a1027-tls-certs\") pod \"auth-test-model-669fee-kserve-796fc59cdc-kq9g7\" (UID: \"358fd68e-d95f-49eb-af9e-2fbb084a1027\") " pod="ai-tenant-e2e-shared-a-669fee/auth-test-model-669fee-kserve-796fc59cdc-kq9g7"
Jul 06 23:21:46.627682 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:46.627594    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/358fd68e-d95f-49eb-af9e-2fbb084a1027-tmp-dir\") pod \"auth-test-model-669fee-kserve-796fc59cdc-kq9g7\" (UID: \"358fd68e-d95f-49eb-af9e-2fbb084a1027\") " pod="ai-tenant-e2e-shared-a-669fee/auth-test-model-669fee-kserve-796fc59cdc-kq9g7"
Jul 06 23:21:46.627682 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:46.627620    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/358fd68e-d95f-49eb-af9e-2fbb084a1027-home\") pod \"auth-test-model-669fee-kserve-796fc59cdc-kq9g7\" (UID: \"358fd68e-d95f-49eb-af9e-2fbb084a1027\") " pod="ai-tenant-e2e-shared-a-669fee/auth-test-model-669fee-kserve-796fc59cdc-kq9g7"
Jul 06 23:21:46.627992 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:46.627969    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/358fd68e-d95f-49eb-af9e-2fbb084a1027-tmp-dir\") pod \"auth-test-model-669fee-kserve-796fc59cdc-kq9g7\" (UID: \"358fd68e-d95f-49eb-af9e-2fbb084a1027\") " pod="ai-tenant-e2e-shared-a-669fee/auth-test-model-669fee-kserve-796fc59cdc-kq9g7"
Jul 06 23:21:46.627992 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:46.627984    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/358fd68e-d95f-49eb-af9e-2fbb084a1027-home\") pod \"auth-test-model-669fee-kserve-796fc59cdc-kq9g7\" (UID: \"358fd68e-d95f-49eb-af9e-2fbb084a1027\") " pod="ai-tenant-e2e-shared-a-669fee/auth-test-model-669fee-kserve-796fc59cdc-kq9g7"
Jul 06 23:21:46.628100 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:46.628041    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/358fd68e-d95f-49eb-af9e-2fbb084a1027-model-cache\") pod \"auth-test-model-669fee-kserve-796fc59cdc-kq9g7\" (UID: \"358fd68e-d95f-49eb-af9e-2fbb084a1027\") " pod="ai-tenant-e2e-shared-a-669fee/auth-test-model-669fee-kserve-796fc59cdc-kq9g7"
Jul 06 23:21:46.630002 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:46.629668    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/358fd68e-d95f-49eb-af9e-2fbb084a1027-dshm\") pod \"auth-test-model-669fee-kserve-796fc59cdc-kq9g7\" (UID: \"358fd68e-d95f-49eb-af9e-2fbb084a1027\") " pod="ai-tenant-e2e-shared-a-669fee/auth-test-model-669fee-kserve-796fc59cdc-kq9g7"
Jul 06 23:21:46.630586 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:46.630558    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/358fd68e-d95f-49eb-af9e-2fbb084a1027-tls-certs\") pod \"auth-test-model-669fee-kserve-796fc59cdc-kq9g7\" (UID: \"358fd68e-d95f-49eb-af9e-2fbb084a1027\") " pod="ai-tenant-e2e-shared-a-669fee/auth-test-model-669fee-kserve-796fc59cdc-kq9g7"
Jul 06 23:21:46.636231 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:46.636208    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gmnf\" (UniqueName: \"kubernetes.io/projected/358fd68e-d95f-49eb-af9e-2fbb084a1027-kube-api-access-9gmnf\") pod \"auth-test-model-669fee-kserve-796fc59cdc-kq9g7\" (UID: \"358fd68e-d95f-49eb-af9e-2fbb084a1027\") " pod="ai-tenant-e2e-shared-a-669fee/auth-test-model-669fee-kserve-796fc59cdc-kq9g7"
Jul 06 23:21:46.704267 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:21:46.704246    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-669fee/auth-test-model-669fee-kserve-796fc59cdc-kq9g7"
Jul 06 23:22:27.821630 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:22:27.821558    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-669fee/auth-test-model-669fee-kserve-796fc59cdc-kq9g7"]
Jul 06 23:22:28.075388 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:22:28.075366    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-669fee/auth-test-model-669fee-kserve-796fc59cdc-kq9g7"]
Jul 06 23:22:28.077633 ip-10-0-134-116 kubenswrapper[2569]: W0706 23:22:28.077606    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod358fd68e_d95f_49eb_af9e_2fbb084a1027.slice/crio-b04e0be09f15e1d5617e2a46d72ce45d9d275c542d516f6b5814adb8a9ddfe6b WatchSource:0}: Error finding container b04e0be09f15e1d5617e2a46d72ce45d9d275c542d516f6b5814adb8a9ddfe6b: Status 404 returned error can't find the container with id b04e0be09f15e1d5617e2a46d72ce45d9d275c542d516f6b5814adb8a9ddfe6b
Jul 06 23:22:28.370512 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:22:28.370433    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-669fee/auth-test-model-669fee-kserve-796fc59cdc-kq9g7" event={"ID":"358fd68e-d95f-49eb-af9e-2fbb084a1027","Type":"ContainerStarted","Data":"2475c98e6a2aa6605f5987c1dbd2cdfbbfe021146355bd4c57b083f4c0786640"}
Jul 06 23:22:28.370512 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:22:28.370478    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-669fee/auth-test-model-669fee-kserve-796fc59cdc-kq9g7" event={"ID":"358fd68e-d95f-49eb-af9e-2fbb084a1027","Type":"ContainerStarted","Data":"b04e0be09f15e1d5617e2a46d72ce45d9d275c542d516f6b5814adb8a9ddfe6b"}
Jul 06 23:22:28.388390 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:22:28.388349    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-669fee/auth-test-model-669fee-kserve-796fc59cdc-kq9g7" podStartSLOduration=42.388334926 podStartE2EDuration="42.388334926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:21:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:22:28.387228747 +0000 UTC m=+2160.791478648" watchObservedRunningTime="2026-07-06 23:22:28.388334926 +0000 UTC m=+2160.792584829"
Jul 06 23:22:36.705118 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:22:36.705080    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-669fee/auth-test-model-669fee-kserve-796fc59cdc-kq9g7"
Jul 06 23:22:36.705666 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:22:36.705154    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-669fee/auth-test-model-669fee-kserve-796fc59cdc-kq9g7"
Jul 06 23:22:36.717062 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:22:36.717043    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-669fee/auth-test-model-669fee-kserve-796fc59cdc-kq9g7"
Jul 06 23:22:37.423882 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:22:37.423847    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-669fee/auth-test-model-669fee-kserve-796fc59cdc-kq9g7"
Jul 06 23:22:58.652090 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:22:58.652051    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-d3a5b8/auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq"]
Jul 06 23:22:58.654441 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:22:58.654421    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d3a5b8/auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq"
Jul 06 23:22:58.657426 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:22:58.657403    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d3a5b8\"/\"auth-test-model-d3a5b8-kserve-self-signed-certs\""
Jul 06 23:22:58.658588 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:22:58.658567    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d3a5b8\"/\"kube-root-ca.crt\""
Jul 06 23:22:58.658707 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:22:58.658595    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d3a5b8\"/\"openshift-service-ca.crt\""
Jul 06 23:22:58.658707 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:22:58.658596    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d3a5b8\"/\"default-dockercfg-kqhdj\""
Jul 06 23:22:58.666880 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:22:58.666860    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d3a5b8/auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq"]
Jul 06 23:22:58.779141 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:22:58.779106    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/902d0bec-c6fe-4a66-8fa6-513172f37b82-dshm\") pod \"auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq\" (UID: \"902d0bec-c6fe-4a66-8fa6-513172f37b82\") " pod="ai-tenant-e2e-shared-b-d3a5b8/auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq"
Jul 06 23:22:58.779290 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:22:58.779154    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/902d0bec-c6fe-4a66-8fa6-513172f37b82-model-cache\") pod \"auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq\" (UID: \"902d0bec-c6fe-4a66-8fa6-513172f37b82\") " pod="ai-tenant-e2e-shared-b-d3a5b8/auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq"
Jul 06 23:22:58.779290 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:22:58.779253    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/902d0bec-c6fe-4a66-8fa6-513172f37b82-tmp-dir\") pod \"auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq\" (UID: \"902d0bec-c6fe-4a66-8fa6-513172f37b82\") " pod="ai-tenant-e2e-shared-b-d3a5b8/auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq"
Jul 06 23:22:58.779290 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:22:58.779270    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpsjg\" (UniqueName: \"kubernetes.io/projected/902d0bec-c6fe-4a66-8fa6-513172f37b82-kube-api-access-jpsjg\") pod \"auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq\" (UID: \"902d0bec-c6fe-4a66-8fa6-513172f37b82\") " pod="ai-tenant-e2e-shared-b-d3a5b8/auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq"
Jul 06 23:22:58.779429 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:22:58.779301    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/902d0bec-c6fe-4a66-8fa6-513172f37b82-tls-certs\") pod \"auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq\" (UID: \"902d0bec-c6fe-4a66-8fa6-513172f37b82\") " pod="ai-tenant-e2e-shared-b-d3a5b8/auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq"
Jul 06 23:22:58.779429 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:22:58.779379    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/902d0bec-c6fe-4a66-8fa6-513172f37b82-home\") pod \"auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq\" (UID: \"902d0bec-c6fe-4a66-8fa6-513172f37b82\") " pod="ai-tenant-e2e-shared-b-d3a5b8/auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq"
Jul 06 23:22:58.880104 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:22:58.880076    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/902d0bec-c6fe-4a66-8fa6-513172f37b82-tmp-dir\") pod \"auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq\" (UID: \"902d0bec-c6fe-4a66-8fa6-513172f37b82\") " pod="ai-tenant-e2e-shared-b-d3a5b8/auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq"
Jul 06 23:22:58.880308 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:22:58.880114    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpsjg\" (UniqueName: \"kubernetes.io/projected/902d0bec-c6fe-4a66-8fa6-513172f37b82-kube-api-access-jpsjg\") pod \"auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq\" (UID: \"902d0bec-c6fe-4a66-8fa6-513172f37b82\") " pod="ai-tenant-e2e-shared-b-d3a5b8/auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq"
Jul 06 23:22:58.880308 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:22:58.880154    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/902d0bec-c6fe-4a66-8fa6-513172f37b82-tls-certs\") pod \"auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq\" (UID: \"902d0bec-c6fe-4a66-8fa6-513172f37b82\") " pod="ai-tenant-e2e-shared-b-d3a5b8/auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq"
Jul 06 23:22:58.880308 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:22:58.880272    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/902d0bec-c6fe-4a66-8fa6-513172f37b82-home\") pod \"auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq\" (UID: \"902d0bec-c6fe-4a66-8fa6-513172f37b82\") " pod="ai-tenant-e2e-shared-b-d3a5b8/auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq"
Jul 06 23:22:58.880481 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:22:58.880311    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/902d0bec-c6fe-4a66-8fa6-513172f37b82-dshm\") pod \"auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq\" (UID: \"902d0bec-c6fe-4a66-8fa6-513172f37b82\") " pod="ai-tenant-e2e-shared-b-d3a5b8/auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq"
Jul 06 23:22:58.880481 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:22:58.880345    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/902d0bec-c6fe-4a66-8fa6-513172f37b82-model-cache\") pod \"auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq\" (UID: \"902d0bec-c6fe-4a66-8fa6-513172f37b82\") " pod="ai-tenant-e2e-shared-b-d3a5b8/auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq"
Jul 06 23:22:58.880602 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:22:58.880540    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/902d0bec-c6fe-4a66-8fa6-513172f37b82-tmp-dir\") pod \"auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq\" (UID: \"902d0bec-c6fe-4a66-8fa6-513172f37b82\") " pod="ai-tenant-e2e-shared-b-d3a5b8/auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq"
Jul 06 23:22:58.880602 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:22:58.880552    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/902d0bec-c6fe-4a66-8fa6-513172f37b82-home\") pod \"auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq\" (UID: \"902d0bec-c6fe-4a66-8fa6-513172f37b82\") " pod="ai-tenant-e2e-shared-b-d3a5b8/auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq"
Jul 06 23:22:58.880706 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:22:58.880651    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/902d0bec-c6fe-4a66-8fa6-513172f37b82-model-cache\") pod \"auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq\" (UID: \"902d0bec-c6fe-4a66-8fa6-513172f37b82\") " pod="ai-tenant-e2e-shared-b-d3a5b8/auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq"
Jul 06 23:22:58.882488 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:22:58.882464    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/902d0bec-c6fe-4a66-8fa6-513172f37b82-dshm\") pod \"auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq\" (UID: \"902d0bec-c6fe-4a66-8fa6-513172f37b82\") " pod="ai-tenant-e2e-shared-b-d3a5b8/auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq"
Jul 06 23:22:58.882720 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:22:58.882704    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/902d0bec-c6fe-4a66-8fa6-513172f37b82-tls-certs\") pod \"auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq\" (UID: \"902d0bec-c6fe-4a66-8fa6-513172f37b82\") " pod="ai-tenant-e2e-shared-b-d3a5b8/auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq"
Jul 06 23:22:58.890875 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:22:58.890855    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpsjg\" (UniqueName: \"kubernetes.io/projected/902d0bec-c6fe-4a66-8fa6-513172f37b82-kube-api-access-jpsjg\") pod \"auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq\" (UID: \"902d0bec-c6fe-4a66-8fa6-513172f37b82\") " pod="ai-tenant-e2e-shared-b-d3a5b8/auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq"
Jul 06 23:22:58.965722 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:22:58.965698    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d3a5b8/auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq"
Jul 06 23:22:59.118305 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:22:59.118281    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d3a5b8/auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq"]
Jul 06 23:22:59.120462 ip-10-0-134-116 kubenswrapper[2569]: W0706 23:22:59.120432    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod902d0bec_c6fe_4a66_8fa6_513172f37b82.slice/crio-d6727796c0bac8dc3abe797c409440d791f133c6582c9c0802ec64bd72d1cb93 WatchSource:0}: Error finding container d6727796c0bac8dc3abe797c409440d791f133c6582c9c0802ec64bd72d1cb93: Status 404 returned error can't find the container with id d6727796c0bac8dc3abe797c409440d791f133c6582c9c0802ec64bd72d1cb93
Jul 06 23:22:59.523408 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:22:59.523377    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d3a5b8/auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq" event={"ID":"902d0bec-c6fe-4a66-8fa6-513172f37b82","Type":"ContainerStarted","Data":"a63f5647545fd8cdc9515e2bcc86061d2b49af2b0088e096756a227a6be1cf1b"}
Jul 06 23:22:59.523606 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:22:59.523416    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d3a5b8/auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq" event={"ID":"902d0bec-c6fe-4a66-8fa6-513172f37b82","Type":"ContainerStarted","Data":"d6727796c0bac8dc3abe797c409440d791f133c6582c9c0802ec64bd72d1cb93"}
Jul 06 23:22:59.564525 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:22:59.564476    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-d3a5b8/auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq" podStartSLOduration=1.564461447 podStartE2EDuration="1.564461447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:22:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:22:59.560881712 +0000 UTC m=+2191.965131616" watchObservedRunningTime="2026-07-06 23:22:59.564461447 +0000 UTC m=+2191.968711350"
Jul 06 23:23:08.965967 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:23:08.965930    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-d3a5b8/auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq"
Jul 06 23:23:08.966493 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:23:08.966152    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-d3a5b8/auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq"
Jul 06 23:23:08.978679 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:23:08.978656    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-d3a5b8/auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq"
Jul 06 23:23:09.581822 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:23:09.581793    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-d3a5b8/auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq"
Jul 06 23:24:59.097554 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:24:59.097520    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-669fee/auth-test-model-669fee-kserve-796fc59cdc-kq9g7"]
Jul 06 23:25:00.016742 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:00.016709    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d3a5b8/auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq"]
Jul 06 23:25:14.117885 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:14.117823    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-669fee/auth-test-model-669fee-kserve-796fc59cdc-kq9g7" podUID="358fd68e-d95f-49eb-af9e-2fbb084a1027" containerName="main" containerID="cri-o://2475c98e6a2aa6605f5987c1dbd2cdfbbfe021146355bd4c57b083f4c0786640" gracePeriod=45
Jul 06 23:25:14.371088 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:14.371036    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-669fee/auth-test-model-669fee-kserve-796fc59cdc-kq9g7"
Jul 06 23:25:14.478460 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:14.478435    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/358fd68e-d95f-49eb-af9e-2fbb084a1027-home\") pod \"358fd68e-d95f-49eb-af9e-2fbb084a1027\" (UID: \"358fd68e-d95f-49eb-af9e-2fbb084a1027\") "
Jul 06 23:25:14.478679 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:14.478472    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/358fd68e-d95f-49eb-af9e-2fbb084a1027-model-cache\") pod \"358fd68e-d95f-49eb-af9e-2fbb084a1027\" (UID: \"358fd68e-d95f-49eb-af9e-2fbb084a1027\") "
Jul 06 23:25:14.478679 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:14.478518    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/358fd68e-d95f-49eb-af9e-2fbb084a1027-tls-certs\") pod \"358fd68e-d95f-49eb-af9e-2fbb084a1027\" (UID: \"358fd68e-d95f-49eb-af9e-2fbb084a1027\") "
Jul 06 23:25:14.478679 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:14.478550    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/358fd68e-d95f-49eb-af9e-2fbb084a1027-dshm\") pod \"358fd68e-d95f-49eb-af9e-2fbb084a1027\" (UID: \"358fd68e-d95f-49eb-af9e-2fbb084a1027\") "
Jul 06 23:25:14.478679 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:14.478568    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gmnf\" (UniqueName: \"kubernetes.io/projected/358fd68e-d95f-49eb-af9e-2fbb084a1027-kube-api-access-9gmnf\") pod \"358fd68e-d95f-49eb-af9e-2fbb084a1027\" (UID: \"358fd68e-d95f-49eb-af9e-2fbb084a1027\") "
Jul 06 23:25:14.478679 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:14.478601    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/358fd68e-d95f-49eb-af9e-2fbb084a1027-tmp-dir\") pod \"358fd68e-d95f-49eb-af9e-2fbb084a1027\" (UID: \"358fd68e-d95f-49eb-af9e-2fbb084a1027\") "
Jul 06 23:25:14.478956 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:14.478692    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/358fd68e-d95f-49eb-af9e-2fbb084a1027-home" (OuterVolumeSpecName: "home") pod "358fd68e-d95f-49eb-af9e-2fbb084a1027" (UID: "358fd68e-d95f-49eb-af9e-2fbb084a1027"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:25:14.478956 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:14.478870    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/358fd68e-d95f-49eb-af9e-2fbb084a1027-model-cache" (OuterVolumeSpecName: "model-cache") pod "358fd68e-d95f-49eb-af9e-2fbb084a1027" (UID: "358fd68e-d95f-49eb-af9e-2fbb084a1027"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:25:14.478956 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:14.478884    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/358fd68e-d95f-49eb-af9e-2fbb084a1027-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "358fd68e-d95f-49eb-af9e-2fbb084a1027" (UID: "358fd68e-d95f-49eb-af9e-2fbb084a1027"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:25:14.478956 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:14.478951    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/358fd68e-d95f-49eb-af9e-2fbb084a1027-home\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:25:14.480752 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:14.480732    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/358fd68e-d95f-49eb-af9e-2fbb084a1027-dshm" (OuterVolumeSpecName: "dshm") pod "358fd68e-d95f-49eb-af9e-2fbb084a1027" (UID: "358fd68e-d95f-49eb-af9e-2fbb084a1027"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:25:14.481191 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:14.481161    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/358fd68e-d95f-49eb-af9e-2fbb084a1027-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "358fd68e-d95f-49eb-af9e-2fbb084a1027" (UID: "358fd68e-d95f-49eb-af9e-2fbb084a1027"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 23:25:14.481250 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:14.481162    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/358fd68e-d95f-49eb-af9e-2fbb084a1027-kube-api-access-9gmnf" (OuterVolumeSpecName: "kube-api-access-9gmnf") pod "358fd68e-d95f-49eb-af9e-2fbb084a1027" (UID: "358fd68e-d95f-49eb-af9e-2fbb084a1027"). InnerVolumeSpecName "kube-api-access-9gmnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 23:25:14.579349 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:14.579322    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/358fd68e-d95f-49eb-af9e-2fbb084a1027-model-cache\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:25:14.579349 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:14.579348    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/358fd68e-d95f-49eb-af9e-2fbb084a1027-tls-certs\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:25:14.579501 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:14.579360    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/358fd68e-d95f-49eb-af9e-2fbb084a1027-dshm\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:25:14.579501 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:14.579372    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gmnf\" (UniqueName: \"kubernetes.io/projected/358fd68e-d95f-49eb-af9e-2fbb084a1027-kube-api-access-9gmnf\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:25:14.579501 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:14.579386    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/358fd68e-d95f-49eb-af9e-2fbb084a1027-tmp-dir\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:25:15.037242 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:15.037153    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-d3a5b8/auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq" podUID="902d0bec-c6fe-4a66-8fa6-513172f37b82" containerName="main" containerID="cri-o://a63f5647545fd8cdc9515e2bcc86061d2b49af2b0088e096756a227a6be1cf1b" gracePeriod=45
Jul 06 23:25:15.198537 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:15.198505    2569 generic.go:356] "Generic (PLEG): container finished" podID="902d0bec-c6fe-4a66-8fa6-513172f37b82" containerID="a63f5647545fd8cdc9515e2bcc86061d2b49af2b0088e096756a227a6be1cf1b" exitCode=0
Jul 06 23:25:15.198902 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:15.198578    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d3a5b8/auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq" event={"ID":"902d0bec-c6fe-4a66-8fa6-513172f37b82","Type":"ContainerDied","Data":"a63f5647545fd8cdc9515e2bcc86061d2b49af2b0088e096756a227a6be1cf1b"}
Jul 06 23:25:15.199893 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:15.199869    2569 generic.go:356] "Generic (PLEG): container finished" podID="358fd68e-d95f-49eb-af9e-2fbb084a1027" containerID="2475c98e6a2aa6605f5987c1dbd2cdfbbfe021146355bd4c57b083f4c0786640" exitCode=0
Jul 06 23:25:15.199997 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:15.199934    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-669fee/auth-test-model-669fee-kserve-796fc59cdc-kq9g7"
Jul 06 23:25:15.199997 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:15.199951    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-669fee/auth-test-model-669fee-kserve-796fc59cdc-kq9g7" event={"ID":"358fd68e-d95f-49eb-af9e-2fbb084a1027","Type":"ContainerDied","Data":"2475c98e6a2aa6605f5987c1dbd2cdfbbfe021146355bd4c57b083f4c0786640"}
Jul 06 23:25:15.199997 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:15.199990    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-669fee/auth-test-model-669fee-kserve-796fc59cdc-kq9g7" event={"ID":"358fd68e-d95f-49eb-af9e-2fbb084a1027","Type":"ContainerDied","Data":"b04e0be09f15e1d5617e2a46d72ce45d9d275c542d516f6b5814adb8a9ddfe6b"}
Jul 06 23:25:15.200117 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:15.200008    2569 scope.go:117] "RemoveContainer" containerID="2475c98e6a2aa6605f5987c1dbd2cdfbbfe021146355bd4c57b083f4c0786640"
Jul 06 23:25:15.210174 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:15.210150    2569 scope.go:117] "RemoveContainer" containerID="2475c98e6a2aa6605f5987c1dbd2cdfbbfe021146355bd4c57b083f4c0786640"
Jul 06 23:25:15.210441 ip-10-0-134-116 kubenswrapper[2569]: E0706 23:25:15.210419    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2475c98e6a2aa6605f5987c1dbd2cdfbbfe021146355bd4c57b083f4c0786640\": container with ID starting with 2475c98e6a2aa6605f5987c1dbd2cdfbbfe021146355bd4c57b083f4c0786640 not found: ID does not exist" containerID="2475c98e6a2aa6605f5987c1dbd2cdfbbfe021146355bd4c57b083f4c0786640"
Jul 06 23:25:15.210514 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:15.210448    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2475c98e6a2aa6605f5987c1dbd2cdfbbfe021146355bd4c57b083f4c0786640"} err="failed to get container status \"2475c98e6a2aa6605f5987c1dbd2cdfbbfe021146355bd4c57b083f4c0786640\": rpc error: code = NotFound desc = could not find container \"2475c98e6a2aa6605f5987c1dbd2cdfbbfe021146355bd4c57b083f4c0786640\": container with ID starting with 2475c98e6a2aa6605f5987c1dbd2cdfbbfe021146355bd4c57b083f4c0786640 not found: ID does not exist"
Jul 06 23:25:15.225629 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:15.225601    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-669fee/auth-test-model-669fee-kserve-796fc59cdc-kq9g7"]
Jul 06 23:25:15.229553 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:15.229532    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-669fee/auth-test-model-669fee-kserve-796fc59cdc-kq9g7"]
Jul 06 23:25:15.290120 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:15.290067    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d3a5b8/auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq"
Jul 06 23:25:15.386751 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:15.386727    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/902d0bec-c6fe-4a66-8fa6-513172f37b82-tmp-dir\") pod \"902d0bec-c6fe-4a66-8fa6-513172f37b82\" (UID: \"902d0bec-c6fe-4a66-8fa6-513172f37b82\") "
Jul 06 23:25:15.386860 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:15.386759    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpsjg\" (UniqueName: \"kubernetes.io/projected/902d0bec-c6fe-4a66-8fa6-513172f37b82-kube-api-access-jpsjg\") pod \"902d0bec-c6fe-4a66-8fa6-513172f37b82\" (UID: \"902d0bec-c6fe-4a66-8fa6-513172f37b82\") "
Jul 06 23:25:15.386860 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:15.386776    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/902d0bec-c6fe-4a66-8fa6-513172f37b82-tls-certs\") pod \"902d0bec-c6fe-4a66-8fa6-513172f37b82\" (UID: \"902d0bec-c6fe-4a66-8fa6-513172f37b82\") "
Jul 06 23:25:15.386940 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:15.386905    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/902d0bec-c6fe-4a66-8fa6-513172f37b82-model-cache\") pod \"902d0bec-c6fe-4a66-8fa6-513172f37b82\" (UID: \"902d0bec-c6fe-4a66-8fa6-513172f37b82\") "
Jul 06 23:25:15.386940 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:15.386932    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/902d0bec-c6fe-4a66-8fa6-513172f37b82-home\") pod \"902d0bec-c6fe-4a66-8fa6-513172f37b82\" (UID: \"902d0bec-c6fe-4a66-8fa6-513172f37b82\") "
Jul 06 23:25:15.387028 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:15.386975    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/902d0bec-c6fe-4a66-8fa6-513172f37b82-dshm\") pod \"902d0bec-c6fe-4a66-8fa6-513172f37b82\" (UID: \"902d0bec-c6fe-4a66-8fa6-513172f37b82\") "
Jul 06 23:25:15.387028 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:15.387012    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902d0bec-c6fe-4a66-8fa6-513172f37b82-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "902d0bec-c6fe-4a66-8fa6-513172f37b82" (UID: "902d0bec-c6fe-4a66-8fa6-513172f37b82"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:25:15.387243 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:15.387200    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902d0bec-c6fe-4a66-8fa6-513172f37b82-model-cache" (OuterVolumeSpecName: "model-cache") pod "902d0bec-c6fe-4a66-8fa6-513172f37b82" (UID: "902d0bec-c6fe-4a66-8fa6-513172f37b82"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:25:15.387243 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:15.387220    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902d0bec-c6fe-4a66-8fa6-513172f37b82-home" (OuterVolumeSpecName: "home") pod "902d0bec-c6fe-4a66-8fa6-513172f37b82" (UID: "902d0bec-c6fe-4a66-8fa6-513172f37b82"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:25:15.387394 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:15.387338    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/902d0bec-c6fe-4a66-8fa6-513172f37b82-tmp-dir\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:25:15.387394 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:15.387355    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/902d0bec-c6fe-4a66-8fa6-513172f37b82-model-cache\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:25:15.387394 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:15.387363    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/902d0bec-c6fe-4a66-8fa6-513172f37b82-home\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:25:15.389007 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:15.388983    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/902d0bec-c6fe-4a66-8fa6-513172f37b82-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "902d0bec-c6fe-4a66-8fa6-513172f37b82" (UID: "902d0bec-c6fe-4a66-8fa6-513172f37b82"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 23:25:15.389093 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:15.389025    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902d0bec-c6fe-4a66-8fa6-513172f37b82-dshm" (OuterVolumeSpecName: "dshm") pod "902d0bec-c6fe-4a66-8fa6-513172f37b82" (UID: "902d0bec-c6fe-4a66-8fa6-513172f37b82"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:25:15.389093 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:15.389042    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/902d0bec-c6fe-4a66-8fa6-513172f37b82-kube-api-access-jpsjg" (OuterVolumeSpecName: "kube-api-access-jpsjg") pod "902d0bec-c6fe-4a66-8fa6-513172f37b82" (UID: "902d0bec-c6fe-4a66-8fa6-513172f37b82"). InnerVolumeSpecName "kube-api-access-jpsjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 23:25:15.487873 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:15.487849    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/902d0bec-c6fe-4a66-8fa6-513172f37b82-dshm\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:25:15.487873 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:15.487871    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpsjg\" (UniqueName: \"kubernetes.io/projected/902d0bec-c6fe-4a66-8fa6-513172f37b82-kube-api-access-jpsjg\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:25:15.488011 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:15.487880    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/902d0bec-c6fe-4a66-8fa6-513172f37b82-tls-certs\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:25:16.170807 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:16.170776    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="358fd68e-d95f-49eb-af9e-2fbb084a1027" path="/var/lib/kubelet/pods/358fd68e-d95f-49eb-af9e-2fbb084a1027/volumes"
Jul 06 23:25:16.206519 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:16.206497    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d3a5b8/auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq"
Jul 06 23:25:16.206891 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:16.206532    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d3a5b8/auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq" event={"ID":"902d0bec-c6fe-4a66-8fa6-513172f37b82","Type":"ContainerDied","Data":"d6727796c0bac8dc3abe797c409440d791f133c6582c9c0802ec64bd72d1cb93"}
Jul 06 23:25:16.206891 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:16.206575    2569 scope.go:117] "RemoveContainer" containerID="a63f5647545fd8cdc9515e2bcc86061d2b49af2b0088e096756a227a6be1cf1b"
Jul 06 23:25:16.221932 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:16.221910    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d3a5b8/auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq"]
Jul 06 23:25:16.227360 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:16.227341    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-d3a5b8/auth-test-model-d3a5b8-kserve-549746f6cf-vhxbq"]
Jul 06 23:25:18.170493 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:18.170461    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="902d0bec-c6fe-4a66-8fa6-513172f37b82" path="/var/lib/kubelet/pods/902d0bec-c6fe-4a66-8fa6-513172f37b82/volumes"
Jul 06 23:25:36.431424 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:36.431388    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-669fee/sub-test-model-669fee-kserve-6c46696bcc-w9rsz"]
Jul 06 23:25:36.432011 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:36.431948    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="358fd68e-d95f-49eb-af9e-2fbb084a1027" containerName="main"
Jul 06 23:25:36.432011 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:36.431968    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="358fd68e-d95f-49eb-af9e-2fbb084a1027" containerName="main"
Jul 06 23:25:36.432011 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:36.431994    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="902d0bec-c6fe-4a66-8fa6-513172f37b82" containerName="main"
Jul 06 23:25:36.432011 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:36.432003    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="902d0bec-c6fe-4a66-8fa6-513172f37b82" containerName="main"
Jul 06 23:25:36.432304 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:36.432087    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="902d0bec-c6fe-4a66-8fa6-513172f37b82" containerName="main"
Jul 06 23:25:36.432304 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:36.432104    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="358fd68e-d95f-49eb-af9e-2fbb084a1027" containerName="main"
Jul 06 23:25:36.434267 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:36.434247    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-669fee/sub-test-model-669fee-kserve-6c46696bcc-w9rsz"
Jul 06 23:25:36.438763 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:36.438716    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-669fee\"/\"openshift-service-ca.crt\""
Jul 06 23:25:36.438900 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:36.438797    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-669fee\"/\"kube-root-ca.crt\""
Jul 06 23:25:36.438900 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:36.438872    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-669fee\"/\"sub-test-model-669fee-kserve-self-signed-certs\""
Jul 06 23:25:36.439097 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:36.438720    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-669fee\"/\"default-dockercfg-6ncrl\""
Jul 06 23:25:36.443683 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:36.443656    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-669fee/sub-test-model-669fee-kserve-6c46696bcc-w9rsz"]
Jul 06 23:25:36.443939 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:36.443919    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecb44e3c-3911-4199-b713-dc85e3386ad4-home\") pod \"sub-test-model-669fee-kserve-6c46696bcc-w9rsz\" (UID: \"ecb44e3c-3911-4199-b713-dc85e3386ad4\") " pod="ai-tenant-e2e-shared-a-669fee/sub-test-model-669fee-kserve-6c46696bcc-w9rsz"
Jul 06 23:25:36.444030 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:36.443950    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecb44e3c-3911-4199-b713-dc85e3386ad4-model-cache\") pod \"sub-test-model-669fee-kserve-6c46696bcc-w9rsz\" (UID: \"ecb44e3c-3911-4199-b713-dc85e3386ad4\") " pod="ai-tenant-e2e-shared-a-669fee/sub-test-model-669fee-kserve-6c46696bcc-w9rsz"
Jul 06 23:25:36.444030 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:36.443995    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecb44e3c-3911-4199-b713-dc85e3386ad4-tmp-dir\") pod \"sub-test-model-669fee-kserve-6c46696bcc-w9rsz\" (UID: \"ecb44e3c-3911-4199-b713-dc85e3386ad4\") " pod="ai-tenant-e2e-shared-a-669fee/sub-test-model-669fee-kserve-6c46696bcc-w9rsz"
Jul 06 23:25:36.444030 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:36.444010    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecb44e3c-3911-4199-b713-dc85e3386ad4-tls-certs\") pod \"sub-test-model-669fee-kserve-6c46696bcc-w9rsz\" (UID: \"ecb44e3c-3911-4199-b713-dc85e3386ad4\") " pod="ai-tenant-e2e-shared-a-669fee/sub-test-model-669fee-kserve-6c46696bcc-w9rsz"
Jul 06 23:25:36.444167 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:36.444031    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecb44e3c-3911-4199-b713-dc85e3386ad4-dshm\") pod \"sub-test-model-669fee-kserve-6c46696bcc-w9rsz\" (UID: \"ecb44e3c-3911-4199-b713-dc85e3386ad4\") " pod="ai-tenant-e2e-shared-a-669fee/sub-test-model-669fee-kserve-6c46696bcc-w9rsz"
Jul 06 23:25:36.444167 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:36.444113    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lmjt\" (UniqueName: \"kubernetes.io/projected/ecb44e3c-3911-4199-b713-dc85e3386ad4-kube-api-access-8lmjt\") pod \"sub-test-model-669fee-kserve-6c46696bcc-w9rsz\" (UID: \"ecb44e3c-3911-4199-b713-dc85e3386ad4\") " pod="ai-tenant-e2e-shared-a-669fee/sub-test-model-669fee-kserve-6c46696bcc-w9rsz"
Jul 06 23:25:36.545240 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:36.545208    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lmjt\" (UniqueName: \"kubernetes.io/projected/ecb44e3c-3911-4199-b713-dc85e3386ad4-kube-api-access-8lmjt\") pod \"sub-test-model-669fee-kserve-6c46696bcc-w9rsz\" (UID: \"ecb44e3c-3911-4199-b713-dc85e3386ad4\") " pod="ai-tenant-e2e-shared-a-669fee/sub-test-model-669fee-kserve-6c46696bcc-w9rsz"
Jul 06 23:25:36.545402 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:36.545263    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecb44e3c-3911-4199-b713-dc85e3386ad4-home\") pod \"sub-test-model-669fee-kserve-6c46696bcc-w9rsz\" (UID: \"ecb44e3c-3911-4199-b713-dc85e3386ad4\") " pod="ai-tenant-e2e-shared-a-669fee/sub-test-model-669fee-kserve-6c46696bcc-w9rsz"
Jul 06 23:25:36.545402 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:36.545294    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecb44e3c-3911-4199-b713-dc85e3386ad4-model-cache\") pod \"sub-test-model-669fee-kserve-6c46696bcc-w9rsz\" (UID: \"ecb44e3c-3911-4199-b713-dc85e3386ad4\") " pod="ai-tenant-e2e-shared-a-669fee/sub-test-model-669fee-kserve-6c46696bcc-w9rsz"
Jul 06 23:25:36.545402 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:36.545351    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecb44e3c-3911-4199-b713-dc85e3386ad4-tmp-dir\") pod \"sub-test-model-669fee-kserve-6c46696bcc-w9rsz\" (UID: \"ecb44e3c-3911-4199-b713-dc85e3386ad4\") " pod="ai-tenant-e2e-shared-a-669fee/sub-test-model-669fee-kserve-6c46696bcc-w9rsz"
Jul 06 23:25:36.545402 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:36.545373    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecb44e3c-3911-4199-b713-dc85e3386ad4-tls-certs\") pod \"sub-test-model-669fee-kserve-6c46696bcc-w9rsz\" (UID: \"ecb44e3c-3911-4199-b713-dc85e3386ad4\") " pod="ai-tenant-e2e-shared-a-669fee/sub-test-model-669fee-kserve-6c46696bcc-w9rsz"
Jul 06 23:25:36.545614 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:36.545408    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecb44e3c-3911-4199-b713-dc85e3386ad4-dshm\") pod \"sub-test-model-669fee-kserve-6c46696bcc-w9rsz\" (UID: \"ecb44e3c-3911-4199-b713-dc85e3386ad4\") " pod="ai-tenant-e2e-shared-a-669fee/sub-test-model-669fee-kserve-6c46696bcc-w9rsz"
Jul 06 23:25:36.545753 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:36.545727    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecb44e3c-3911-4199-b713-dc85e3386ad4-home\") pod \"sub-test-model-669fee-kserve-6c46696bcc-w9rsz\" (UID: \"ecb44e3c-3911-4199-b713-dc85e3386ad4\") " pod="ai-tenant-e2e-shared-a-669fee/sub-test-model-669fee-kserve-6c46696bcc-w9rsz"
Jul 06 23:25:36.545753 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:36.545745    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecb44e3c-3911-4199-b713-dc85e3386ad4-tmp-dir\") pod \"sub-test-model-669fee-kserve-6c46696bcc-w9rsz\" (UID: \"ecb44e3c-3911-4199-b713-dc85e3386ad4\") " pod="ai-tenant-e2e-shared-a-669fee/sub-test-model-669fee-kserve-6c46696bcc-w9rsz"
Jul 06 23:25:36.545858 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:36.545776    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecb44e3c-3911-4199-b713-dc85e3386ad4-model-cache\") pod \"sub-test-model-669fee-kserve-6c46696bcc-w9rsz\" (UID: \"ecb44e3c-3911-4199-b713-dc85e3386ad4\") " pod="ai-tenant-e2e-shared-a-669fee/sub-test-model-669fee-kserve-6c46696bcc-w9rsz"
Jul 06 23:25:36.547670 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:36.547645    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecb44e3c-3911-4199-b713-dc85e3386ad4-dshm\") pod \"sub-test-model-669fee-kserve-6c46696bcc-w9rsz\" (UID: \"ecb44e3c-3911-4199-b713-dc85e3386ad4\") " pod="ai-tenant-e2e-shared-a-669fee/sub-test-model-669fee-kserve-6c46696bcc-w9rsz"
Jul 06 23:25:36.547866 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:36.547845    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecb44e3c-3911-4199-b713-dc85e3386ad4-tls-certs\") pod \"sub-test-model-669fee-kserve-6c46696bcc-w9rsz\" (UID: \"ecb44e3c-3911-4199-b713-dc85e3386ad4\") " pod="ai-tenant-e2e-shared-a-669fee/sub-test-model-669fee-kserve-6c46696bcc-w9rsz"
Jul 06 23:25:36.552835 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:36.552814    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lmjt\" (UniqueName: \"kubernetes.io/projected/ecb44e3c-3911-4199-b713-dc85e3386ad4-kube-api-access-8lmjt\") pod \"sub-test-model-669fee-kserve-6c46696bcc-w9rsz\" (UID: \"ecb44e3c-3911-4199-b713-dc85e3386ad4\") " pod="ai-tenant-e2e-shared-a-669fee/sub-test-model-669fee-kserve-6c46696bcc-w9rsz"
Jul 06 23:25:36.745165 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:36.745084    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-669fee/sub-test-model-669fee-kserve-6c46696bcc-w9rsz"
Jul 06 23:25:36.872730 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:36.872703    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-669fee/sub-test-model-669fee-kserve-6c46696bcc-w9rsz"]
Jul 06 23:25:36.874980 ip-10-0-134-116 kubenswrapper[2569]: W0706 23:25:36.874954    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podecb44e3c_3911_4199_b713_dc85e3386ad4.slice/crio-952a6742f36fa538337723d0d4b529b83322f7f3d36a21fce08fc96162fafd9d WatchSource:0}: Error finding container 952a6742f36fa538337723d0d4b529b83322f7f3d36a21fce08fc96162fafd9d: Status 404 returned error can't find the container with id 952a6742f36fa538337723d0d4b529b83322f7f3d36a21fce08fc96162fafd9d
Jul 06 23:25:37.313642 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:37.313608    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-669fee/sub-test-model-669fee-kserve-6c46696bcc-w9rsz" event={"ID":"ecb44e3c-3911-4199-b713-dc85e3386ad4","Type":"ContainerStarted","Data":"f005800fd6a223ba97302fc1621e942b4342bfa2f9601e07d594e98b52e2b2ed"}
Jul 06 23:25:37.313642 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:37.313646    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-669fee/sub-test-model-669fee-kserve-6c46696bcc-w9rsz" event={"ID":"ecb44e3c-3911-4199-b713-dc85e3386ad4","Type":"ContainerStarted","Data":"952a6742f36fa538337723d0d4b529b83322f7f3d36a21fce08fc96162fafd9d"}
Jul 06 23:25:37.331932 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:37.331888    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-669fee/sub-test-model-669fee-kserve-6c46696bcc-w9rsz" podStartSLOduration=1.3318729249999999 podStartE2EDuration="1.331872925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:25:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:25:37.329998126 +0000 UTC m=+2349.734248031" watchObservedRunningTime="2026-07-06 23:25:37.331872925 +0000 UTC m=+2349.736122829"
Jul 06 23:25:46.745216 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:46.745169    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-669fee/sub-test-model-669fee-kserve-6c46696bcc-w9rsz"
Jul 06 23:25:46.745586 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:46.745251    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-669fee/sub-test-model-669fee-kserve-6c46696bcc-w9rsz"
Jul 06 23:25:46.757408 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:46.757389    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-669fee/sub-test-model-669fee-kserve-6c46696bcc-w9rsz"
Jul 06 23:25:47.372029 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:25:47.372003    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-669fee/sub-test-model-669fee-kserve-6c46696bcc-w9rsz"
Jul 06 23:26:08.938212 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:26:08.938163    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-d3a5b8/sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n"]
Jul 06 23:26:08.940720 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:26:08.940700    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d3a5b8/sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n"
Jul 06 23:26:08.943345 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:26:08.943319    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d3a5b8\"/\"openshift-service-ca.crt\""
Jul 06 23:26:08.943454 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:26:08.943376    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d3a5b8\"/\"kube-root-ca.crt\""
Jul 06 23:26:08.944710 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:26:08.944691    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d3a5b8\"/\"sub-test-model-d3a5b8-kserve-self-signed-certs\""
Jul 06 23:26:08.944807 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:26:08.944695    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d3a5b8\"/\"default-dockercfg-kqhdj\""
Jul 06 23:26:08.950732 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:26:08.950711    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d3a5b8/sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n"]
Jul 06 23:26:09.110995 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:26:09.110967    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27f97c34-55d1-4b97-a5d9-cfd4a465c9af-model-cache\") pod \"sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n\" (UID: \"27f97c34-55d1-4b97-a5d9-cfd4a465c9af\") " pod="ai-tenant-e2e-shared-b-d3a5b8/sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n"
Jul 06 23:26:09.111118 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:26:09.111012    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27f97c34-55d1-4b97-a5d9-cfd4a465c9af-tmp-dir\") pod \"sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n\" (UID: \"27f97c34-55d1-4b97-a5d9-cfd4a465c9af\") " pod="ai-tenant-e2e-shared-b-d3a5b8/sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n"
Jul 06 23:26:09.111118 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:26:09.111032    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btwpd\" (UniqueName: \"kubernetes.io/projected/27f97c34-55d1-4b97-a5d9-cfd4a465c9af-kube-api-access-btwpd\") pod \"sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n\" (UID: \"27f97c34-55d1-4b97-a5d9-cfd4a465c9af\") " pod="ai-tenant-e2e-shared-b-d3a5b8/sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n"
Jul 06 23:26:09.111118 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:26:09.111050    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27f97c34-55d1-4b97-a5d9-cfd4a465c9af-dshm\") pod \"sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n\" (UID: \"27f97c34-55d1-4b97-a5d9-cfd4a465c9af\") " pod="ai-tenant-e2e-shared-b-d3a5b8/sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n"
Jul 06 23:26:09.111315 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:26:09.111139    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27f97c34-55d1-4b97-a5d9-cfd4a465c9af-home\") pod \"sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n\" (UID: \"27f97c34-55d1-4b97-a5d9-cfd4a465c9af\") " pod="ai-tenant-e2e-shared-b-d3a5b8/sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n"
Jul 06 23:26:09.111315 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:26:09.111228    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27f97c34-55d1-4b97-a5d9-cfd4a465c9af-tls-certs\") pod \"sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n\" (UID: \"27f97c34-55d1-4b97-a5d9-cfd4a465c9af\") " pod="ai-tenant-e2e-shared-b-d3a5b8/sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n"
Jul 06 23:26:09.212056 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:26:09.211966    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27f97c34-55d1-4b97-a5d9-cfd4a465c9af-model-cache\") pod \"sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n\" (UID: \"27f97c34-55d1-4b97-a5d9-cfd4a465c9af\") " pod="ai-tenant-e2e-shared-b-d3a5b8/sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n"
Jul 06 23:26:09.212056 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:26:09.212011    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27f97c34-55d1-4b97-a5d9-cfd4a465c9af-tmp-dir\") pod \"sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n\" (UID: \"27f97c34-55d1-4b97-a5d9-cfd4a465c9af\") " pod="ai-tenant-e2e-shared-b-d3a5b8/sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n"
Jul 06 23:26:09.212056 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:26:09.212028    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btwpd\" (UniqueName: \"kubernetes.io/projected/27f97c34-55d1-4b97-a5d9-cfd4a465c9af-kube-api-access-btwpd\") pod \"sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n\" (UID: \"27f97c34-55d1-4b97-a5d9-cfd4a465c9af\") " pod="ai-tenant-e2e-shared-b-d3a5b8/sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n"
Jul 06 23:26:09.212056 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:26:09.212045    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27f97c34-55d1-4b97-a5d9-cfd4a465c9af-dshm\") pod \"sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n\" (UID: \"27f97c34-55d1-4b97-a5d9-cfd4a465c9af\") " pod="ai-tenant-e2e-shared-b-d3a5b8/sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n"
Jul 06 23:26:09.212411 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:26:09.212123    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27f97c34-55d1-4b97-a5d9-cfd4a465c9af-home\") pod \"sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n\" (UID: \"27f97c34-55d1-4b97-a5d9-cfd4a465c9af\") " pod="ai-tenant-e2e-shared-b-d3a5b8/sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n"
Jul 06 23:26:09.212411 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:26:09.212219    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27f97c34-55d1-4b97-a5d9-cfd4a465c9af-tls-certs\") pod \"sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n\" (UID: \"27f97c34-55d1-4b97-a5d9-cfd4a465c9af\") " pod="ai-tenant-e2e-shared-b-d3a5b8/sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n"
Jul 06 23:26:09.212411 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:26:09.212378    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27f97c34-55d1-4b97-a5d9-cfd4a465c9af-tmp-dir\") pod \"sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n\" (UID: \"27f97c34-55d1-4b97-a5d9-cfd4a465c9af\") " pod="ai-tenant-e2e-shared-b-d3a5b8/sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n"
Jul 06 23:26:09.212556 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:26:09.212448    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27f97c34-55d1-4b97-a5d9-cfd4a465c9af-model-cache\") pod \"sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n\" (UID: \"27f97c34-55d1-4b97-a5d9-cfd4a465c9af\") " pod="ai-tenant-e2e-shared-b-d3a5b8/sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n"
Jul 06 23:26:09.212671 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:26:09.212651    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27f97c34-55d1-4b97-a5d9-cfd4a465c9af-home\") pod \"sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n\" (UID: \"27f97c34-55d1-4b97-a5d9-cfd4a465c9af\") " pod="ai-tenant-e2e-shared-b-d3a5b8/sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n"
Jul 06 23:26:09.214307 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:26:09.214281    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27f97c34-55d1-4b97-a5d9-cfd4a465c9af-dshm\") pod \"sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n\" (UID: \"27f97c34-55d1-4b97-a5d9-cfd4a465c9af\") " pod="ai-tenant-e2e-shared-b-d3a5b8/sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n"
Jul 06 23:26:09.214781 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:26:09.214761    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27f97c34-55d1-4b97-a5d9-cfd4a465c9af-tls-certs\") pod \"sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n\" (UID: \"27f97c34-55d1-4b97-a5d9-cfd4a465c9af\") " pod="ai-tenant-e2e-shared-b-d3a5b8/sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n"
Jul 06 23:26:09.220431 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:26:09.220408    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btwpd\" (UniqueName: \"kubernetes.io/projected/27f97c34-55d1-4b97-a5d9-cfd4a465c9af-kube-api-access-btwpd\") pod \"sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n\" (UID: \"27f97c34-55d1-4b97-a5d9-cfd4a465c9af\") " pod="ai-tenant-e2e-shared-b-d3a5b8/sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n"
Jul 06 23:26:09.252409 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:26:09.252383    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d3a5b8/sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n"
Jul 06 23:26:09.385514 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:26:09.385485    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d3a5b8/sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n"]
Jul 06 23:26:09.387265 ip-10-0-134-116 kubenswrapper[2569]: W0706 23:26:09.387235    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27f97c34_55d1_4b97_a5d9_cfd4a465c9af.slice/crio-5de4b4b031ed019c23c02e10efbdd69ed6f517501f30299b8c77641c61cfd1b3 WatchSource:0}: Error finding container 5de4b4b031ed019c23c02e10efbdd69ed6f517501f30299b8c77641c61cfd1b3: Status 404 returned error can't find the container with id 5de4b4b031ed019c23c02e10efbdd69ed6f517501f30299b8c77641c61cfd1b3
Jul 06 23:26:09.472302 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:26:09.472218    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d3a5b8/sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n" event={"ID":"27f97c34-55d1-4b97-a5d9-cfd4a465c9af","Type":"ContainerStarted","Data":"ac105b2b1b5c8c70432743f702f514901e04475a185591ccd89590259a34c66a"}
Jul 06 23:26:09.472302 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:26:09.472275    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d3a5b8/sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n" event={"ID":"27f97c34-55d1-4b97-a5d9-cfd4a465c9af","Type":"ContainerStarted","Data":"5de4b4b031ed019c23c02e10efbdd69ed6f517501f30299b8c77641c61cfd1b3"}
Jul 06 23:26:09.488625 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:26:09.488565    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-d3a5b8/sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n" podStartSLOduration=1.488549041 podStartE2EDuration="1.488549041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:26:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:26:09.488209822 +0000 UTC m=+2381.892459717" watchObservedRunningTime="2026-07-06 23:26:09.488549041 +0000 UTC m=+2381.892798947"
Jul 06 23:26:19.253109 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:26:19.253059    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-d3a5b8/sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n"
Jul 06 23:26:19.253636 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:26:19.253125    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-d3a5b8/sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n"
Jul 06 23:26:19.265934 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:26:19.265913    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-d3a5b8/sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n"
Jul 06 23:26:19.531710 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:26:19.531641    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-d3a5b8/sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n"
Jul 06 23:26:28.204928 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:26:28.204895    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bvsb_571f407a-249a-4fd9-8c15-53a6541db635/ovn-acl-logging/0.log"
Jul 06 23:26:28.212690 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:26:28.212671    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bvsb_571f407a-249a-4fd9-8c15-53a6541db635/ovn-acl-logging/0.log"
Jul 06 23:26:53.653760 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:26:53.653670    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-669fee/sub-test-model-669fee-kserve-6c46696bcc-w9rsz"]
Jul 06 23:26:54.439561 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:26:54.439530    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d3a5b8/sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n"]
Jul 06 23:27:08.673933 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:08.673894    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-669fee/sub-test-model-669fee-kserve-6c46696bcc-w9rsz" podUID="ecb44e3c-3911-4199-b713-dc85e3386ad4" containerName="main" containerID="cri-o://f005800fd6a223ba97302fc1621e942b4342bfa2f9601e07d594e98b52e2b2ed" gracePeriod=45
Jul 06 23:27:08.928509 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:08.928449    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-669fee/sub-test-model-669fee-kserve-6c46696bcc-w9rsz"
Jul 06 23:27:09.105543 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.105515    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecb44e3c-3911-4199-b713-dc85e3386ad4-model-cache\") pod \"ecb44e3c-3911-4199-b713-dc85e3386ad4\" (UID: \"ecb44e3c-3911-4199-b713-dc85e3386ad4\") "
Jul 06 23:27:09.105668 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.105553    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lmjt\" (UniqueName: \"kubernetes.io/projected/ecb44e3c-3911-4199-b713-dc85e3386ad4-kube-api-access-8lmjt\") pod \"ecb44e3c-3911-4199-b713-dc85e3386ad4\" (UID: \"ecb44e3c-3911-4199-b713-dc85e3386ad4\") "
Jul 06 23:27:09.105668 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.105592    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecb44e3c-3911-4199-b713-dc85e3386ad4-dshm\") pod \"ecb44e3c-3911-4199-b713-dc85e3386ad4\" (UID: \"ecb44e3c-3911-4199-b713-dc85e3386ad4\") "
Jul 06 23:27:09.105668 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.105642    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecb44e3c-3911-4199-b713-dc85e3386ad4-home\") pod \"ecb44e3c-3911-4199-b713-dc85e3386ad4\" (UID: \"ecb44e3c-3911-4199-b713-dc85e3386ad4\") "
Jul 06 23:27:09.105668 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.105665    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecb44e3c-3911-4199-b713-dc85e3386ad4-tls-certs\") pod \"ecb44e3c-3911-4199-b713-dc85e3386ad4\" (UID: \"ecb44e3c-3911-4199-b713-dc85e3386ad4\") "
Jul 06 23:27:09.105879 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.105692    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecb44e3c-3911-4199-b713-dc85e3386ad4-tmp-dir\") pod \"ecb44e3c-3911-4199-b713-dc85e3386ad4\" (UID: \"ecb44e3c-3911-4199-b713-dc85e3386ad4\") "
Jul 06 23:27:09.105879 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.105805    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb44e3c-3911-4199-b713-dc85e3386ad4-model-cache" (OuterVolumeSpecName: "model-cache") pod "ecb44e3c-3911-4199-b713-dc85e3386ad4" (UID: "ecb44e3c-3911-4199-b713-dc85e3386ad4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:27:09.105994 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.105894    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb44e3c-3911-4199-b713-dc85e3386ad4-home" (OuterVolumeSpecName: "home") pod "ecb44e3c-3911-4199-b713-dc85e3386ad4" (UID: "ecb44e3c-3911-4199-b713-dc85e3386ad4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:27:09.106061 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.106036    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb44e3c-3911-4199-b713-dc85e3386ad4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ecb44e3c-3911-4199-b713-dc85e3386ad4" (UID: "ecb44e3c-3911-4199-b713-dc85e3386ad4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:27:09.106113 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.106043    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecb44e3c-3911-4199-b713-dc85e3386ad4-home\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:27:09.106113 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.106085    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecb44e3c-3911-4199-b713-dc85e3386ad4-model-cache\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:27:09.107715 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.107693    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecb44e3c-3911-4199-b713-dc85e3386ad4-kube-api-access-8lmjt" (OuterVolumeSpecName: "kube-api-access-8lmjt") pod "ecb44e3c-3911-4199-b713-dc85e3386ad4" (UID: "ecb44e3c-3911-4199-b713-dc85e3386ad4"). InnerVolumeSpecName "kube-api-access-8lmjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 23:27:09.107715 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.107705    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb44e3c-3911-4199-b713-dc85e3386ad4-dshm" (OuterVolumeSpecName: "dshm") pod "ecb44e3c-3911-4199-b713-dc85e3386ad4" (UID: "ecb44e3c-3911-4199-b713-dc85e3386ad4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:27:09.107844 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.107711    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecb44e3c-3911-4199-b713-dc85e3386ad4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ecb44e3c-3911-4199-b713-dc85e3386ad4" (UID: "ecb44e3c-3911-4199-b713-dc85e3386ad4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 23:27:09.207237 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.207200    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lmjt\" (UniqueName: \"kubernetes.io/projected/ecb44e3c-3911-4199-b713-dc85e3386ad4-kube-api-access-8lmjt\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:27:09.207237 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.207237    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecb44e3c-3911-4199-b713-dc85e3386ad4-dshm\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:27:09.207453 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.207253    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecb44e3c-3911-4199-b713-dc85e3386ad4-tls-certs\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:27:09.207453 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.207294    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecb44e3c-3911-4199-b713-dc85e3386ad4-tmp-dir\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:27:09.459233 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.459113    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-d3a5b8/sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n" podUID="27f97c34-55d1-4b97-a5d9-cfd4a465c9af" containerName="main" containerID="cri-o://ac105b2b1b5c8c70432743f702f514901e04475a185591ccd89590259a34c66a" gracePeriod=45
Jul 06 23:27:09.521263 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.521230    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-d3a5b8/sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n" podUID="27f97c34-55d1-4b97-a5d9-cfd4a465c9af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": dial tcp 10.134.0.63:8000: connect: connection refused"
Jul 06 23:27:09.716432 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.716378    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d3a5b8/sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n"
Jul 06 23:27:09.776099 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.776067    2569 generic.go:356] "Generic (PLEG): container finished" podID="27f97c34-55d1-4b97-a5d9-cfd4a465c9af" containerID="ac105b2b1b5c8c70432743f702f514901e04475a185591ccd89590259a34c66a" exitCode=0
Jul 06 23:27:09.776286 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.776122    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d3a5b8/sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n"
Jul 06 23:27:09.776286 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.776149    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d3a5b8/sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n" event={"ID":"27f97c34-55d1-4b97-a5d9-cfd4a465c9af","Type":"ContainerDied","Data":"ac105b2b1b5c8c70432743f702f514901e04475a185591ccd89590259a34c66a"}
Jul 06 23:27:09.776286 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.776230    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d3a5b8/sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n" event={"ID":"27f97c34-55d1-4b97-a5d9-cfd4a465c9af","Type":"ContainerDied","Data":"5de4b4b031ed019c23c02e10efbdd69ed6f517501f30299b8c77641c61cfd1b3"}
Jul 06 23:27:09.776286 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.776254    2569 scope.go:117] "RemoveContainer" containerID="ac105b2b1b5c8c70432743f702f514901e04475a185591ccd89590259a34c66a"
Jul 06 23:27:09.777640 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.777616    2569 generic.go:356] "Generic (PLEG): container finished" podID="ecb44e3c-3911-4199-b713-dc85e3386ad4" containerID="f005800fd6a223ba97302fc1621e942b4342bfa2f9601e07d594e98b52e2b2ed" exitCode=0
Jul 06 23:27:09.777769 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.777656    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-669fee/sub-test-model-669fee-kserve-6c46696bcc-w9rsz"
Jul 06 23:27:09.777769 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.777692    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-669fee/sub-test-model-669fee-kserve-6c46696bcc-w9rsz" event={"ID":"ecb44e3c-3911-4199-b713-dc85e3386ad4","Type":"ContainerDied","Data":"f005800fd6a223ba97302fc1621e942b4342bfa2f9601e07d594e98b52e2b2ed"}
Jul 06 23:27:09.777769 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.777730    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-669fee/sub-test-model-669fee-kserve-6c46696bcc-w9rsz" event={"ID":"ecb44e3c-3911-4199-b713-dc85e3386ad4","Type":"ContainerDied","Data":"952a6742f36fa538337723d0d4b529b83322f7f3d36a21fce08fc96162fafd9d"}
Jul 06 23:27:09.786060 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.786045    2569 scope.go:117] "RemoveContainer" containerID="ac105b2b1b5c8c70432743f702f514901e04475a185591ccd89590259a34c66a"
Jul 06 23:27:09.786321 ip-10-0-134-116 kubenswrapper[2569]: E0706 23:27:09.786297    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac105b2b1b5c8c70432743f702f514901e04475a185591ccd89590259a34c66a\": container with ID starting with ac105b2b1b5c8c70432743f702f514901e04475a185591ccd89590259a34c66a not found: ID does not exist" containerID="ac105b2b1b5c8c70432743f702f514901e04475a185591ccd89590259a34c66a"
Jul 06 23:27:09.786391 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.786333    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac105b2b1b5c8c70432743f702f514901e04475a185591ccd89590259a34c66a"} err="failed to get container status \"ac105b2b1b5c8c70432743f702f514901e04475a185591ccd89590259a34c66a\": rpc error: code = NotFound desc = could not find container \"ac105b2b1b5c8c70432743f702f514901e04475a185591ccd89590259a34c66a\": container with ID starting with ac105b2b1b5c8c70432743f702f514901e04475a185591ccd89590259a34c66a not found: ID does not exist"
Jul 06 23:27:09.786391 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.786350    2569 scope.go:117] "RemoveContainer" containerID="f005800fd6a223ba97302fc1621e942b4342bfa2f9601e07d594e98b52e2b2ed"
Jul 06 23:27:09.794068 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.794051    2569 scope.go:117] "RemoveContainer" containerID="f005800fd6a223ba97302fc1621e942b4342bfa2f9601e07d594e98b52e2b2ed"
Jul 06 23:27:09.794360 ip-10-0-134-116 kubenswrapper[2569]: E0706 23:27:09.794338    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f005800fd6a223ba97302fc1621e942b4342bfa2f9601e07d594e98b52e2b2ed\": container with ID starting with f005800fd6a223ba97302fc1621e942b4342bfa2f9601e07d594e98b52e2b2ed not found: ID does not exist" containerID="f005800fd6a223ba97302fc1621e942b4342bfa2f9601e07d594e98b52e2b2ed"
Jul 06 23:27:09.794432 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.794373    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f005800fd6a223ba97302fc1621e942b4342bfa2f9601e07d594e98b52e2b2ed"} err="failed to get container status \"f005800fd6a223ba97302fc1621e942b4342bfa2f9601e07d594e98b52e2b2ed\": rpc error: code = NotFound desc = could not find container \"f005800fd6a223ba97302fc1621e942b4342bfa2f9601e07d594e98b52e2b2ed\": container with ID starting with f005800fd6a223ba97302fc1621e942b4342bfa2f9601e07d594e98b52e2b2ed not found: ID does not exist"
Jul 06 23:27:09.799499 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.799478    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-669fee/sub-test-model-669fee-kserve-6c46696bcc-w9rsz"]
Jul 06 23:27:09.805013 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.804993    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-669fee/sub-test-model-669fee-kserve-6c46696bcc-w9rsz"]
Jul 06 23:27:09.811446 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.811428    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27f97c34-55d1-4b97-a5d9-cfd4a465c9af-home\") pod \"27f97c34-55d1-4b97-a5d9-cfd4a465c9af\" (UID: \"27f97c34-55d1-4b97-a5d9-cfd4a465c9af\") "
Jul 06 23:27:09.811545 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.811468    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27f97c34-55d1-4b97-a5d9-cfd4a465c9af-tmp-dir\") pod \"27f97c34-55d1-4b97-a5d9-cfd4a465c9af\" (UID: \"27f97c34-55d1-4b97-a5d9-cfd4a465c9af\") "
Jul 06 23:27:09.811545 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.811525    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btwpd\" (UniqueName: \"kubernetes.io/projected/27f97c34-55d1-4b97-a5d9-cfd4a465c9af-kube-api-access-btwpd\") pod \"27f97c34-55d1-4b97-a5d9-cfd4a465c9af\" (UID: \"27f97c34-55d1-4b97-a5d9-cfd4a465c9af\") "
Jul 06 23:27:09.811659 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.811550    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27f97c34-55d1-4b97-a5d9-cfd4a465c9af-tls-certs\") pod \"27f97c34-55d1-4b97-a5d9-cfd4a465c9af\" (UID: \"27f97c34-55d1-4b97-a5d9-cfd4a465c9af\") "
Jul 06 23:27:09.811659 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.811577    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27f97c34-55d1-4b97-a5d9-cfd4a465c9af-dshm\") pod \"27f97c34-55d1-4b97-a5d9-cfd4a465c9af\" (UID: \"27f97c34-55d1-4b97-a5d9-cfd4a465c9af\") "
Jul 06 23:27:09.811659 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.811637    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27f97c34-55d1-4b97-a5d9-cfd4a465c9af-model-cache\") pod \"27f97c34-55d1-4b97-a5d9-cfd4a465c9af\" (UID: \"27f97c34-55d1-4b97-a5d9-cfd4a465c9af\") "
Jul 06 23:27:09.811659 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.811648    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f97c34-55d1-4b97-a5d9-cfd4a465c9af-home" (OuterVolumeSpecName: "home") pod "27f97c34-55d1-4b97-a5d9-cfd4a465c9af" (UID: "27f97c34-55d1-4b97-a5d9-cfd4a465c9af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:27:09.811838 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.811782    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f97c34-55d1-4b97-a5d9-cfd4a465c9af-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "27f97c34-55d1-4b97-a5d9-cfd4a465c9af" (UID: "27f97c34-55d1-4b97-a5d9-cfd4a465c9af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:27:09.811921 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.811901    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f97c34-55d1-4b97-a5d9-cfd4a465c9af-model-cache" (OuterVolumeSpecName: "model-cache") pod "27f97c34-55d1-4b97-a5d9-cfd4a465c9af" (UID: "27f97c34-55d1-4b97-a5d9-cfd4a465c9af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:27:09.812032 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.812017    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27f97c34-55d1-4b97-a5d9-cfd4a465c9af-tmp-dir\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:27:09.812091 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.812033    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27f97c34-55d1-4b97-a5d9-cfd4a465c9af-model-cache\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:27:09.812091 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.812044    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27f97c34-55d1-4b97-a5d9-cfd4a465c9af-home\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:27:09.813581 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.813562    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f97c34-55d1-4b97-a5d9-cfd4a465c9af-dshm" (OuterVolumeSpecName: "dshm") pod "27f97c34-55d1-4b97-a5d9-cfd4a465c9af" (UID: "27f97c34-55d1-4b97-a5d9-cfd4a465c9af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:27:09.813667 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.813587    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27f97c34-55d1-4b97-a5d9-cfd4a465c9af-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "27f97c34-55d1-4b97-a5d9-cfd4a465c9af" (UID: "27f97c34-55d1-4b97-a5d9-cfd4a465c9af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 23:27:09.813713 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.813677    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f97c34-55d1-4b97-a5d9-cfd4a465c9af-kube-api-access-btwpd" (OuterVolumeSpecName: "kube-api-access-btwpd") pod "27f97c34-55d1-4b97-a5d9-cfd4a465c9af" (UID: "27f97c34-55d1-4b97-a5d9-cfd4a465c9af"). InnerVolumeSpecName "kube-api-access-btwpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 23:27:09.912728 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.912703    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btwpd\" (UniqueName: \"kubernetes.io/projected/27f97c34-55d1-4b97-a5d9-cfd4a465c9af-kube-api-access-btwpd\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:27:09.912728 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.912724    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27f97c34-55d1-4b97-a5d9-cfd4a465c9af-tls-certs\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:27:09.912874 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:09.912734    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27f97c34-55d1-4b97-a5d9-cfd4a465c9af-dshm\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:27:10.098195 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:10.098147    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d3a5b8/sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n"]
Jul 06 23:27:10.099815 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:10.099795    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-d3a5b8/sub-test-model-d3a5b8-kserve-58b87d5b49-2w97n"]
Jul 06 23:27:10.170168 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:10.170142    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27f97c34-55d1-4b97-a5d9-cfd4a465c9af" path="/var/lib/kubelet/pods/27f97c34-55d1-4b97-a5d9-cfd4a465c9af/volumes"
Jul 06 23:27:10.170567 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:10.170552    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecb44e3c-3911-4199-b713-dc85e3386ad4" path="/var/lib/kubelet/pods/ecb44e3c-3911-4199-b713-dc85e3386ad4/volumes"
Jul 06 23:27:32.335900 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:32.335864    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-d3a5b8/rate-test-model-d3a5b8-kserve-fbfb49947-798s6"]
Jul 06 23:27:32.336332 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:32.336244    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27f97c34-55d1-4b97-a5d9-cfd4a465c9af" containerName="main"
Jul 06 23:27:32.336332 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:32.336256    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="27f97c34-55d1-4b97-a5d9-cfd4a465c9af" containerName="main"
Jul 06 23:27:32.336332 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:32.336269    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecb44e3c-3911-4199-b713-dc85e3386ad4" containerName="main"
Jul 06 23:27:32.336332 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:32.336275    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecb44e3c-3911-4199-b713-dc85e3386ad4" containerName="main"
Jul 06 23:27:32.336580 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:32.336348    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecb44e3c-3911-4199-b713-dc85e3386ad4" containerName="main"
Jul 06 23:27:32.336580 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:32.336359    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="27f97c34-55d1-4b97-a5d9-cfd4a465c9af" containerName="main"
Jul 06 23:27:32.341057 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:32.341039    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d3a5b8/rate-test-model-d3a5b8-kserve-fbfb49947-798s6"
Jul 06 23:27:32.345005 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:32.344983    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d3a5b8\"/\"default-dockercfg-kqhdj\""
Jul 06 23:27:32.345115 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:32.345012    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d3a5b8\"/\"kube-root-ca.crt\""
Jul 06 23:27:32.345115 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:32.345017    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d3a5b8\"/\"openshift-service-ca.crt\""
Jul 06 23:27:32.345115 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:32.345070    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d3a5b8\"/\"rate-test-model-d3a5b8-kserve-self-signed-certs\""
Jul 06 23:27:32.351763 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:32.351743    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d3a5b8/rate-test-model-d3a5b8-kserve-fbfb49947-798s6"]
Jul 06 23:27:32.395555 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:32.395526    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfaa4804-aba7-45bd-99ca-7dfac726232e-tmp-dir\") pod \"rate-test-model-d3a5b8-kserve-fbfb49947-798s6\" (UID: \"bfaa4804-aba7-45bd-99ca-7dfac726232e\") " pod="ai-tenant-e2e-shared-b-d3a5b8/rate-test-model-d3a5b8-kserve-fbfb49947-798s6"
Jul 06 23:27:32.395709 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:32.395580    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74nxg\" (UniqueName: \"kubernetes.io/projected/bfaa4804-aba7-45bd-99ca-7dfac726232e-kube-api-access-74nxg\") pod \"rate-test-model-d3a5b8-kserve-fbfb49947-798s6\" (UID: \"bfaa4804-aba7-45bd-99ca-7dfac726232e\") " pod="ai-tenant-e2e-shared-b-d3a5b8/rate-test-model-d3a5b8-kserve-fbfb49947-798s6"
Jul 06 23:27:32.395709 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:32.395644    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfaa4804-aba7-45bd-99ca-7dfac726232e-dshm\") pod \"rate-test-model-d3a5b8-kserve-fbfb49947-798s6\" (UID: \"bfaa4804-aba7-45bd-99ca-7dfac726232e\") " pod="ai-tenant-e2e-shared-b-d3a5b8/rate-test-model-d3a5b8-kserve-fbfb49947-798s6"
Jul 06 23:27:32.395709 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:32.395689    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfaa4804-aba7-45bd-99ca-7dfac726232e-home\") pod \"rate-test-model-d3a5b8-kserve-fbfb49947-798s6\" (UID: \"bfaa4804-aba7-45bd-99ca-7dfac726232e\") " pod="ai-tenant-e2e-shared-b-d3a5b8/rate-test-model-d3a5b8-kserve-fbfb49947-798s6"
Jul 06 23:27:32.395897 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:32.395760    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfaa4804-aba7-45bd-99ca-7dfac726232e-model-cache\") pod \"rate-test-model-d3a5b8-kserve-fbfb49947-798s6\" (UID: \"bfaa4804-aba7-45bd-99ca-7dfac726232e\") " pod="ai-tenant-e2e-shared-b-d3a5b8/rate-test-model-d3a5b8-kserve-fbfb49947-798s6"
Jul 06 23:27:32.395897 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:32.395817    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfaa4804-aba7-45bd-99ca-7dfac726232e-tls-certs\") pod \"rate-test-model-d3a5b8-kserve-fbfb49947-798s6\" (UID: \"bfaa4804-aba7-45bd-99ca-7dfac726232e\") " pod="ai-tenant-e2e-shared-b-d3a5b8/rate-test-model-d3a5b8-kserve-fbfb49947-798s6"
Jul 06 23:27:32.496710 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:32.496683    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfaa4804-aba7-45bd-99ca-7dfac726232e-home\") pod \"rate-test-model-d3a5b8-kserve-fbfb49947-798s6\" (UID: \"bfaa4804-aba7-45bd-99ca-7dfac726232e\") " pod="ai-tenant-e2e-shared-b-d3a5b8/rate-test-model-d3a5b8-kserve-fbfb49947-798s6"
Jul 06 23:27:32.496840 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:32.496721    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfaa4804-aba7-45bd-99ca-7dfac726232e-model-cache\") pod \"rate-test-model-d3a5b8-kserve-fbfb49947-798s6\" (UID: \"bfaa4804-aba7-45bd-99ca-7dfac726232e\") " pod="ai-tenant-e2e-shared-b-d3a5b8/rate-test-model-d3a5b8-kserve-fbfb49947-798s6"
Jul 06 23:27:32.496840 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:32.496762    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfaa4804-aba7-45bd-99ca-7dfac726232e-tls-certs\") pod \"rate-test-model-d3a5b8-kserve-fbfb49947-798s6\" (UID: \"bfaa4804-aba7-45bd-99ca-7dfac726232e\") " pod="ai-tenant-e2e-shared-b-d3a5b8/rate-test-model-d3a5b8-kserve-fbfb49947-798s6"
Jul 06 23:27:32.496840 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:32.496788    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfaa4804-aba7-45bd-99ca-7dfac726232e-tmp-dir\") pod \"rate-test-model-d3a5b8-kserve-fbfb49947-798s6\" (UID: \"bfaa4804-aba7-45bd-99ca-7dfac726232e\") " pod="ai-tenant-e2e-shared-b-d3a5b8/rate-test-model-d3a5b8-kserve-fbfb49947-798s6"
Jul 06 23:27:32.496840 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:32.496831    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74nxg\" (UniqueName: \"kubernetes.io/projected/bfaa4804-aba7-45bd-99ca-7dfac726232e-kube-api-access-74nxg\") pod \"rate-test-model-d3a5b8-kserve-fbfb49947-798s6\" (UID: \"bfaa4804-aba7-45bd-99ca-7dfac726232e\") " pod="ai-tenant-e2e-shared-b-d3a5b8/rate-test-model-d3a5b8-kserve-fbfb49947-798s6"
Jul 06 23:27:32.497026 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:32.496955    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfaa4804-aba7-45bd-99ca-7dfac726232e-dshm\") pod \"rate-test-model-d3a5b8-kserve-fbfb49947-798s6\" (UID: \"bfaa4804-aba7-45bd-99ca-7dfac726232e\") " pod="ai-tenant-e2e-shared-b-d3a5b8/rate-test-model-d3a5b8-kserve-fbfb49947-798s6"
Jul 06 23:27:32.497119 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:32.497094    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfaa4804-aba7-45bd-99ca-7dfac726232e-home\") pod \"rate-test-model-d3a5b8-kserve-fbfb49947-798s6\" (UID: \"bfaa4804-aba7-45bd-99ca-7dfac726232e\") " pod="ai-tenant-e2e-shared-b-d3a5b8/rate-test-model-d3a5b8-kserve-fbfb49947-798s6"
Jul 06 23:27:32.497209 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:32.497123    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfaa4804-aba7-45bd-99ca-7dfac726232e-model-cache\") pod \"rate-test-model-d3a5b8-kserve-fbfb49947-798s6\" (UID: \"bfaa4804-aba7-45bd-99ca-7dfac726232e\") " pod="ai-tenant-e2e-shared-b-d3a5b8/rate-test-model-d3a5b8-kserve-fbfb49947-798s6"
Jul 06 23:27:32.497209 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:32.497164    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfaa4804-aba7-45bd-99ca-7dfac726232e-tmp-dir\") pod \"rate-test-model-d3a5b8-kserve-fbfb49947-798s6\" (UID: \"bfaa4804-aba7-45bd-99ca-7dfac726232e\") " pod="ai-tenant-e2e-shared-b-d3a5b8/rate-test-model-d3a5b8-kserve-fbfb49947-798s6"
Jul 06 23:27:32.499046 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:32.499026    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfaa4804-aba7-45bd-99ca-7dfac726232e-dshm\") pod \"rate-test-model-d3a5b8-kserve-fbfb49947-798s6\" (UID: \"bfaa4804-aba7-45bd-99ca-7dfac726232e\") " pod="ai-tenant-e2e-shared-b-d3a5b8/rate-test-model-d3a5b8-kserve-fbfb49947-798s6"
Jul 06 23:27:32.499136 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:32.499125    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfaa4804-aba7-45bd-99ca-7dfac726232e-tls-certs\") pod \"rate-test-model-d3a5b8-kserve-fbfb49947-798s6\" (UID: \"bfaa4804-aba7-45bd-99ca-7dfac726232e\") " pod="ai-tenant-e2e-shared-b-d3a5b8/rate-test-model-d3a5b8-kserve-fbfb49947-798s6"
Jul 06 23:27:32.505301 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:32.505278    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74nxg\" (UniqueName: \"kubernetes.io/projected/bfaa4804-aba7-45bd-99ca-7dfac726232e-kube-api-access-74nxg\") pod \"rate-test-model-d3a5b8-kserve-fbfb49947-798s6\" (UID: \"bfaa4804-aba7-45bd-99ca-7dfac726232e\") " pod="ai-tenant-e2e-shared-b-d3a5b8/rate-test-model-d3a5b8-kserve-fbfb49947-798s6"
Jul 06 23:27:32.651373 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:32.651296    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d3a5b8/rate-test-model-d3a5b8-kserve-fbfb49947-798s6"
Jul 06 23:27:32.780337 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:32.780312    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d3a5b8/rate-test-model-d3a5b8-kserve-fbfb49947-798s6"]
Jul 06 23:27:32.781882 ip-10-0-134-116 kubenswrapper[2569]: W0706 23:27:32.781855    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbfaa4804_aba7_45bd_99ca_7dfac726232e.slice/crio-f652886e00df025541fa9b8524292f2feb7e6e5591d9bf51fb8e1a282c090e5d WatchSource:0}: Error finding container f652886e00df025541fa9b8524292f2feb7e6e5591d9bf51fb8e1a282c090e5d: Status 404 returned error can't find the container with id f652886e00df025541fa9b8524292f2feb7e6e5591d9bf51fb8e1a282c090e5d
Jul 06 23:27:32.887928 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:32.887897    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d3a5b8/rate-test-model-d3a5b8-kserve-fbfb49947-798s6" event={"ID":"bfaa4804-aba7-45bd-99ca-7dfac726232e","Type":"ContainerStarted","Data":"867cda61c5088d5b7523ac22e60b65925c20479c55611149a98e80265b868e30"}
Jul 06 23:27:32.888071 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:32.887940    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d3a5b8/rate-test-model-d3a5b8-kserve-fbfb49947-798s6" event={"ID":"bfaa4804-aba7-45bd-99ca-7dfac726232e","Type":"ContainerStarted","Data":"f652886e00df025541fa9b8524292f2feb7e6e5591d9bf51fb8e1a282c090e5d"}
Jul 06 23:27:32.915634 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:32.915552    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-d3a5b8/rate-test-model-d3a5b8-kserve-fbfb49947-798s6" podStartSLOduration=0.915537574 podStartE2EDuration="915.537574ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:27:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:27:32.913540208 +0000 UTC m=+2465.317790114" watchObservedRunningTime="2026-07-06 23:27:32.915537574 +0000 UTC m=+2465.319787477"
Jul 06 23:27:42.651730 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:42.651700    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-d3a5b8/rate-test-model-d3a5b8-kserve-fbfb49947-798s6"
Jul 06 23:27:42.652110 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:42.651769    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-d3a5b8/rate-test-model-d3a5b8-kserve-fbfb49947-798s6"
Jul 06 23:27:42.663846 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:42.663826    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-d3a5b8/rate-test-model-d3a5b8-kserve-fbfb49947-798s6"
Jul 06 23:27:42.949463 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:27:42.949441    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-d3a5b8/rate-test-model-d3a5b8-kserve-fbfb49947-798s6"
Jul 06 23:28:17.875636 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:28:17.875607    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d3a5b8/rate-test-model-d3a5b8-kserve-fbfb49947-798s6"]
Jul 06 23:28:32.895351 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:28:32.895253    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-d3a5b8/rate-test-model-d3a5b8-kserve-fbfb49947-798s6" podUID="bfaa4804-aba7-45bd-99ca-7dfac726232e" containerName="main" containerID="cri-o://867cda61c5088d5b7523ac22e60b65925c20479c55611149a98e80265b868e30" gracePeriod=45
Jul 06 23:28:32.939370 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:28:32.939336    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-d3a5b8/rate-test-model-d3a5b8-kserve-fbfb49947-798s6" podUID="bfaa4804-aba7-45bd-99ca-7dfac726232e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": dial tcp 10.134.0.64:8000: connect: connection refused"
Jul 06 23:28:33.144683 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:28:33.144661    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d3a5b8/rate-test-model-d3a5b8-kserve-fbfb49947-798s6"
Jul 06 23:28:33.178522 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:28:33.178447    2569 generic.go:356] "Generic (PLEG): container finished" podID="bfaa4804-aba7-45bd-99ca-7dfac726232e" containerID="867cda61c5088d5b7523ac22e60b65925c20479c55611149a98e80265b868e30" exitCode=0
Jul 06 23:28:33.178522 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:28:33.178485    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d3a5b8/rate-test-model-d3a5b8-kserve-fbfb49947-798s6"
Jul 06 23:28:33.178522 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:28:33.178484    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d3a5b8/rate-test-model-d3a5b8-kserve-fbfb49947-798s6" event={"ID":"bfaa4804-aba7-45bd-99ca-7dfac726232e","Type":"ContainerDied","Data":"867cda61c5088d5b7523ac22e60b65925c20479c55611149a98e80265b868e30"}
Jul 06 23:28:33.178759 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:28:33.178538    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d3a5b8/rate-test-model-d3a5b8-kserve-fbfb49947-798s6" event={"ID":"bfaa4804-aba7-45bd-99ca-7dfac726232e","Type":"ContainerDied","Data":"f652886e00df025541fa9b8524292f2feb7e6e5591d9bf51fb8e1a282c090e5d"}
Jul 06 23:28:33.178759 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:28:33.178562    2569 scope.go:117] "RemoveContainer" containerID="867cda61c5088d5b7523ac22e60b65925c20479c55611149a98e80265b868e30"
Jul 06 23:28:33.187494 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:28:33.187479    2569 scope.go:117] "RemoveContainer" containerID="867cda61c5088d5b7523ac22e60b65925c20479c55611149a98e80265b868e30"
Jul 06 23:28:33.187744 ip-10-0-134-116 kubenswrapper[2569]: E0706 23:28:33.187725    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"867cda61c5088d5b7523ac22e60b65925c20479c55611149a98e80265b868e30\": container with ID starting with 867cda61c5088d5b7523ac22e60b65925c20479c55611149a98e80265b868e30 not found: ID does not exist" containerID="867cda61c5088d5b7523ac22e60b65925c20479c55611149a98e80265b868e30"
Jul 06 23:28:33.187818 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:28:33.187756    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"867cda61c5088d5b7523ac22e60b65925c20479c55611149a98e80265b868e30"} err="failed to get container status \"867cda61c5088d5b7523ac22e60b65925c20479c55611149a98e80265b868e30\": rpc error: code = NotFound desc = could not find container \"867cda61c5088d5b7523ac22e60b65925c20479c55611149a98e80265b868e30\": container with ID starting with 867cda61c5088d5b7523ac22e60b65925c20479c55611149a98e80265b868e30 not found: ID does not exist"
Jul 06 23:28:33.215140 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:28:33.215118    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfaa4804-aba7-45bd-99ca-7dfac726232e-dshm\") pod \"bfaa4804-aba7-45bd-99ca-7dfac726232e\" (UID: \"bfaa4804-aba7-45bd-99ca-7dfac726232e\") "
Jul 06 23:28:33.215248 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:28:33.215162    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfaa4804-aba7-45bd-99ca-7dfac726232e-home\") pod \"bfaa4804-aba7-45bd-99ca-7dfac726232e\" (UID: \"bfaa4804-aba7-45bd-99ca-7dfac726232e\") "
Jul 06 23:28:33.215299 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:28:33.215247    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfaa4804-aba7-45bd-99ca-7dfac726232e-tmp-dir\") pod \"bfaa4804-aba7-45bd-99ca-7dfac726232e\" (UID: \"bfaa4804-aba7-45bd-99ca-7dfac726232e\") "
Jul 06 23:28:33.215346 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:28:33.215306    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74nxg\" (UniqueName: \"kubernetes.io/projected/bfaa4804-aba7-45bd-99ca-7dfac726232e-kube-api-access-74nxg\") pod \"bfaa4804-aba7-45bd-99ca-7dfac726232e\" (UID: \"bfaa4804-aba7-45bd-99ca-7dfac726232e\") "
Jul 06 23:28:33.215346 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:28:33.215340    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfaa4804-aba7-45bd-99ca-7dfac726232e-tls-certs\") pod \"bfaa4804-aba7-45bd-99ca-7dfac726232e\" (UID: \"bfaa4804-aba7-45bd-99ca-7dfac726232e\") "
Jul 06 23:28:33.215745 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:28:33.215463    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfaa4804-aba7-45bd-99ca-7dfac726232e-model-cache\") pod \"bfaa4804-aba7-45bd-99ca-7dfac726232e\" (UID: \"bfaa4804-aba7-45bd-99ca-7dfac726232e\") "
Jul 06 23:28:33.215745 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:28:33.215502    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfaa4804-aba7-45bd-99ca-7dfac726232e-home" (OuterVolumeSpecName: "home") pod "bfaa4804-aba7-45bd-99ca-7dfac726232e" (UID: "bfaa4804-aba7-45bd-99ca-7dfac726232e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:28:33.215745 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:28:33.215528    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfaa4804-aba7-45bd-99ca-7dfac726232e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bfaa4804-aba7-45bd-99ca-7dfac726232e" (UID: "bfaa4804-aba7-45bd-99ca-7dfac726232e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:28:33.215745 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:28:33.215715    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfaa4804-aba7-45bd-99ca-7dfac726232e-model-cache" (OuterVolumeSpecName: "model-cache") pod "bfaa4804-aba7-45bd-99ca-7dfac726232e" (UID: "bfaa4804-aba7-45bd-99ca-7dfac726232e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:28:33.215954 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:28:33.215800    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfaa4804-aba7-45bd-99ca-7dfac726232e-home\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:28:33.215954 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:28:33.215817    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfaa4804-aba7-45bd-99ca-7dfac726232e-tmp-dir\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:28:33.215954 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:28:33.215826    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfaa4804-aba7-45bd-99ca-7dfac726232e-model-cache\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:28:33.217298 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:28:33.217277    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfaa4804-aba7-45bd-99ca-7dfac726232e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bfaa4804-aba7-45bd-99ca-7dfac726232e" (UID: "bfaa4804-aba7-45bd-99ca-7dfac726232e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 23:28:33.217664 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:28:33.217648    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfaa4804-aba7-45bd-99ca-7dfac726232e-dshm" (OuterVolumeSpecName: "dshm") pod "bfaa4804-aba7-45bd-99ca-7dfac726232e" (UID: "bfaa4804-aba7-45bd-99ca-7dfac726232e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:28:33.217664 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:28:33.217655    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfaa4804-aba7-45bd-99ca-7dfac726232e-kube-api-access-74nxg" (OuterVolumeSpecName: "kube-api-access-74nxg") pod "bfaa4804-aba7-45bd-99ca-7dfac726232e" (UID: "bfaa4804-aba7-45bd-99ca-7dfac726232e"). InnerVolumeSpecName "kube-api-access-74nxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 23:28:33.316549 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:28:33.316524    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74nxg\" (UniqueName: \"kubernetes.io/projected/bfaa4804-aba7-45bd-99ca-7dfac726232e-kube-api-access-74nxg\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:28:33.316549 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:28:33.316546    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfaa4804-aba7-45bd-99ca-7dfac726232e-tls-certs\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:28:33.316678 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:28:33.316555    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfaa4804-aba7-45bd-99ca-7dfac726232e-dshm\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:28:33.501935 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:28:33.501908    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d3a5b8/rate-test-model-d3a5b8-kserve-fbfb49947-798s6"]
Jul 06 23:28:33.506234 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:28:33.506208    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-d3a5b8/rate-test-model-d3a5b8-kserve-fbfb49947-798s6"]
Jul 06 23:28:34.170824 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:28:34.170794    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfaa4804-aba7-45bd-99ca-7dfac726232e" path="/var/lib/kubelet/pods/bfaa4804-aba7-45bd-99ca-7dfac726232e/volumes"
Jul 06 23:29:01.578173 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:01.578140    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"]
Jul 06 23:29:01.578813 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:01.578411    2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47" podUID="80e57fe7-6888-4ff4-bd85-a541c58164ad" containerName="istio-proxy" containerID="cri-o://1c2c45e7139560fa01f88aebe8fca0e47663e582a5a4b5abaa276faa86905e26" gracePeriod=30
Jul 06 23:29:06.836093 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:06.836068    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"
Jul 06 23:29:06.874268 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:06.874206    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e57fe7-6888-4ff4-bd85-a541c58164ad-istio-data\") pod \"80e57fe7-6888-4ff4-bd85-a541c58164ad\" (UID: \"80e57fe7-6888-4ff4-bd85-a541c58164ad\") "
Jul 06 23:29:06.874268 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:06.874248    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e57fe7-6888-4ff4-bd85-a541c58164ad-istio-podinfo\") pod \"80e57fe7-6888-4ff4-bd85-a541c58164ad\" (UID: \"80e57fe7-6888-4ff4-bd85-a541c58164ad\") "
Jul 06 23:29:06.874268 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:06.874267    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nkkjw\" (UniqueName: \"kubernetes.io/projected/80e57fe7-6888-4ff4-bd85-a541c58164ad-kube-api-access-nkkjw\") pod \"80e57fe7-6888-4ff4-bd85-a541c58164ad\" (UID: \"80e57fe7-6888-4ff4-bd85-a541c58164ad\") "
Jul 06 23:29:06.874453 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:06.874288    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e57fe7-6888-4ff4-bd85-a541c58164ad-istio-envoy\") pod \"80e57fe7-6888-4ff4-bd85-a541c58164ad\" (UID: \"80e57fe7-6888-4ff4-bd85-a541c58164ad\") "
Jul 06 23:29:06.874453 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:06.874329    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e57fe7-6888-4ff4-bd85-a541c58164ad-workload-socket\") pod \"80e57fe7-6888-4ff4-bd85-a541c58164ad\" (UID: \"80e57fe7-6888-4ff4-bd85-a541c58164ad\") "
Jul 06 23:29:06.874453 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:06.874350    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e57fe7-6888-4ff4-bd85-a541c58164ad-credential-socket\") pod \"80e57fe7-6888-4ff4-bd85-a541c58164ad\" (UID: \"80e57fe7-6888-4ff4-bd85-a541c58164ad\") "
Jul 06 23:29:06.874453 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:06.874390    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e57fe7-6888-4ff4-bd85-a541c58164ad-istiod-ca-cert\") pod \"80e57fe7-6888-4ff4-bd85-a541c58164ad\" (UID: \"80e57fe7-6888-4ff4-bd85-a541c58164ad\") "
Jul 06 23:29:06.874453 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:06.874437    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80e57fe7-6888-4ff4-bd85-a541c58164ad-istio-token\") pod \"80e57fe7-6888-4ff4-bd85-a541c58164ad\" (UID: \"80e57fe7-6888-4ff4-bd85-a541c58164ad\") "
Jul 06 23:29:06.874658 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:06.874465    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e57fe7-6888-4ff4-bd85-a541c58164ad-workload-certs\") pod \"80e57fe7-6888-4ff4-bd85-a541c58164ad\" (UID: \"80e57fe7-6888-4ff4-bd85-a541c58164ad\") "
Jul 06 23:29:06.874836 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:06.874807    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e57fe7-6888-4ff4-bd85-a541c58164ad-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "80e57fe7-6888-4ff4-bd85-a541c58164ad" (UID: "80e57fe7-6888-4ff4-bd85-a541c58164ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:29:06.874980 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:06.874941    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e57fe7-6888-4ff4-bd85-a541c58164ad-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "80e57fe7-6888-4ff4-bd85-a541c58164ad" (UID: "80e57fe7-6888-4ff4-bd85-a541c58164ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:29:06.875253 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:06.875226    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e57fe7-6888-4ff4-bd85-a541c58164ad-istio-data" (OuterVolumeSpecName: "istio-data") pod "80e57fe7-6888-4ff4-bd85-a541c58164ad" (UID: "80e57fe7-6888-4ff4-bd85-a541c58164ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:29:06.875484 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:06.875458    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80e57fe7-6888-4ff4-bd85-a541c58164ad-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "80e57fe7-6888-4ff4-bd85-a541c58164ad" (UID: "80e57fe7-6888-4ff4-bd85-a541c58164ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 23:29:06.875564 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:06.875529    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e57fe7-6888-4ff4-bd85-a541c58164ad-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "80e57fe7-6888-4ff4-bd85-a541c58164ad" (UID: "80e57fe7-6888-4ff4-bd85-a541c58164ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:29:06.876617 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:06.876577    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e57fe7-6888-4ff4-bd85-a541c58164ad-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "80e57fe7-6888-4ff4-bd85-a541c58164ad" (UID: "80e57fe7-6888-4ff4-bd85-a541c58164ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 23:29:06.876714 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:06.876650    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80e57fe7-6888-4ff4-bd85-a541c58164ad-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "80e57fe7-6888-4ff4-bd85-a541c58164ad" (UID: "80e57fe7-6888-4ff4-bd85-a541c58164ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 06 23:29:06.876788 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:06.876764    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e57fe7-6888-4ff4-bd85-a541c58164ad-kube-api-access-nkkjw" (OuterVolumeSpecName: "kube-api-access-nkkjw") pod "80e57fe7-6888-4ff4-bd85-a541c58164ad" (UID: "80e57fe7-6888-4ff4-bd85-a541c58164ad"). InnerVolumeSpecName "kube-api-access-nkkjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 23:29:06.876917 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:06.876894    2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e57fe7-6888-4ff4-bd85-a541c58164ad-istio-token" (OuterVolumeSpecName: "istio-token") pod "80e57fe7-6888-4ff4-bd85-a541c58164ad" (UID: "80e57fe7-6888-4ff4-bd85-a541c58164ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 23:29:06.975412 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:06.975390    2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e57fe7-6888-4ff4-bd85-a541c58164ad-istio-data\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:29:06.975412 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:06.975408    2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e57fe7-6888-4ff4-bd85-a541c58164ad-istio-podinfo\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:29:06.975547 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:06.975417    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkkjw\" (UniqueName: \"kubernetes.io/projected/80e57fe7-6888-4ff4-bd85-a541c58164ad-kube-api-access-nkkjw\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:29:06.975547 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:06.975427    2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e57fe7-6888-4ff4-bd85-a541c58164ad-istio-envoy\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:29:06.975547 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:06.975435    2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e57fe7-6888-4ff4-bd85-a541c58164ad-workload-socket\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:29:06.975547 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:06.975442    2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e57fe7-6888-4ff4-bd85-a541c58164ad-credential-socket\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:29:06.975547 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:06.975450    2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e57fe7-6888-4ff4-bd85-a541c58164ad-istiod-ca-cert\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:29:06.975547 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:06.975458    2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80e57fe7-6888-4ff4-bd85-a541c58164ad-istio-token\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:29:06.975547 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:06.975466    2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e57fe7-6888-4ff4-bd85-a541c58164ad-workload-certs\") on node \"ip-10-0-134-116.ec2.internal\" DevicePath \"\""
Jul 06 23:29:07.346947 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:07.346915    2569 generic.go:356] "Generic (PLEG): container finished" podID="80e57fe7-6888-4ff4-bd85-a541c58164ad" containerID="1c2c45e7139560fa01f88aebe8fca0e47663e582a5a4b5abaa276faa86905e26" exitCode=0
Jul 06 23:29:07.347110 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:07.346964    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"
Jul 06 23:29:07.347110 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:07.347008    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47" event={"ID":"80e57fe7-6888-4ff4-bd85-a541c58164ad","Type":"ContainerDied","Data":"1c2c45e7139560fa01f88aebe8fca0e47663e582a5a4b5abaa276faa86905e26"}
Jul 06 23:29:07.347110 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:07.347077    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47" event={"ID":"80e57fe7-6888-4ff4-bd85-a541c58164ad","Type":"ContainerDied","Data":"948ec83327a8ec25dd2a224e0d1dc1ea94dbcded8efa08b533ebb458d39fb904"}
Jul 06 23:29:07.347315 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:07.347111    2569 scope.go:117] "RemoveContainer" containerID="1c2c45e7139560fa01f88aebe8fca0e47663e582a5a4b5abaa276faa86905e26"
Jul 06 23:29:07.358364 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:07.358346    2569 scope.go:117] "RemoveContainer" containerID="8a235296c09b3bae1525211802187255296d81f1bf39999ab80a00a246a34314"
Jul 06 23:29:07.366389 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:07.366372    2569 scope.go:117] "RemoveContainer" containerID="1c2c45e7139560fa01f88aebe8fca0e47663e582a5a4b5abaa276faa86905e26"
Jul 06 23:29:07.366657 ip-10-0-134-116 kubenswrapper[2569]: E0706 23:29:07.366637    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c2c45e7139560fa01f88aebe8fca0e47663e582a5a4b5abaa276faa86905e26\": container with ID starting with 1c2c45e7139560fa01f88aebe8fca0e47663e582a5a4b5abaa276faa86905e26 not found: ID does not exist" containerID="1c2c45e7139560fa01f88aebe8fca0e47663e582a5a4b5abaa276faa86905e26"
Jul 06 23:29:07.366711 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:07.366665    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c2c45e7139560fa01f88aebe8fca0e47663e582a5a4b5abaa276faa86905e26"} err="failed to get container status \"1c2c45e7139560fa01f88aebe8fca0e47663e582a5a4b5abaa276faa86905e26\": rpc error: code = NotFound desc = could not find container \"1c2c45e7139560fa01f88aebe8fca0e47663e582a5a4b5abaa276faa86905e26\": container with ID starting with 1c2c45e7139560fa01f88aebe8fca0e47663e582a5a4b5abaa276faa86905e26 not found: ID does not exist"
Jul 06 23:29:07.366711 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:07.366682    2569 scope.go:117] "RemoveContainer" containerID="8a235296c09b3bae1525211802187255296d81f1bf39999ab80a00a246a34314"
Jul 06 23:29:07.366925 ip-10-0-134-116 kubenswrapper[2569]: E0706 23:29:07.366906    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a235296c09b3bae1525211802187255296d81f1bf39999ab80a00a246a34314\": container with ID starting with 8a235296c09b3bae1525211802187255296d81f1bf39999ab80a00a246a34314 not found: ID does not exist" containerID="8a235296c09b3bae1525211802187255296d81f1bf39999ab80a00a246a34314"
Jul 06 23:29:07.366986 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:07.366934    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a235296c09b3bae1525211802187255296d81f1bf39999ab80a00a246a34314"} err="failed to get container status \"8a235296c09b3bae1525211802187255296d81f1bf39999ab80a00a246a34314\": rpc error: code = NotFound desc = could not find container \"8a235296c09b3bae1525211802187255296d81f1bf39999ab80a00a246a34314\": container with ID starting with 8a235296c09b3bae1525211802187255296d81f1bf39999ab80a00a246a34314 not found: ID does not exist"
Jul 06 23:29:07.373023 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:07.373003    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"]
Jul 06 23:29:07.378642 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:07.378622    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-669fee-openshift-default-77b88cd47-ngx47"]
Jul 06 23:29:08.170279 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:08.170250    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80e57fe7-6888-4ff4-bd85-a541c58164ad" path="/var/lib/kubelet/pods/80e57fe7-6888-4ff4-bd85-a541c58164ad/volumes"
Jul 06 23:29:09.276000 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:09.275973    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d66d76884-trqqr_3c96c19d-0c87-49e1-a550-361c2ce200d9/authorino/0.log"
Jul 06 23:29:14.192735 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:14.192702    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-dhcx5_4ec333e7-07df-46b2-825f-91b729228c0b/manager/0.log"
Jul 06 23:29:14.318306 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:14.318281    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-dmnld_1753a9aa-4104-4f17-b295-abeea36a2262/manager/0.log"
Jul 06 23:29:14.450458 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:14.450374    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-7cc74d4bd-wz876_11f382ed-5d3a-4006-bac8-c39fd893e6a2/maas-api/0.log"
Jul 06 23:29:14.846716 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:14.846645    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5dfdd7fcb5-m6mtz_02009b0e-a948-4de4-9eef-9a963f017c5a/manager/0.log"
Jul 06 23:29:14.966763 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:14.966737    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-vqgqh_d1cfb9e6-e14a-4ba6-a715-3af91a5f38d6/server/0.log"
Jul 06 23:29:15.347838 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:15.347814    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b8c6df956-qbpzs_c906c6f8-114c-4e69-a05f-db2147e435ff/copy-manifests/0.log"
Jul 06 23:29:15.366727 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:15.366686    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b8c6df956-qbpzs_c906c6f8-114c-4e69-a05f-db2147e435ff/manager/0.log"
Jul 06 23:29:15.615025 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:15.614947    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-l29j5_67be9ca0-f203-4586-968f-3b06828f4899/postgres/0.log"
Jul 06 23:29:16.988786 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:16.988739    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d66d76884-trqqr_3c96c19d-0c87-49e1-a550-361c2ce200d9/authorino/0.log"
Jul 06 23:29:17.121514 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:17.121481    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-bvl49_eac3b1e8-3fdc-4930-84f4-bf118dbc353d/manager/0.log"
Jul 06 23:29:17.235783 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:17.235756    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-5frkz_5313328e-7735-45c7-ad8f-e72407ff30df/manager/0.log"
Jul 06 23:29:17.369711 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:17.369650    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-k5xcs_fe3069f0-f612-4413-bc74-2cfd710ccf6b/kuadrant-console-plugin/0.log"
Jul 06 23:29:17.499076 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:17.499048    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-jkmbd_3559ecd8-3e89-4132-812a-461d37c05b2e/registry-server/0.log"
Jul 06 23:29:17.904790 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:17.904762    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-lf6v7_eef1a6cf-89f8-4e69-a5b6-a54fdda4c483/manager/0.log"
Jul 06 23:29:18.629195 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:18.629154    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-698448cf7f-ll8cq_73f21974-debc-40ab-8b7d-18ca71bd3082/kube-auth-proxy/0.log"
Jul 06 23:29:18.755690 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:18.755654    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-tzqx6_404a2601-0b38-4fe9-81dc-23ae8fe26c6d/istio-proxy/1.log"
Jul 06 23:29:18.882878 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:18.882791    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-qzkbl_fe012d97-8ca1-4fd7-a0d1-afdccd427929/payload-pre-processing/0.log"
Jul 06 23:29:19.014440 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:19.014413    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-jd8zt_02af0679-599a-44a8-b470-a7e480ddf47d/payload-processing/0.log"
Jul 06 23:29:19.135657 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:19.135578    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-65b4d9bcf5-kkhtq_3e6a1480-5c80-484b-87e0-78d5a855594e/router/0.log"
Jul 06 23:29:19.502714 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:19.502687    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-x45bv_9578aaef-e242-4fe1-a9ca-6f00b96bc360/main/0.log"
Jul 06 23:29:19.620564 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:19.620536    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-7lc2v_b1f8b9a7-86fa-4557-a90e-f3c2cf5a879b/main/0.log"
Jul 06 23:29:24.125054 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:24.125026    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-r9kzh/must-gather-p24jw"]
Jul 06 23:29:24.125442 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:24.125388    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80e57fe7-6888-4ff4-bd85-a541c58164ad" containerName="istio-proxy"
Jul 06 23:29:24.125442 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:24.125400    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="80e57fe7-6888-4ff4-bd85-a541c58164ad" containerName="istio-proxy"
Jul 06 23:29:24.125442 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:24.125416    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80e57fe7-6888-4ff4-bd85-a541c58164ad" containerName="istio-proxy"
Jul 06 23:29:24.125442 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:24.125421    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="80e57fe7-6888-4ff4-bd85-a541c58164ad" containerName="istio-proxy"
Jul 06 23:29:24.125442 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:24.125431    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfaa4804-aba7-45bd-99ca-7dfac726232e" containerName="main"
Jul 06 23:29:24.125442 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:24.125437    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfaa4804-aba7-45bd-99ca-7dfac726232e" containerName="main"
Jul 06 23:29:24.125653 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:24.125502    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="80e57fe7-6888-4ff4-bd85-a541c58164ad" containerName="istio-proxy"
Jul 06 23:29:24.125653 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:24.125511    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfaa4804-aba7-45bd-99ca-7dfac726232e" containerName="main"
Jul 06 23:29:24.125653 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:24.125631    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="80e57fe7-6888-4ff4-bd85-a541c58164ad" containerName="istio-proxy"
Jul 06 23:29:24.130202 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:24.130163    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r9kzh/must-gather-p24jw"
Jul 06 23:29:24.133144 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:24.133044    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r9kzh\"/\"kube-root-ca.crt\""
Jul 06 23:29:24.134580 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:24.134555    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r9kzh\"/\"openshift-service-ca.crt\""
Jul 06 23:29:24.134712 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:24.134584    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-r9kzh\"/\"default-dockercfg-v45ts\""
Jul 06 23:29:24.136484 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:24.136464    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r9kzh/must-gather-p24jw"]
Jul 06 23:29:24.207290 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:24.207266    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b272f56d-a8e2-4aaa-8447-451ef0093de0-must-gather-output\") pod \"must-gather-p24jw\" (UID: \"b272f56d-a8e2-4aaa-8447-451ef0093de0\") " pod="openshift-must-gather-r9kzh/must-gather-p24jw"
Jul 06 23:29:24.207420 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:24.207314    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c999w\" (UniqueName: \"kubernetes.io/projected/b272f56d-a8e2-4aaa-8447-451ef0093de0-kube-api-access-c999w\") pod \"must-gather-p24jw\" (UID: \"b272f56d-a8e2-4aaa-8447-451ef0093de0\") " pod="openshift-must-gather-r9kzh/must-gather-p24jw"
Jul 06 23:29:24.308554 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:24.308527    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b272f56d-a8e2-4aaa-8447-451ef0093de0-must-gather-output\") pod \"must-gather-p24jw\" (UID: \"b272f56d-a8e2-4aaa-8447-451ef0093de0\") " pod="openshift-must-gather-r9kzh/must-gather-p24jw"
Jul 06 23:29:24.308656 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:24.308574    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c999w\" (UniqueName: \"kubernetes.io/projected/b272f56d-a8e2-4aaa-8447-451ef0093de0-kube-api-access-c999w\") pod \"must-gather-p24jw\" (UID: \"b272f56d-a8e2-4aaa-8447-451ef0093de0\") " pod="openshift-must-gather-r9kzh/must-gather-p24jw"
Jul 06 23:29:24.308850 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:24.308828    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b272f56d-a8e2-4aaa-8447-451ef0093de0-must-gather-output\") pod \"must-gather-p24jw\" (UID: \"b272f56d-a8e2-4aaa-8447-451ef0093de0\") " pod="openshift-must-gather-r9kzh/must-gather-p24jw"
Jul 06 23:29:24.316564 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:24.316540    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c999w\" (UniqueName: \"kubernetes.io/projected/b272f56d-a8e2-4aaa-8447-451ef0093de0-kube-api-access-c999w\") pod \"must-gather-p24jw\" (UID: \"b272f56d-a8e2-4aaa-8447-451ef0093de0\") " pod="openshift-must-gather-r9kzh/must-gather-p24jw"
Jul 06 23:29:24.440206 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:24.440169    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r9kzh/must-gather-p24jw"
Jul 06 23:29:24.568573 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:24.568543    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r9kzh/must-gather-p24jw"]
Jul 06 23:29:24.570324 ip-10-0-134-116 kubenswrapper[2569]: W0706 23:29:24.570296    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb272f56d_a8e2_4aaa_8447_451ef0093de0.slice/crio-05677dcab7df1a214832110c34e5a2e82d736ef48b3f9d03df41bd7b62dc06fe WatchSource:0}: Error finding container 05677dcab7df1a214832110c34e5a2e82d736ef48b3f9d03df41bd7b62dc06fe: Status 404 returned error can't find the container with id 05677dcab7df1a214832110c34e5a2e82d736ef48b3f9d03df41bd7b62dc06fe
Jul 06 23:29:24.572328 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:24.572312    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 23:29:25.435565 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:25.435480    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r9kzh/must-gather-p24jw" event={"ID":"b272f56d-a8e2-4aaa-8447-451ef0093de0","Type":"ContainerStarted","Data":"074a071d6f6832e43f034d7175b494646582cffe78d0fcc2fad4c3083cd6bb2a"}
Jul 06 23:29:25.435565 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:25.435538    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r9kzh/must-gather-p24jw" event={"ID":"b272f56d-a8e2-4aaa-8447-451ef0093de0","Type":"ContainerStarted","Data":"05677dcab7df1a214832110c34e5a2e82d736ef48b3f9d03df41bd7b62dc06fe"}
Jul 06 23:29:26.443493 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:26.443450    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r9kzh/must-gather-p24jw" event={"ID":"b272f56d-a8e2-4aaa-8447-451ef0093de0","Type":"ContainerStarted","Data":"5cc06ce3c3de788f0a2574e9965284d3d214b6ffffa9309da12f1e54969f964f"}
Jul 06 23:29:26.461534 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:26.461474    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-r9kzh/must-gather-p24jw" podStartSLOduration=1.746790395 podStartE2EDuration="2.461451472s" totalImagesPullingTime="714.661077ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:29:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:29:26.459114116 +0000 UTC m=+2578.863364022" watchObservedRunningTime="2026-07-06 23:29:26.461451472 +0000 UTC m=+2578.865701377"
Jul 06 23:29:26.790918 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:26.790840    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-l6fbb_8cc9b77e-7a6d-4d5b-83a2-8f24359ec453/global-pull-secret-syncer/0.log"
Jul 06 23:29:26.936492 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:26.936460    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jtsxh_17c837d3-15da-4a3b-97ba-c035de2e4d38/konnectivity-agent/0.log"
Jul 06 23:29:26.978267 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:26.978238    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-116.ec2.internal_af57faf3951e9e93eefc26772d49ee48/haproxy/0.log"
Jul 06 23:29:31.953653 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:31.953613    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d66d76884-trqqr_3c96c19d-0c87-49e1-a550-361c2ce200d9/authorino/0.log"
Jul 06 23:29:31.992159 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:31.990611    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-bvl49_eac3b1e8-3fdc-4930-84f4-bf118dbc353d/manager/0.log"
Jul 06 23:29:32.021244 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:32.021094    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-5frkz_5313328e-7735-45c7-ad8f-e72407ff30df/manager/0.log"
Jul 06 23:29:32.055982 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:32.055948    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-k5xcs_fe3069f0-f612-4413-bc74-2cfd710ccf6b/kuadrant-console-plugin/0.log"
Jul 06 23:29:32.101830 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:32.101782    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-jkmbd_3559ecd8-3e89-4132-812a-461d37c05b2e/registry-server/0.log"
Jul 06 23:29:32.355782 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:32.355690    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-lf6v7_eef1a6cf-89f8-4e69-a5b6-a54fdda4c483/manager/0.log"
Jul 06 23:29:34.015369 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:34.015324    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-dpdfh_ad24049a-6893-4d5e-bb79-0e176fc731c3/cluster-monitoring-operator/0.log"
Jul 06 23:29:34.371751 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:34.371677    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nvtn5_97cb756f-f621-419b-8f56-1b144ccdade3/node-exporter/0.log"
Jul 06 23:29:34.393685 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:34.393659    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nvtn5_97cb756f-f621-419b-8f56-1b144ccdade3/kube-rbac-proxy/0.log"
Jul 06 23:29:34.417586 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:34.417562    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nvtn5_97cb756f-f621-419b-8f56-1b144ccdade3/init-textfile/0.log"
Jul 06 23:29:36.810553 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:36.810504    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-r9kzh/perf-node-gather-daemonset-45sld"]
Jul 06 23:29:36.818112 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:36.818080    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r9kzh/perf-node-gather-daemonset-45sld"
Jul 06 23:29:36.822071 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:36.822039    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r9kzh/perf-node-gather-daemonset-45sld"]
Jul 06 23:29:36.930707 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:36.930668    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/74832bee-e02f-47e4-9e82-6e3b95ae33dc-lib-modules\") pod \"perf-node-gather-daemonset-45sld\" (UID: \"74832bee-e02f-47e4-9e82-6e3b95ae33dc\") " pod="openshift-must-gather-r9kzh/perf-node-gather-daemonset-45sld"
Jul 06 23:29:36.931095 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:36.931075    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/74832bee-e02f-47e4-9e82-6e3b95ae33dc-proc\") pod \"perf-node-gather-daemonset-45sld\" (UID: \"74832bee-e02f-47e4-9e82-6e3b95ae33dc\") " pod="openshift-must-gather-r9kzh/perf-node-gather-daemonset-45sld"
Jul 06 23:29:36.931286 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:36.931270    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/74832bee-e02f-47e4-9e82-6e3b95ae33dc-podres\") pod \"perf-node-gather-daemonset-45sld\" (UID: \"74832bee-e02f-47e4-9e82-6e3b95ae33dc\") " pod="openshift-must-gather-r9kzh/perf-node-gather-daemonset-45sld"
Jul 06 23:29:36.931452 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:36.931434    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6fht\" (UniqueName: \"kubernetes.io/projected/74832bee-e02f-47e4-9e82-6e3b95ae33dc-kube-api-access-s6fht\") pod \"perf-node-gather-daemonset-45sld\" (UID: \"74832bee-e02f-47e4-9e82-6e3b95ae33dc\") " pod="openshift-must-gather-r9kzh/perf-node-gather-daemonset-45sld"
Jul 06 23:29:36.931610 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:36.931595    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74832bee-e02f-47e4-9e82-6e3b95ae33dc-sys\") pod \"perf-node-gather-daemonset-45sld\" (UID: \"74832bee-e02f-47e4-9e82-6e3b95ae33dc\") " pod="openshift-must-gather-r9kzh/perf-node-gather-daemonset-45sld"
Jul 06 23:29:37.032864 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:37.032821    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/74832bee-e02f-47e4-9e82-6e3b95ae33dc-lib-modules\") pod \"perf-node-gather-daemonset-45sld\" (UID: \"74832bee-e02f-47e4-9e82-6e3b95ae33dc\") " pod="openshift-must-gather-r9kzh/perf-node-gather-daemonset-45sld"
Jul 06 23:29:37.033058 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:37.032940    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/74832bee-e02f-47e4-9e82-6e3b95ae33dc-proc\") pod \"perf-node-gather-daemonset-45sld\" (UID: \"74832bee-e02f-47e4-9e82-6e3b95ae33dc\") " pod="openshift-must-gather-r9kzh/perf-node-gather-daemonset-45sld"
Jul 06 23:29:37.033058 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:37.032966    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/74832bee-e02f-47e4-9e82-6e3b95ae33dc-podres\") pod \"perf-node-gather-daemonset-45sld\" (UID: \"74832bee-e02f-47e4-9e82-6e3b95ae33dc\") " pod="openshift-must-gather-r9kzh/perf-node-gather-daemonset-45sld"
Jul 06 23:29:37.033058 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:37.032991    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6fht\" (UniqueName: \"kubernetes.io/projected/74832bee-e02f-47e4-9e82-6e3b95ae33dc-kube-api-access-s6fht\") pod \"perf-node-gather-daemonset-45sld\" (UID: \"74832bee-e02f-47e4-9e82-6e3b95ae33dc\") " pod="openshift-must-gather-r9kzh/perf-node-gather-daemonset-45sld"
Jul 06 23:29:37.033058 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:37.033032    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74832bee-e02f-47e4-9e82-6e3b95ae33dc-sys\") pod \"perf-node-gather-daemonset-45sld\" (UID: \"74832bee-e02f-47e4-9e82-6e3b95ae33dc\") " pod="openshift-must-gather-r9kzh/perf-node-gather-daemonset-45sld"
Jul 06 23:29:37.033330 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:37.033134    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74832bee-e02f-47e4-9e82-6e3b95ae33dc-sys\") pod \"perf-node-gather-daemonset-45sld\" (UID: \"74832bee-e02f-47e4-9e82-6e3b95ae33dc\") " pod="openshift-must-gather-r9kzh/perf-node-gather-daemonset-45sld"
Jul 06 23:29:37.033330 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:37.033267    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/74832bee-e02f-47e4-9e82-6e3b95ae33dc-lib-modules\") pod \"perf-node-gather-daemonset-45sld\" (UID: \"74832bee-e02f-47e4-9e82-6e3b95ae33dc\") " pod="openshift-must-gather-r9kzh/perf-node-gather-daemonset-45sld"
Jul 06 23:29:37.033330 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:37.033316    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/74832bee-e02f-47e4-9e82-6e3b95ae33dc-proc\") pod \"perf-node-gather-daemonset-45sld\" (UID: \"74832bee-e02f-47e4-9e82-6e3b95ae33dc\") " pod="openshift-must-gather-r9kzh/perf-node-gather-daemonset-45sld"
Jul 06 23:29:37.033508 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:37.033385    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/74832bee-e02f-47e4-9e82-6e3b95ae33dc-podres\") pod \"perf-node-gather-daemonset-45sld\" (UID: \"74832bee-e02f-47e4-9e82-6e3b95ae33dc\") " pod="openshift-must-gather-r9kzh/perf-node-gather-daemonset-45sld"
Jul 06 23:29:37.041610 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:37.041581    2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6fht\" (UniqueName: \"kubernetes.io/projected/74832bee-e02f-47e4-9e82-6e3b95ae33dc-kube-api-access-s6fht\") pod \"perf-node-gather-daemonset-45sld\" (UID: \"74832bee-e02f-47e4-9e82-6e3b95ae33dc\") " pod="openshift-must-gather-r9kzh/perf-node-gather-daemonset-45sld"
Jul 06 23:29:37.140232 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:37.140123    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r9kzh/perf-node-gather-daemonset-45sld"
Jul 06 23:29:37.320861 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:37.320836    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r9kzh/perf-node-gather-daemonset-45sld"]
Jul 06 23:29:37.323356 ip-10-0-134-116 kubenswrapper[2569]: W0706 23:29:37.323319    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod74832bee_e02f_47e4_9e82_6e3b95ae33dc.slice/crio-97caa9c0650282a5f1c732abe8c16e0b5cdc1003e32dce52048508e4ad3afb33 WatchSource:0}: Error finding container 97caa9c0650282a5f1c732abe8c16e0b5cdc1003e32dce52048508e4ad3afb33: Status 404 returned error can't find the container with id 97caa9c0650282a5f1c732abe8c16e0b5cdc1003e32dce52048508e4ad3afb33
Jul 06 23:29:37.524340 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:37.524301    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r9kzh/perf-node-gather-daemonset-45sld" event={"ID":"74832bee-e02f-47e4-9e82-6e3b95ae33dc","Type":"ContainerStarted","Data":"63c02ed29881efa1e4628eb4ba1bfce4ff0b3513aebe3ae79e022a1accc18753"}
Jul 06 23:29:37.524503 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:37.524389    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-r9kzh/perf-node-gather-daemonset-45sld"
Jul 06 23:29:37.524503 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:37.524417    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r9kzh/perf-node-gather-daemonset-45sld" event={"ID":"74832bee-e02f-47e4-9e82-6e3b95ae33dc","Type":"ContainerStarted","Data":"97caa9c0650282a5f1c732abe8c16e0b5cdc1003e32dce52048508e4ad3afb33"}
Jul 06 23:29:37.541818 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:37.541758    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-r9kzh/perf-node-gather-daemonset-45sld" podStartSLOduration=1.541740564 podStartE2EDuration="1.541740564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:29:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:29:37.54106965 +0000 UTC m=+2589.945319576" watchObservedRunningTime="2026-07-06 23:29:37.541740564 +0000 UTC m=+2589.945990471"
Jul 06 23:29:38.900690 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:38.900649    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2xhr2_a674edfd-a962-4327-b319-320fb62922c0/dns/0.log"
Jul 06 23:29:38.921475 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:38.921444    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2xhr2_a674edfd-a962-4327-b319-320fb62922c0/kube-rbac-proxy/0.log"
Jul 06 23:29:39.057065 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:39.057032    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wdchz_bc58eac9-19c3-4a2c-be0f-2f5211fc5f39/dns-node-resolver/0.log"
Jul 06 23:29:39.625477 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:39.625450    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2zsx8_6b66febe-a59f-40a7-bff3-d248315b326e/node-ca/0.log"
Jul 06 23:29:40.750896 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:40.750866    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-698448cf7f-ll8cq_73f21974-debc-40ab-8b7d-18ca71bd3082/kube-auth-proxy/0.log"
Jul 06 23:29:40.776401 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:40.776372    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-tzqx6_404a2601-0b38-4fe9-81dc-23ae8fe26c6d/istio-proxy/0.log"
Jul 06 23:29:40.840572 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:40.840542    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-tzqx6_404a2601-0b38-4fe9-81dc-23ae8fe26c6d/istio-proxy/1.log"
Jul 06 23:29:40.879440 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:40.879411    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-qzkbl_fe012d97-8ca1-4fd7-a0d1-afdccd427929/payload-pre-processing/0.log"
Jul 06 23:29:40.965037 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:40.965011    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-jd8zt_02af0679-599a-44a8-b470-a7e480ddf47d/payload-processing/0.log"
Jul 06 23:29:40.986516 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:40.986486    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-65b4d9bcf5-kkhtq_3e6a1480-5c80-484b-87e0-78d5a855594e/router/0.log"
Jul 06 23:29:41.650976 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:41.650947    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dwnzz_8624c1c4-bd8a-4e14-9445-3c7352c1cd60/serve-healthcheck-canary/0.log"
Jul 06 23:29:42.138167 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:42.138137    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-z62hq_6e00d4ca-8ce5-4ec1-8895-fe762a2d9e4c/insights-operator/0.log"
Jul 06 23:29:42.138691 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:42.138672    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-z62hq_6e00d4ca-8ce5-4ec1-8895-fe762a2d9e4c/insights-operator/1.log"
Jul 06 23:29:42.157852 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:42.157828    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8bd6t_c13e679b-e3c2-432e-b81d-619f3e346e92/kube-rbac-proxy/0.log"
Jul 06 23:29:42.176158 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:42.176115    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8bd6t_c13e679b-e3c2-432e-b81d-619f3e346e92/exporter/0.log"
Jul 06 23:29:42.198598 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:42.198571    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8bd6t_c13e679b-e3c2-432e-b81d-619f3e346e92/extractor/0.log"
Jul 06 23:29:43.539110 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:43.539074    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-r9kzh/perf-node-gather-daemonset-45sld"
Jul 06 23:29:44.388542 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:44.388509    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-dhcx5_4ec333e7-07df-46b2-825f-91b729228c0b/manager/0.log"
Jul 06 23:29:44.630332 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:44.630296    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-dmnld_1753a9aa-4104-4f17-b295-abeea36a2262/manager/0.log"
Jul 06 23:29:44.720581 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:44.720538    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-7cc74d4bd-wz876_11f382ed-5d3a-4006-bac8-c39fd893e6a2/maas-api/0.log"
Jul 06 23:29:44.936366 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:44.936326    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5dfdd7fcb5-m6mtz_02009b0e-a948-4de4-9eef-9a963f017c5a/manager/0.log"
Jul 06 23:29:44.954843 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:44.954813    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-vqgqh_d1cfb9e6-e14a-4ba6-a715-3af91a5f38d6/server/0.log"
Jul 06 23:29:45.203864 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:45.203838    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b8c6df956-qbpzs_c906c6f8-114c-4e69-a05f-db2147e435ff/manager/0.log"
Jul 06 23:29:45.220224 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:45.220195    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b8c6df956-qbpzs_c906c6f8-114c-4e69-a05f-db2147e435ff/copy-manifests/0.log"
Jul 06 23:29:45.290924 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:45.290867    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-l29j5_67be9ca0-f203-4586-968f-3b06828f4899/postgres/0.log"
Jul 06 23:29:46.885564 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:46.885526    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-z6drs_448200ad-4175-4440-b1c1-dd87168693c6/openshift-lws-operator/0.log"
Jul 06 23:29:47.309896 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:47.309863    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-698448cf7f-ll8cq_73f21974-debc-40ab-8b7d-18ca71bd3082/kube-auth-proxy/0.log"
Jul 06 23:29:47.327744 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:47.327721    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-tzqx6_404a2601-0b38-4fe9-81dc-23ae8fe26c6d/istio-proxy/0.log"
Jul 06 23:29:47.370815 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:47.370761    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-tzqx6_404a2601-0b38-4fe9-81dc-23ae8fe26c6d/istio-proxy/1.log"
Jul 06 23:29:47.403005 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:47.402972    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-qzkbl_fe012d97-8ca1-4fd7-a0d1-afdccd427929/payload-pre-processing/0.log"
Jul 06 23:29:47.477624 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:47.477596    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-jd8zt_02af0679-599a-44a8-b470-a7e480ddf47d/payload-processing/0.log"
Jul 06 23:29:47.493702 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:47.493670    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-65b4d9bcf5-kkhtq_3e6a1480-5c80-484b-87e0-78d5a855594e/router/0.log"
Jul 06 23:29:48.808988 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:48.808961    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-698448cf7f-ll8cq_73f21974-debc-40ab-8b7d-18ca71bd3082/kube-auth-proxy/0.log"
Jul 06 23:29:48.827686 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:48.827662    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-tzqx6_404a2601-0b38-4fe9-81dc-23ae8fe26c6d/istio-proxy/0.log"
Jul 06 23:29:48.881457 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:48.881427    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-tzqx6_404a2601-0b38-4fe9-81dc-23ae8fe26c6d/istio-proxy/1.log"
Jul 06 23:29:48.911069 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:48.910997    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-qzkbl_fe012d97-8ca1-4fd7-a0d1-afdccd427929/payload-pre-processing/0.log"
Jul 06 23:29:48.979204 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:48.978935    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-jd8zt_02af0679-599a-44a8-b470-a7e480ddf47d/payload-processing/0.log"
Jul 06 23:29:48.994336 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:48.994306    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-65b4d9bcf5-kkhtq_3e6a1480-5c80-484b-87e0-78d5a855594e/router/0.log"
Jul 06 23:29:51.878336 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:51.878260    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-v2lzx_6ab91bcb-c3c8-47ef-a1b8-2fb8cafac398/migrator/0.log"
Jul 06 23:29:51.895939 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:51.895914    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-v2lzx_6ab91bcb-c3c8-47ef-a1b8-2fb8cafac398/graceful-termination/0.log"
Jul 06 23:29:53.575361 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:53.575332    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qgqcr_3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90/kube-multus-additional-cni-plugins/0.log"
Jul 06 23:29:53.593037 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:53.593018    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qgqcr_3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90/egress-router-binary-copy/0.log"
Jul 06 23:29:53.611462 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:53.611437    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qgqcr_3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90/cni-plugins/0.log"
Jul 06 23:29:53.629072 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:53.629052    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qgqcr_3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90/bond-cni-plugin/0.log"
Jul 06 23:29:53.646830 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:53.646809    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qgqcr_3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90/routeoverride-cni/0.log"
Jul 06 23:29:53.664377 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:53.664328    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qgqcr_3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90/whereabouts-cni-bincopy/0.log"
Jul 06 23:29:53.682216 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:53.682197    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qgqcr_3e2c05e5-ad3c-4568-99a3-3cfb9b42cd90/whereabouts-cni/0.log"
Jul 06 23:29:53.757091 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:53.757071    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hwhh7_345357e4-a764-47c9-8ef9-acfc5b4e8cf6/kube-multus/0.log"
Jul 06 23:29:53.777728 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:53.777704    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mfmlk_107658f0-a3ae-4883-99dd-ddc62057a8a0/network-metrics-daemon/0.log"
Jul 06 23:29:53.793725 ip-10-0-134-116 kubenswrapper[2569]: I0706 23:29:53.793700    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mfmlk_107658f0-a3ae-4883-99dd-ddc62057a8a0/kube-rbac-proxy/0.log"