Jul 14 22:55:08.137108 ip-10-0-134-8 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 14 22:55:08.137118 ip-10-0-134-8 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 14 22:55:08.137125 ip-10-0-134-8 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 14 22:55:08.137391 ip-10-0-134-8 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 14 22:55:18.369456 ip-10-0-134-8 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 14 22:55:18.369473 ip-10-0-134-8 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot b8df1414cba043eea399618c541f40b4 --
Jul 14 22:56:52.567545 ip-10-0-134-8 systemd[1]: Starting Kubernetes Kubelet...
Jul 14 22:56:53.023960 ip-10-0-134-8 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 14 22:56:53.023960 ip-10-0-134-8 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 14 22:56:53.023960 ip-10-0-134-8 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 14 22:56:53.023960 ip-10-0-134-8 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 14 22:56:53.023960 ip-10-0-134-8 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 14 22:56:53.026019 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.025932    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 14 22:56:53.033102 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033078    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 22:56:53.033102 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033098    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 22:56:53.033102 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033102    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 22:56:53.033102 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033106    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 22:56:53.033102 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033109    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 22:56:53.033320 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033113    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 22:56:53.033320 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033115    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 22:56:53.033320 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033118    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 22:56:53.033320 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033121    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 22:56:53.033320 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033123    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 22:56:53.033320 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033126    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 22:56:53.033320 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033129    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 22:56:53.033320 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033131    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 22:56:53.033320 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033134    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 22:56:53.033320 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033136    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 22:56:53.033320 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033139    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 22:56:53.033320 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033142    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 22:56:53.033320 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033145    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 22:56:53.033320 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033147    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 22:56:53.033320 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033149    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 22:56:53.033320 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033152    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 22:56:53.033320 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033155    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 22:56:53.033320 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033157    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 22:56:53.033320 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033159    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 22:56:53.033320 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033162    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 22:56:53.033783 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033164    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 22:56:53.033783 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033167    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 22:56:53.033783 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033169    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 22:56:53.033783 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033172    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 22:56:53.033783 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033174    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 22:56:53.033783 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033177    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 22:56:53.033783 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033182    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 22:56:53.033783 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033185    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 22:56:53.033783 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033187    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 22:56:53.033783 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033190    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 22:56:53.033783 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033193    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 22:56:53.033783 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033195    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 22:56:53.033783 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033198    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 22:56:53.033783 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033201    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 22:56:53.033783 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033204    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 22:56:53.033783 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033206    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 22:56:53.033783 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033208    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 22:56:53.033783 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033211    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 22:56:53.033783 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033213    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 22:56:53.033783 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033216    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 22:56:53.034259 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033218    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 22:56:53.034259 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033221    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 22:56:53.034259 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033224    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 22:56:53.034259 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033226    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 22:56:53.034259 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033229    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 22:56:53.034259 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033232    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 22:56:53.034259 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033234    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 22:56:53.034259 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033237    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 22:56:53.034259 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033239    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 22:56:53.034259 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033241    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 22:56:53.034259 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033244    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 22:56:53.034259 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033246    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 22:56:53.034259 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033248    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 22:56:53.034259 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033251    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 22:56:53.034259 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033253    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 22:56:53.034259 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033256    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 22:56:53.034259 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033258    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 22:56:53.034259 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033261    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 22:56:53.034259 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033264    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 22:56:53.034724 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033268    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 22:56:53.034724 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033272    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 22:56:53.034724 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033275    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 22:56:53.034724 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033294    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 22:56:53.034724 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033297    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 22:56:53.034724 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033301    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 22:56:53.034724 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033304    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 22:56:53.034724 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033306    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 22:56:53.034724 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033309    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 22:56:53.034724 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033312    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 22:56:53.034724 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033314    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 22:56:53.034724 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033317    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 22:56:53.034724 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033320    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 22:56:53.034724 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033322    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 22:56:53.034724 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033326    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 22:56:53.034724 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033330    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 22:56:53.034724 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033333    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 22:56:53.034724 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033336    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 22:56:53.034724 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033339    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 22:56:53.035172 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033345    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 22:56:53.035172 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033349    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 22:56:53.035172 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033352    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 22:56:53.035172 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033354    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 22:56:53.035172 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033357    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 22:56:53.035172 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033360    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 22:56:53.035172 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033363    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 22:56:53.035172 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033365    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 22:56:53.035172 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033368    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 22:56:53.035172 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033371    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 22:56:53.035172 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033373    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 22:56:53.035172 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033376    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 22:56:53.035172 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033378    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 22:56:53.035172 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033381    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 22:56:53.035172 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033383    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 22:56:53.035172 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.033386    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 22:56:53.035172 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034081    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 22:56:53.035172 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034090    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 22:56:53.035172 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034095    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 22:56:53.035640 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034098    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 22:56:53.035640 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034101    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 22:56:53.035640 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034104    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 22:56:53.035640 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034107    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 22:56:53.035640 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034110    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 22:56:53.035640 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034113    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 22:56:53.035640 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034117    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 22:56:53.035640 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034120    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 22:56:53.035640 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034123    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 22:56:53.035640 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034126    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 22:56:53.035640 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034129    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 22:56:53.035640 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034131    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 22:56:53.035640 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034134    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 22:56:53.035640 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034136    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 22:56:53.035640 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034140    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 22:56:53.035640 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034142    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 22:56:53.035640 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034144    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 22:56:53.035640 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034147    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 22:56:53.035640 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034149    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 22:56:53.036107 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034151    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 22:56:53.036107 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034154    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 22:56:53.036107 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034156    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 22:56:53.036107 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034159    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 22:56:53.036107 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034161    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 22:56:53.036107 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034164    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 22:56:53.036107 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034166    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 22:56:53.036107 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034168    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 22:56:53.036107 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034171    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 22:56:53.036107 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034173    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 22:56:53.036107 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034176    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 22:56:53.036107 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034179    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 22:56:53.036107 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034182    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 22:56:53.036107 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034185    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 22:56:53.036107 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034187    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 22:56:53.036107 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034189    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 22:56:53.036107 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034192    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 22:56:53.036107 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034194    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 22:56:53.036107 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034197    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 22:56:53.036107 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034199    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 22:56:53.036619 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034202    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 22:56:53.036619 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034204    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 22:56:53.036619 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034206    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 22:56:53.036619 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034209    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 22:56:53.036619 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034211    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 22:56:53.036619 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034213    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 22:56:53.036619 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034216    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 22:56:53.036619 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034218    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 22:56:53.036619 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034221    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 22:56:53.036619 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034225    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 22:56:53.036619 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034227    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 22:56:53.036619 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034230    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 22:56:53.036619 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034232    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 22:56:53.036619 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034235    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 22:56:53.036619 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034237    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 22:56:53.036619 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034240    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 22:56:53.036619 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034242    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 22:56:53.036619 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034244    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 22:56:53.036619 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034246    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 22:56:53.036619 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034249    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 22:56:53.037114 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034252    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 22:56:53.037114 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034255    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 22:56:53.037114 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034259    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 22:56:53.037114 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034262    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 22:56:53.037114 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034266    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 22:56:53.037114 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034269    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 22:56:53.037114 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034272    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 22:56:53.037114 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034275    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 22:56:53.037114 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034294    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 22:56:53.037114 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034297    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 22:56:53.037114 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034299    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 22:56:53.037114 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034302    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 22:56:53.037114 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034304    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 22:56:53.037114 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034307    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 22:56:53.037114 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034309    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 22:56:53.037114 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034312    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 22:56:53.037114 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034314    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 22:56:53.037114 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034317    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 22:56:53.037114 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034319    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 22:56:53.037622 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034322    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 22:56:53.037622 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034324    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 22:56:53.037622 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034327    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 22:56:53.037622 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034330    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 22:56:53.037622 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034332    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 22:56:53.037622 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034335    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 22:56:53.037622 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034337    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 22:56:53.037622 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034340    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 22:56:53.037622 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034342    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 22:56:53.037622 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034345    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 22:56:53.037622 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034347    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 22:56:53.037622 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034350    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 22:56:53.037622 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034352    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 22:56:53.037622 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034355    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 22:56:53.037622 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034357    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 22:56:53.037622 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034360    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 22:56:53.037622 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034362    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 22:56:53.037622 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034365    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 22:56:53.037622 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034803    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 22:56:53.037622 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034809    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 22:56:53.038096 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034813    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 22:56:53.038096 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034817    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 22:56:53.038096 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034820    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 22:56:53.038096 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034822    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 22:56:53.038096 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034825    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 22:56:53.038096 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034828    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 22:56:53.038096 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034830    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 22:56:53.038096 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034833    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 22:56:53.038096 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034835    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 22:56:53.038096 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034838    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 22:56:53.038096 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034840    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 22:56:53.038096 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034843    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 22:56:53.038096 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034845    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 22:56:53.038096 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034848    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 22:56:53.038096 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034851    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 22:56:53.038096 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034853    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 22:56:53.038096 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034856    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 22:56:53.038096 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034858    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 22:56:53.038096 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034861    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 22:56:53.038096 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034863    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 22:56:53.038578 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034866    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 22:56:53.038578 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034868    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 22:56:53.038578 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034871    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 22:56:53.038578 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034873    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 22:56:53.038578 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034876    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 22:56:53.038578 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034878    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 22:56:53.038578 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034880    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 22:56:53.038578 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034883    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 22:56:53.038578 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034886    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 22:56:53.038578 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034889    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 22:56:53.038578 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034891    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 22:56:53.038578 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034894    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 22:56:53.038578 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034896    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 22:56:53.038578 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034899    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 22:56:53.038578 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034902    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 22:56:53.038578 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034905    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 22:56:53.038578 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034908    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 22:56:53.038578 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034911    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 22:56:53.038578 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034913    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 22:56:53.039066 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034916    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 22:56:53.039066 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034918    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 22:56:53.039066 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034920    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 22:56:53.039066 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034923    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 22:56:53.039066 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034925    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 22:56:53.039066 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034928    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 22:56:53.039066 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034930    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 22:56:53.039066 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034933    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 22:56:53.039066 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034936    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 22:56:53.039066 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034938    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 22:56:53.039066 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034941    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 22:56:53.039066 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034943    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 22:56:53.039066 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034947    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 22:56:53.039066 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034949    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 22:56:53.039066 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034951    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 22:56:53.039066 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034954    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 22:56:53.039066 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034956    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 22:56:53.039066 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034959    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 22:56:53.039066 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034961    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 22:56:53.039066 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034964    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 22:56:53.039562 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034966    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 22:56:53.039562 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034969    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 22:56:53.039562 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034972    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 22:56:53.039562 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034974    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 22:56:53.039562 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034977    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 22:56:53.039562 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034979    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 22:56:53.039562 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034982    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 22:56:53.039562 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034984    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 22:56:53.039562 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034987    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 22:56:53.039562 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034991    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 22:56:53.039562 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034993    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 22:56:53.039562 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034996    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 22:56:53.039562 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.034998    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 22:56:53.039562 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035001    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 22:56:53.039562 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035004    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 22:56:53.039562 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035007    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 22:56:53.039562 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035010    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 22:56:53.039562 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035013    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 22:56:53.039562 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035016    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 22:56:53.039562 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035018    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 22:56:53.040034 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035021    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 22:56:53.040034 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035024    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 22:56:53.040034 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035027    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 22:56:53.040034 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035030    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 22:56:53.040034 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035032    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 22:56:53.040034 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035035    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 22:56:53.040034 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035037    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 22:56:53.040034 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035039    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 22:56:53.040034 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035041    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 22:56:53.040034 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035044    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 22:56:53.040034 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035047    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 22:56:53.040034 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035050    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 22:56:53.040034 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035053    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 22:56:53.040034 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035056    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 22:56:53.040034 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035059    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 22:56:53.040034 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035062    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 22:56:53.040034 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035064    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 22:56:53.040034 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035067    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 22:56:53.040034 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035143    2563 flags.go:64] FLAG: --address="0.0.0.0"
Jul 14 22:56:53.040034 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035153    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 14 22:56:53.040567 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035162    2563 flags.go:64] FLAG: --anonymous-auth="true"
Jul 14 22:56:53.040567 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035169    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 14 22:56:53.040567 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035176    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 14 22:56:53.040567 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035181    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 14 22:56:53.040567 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035185    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 14 22:56:53.040567 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035190    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 14 22:56:53.040567 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035193    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 14 22:56:53.040567 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035196    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 14 22:56:53.040567 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035199    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 14 22:56:53.040567 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035202    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 14 22:56:53.040567 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035205    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 14 22:56:53.040567 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035208    2563 flags.go:64] FLAG: --cgroup-root=""
Jul 14 22:56:53.040567 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035211    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 14 22:56:53.040567 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035214    2563 flags.go:64] FLAG: --client-ca-file=""
Jul 14 22:56:53.040567 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035218    2563 flags.go:64] FLAG: --cloud-provider="external"
Jul 14 22:56:53.040567 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035221    2563 flags.go:64] FLAG: --cluster-dns="[]"
Jul 14 22:56:53.040567 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035226    2563 flags.go:64] FLAG: --cluster-domain=""
Jul 14 22:56:53.040567 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035229    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 14 22:56:53.040567 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035232    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 14 22:56:53.040567 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035235    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 14 22:56:53.040567 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035239    2563 flags.go:64] FLAG: --container-log-max-files="5"
Jul 14 22:56:53.040567 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035242    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 14 22:56:53.040567 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035245    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 14 22:56:53.041112 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035248    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 14 22:56:53.041112 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035251    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 14 22:56:53.041112 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035254    2563 flags.go:64] FLAG: --contention-profiling="false"
Jul 14 22:56:53.041112 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035257    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 14 22:56:53.041112 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035260    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 14 22:56:53.041112 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035263    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 14 22:56:53.041112 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035265    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 14 22:56:53.041112 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035269    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 14 22:56:53.041112 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035272    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 14 22:56:53.041112 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035275    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 14 22:56:53.041112 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035291    2563 flags.go:64] FLAG: --enable-load-reader="false"
Jul 14 22:56:53.041112 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035294    2563 flags.go:64] FLAG: --enable-server="true"
Jul 14 22:56:53.041112 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035297    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 14 22:56:53.041112 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035301    2563 flags.go:64] FLAG: --event-burst="100"
Jul 14 22:56:53.041112 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035304    2563 flags.go:64] FLAG: --event-qps="50"
Jul 14 22:56:53.041112 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035307    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 14 22:56:53.041112 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035310    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 14 22:56:53.041112 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035312    2563 flags.go:64] FLAG: --eviction-hard=""
Jul 14 22:56:53.041112 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035316    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 14 22:56:53.041112 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035321    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 14 22:56:53.041112 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035324    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 14 22:56:53.041112 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035327    2563 flags.go:64] FLAG: --eviction-soft=""
Jul 14 22:56:53.041112 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035331    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 14 22:56:53.041112 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035333    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 14 22:56:53.041112 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035337    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 14 22:56:53.041743 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035340    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 14 22:56:53.041743 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035342    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 14 22:56:53.041743 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035345    2563 flags.go:64] FLAG: --fail-swap-on="true"
Jul 14 22:56:53.041743 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035348    2563 flags.go:64] FLAG: --feature-gates=""
Jul 14 22:56:53.041743 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035352    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 14 22:56:53.041743 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035355    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 14 22:56:53.041743 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035358    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 14 22:56:53.041743 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035361    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 14 22:56:53.041743 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035364    2563 flags.go:64] FLAG: --healthz-port="10248"
Jul 14 22:56:53.041743 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035367    2563 flags.go:64] FLAG: --help="false"
Jul 14 22:56:53.041743 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035370    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-134-8.ec2.internal"
Jul 14 22:56:53.041743 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035373    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 14 22:56:53.041743 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035376    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 14 22:56:53.041743 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035378    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 14 22:56:53.041743 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035381    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 14 22:56:53.041743 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035385    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 14 22:56:53.041743 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035388    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 14 22:56:53.041743 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035390    2563 flags.go:64] FLAG: --image-service-endpoint=""
Jul 14 22:56:53.041743 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035393    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 14 22:56:53.041743 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035396    2563 flags.go:64] FLAG: --kube-api-burst="100"
Jul 14 22:56:53.041743 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035398    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 14 22:56:53.041743 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035401    2563 flags.go:64] FLAG: --kube-api-qps="50"
Jul 14 22:56:53.041743 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035404    2563 flags.go:64] FLAG: --kube-reserved=""
Jul 14 22:56:53.041743 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035406    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 14 22:56:53.042331 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035409    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 14 22:56:53.042331 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035412    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 14 22:56:53.042331 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035416    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 14 22:56:53.042331 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035419    2563 flags.go:64] FLAG: --lock-file=""
Jul 14 22:56:53.042331 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035422    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 14 22:56:53.042331 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035425    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 14 22:56:53.042331 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035428    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 14 22:56:53.042331 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035433    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 14 22:56:53.042331 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035436    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 14 22:56:53.042331 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035439    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 14 22:56:53.042331 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035442    2563 flags.go:64] FLAG: --logging-format="text"
Jul 14 22:56:53.042331 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035444    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 14 22:56:53.042331 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035448    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 14 22:56:53.042331 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035450    2563 flags.go:64] FLAG: --manifest-url=""
Jul 14 22:56:53.042331 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035453    2563 flags.go:64] FLAG: --manifest-url-header=""
Jul 14 22:56:53.042331 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035457    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 14 22:56:53.042331 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035460    2563 flags.go:64] FLAG: --max-open-files="1000000"
Jul 14 22:56:53.042331 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035464    2563 flags.go:64] FLAG: --max-pods="110"
Jul 14 22:56:53.042331 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035471    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 14 22:56:53.042331 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035474    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 14 22:56:53.042331 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035477    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 14 22:56:53.042331 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035480    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 14 22:56:53.042331 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035482    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 14 22:56:53.042331 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035485    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 14 22:56:53.042331 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035488    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 14 22:56:53.042989 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035495    2563 flags.go:64] FLAG: --node-status-max-images="50"
Jul 14 22:56:53.042989 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035498    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 14 22:56:53.042989 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035500    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 14 22:56:53.042989 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035504    2563 flags.go:64] FLAG: --pod-cidr=""
Jul 14 22:56:53.042989 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035508    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 14 22:56:53.042989 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035516    2563 flags.go:64] FLAG: --pod-manifest-path=""
Jul 14 22:56:53.042989 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035520    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 14 22:56:53.042989 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035523    2563 flags.go:64] FLAG: --pods-per-core="0"
Jul 14 22:56:53.042989 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035525    2563 flags.go:64] FLAG: --port="10250"
Jul 14 22:56:53.042989 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035529    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 14 22:56:53.042989 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035532    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0cd57c35c24225abd"
Jul 14 22:56:53.042989 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035535    2563 flags.go:64] FLAG: --qos-reserved=""
Jul 14 22:56:53.042989 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035538    2563 flags.go:64] FLAG: --read-only-port="10255"
Jul 14 22:56:53.042989 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035541    2563 flags.go:64] FLAG: --register-node="true"
Jul 14 22:56:53.042989 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035543    2563 flags.go:64] FLAG: --register-with-taints=""
Jul 14 22:56:53.042989 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035547    2563 flags.go:64] FLAG: --registry-burst="10"
Jul 14 22:56:53.042989 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035551    2563 flags.go:64] FLAG: --registry-qps="5"
Jul 14 22:56:53.042989 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035553    2563 flags.go:64] FLAG: --reserved-cpus=""
Jul 14 22:56:53.042989 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035556    2563 flags.go:64] FLAG: --reserved-memory=""
Jul 14 22:56:53.042989 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035560    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 14 22:56:53.042989 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035563    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 14 22:56:53.042989 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035566    2563 flags.go:64] FLAG: --rotate-certificates="false"
Jul 14 22:56:53.042989 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035568    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 14 22:56:53.042989 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035571    2563 flags.go:64] FLAG: --runonce="false"
Jul 14 22:56:53.042989 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035574    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 14 22:56:53.044040 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035576    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 14 22:56:53.044040 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035579    2563 flags.go:64] FLAG: --seccomp-default="false"
Jul 14 22:56:53.044040 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035582    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 14 22:56:53.044040 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035584    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 14 22:56:53.044040 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035587    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 14 22:56:53.044040 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035590    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 14 22:56:53.044040 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035593    2563 flags.go:64] FLAG: --storage-driver-password="root"
Jul 14 22:56:53.044040 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035595    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 14 22:56:53.044040 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035598    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 14 22:56:53.044040 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035601    2563 flags.go:64] FLAG: --storage-driver-user="root"
Jul 14 22:56:53.044040 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035603    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 14 22:56:53.044040 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035606    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 14 22:56:53.044040 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035609    2563 flags.go:64] FLAG: --system-cgroups=""
Jul 14 22:56:53.044040 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035611    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 14 22:56:53.044040 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035617    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 14 22:56:53.044040 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035619    2563 flags.go:64] FLAG: --tls-cert-file=""
Jul 14 22:56:53.044040 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035623    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 14 22:56:53.044040 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035628    2563 flags.go:64] FLAG: --tls-min-version=""
Jul 14 22:56:53.044040 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035630    2563 flags.go:64] FLAG: --tls-private-key-file=""
Jul 14 22:56:53.044040 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035633    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 14 22:56:53.044040 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035635    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 14 22:56:53.044040 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035638    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 14 22:56:53.044040 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035641    2563 flags.go:64] FLAG: --v="2"
Jul 14 22:56:53.044040 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035645    2563 flags.go:64] FLAG: --version="false"
Jul 14 22:56:53.044040 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035650    2563 flags.go:64] FLAG: --vmodule=""
Jul 14 22:56:53.045130 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035656    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 14 22:56:53.045130 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.035659    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 14 22:56:53.045130 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035795    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 22:56:53.045130 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035799    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 22:56:53.045130 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035802    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 22:56:53.045130 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035805    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 22:56:53.045130 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035807    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 22:56:53.045130 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035810    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 22:56:53.045130 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035812    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 22:56:53.045130 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035815    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 22:56:53.045130 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035817    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 22:56:53.045130 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035820    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 22:56:53.045130 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035822    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 22:56:53.045130 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035824    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 22:56:53.045130 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035827    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 22:56:53.045130 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035830    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 22:56:53.045130 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035834    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 22:56:53.045130 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035837    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 22:56:53.045130 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035840    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 22:56:53.045947 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035843    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 22:56:53.045947 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035846    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 22:56:53.045947 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035848    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 22:56:53.045947 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035851    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 22:56:53.045947 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035856    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 22:56:53.045947 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035859    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 22:56:53.045947 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035861    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 22:56:53.045947 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035864    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 22:56:53.045947 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035867    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 22:56:53.045947 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035869    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 22:56:53.045947 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035872    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 22:56:53.045947 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035874    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 22:56:53.045947 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035877    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 22:56:53.045947 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035887    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 22:56:53.045947 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035890    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 22:56:53.045947 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035894    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 22:56:53.045947 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035897    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 22:56:53.045947 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035900    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 22:56:53.045947 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035903    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 22:56:53.045947 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035905    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 22:56:53.046836 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035908    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 22:56:53.046836 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035911    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 22:56:53.046836 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035913    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 22:56:53.046836 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035916    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 22:56:53.046836 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035919    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 22:56:53.046836 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035921    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 22:56:53.046836 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035924    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 22:56:53.046836 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035926    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 22:56:53.046836 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035929    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 22:56:53.046836 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035931    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 22:56:53.046836 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035933    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 22:56:53.046836 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035936    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 22:56:53.046836 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035940    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 22:56:53.046836 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035943    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 22:56:53.046836 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035946    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 22:56:53.046836 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035948    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 22:56:53.046836 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035952    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 22:56:53.046836 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035954    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 22:56:53.046836 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035957    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 22:56:53.046836 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035959    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 22:56:53.047410 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035961    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 22:56:53.047410 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035964    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 22:56:53.047410 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035967    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 22:56:53.047410 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035969    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 22:56:53.047410 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035971    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 22:56:53.047410 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035975    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 22:56:53.047410 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035978    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 22:56:53.047410 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035980    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 22:56:53.047410 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035982    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 22:56:53.047410 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035985    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 22:56:53.047410 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035987    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 22:56:53.047410 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035989    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 22:56:53.047410 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035992    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 22:56:53.047410 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035995    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 22:56:53.047410 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035997    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 22:56:53.047410 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.035999    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 22:56:53.047410 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.036002    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 22:56:53.047410 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.036004    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 22:56:53.047410 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.036007    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 22:56:53.048179 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.036010    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 22:56:53.048179 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.036012    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 22:56:53.048179 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.036014    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 22:56:53.048179 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.036017    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 22:56:53.048179 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.036019    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 22:56:53.048179 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.036022    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 22:56:53.048179 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.036024    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 22:56:53.048179 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.036026    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 22:56:53.048179 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.036029    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 22:56:53.048179 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.036034    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 22:56:53.048179 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.036036    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 22:56:53.048179 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.036038    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 22:56:53.048179 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.036041    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 22:56:53.048179 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.036044    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 22:56:53.048179 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.036046    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 22:56:53.048179 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.036049    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 22:56:53.048179 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.036052    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 22:56:53.048179 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.036054    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 22:56:53.048179 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.036058    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 22:56:53.048179 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.036060    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 22:56:53.048945 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.036063    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 22:56:53.048945 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.036066    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 22:56:53.048945 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.036068    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 22:56:53.048945 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.036977    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 14 22:56:53.048945 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.045190    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 14 22:56:53.048945 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.045210    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 14 22:56:53.048945 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.045239    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 14 22:56:53.048945 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.045252    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 14 22:56:53.048945 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045365    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 22:56:53.048945 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045374    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 22:56:53.048945 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045379    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 22:56:53.048945 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045383    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 22:56:53.048945 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045388    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 22:56:53.048945 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045393    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 22:56:53.048945 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045397    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 22:56:53.049335 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045401    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 22:56:53.049335 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045405    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 22:56:53.049335 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045410    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 22:56:53.049335 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045414    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 22:56:53.049335 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045418    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 22:56:53.049335 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045422    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 22:56:53.049335 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045426    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 22:56:53.049335 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045430    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 22:56:53.049335 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045434    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 22:56:53.049335 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045439    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 22:56:53.049335 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045443    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 22:56:53.049335 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045447    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 22:56:53.049335 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045451    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 22:56:53.049335 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045455    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 22:56:53.049335 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045460    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 22:56:53.049335 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045464    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 22:56:53.049335 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045468    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 22:56:53.049335 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045473    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 22:56:53.049335 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045477    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 22:56:53.049335 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045481    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 22:56:53.049946 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045485    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 22:56:53.049946 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045489    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 22:56:53.049946 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045494    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 22:56:53.049946 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045498    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 22:56:53.049946 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045503    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 22:56:53.049946 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045508    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 22:56:53.049946 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045512    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 22:56:53.049946 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045516    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 22:56:53.049946 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045520    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 22:56:53.049946 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045524    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 22:56:53.049946 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045528    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 22:56:53.049946 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045532    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 22:56:53.049946 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045536    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 22:56:53.049946 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045540    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 22:56:53.049946 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045544    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 22:56:53.049946 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045552    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 22:56:53.049946 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045558    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 22:56:53.049946 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045563    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 22:56:53.049946 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045567    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 22:56:53.049946 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045571    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 22:56:53.050471 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045575    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 22:56:53.050471 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045579    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 22:56:53.050471 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045583    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 22:56:53.050471 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045587    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 22:56:53.050471 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045591    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 22:56:53.050471 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045595    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 22:56:53.050471 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045599    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 22:56:53.050471 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045604    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 22:56:53.050471 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045608    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 22:56:53.050471 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045612    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 22:56:53.050471 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045618    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 22:56:53.050471 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045622    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 22:56:53.050471 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045627    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 22:56:53.050471 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045631    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 22:56:53.050471 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045636    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 22:56:53.050471 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045640    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 22:56:53.050471 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045644    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 22:56:53.050471 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045648    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 22:56:53.050471 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045652    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 22:56:53.050471 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045656    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 22:56:53.050955 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045660    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 22:56:53.050955 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045664    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 22:56:53.050955 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045671    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 22:56:53.050955 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045676    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 22:56:53.050955 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045681    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 22:56:53.050955 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045687    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 22:56:53.050955 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045691    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 22:56:53.050955 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045695    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 22:56:53.050955 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045699    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 22:56:53.050955 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045704    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 22:56:53.050955 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045708    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 22:56:53.050955 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045712    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 22:56:53.050955 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045716    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 22:56:53.050955 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045720    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 22:56:53.050955 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045724    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 22:56:53.050955 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045728    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 22:56:53.050955 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045733    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 22:56:53.050955 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045737    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 22:56:53.050955 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045741    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 22:56:53.051630 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045747    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 22:56:53.051630 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045752    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 22:56:53.051630 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045757    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 22:56:53.051630 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045761    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 22:56:53.051630 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045766    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 22:56:53.051630 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045770    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 22:56:53.051630 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045774    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 22:56:53.051630 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045779    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 22:56:53.051630 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045783    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 22:56:53.051630 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045787    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 22:56:53.051630 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045792    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 22:56:53.051630 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045796    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 22:56:53.051630 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.045800    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 22:56:53.051630 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.045809    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 14 22:56:53.051630 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046011    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 22:56:53.052178 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046019    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 22:56:53.052178 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046024    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 22:56:53.052178 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046028    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 22:56:53.052178 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046033    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 22:56:53.052178 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046037    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 22:56:53.052178 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046041    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 22:56:53.052178 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046045    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 22:56:53.052178 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046049    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 22:56:53.052178 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046053    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 22:56:53.052178 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046058    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 22:56:53.052178 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046062    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 22:56:53.052178 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046065    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 22:56:53.052178 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046069    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 22:56:53.052178 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046076    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 22:56:53.052178 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046082    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 22:56:53.052178 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046086    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 22:56:53.052178 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046091    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 22:56:53.052178 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046095    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 22:56:53.052178 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046099    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 22:56:53.052178 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046103    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 22:56:53.052688 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046107    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 22:56:53.052688 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046111    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 22:56:53.052688 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046116    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 22:56:53.052688 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046120    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 22:56:53.052688 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046124    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 22:56:53.052688 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046129    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 22:56:53.052688 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046133    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 22:56:53.052688 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046137    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 22:56:53.052688 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046141    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 22:56:53.052688 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046145    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 22:56:53.052688 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046149    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 22:56:53.052688 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046153    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 22:56:53.052688 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046157    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 22:56:53.052688 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046162    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 22:56:53.052688 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046166    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 22:56:53.052688 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046170    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 22:56:53.052688 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046174    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 22:56:53.052688 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046178    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 22:56:53.052688 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046182    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 22:56:53.052688 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046186    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 22:56:53.053198 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046190    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 22:56:53.053198 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046194    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 22:56:53.053198 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046198    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 22:56:53.053198 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046202    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 22:56:53.053198 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046206    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 22:56:53.053198 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046213    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 22:56:53.053198 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046218    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 22:56:53.053198 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046222    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 22:56:53.053198 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046227    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 22:56:53.053198 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046230    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 22:56:53.053198 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046235    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 22:56:53.053198 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046239    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 22:56:53.053198 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046243    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 22:56:53.053198 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046247    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 22:56:53.053198 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046251    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 22:56:53.053198 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046256    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 22:56:53.053198 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046261    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 22:56:53.053198 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046265    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 22:56:53.053198 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046270    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 22:56:53.053693 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046273    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 22:56:53.053693 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046297    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 22:56:53.053693 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046301    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 22:56:53.053693 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046305    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 22:56:53.053693 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046309    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 22:56:53.053693 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046313    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 22:56:53.053693 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046316    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 22:56:53.053693 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046320    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 22:56:53.053693 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046323    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 22:56:53.053693 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046327    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 22:56:53.053693 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046331    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 22:56:53.053693 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046335    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 22:56:53.053693 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046338    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 22:56:53.053693 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046342    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 22:56:53.053693 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046346    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 22:56:53.053693 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046349    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 22:56:53.053693 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046353    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 22:56:53.053693 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046357    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 22:56:53.053693 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046361    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 22:56:53.053693 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046365    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 22:56:53.054534 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046369    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 22:56:53.054534 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046373    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 22:56:53.054534 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046378    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 22:56:53.054534 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046382    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 22:56:53.054534 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046387    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 22:56:53.054534 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046391    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 22:56:53.054534 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046395    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 22:56:53.054534 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046399    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 22:56:53.054534 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046403    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 22:56:53.054534 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046408    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 22:56:53.054534 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046413    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 22:56:53.054534 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046417    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 22:56:53.054534 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046422    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 22:56:53.054534 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046429    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 22:56:53.054534 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046434    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 22:56:53.054534 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046439    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 22:56:53.054534 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046444    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 22:56:53.054534 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046448    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 22:56:53.054534 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:53.046452    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 22:56:53.055332 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.046459    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 14 22:56:53.055332 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.046607    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 14 22:56:53.055332 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.049492    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 14 22:56:53.055332 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.051131    2563 server.go:1019] "Starting client certificate rotation"
Jul 14 22:56:53.055332 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.051235    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 14 22:56:53.055332 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.051982    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 14 22:56:53.055332 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.054353    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 14 22:56:53.056412 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.056393    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 14 22:56:53.067794 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.067776    2563 log.go:25] "Validated CRI v1 runtime API"
Jul 14 22:56:53.070322 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.070306    2563 log.go:25] "Validated CRI v1 image API"
Jul 14 22:56:53.071108 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.071093    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 14 22:56:53.073742 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.073719    2563 fs.go:135] Filesystem UUIDs: map[5d5d36d2-56c2-4e47-ae6f-a6eaac88b7da:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c06e536b-6e0b-46af-bd01-bf3239b72c5e:/dev/nvme0n1p4]
Jul 14 22:56:53.073823 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.073742    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 14 22:56:53.079775 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.079651    2563 manager.go:216] Machine: {Timestamp:2026-07-14 22:56:53.078345809 +0000 UTC m=+0.381336611 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099482 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a75b967fc40626ad95de1519d7aa6 SystemUUID:ec2a75b9-67fc-4062-6ad9-5de1519d7aa6 BootID:b8df1414-cba0-43ee-a399-618c541f40b4 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c0:b8:d8:c8:ab Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c0:b8:d8:c8:ab Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ae:1b:a0:b5:ce:63 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 14 22:56:53.079775 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.079766    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 14 22:56:53.079948 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.079930    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 14 22:56:53.082433 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.082417    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 14 22:56:53.083377 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.083349    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 14 22:56:53.083512 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.083381    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-8.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 14 22:56:53.084245 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.084234    2563 topology_manager.go:138] "Creating topology manager with none policy"
Jul 14 22:56:53.084299 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.084248    2563 container_manager_linux.go:309] "Creating device plugin manager"
Jul 14 22:56:53.084299 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.084256    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 14 22:56:53.085188 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.085177    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 14 22:56:53.087058 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.087048    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 14 22:56:53.090933 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.090919    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 14 22:56:53.091069 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.091059    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 14 22:56:53.091107 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.091090    2563 kubelet.go:486] "Attempting to sync node with API server"
Jul 14 22:56:53.091107 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.091100    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 14 22:56:53.091194 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.091117    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 14 22:56:53.091194 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.091127    2563 kubelet.go:398] "Adding apiserver pod source"
Jul 14 22:56:53.091194 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.091138    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 14 22:56:53.091567 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.091555    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 14 22:56:53.091602 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.091576    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 14 22:56:53.094077 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.094061    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 14 22:56:53.096365 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.096353    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 14 22:56:53.096422 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.096382    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 14 22:56:53.096554 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.096544    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 14 22:56:53.096602 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.096559    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 14 22:56:53.096602 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.096566    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 14 22:56:53.096602 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.096571    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 14 22:56:53.096602 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.096579    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 14 22:56:53.096602 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.096584    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 14 22:56:53.097189 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.097181    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 14 22:56:53.097217 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.097189    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 14 22:56:53.097217 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.097196    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 14 22:56:53.097217 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.097201    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 14 22:56:53.097217 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.097217    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 14 22:56:53.097349 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.097225    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 14 22:56:53.098124 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.098113    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 14 22:56:53.098124 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.098124    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 14 22:56:53.101712 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.101695    2563 server.go:1268] "Started kubelet"
Jul 14 22:56:53.101858 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.101803    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 14 22:56:53.101913 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.101883    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 14 22:56:53.102004 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.101966    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 14 22:56:53.102620 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.102597    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-8.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 14 22:56:53.102712 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:53.102654    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-8.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 14 22:56:53.102773 ip-10-0-134-8 systemd[1]: Started Kubernetes Kubelet.
Jul 14 22:56:53.103243 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:53.103091    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 14 22:56:53.103243 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.103228    2563 server.go:310] "Adding debug handlers to kubelet server"
Jul 14 22:56:53.106368 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.106350    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 14 22:56:53.108407 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:53.108379    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 14 22:56:53.111133 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:53.110041    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-8.ec2.internal.18c24a0ae86c2aee  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-8.ec2.internal,UID:ip-10-0-134-8.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-8.ec2.internal,},FirstTimestamp:2026-07-14 22:56:53.101669102 +0000 UTC m=+0.404659886,LastTimestamp:2026-07-14 22:56:53.101669102 +0000 UTC m=+0.404659886,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-8.ec2.internal,}"
Jul 14 22:56:53.111342 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.111326    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qm55n"
Jul 14 22:56:53.114948 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.114923    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 14 22:56:53.114948 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.114935    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 14 22:56:53.115087 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:53.115001    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-8.ec2.internal\" not found"
Jul 14 22:56:53.115810 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.115793    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 14 22:56:53.115810 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.115795    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 14 22:56:53.115937 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.115821    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 14 22:56:53.115937 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.115930    2563 reconstruct.go:97] "Volume reconstruction finished"
Jul 14 22:56:53.115937 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.115938    2563 reconciler.go:29] "Reconciler: start to sync state"
Jul 14 22:56:53.116225 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.116204    2563 factory.go:55] Registering systemd factory
Jul 14 22:56:53.116358 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.116233    2563 factory.go:223] Registration of the systemd container factory successfully
Jul 14 22:56:53.116510 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.116485    2563 factory.go:153] Registering CRI-O factory
Jul 14 22:56:53.116575 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.116514    2563 factory.go:223] Registration of the crio container factory successfully
Jul 14 22:56:53.116575 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.116568    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 14 22:56:53.116670 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.116599    2563 factory.go:103] Registering Raw factory
Jul 14 22:56:53.116670 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.116616    2563 manager.go:1196] Started watching for new ooms in manager
Jul 14 22:56:53.117077 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.117034    2563 manager.go:349] Starting recovery of all containers
Jul 14 22:56:53.117310 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:53.117264    2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-8.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 14 22:56:53.117448 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:53.117423    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 14 22:56:53.118311 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.118270    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qm55n"
Jul 14 22:56:53.123256 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.123225    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 14 22:56:53.127373 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.127345    2563 manager.go:354] Recovery completed
Jul 14 22:56:53.131817 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.131804    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 22:56:53.133820 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.133802    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 22:56:53.133884 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.133839    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 22:56:53.133884 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.133851    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasSufficientPID"
Jul 14 22:56:53.134359 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.134343    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 14 22:56:53.134359 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.134358    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 14 22:56:53.134543 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.134378    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 14 22:56:53.134543 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.134384    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 22:56:53.135072 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.135015    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 22:56:53.135072 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.135061    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 22:56:53.135181 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.135076    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasSufficientPID"
Jul 14 22:56:53.135708 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.135696    2563 policy_none.go:49] "None policy: Start"
Jul 14 22:56:53.135755 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.135712    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 14 22:56:53.135755 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.135722    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 14 22:56:53.136848 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.136837    2563 policy_none.go:47] "Start"
Jul 14 22:56:53.175089 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.174932    2563 manager.go:337] "Starting Device Plugin manager"
Jul 14 22:56:53.175089 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:53.175105    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 14 22:56:53.191843 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.175116    2563 server.go:87] "Starting device plugin registration server"
Jul 14 22:56:53.191843 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.175383    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 14 22:56:53.191843 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.175394    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 14 22:56:53.191843 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.175482    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 14 22:56:53.191843 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.175560    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 14 22:56:53.191843 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.175568    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 14 22:56:53.191843 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:53.177139    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 14 22:56:53.191843 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:53.177185    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-8.ec2.internal\" not found"
Jul 14 22:56:53.223204 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.223173    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 14 22:56:53.223313 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.223221    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 14 22:56:53.223313 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.223245    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 14 22:56:53.223391 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:53.223302    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 14 22:56:53.225873 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.225846    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 22:56:53.276266 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.276180    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 22:56:53.278001 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.277590    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 22:56:53.278001 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.277623    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 22:56:53.278001 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.277638    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasSufficientPID"
Jul 14 22:56:53.278001 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.277667    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-8.ec2.internal"
Jul 14 22:56:53.285390 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.285369    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-8.ec2.internal"
Jul 14 22:56:53.285494 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:53.285396    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-8.ec2.internal\": node \"ip-10-0-134-8.ec2.internal\" not found"
Jul 14 22:56:53.307024 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:53.306987    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-8.ec2.internal\" not found"
Jul 14 22:56:53.323535 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.323503    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-8.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-8.ec2.internal"]
Jul 14 22:56:53.323627 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.323582    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 22:56:53.325297 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.325264    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 22:56:53.325394 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.325322    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 22:56:53.325394 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.325340    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasSufficientPID"
Jul 14 22:56:53.327644 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.327626    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 22:56:53.327768 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.327749    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-8.ec2.internal"
Jul 14 22:56:53.327816 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.327793    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 22:56:53.328442 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.328420    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 22:56:53.328544 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.328450    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 22:56:53.328544 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.328421    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 22:56:53.328544 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.328495    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 22:56:53.328544 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.328463    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasSufficientPID"
Jul 14 22:56:53.328544 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.328516    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasSufficientPID"
Jul 14 22:56:53.330654 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.330640    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-8.ec2.internal"
Jul 14 22:56:53.330750 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.330667    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 22:56:53.331462 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.331446    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 22:56:53.331539 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.331478    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 22:56:53.331539 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.331494    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeHasSufficientPID"
Jul 14 22:56:53.347903 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:53.347872    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-8.ec2.internal\" not found" node="ip-10-0-134-8.ec2.internal"
Jul 14 22:56:53.351686 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:53.351667    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-8.ec2.internal\" not found" node="ip-10-0-134-8.ec2.internal"
Jul 14 22:56:53.407883 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:53.407842    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-8.ec2.internal\" not found"
Jul 14 22:56:53.418295 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.418249    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3aa82a5123efb70e24ea524a0aeee7ed-config\") pod \"kube-apiserver-proxy-ip-10-0-134-8.ec2.internal\" (UID: \"3aa82a5123efb70e24ea524a0aeee7ed\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-8.ec2.internal"
Jul 14 22:56:53.418419 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.418294    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e408b8465ac9f8b60353a9a16a836248-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-8.ec2.internal\" (UID: \"e408b8465ac9f8b60353a9a16a836248\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-8.ec2.internal"
Jul 14 22:56:53.418419 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.418323    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e408b8465ac9f8b60353a9a16a836248-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-8.ec2.internal\" (UID: \"e408b8465ac9f8b60353a9a16a836248\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-8.ec2.internal"
Jul 14 22:56:53.507961 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:53.507921    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-8.ec2.internal\" not found"
Jul 14 22:56:53.519326 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.519277    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e408b8465ac9f8b60353a9a16a836248-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-8.ec2.internal\" (UID: \"e408b8465ac9f8b60353a9a16a836248\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-8.ec2.internal"
Jul 14 22:56:53.519326 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.519306    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e408b8465ac9f8b60353a9a16a836248-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-8.ec2.internal\" (UID: \"e408b8465ac9f8b60353a9a16a836248\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-8.ec2.internal"
Jul 14 22:56:53.519479 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.519353    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e408b8465ac9f8b60353a9a16a836248-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-8.ec2.internal\" (UID: \"e408b8465ac9f8b60353a9a16a836248\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-8.ec2.internal"
Jul 14 22:56:53.519479 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.519388    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e408b8465ac9f8b60353a9a16a836248-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-8.ec2.internal\" (UID: \"e408b8465ac9f8b60353a9a16a836248\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-8.ec2.internal"
Jul 14 22:56:53.519479 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.519393    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3aa82a5123efb70e24ea524a0aeee7ed-config\") pod \"kube-apiserver-proxy-ip-10-0-134-8.ec2.internal\" (UID: \"3aa82a5123efb70e24ea524a0aeee7ed\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-8.ec2.internal"
Jul 14 22:56:53.519479 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.519438    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3aa82a5123efb70e24ea524a0aeee7ed-config\") pod \"kube-apiserver-proxy-ip-10-0-134-8.ec2.internal\" (UID: \"3aa82a5123efb70e24ea524a0aeee7ed\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-8.ec2.internal"
Jul 14 22:56:53.608707 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:53.608638    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-8.ec2.internal\" not found"
Jul 14 22:56:53.651202 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.651163    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-8.ec2.internal"
Jul 14 22:56:53.654689 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:53.654671    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-8.ec2.internal"
Jul 14 22:56:53.709704 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:53.709665    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-8.ec2.internal\" not found"
Jul 14 22:56:53.810229 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:53.810196    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-8.ec2.internal\" not found"
Jul 14 22:56:53.910857 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:53.910765    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-8.ec2.internal\" not found"
Jul 14 22:56:54.011267 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:54.011233    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-8.ec2.internal\" not found"
Jul 14 22:56:54.050826 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:54.050770    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 14 22:56:54.051306 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:54.050947    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 14 22:56:54.106218 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:54.106188    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 22:56:54.111549 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:54.111523    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-8.ec2.internal\" not found"
Jul 14 22:56:54.112439 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:54.112422    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 22:56:54.122207 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:54.122168    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-13 22:51:54 +0000 UTC" deadline="2028-02-28 21:05:44.330631314 +0000 UTC"
Jul 14 22:56:54.122207 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:54.122205    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14254h8m50.208430907s"
Jul 14 22:56:54.190177 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:54.190148    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 22:56:54.215979 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:54.215950    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-8.ec2.internal"
Jul 14 22:56:54.232690 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:54.232663    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 14 22:56:54.233689 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:54.233673    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-8.ec2.internal"
Jul 14 22:56:54.244590 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:54.244559    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 14 22:56:54.302680 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:54.302643    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa82a5123efb70e24ea524a0aeee7ed.slice/crio-742a2b1bca7df551206d4aa117bbd1dde57f15c7b042aea6b9131ef70352b6c2 WatchSource:0}: Error finding container 742a2b1bca7df551206d4aa117bbd1dde57f15c7b042aea6b9131ef70352b6c2: Status 404 returned error can't find the container with id 742a2b1bca7df551206d4aa117bbd1dde57f15c7b042aea6b9131ef70352b6c2
Jul 14 22:56:54.304485 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:54.304456    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode408b8465ac9f8b60353a9a16a836248.slice/crio-0bd3d732e657e45ea5a8adc3cad85677acbb2b7c070b7dde3e203534c8346f03 WatchSource:0}: Error finding container 0bd3d732e657e45ea5a8adc3cad85677acbb2b7c070b7dde3e203534c8346f03: Status 404 returned error can't find the container with id 0bd3d732e657e45ea5a8adc3cad85677acbb2b7c070b7dde3e203534c8346f03
Jul 14 22:56:54.309356 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:54.309336    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 22:56:55.092210 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.092177    2563 apiserver.go:52] "Watching apiserver"
Jul 14 22:56:55.102881 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.102683    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 14 22:56:55.103078 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.103048    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwzlt","openshift-dns/node-resolver-s4zgk","openshift-image-registry/node-ca-lzt25","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-8.ec2.internal","openshift-multus/multus-additional-cni-plugins-r5s76","openshift-multus/multus-vlbvv","kube-system/global-pull-secret-syncer-xjw4z","openshift-cluster-node-tuning-operator/tuned-7wnl4","openshift-multus/network-metrics-daemon-wjpcm","openshift-network-diagnostics/network-check-target-pbtpg","openshift-network-operator/iptables-alerter-c8c42","openshift-ovn-kubernetes/ovnkube-node-j2z4s","kube-system/konnectivity-agent-pthrz","kube-system/kube-apiserver-proxy-ip-10-0-134-8.ec2.internal"]
Jul 14 22:56:55.105681 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.105656    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pbtpg"
Jul 14 22:56:55.105795 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:55.105757    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pbtpg" podUID="39492113-cc47-4511-bc7d-d0785b892ed9"
Jul 14 22:56:55.108196 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.107795    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pthrz"
Jul 14 22:56:55.109984 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.109964    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lzt25"
Jul 14 22:56:55.110553 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.110536    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-lp8pr\""
Jul 14 22:56:55.110647 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.110607    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 14 22:56:55.110875 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.110857    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 14 22:56:55.114758 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.112681    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7wnl4"
Jul 14 22:56:55.114758 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.112796    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 14 22:56:55.114758 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.113356    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 14 22:56:55.114758 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.114367    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 14 22:56:55.115725 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.115555    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jxksr\""
Jul 14 22:56:55.115725 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.115602    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 14 22:56:55.115845 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.115736    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-r5s76"
Jul 14 22:56:55.115910 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.115893    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 14 22:56:55.116361 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.116341    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rcjxj\""
Jul 14 22:56:55.116456 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.116360    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 14 22:56:55.117831 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.117812    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.118727 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.118709    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 14 22:56:55.119129 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.119006    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 14 22:56:55.119214 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.119189    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 14 22:56:55.119672 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.119396    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 14 22:56:55.119672 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.119487    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pxzld\""
Jul 14 22:56:55.119672 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.119586    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 14 22:56:55.120555 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.120531    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 14 22:56:55.120769 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.120751    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4rffc\""
Jul 14 22:56:55.121066 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.121050    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wjpcm"
Jul 14 22:56:55.121136 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:55.121116    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wjpcm" podUID="8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594"
Jul 14 22:56:55.123457 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.123435    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xjw4z"
Jul 14 22:56:55.123588 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:55.123499    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xjw4z" podUID="7e3e444e-1753-442e-8430-24b0022f9418"
Jul 14 22:56:55.125855 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.125838    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s4zgk"
Jul 14 22:56:55.126545 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.126521    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/da7d977f-1dd0-42e6-b3a7-455490097670-cni-binary-copy\") pod \"multus-additional-cni-plugins-r5s76\" (UID: \"da7d977f-1dd0-42e6-b3a7-455490097670\") " pod="openshift-multus/multus-additional-cni-plugins-r5s76"
Jul 14 22:56:55.126641 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.126559    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-multus-conf-dir\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.126641 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.126583    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1ef7f81d-618b-4d9a-b018-9e3346c27bd1-serviceca\") pod \"node-ca-lzt25\" (UID: \"1ef7f81d-618b-4d9a-b018-9e3346c27bd1\") " pod="openshift-image-registry/node-ca-lzt25"
Jul 14 22:56:55.126641 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.126604    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eda746ab-d0ce-4007-9db1-f6ee9cbdc317-etc-sysctl-d\") pod \"tuned-7wnl4\" (UID: \"eda746ab-d0ce-4007-9db1-f6ee9cbdc317\") " pod="openshift-cluster-node-tuning-operator/tuned-7wnl4"
Jul 14 22:56:55.127132 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.127111    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eda746ab-d0ce-4007-9db1-f6ee9cbdc317-etc-sysctl-conf\") pod \"tuned-7wnl4\" (UID: \"eda746ab-d0ce-4007-9db1-f6ee9cbdc317\") " pod="openshift-cluster-node-tuning-operator/tuned-7wnl4"
Jul 14 22:56:55.127208 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.127150    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eda746ab-d0ce-4007-9db1-f6ee9cbdc317-sys\") pod \"tuned-7wnl4\" (UID: \"eda746ab-d0ce-4007-9db1-f6ee9cbdc317\") " pod="openshift-cluster-node-tuning-operator/tuned-7wnl4"
Jul 14 22:56:55.127208 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.127173    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eda746ab-d0ce-4007-9db1-f6ee9cbdc317-host\") pod \"tuned-7wnl4\" (UID: \"eda746ab-d0ce-4007-9db1-f6ee9cbdc317\") " pod="openshift-cluster-node-tuning-operator/tuned-7wnl4"
Jul 14 22:56:55.127330 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.127212    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/da7d977f-1dd0-42e6-b3a7-455490097670-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r5s76\" (UID: \"da7d977f-1dd0-42e6-b3a7-455490097670\") " pod="openshift-multus/multus-additional-cni-plugins-r5s76"
Jul 14 22:56:55.127330 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.127258    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-cnibin\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.127330 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.127311    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-cni-binary-copy\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.127458 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.127339    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-multus-socket-dir-parent\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.127458 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.127366    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-host-var-lib-cni-bin\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.127458 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.127388    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-hostroot\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.127458 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.127411    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-multus-daemon-config\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.127458 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.127449    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/da7d977f-1dd0-42e6-b3a7-455490097670-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r5s76\" (UID: \"da7d977f-1dd0-42e6-b3a7-455490097670\") " pod="openshift-multus/multus-additional-cni-plugins-r5s76"
Jul 14 22:56:55.127691 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.127479    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-host-run-multus-certs\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.127691 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.127505    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/186005d6-1e47-4ea0-816c-469b21cc77d4-konnectivity-ca\") pod \"konnectivity-agent-pthrz\" (UID: \"186005d6-1e47-4ea0-816c-469b21cc77d4\") " pod="kube-system/konnectivity-agent-pthrz"
Jul 14 22:56:55.127691 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.127529    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eda746ab-d0ce-4007-9db1-f6ee9cbdc317-etc-modprobe-d\") pod \"tuned-7wnl4\" (UID: \"eda746ab-d0ce-4007-9db1-f6ee9cbdc317\") " pod="openshift-cluster-node-tuning-operator/tuned-7wnl4"
Jul 14 22:56:55.127691 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.127552    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eda746ab-d0ce-4007-9db1-f6ee9cbdc317-var-lib-kubelet\") pod \"tuned-7wnl4\" (UID: \"eda746ab-d0ce-4007-9db1-f6ee9cbdc317\") " pod="openshift-cluster-node-tuning-operator/tuned-7wnl4"
Jul 14 22:56:55.127691 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.127578    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/da7d977f-1dd0-42e6-b3a7-455490097670-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r5s76\" (UID: \"da7d977f-1dd0-42e6-b3a7-455490097670\") " pod="openshift-multus/multus-additional-cni-plugins-r5s76"
Jul 14 22:56:55.128217 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.128198    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwzlt"
Jul 14 22:56:55.128448 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.128421    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59km6\" (UniqueName: \"kubernetes.io/projected/da7d977f-1dd0-42e6-b3a7-455490097670-kube-api-access-59km6\") pod \"multus-additional-cni-plugins-r5s76\" (UID: \"da7d977f-1dd0-42e6-b3a7-455490097670\") " pod="openshift-multus/multus-additional-cni-plugins-r5s76"
Jul 14 22:56:55.128556 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.128462    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-os-release\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.128556 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.128487    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vg7c\" (UniqueName: \"kubernetes.io/projected/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-kube-api-access-8vg7c\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.128556 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.128520    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eda746ab-d0ce-4007-9db1-f6ee9cbdc317-run\") pod \"tuned-7wnl4\" (UID: \"eda746ab-d0ce-4007-9db1-f6ee9cbdc317\") " pod="openshift-cluster-node-tuning-operator/tuned-7wnl4"
Jul 14 22:56:55.128748 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.128547    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eda746ab-d0ce-4007-9db1-f6ee9cbdc317-etc-tuned\") pod \"tuned-7wnl4\" (UID: \"eda746ab-d0ce-4007-9db1-f6ee9cbdc317\") " pod="openshift-cluster-node-tuning-operator/tuned-7wnl4"
Jul 14 22:56:55.128748 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.128596    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/da7d977f-1dd0-42e6-b3a7-455490097670-cnibin\") pod \"multus-additional-cni-plugins-r5s76\" (UID: \"da7d977f-1dd0-42e6-b3a7-455490097670\") " pod="openshift-multus/multus-additional-cni-plugins-r5s76"
Jul 14 22:56:55.128748 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.128618    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/186005d6-1e47-4ea0-816c-469b21cc77d4-agent-certs\") pod \"konnectivity-agent-pthrz\" (UID: \"186005d6-1e47-4ea0-816c-469b21cc77d4\") " pod="kube-system/konnectivity-agent-pthrz"
Jul 14 22:56:55.128748 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.128683    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-system-cni-dir\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.128748 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.128721    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-multus-cni-dir\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.128748 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.128746    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6bbh\" (UniqueName: \"kubernetes.io/projected/39492113-cc47-4511-bc7d-d0785b892ed9-kube-api-access-p6bbh\") pod \"network-check-target-pbtpg\" (UID: \"39492113-cc47-4511-bc7d-d0785b892ed9\") " pod="openshift-network-diagnostics/network-check-target-pbtpg"
Jul 14 22:56:55.129118 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.128774    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eda746ab-d0ce-4007-9db1-f6ee9cbdc317-etc-sysconfig\") pod \"tuned-7wnl4\" (UID: \"eda746ab-d0ce-4007-9db1-f6ee9cbdc317\") " pod="openshift-cluster-node-tuning-operator/tuned-7wnl4"
Jul 14 22:56:55.129118 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.128803    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eda746ab-d0ce-4007-9db1-f6ee9cbdc317-tmp\") pod \"tuned-7wnl4\" (UID: \"eda746ab-d0ce-4007-9db1-f6ee9cbdc317\") " pod="openshift-cluster-node-tuning-operator/tuned-7wnl4"
Jul 14 22:56:55.129118 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.128827    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-host-run-k8s-cni-cncf-io\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.129118 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.128851    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-host-run-netns\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.129118 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.128873    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-host-var-lib-kubelet\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.129118 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.128894    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-etc-kubernetes\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.129118 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.128899    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fkjb4\""
Jul 14 22:56:55.129118 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.128916    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf4h6\" (UniqueName: \"kubernetes.io/projected/1ef7f81d-618b-4d9a-b018-9e3346c27bd1-kube-api-access-tf4h6\") pod \"node-ca-lzt25\" (UID: \"1ef7f81d-618b-4d9a-b018-9e3346c27bd1\") " pod="openshift-image-registry/node-ca-lzt25"
Jul 14 22:56:55.129118 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.128937    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eda746ab-d0ce-4007-9db1-f6ee9cbdc317-etc-kubernetes\") pod \"tuned-7wnl4\" (UID: \"eda746ab-d0ce-4007-9db1-f6ee9cbdc317\") " pod="openshift-cluster-node-tuning-operator/tuned-7wnl4"
Jul 14 22:56:55.129118 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.128959    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eda746ab-d0ce-4007-9db1-f6ee9cbdc317-etc-systemd\") pod \"tuned-7wnl4\" (UID: \"eda746ab-d0ce-4007-9db1-f6ee9cbdc317\") " pod="openshift-cluster-node-tuning-operator/tuned-7wnl4"
Jul 14 22:56:55.129118 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.128983    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/da7d977f-1dd0-42e6-b3a7-455490097670-os-release\") pod \"multus-additional-cni-plugins-r5s76\" (UID: \"da7d977f-1dd0-42e6-b3a7-455490097670\") " pod="openshift-multus/multus-additional-cni-plugins-r5s76"
Jul 14 22:56:55.129118 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.129006    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-host-var-lib-cni-multus\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.129118 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.129052    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1ef7f81d-618b-4d9a-b018-9e3346c27bd1-host\") pod \"node-ca-lzt25\" (UID: \"1ef7f81d-618b-4d9a-b018-9e3346c27bd1\") " pod="openshift-image-registry/node-ca-lzt25"
Jul 14 22:56:55.129118 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.129097    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eda746ab-d0ce-4007-9db1-f6ee9cbdc317-lib-modules\") pod \"tuned-7wnl4\" (UID: \"eda746ab-d0ce-4007-9db1-f6ee9cbdc317\") " pod="openshift-cluster-node-tuning-operator/tuned-7wnl4"
Jul 14 22:56:55.129118 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.129119    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlckb\" (UniqueName: \"kubernetes.io/projected/eda746ab-d0ce-4007-9db1-f6ee9cbdc317-kube-api-access-qlckb\") pod \"tuned-7wnl4\" (UID: \"eda746ab-d0ce-4007-9db1-f6ee9cbdc317\") " pod="openshift-cluster-node-tuning-operator/tuned-7wnl4"
Jul 14 22:56:55.129118 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.129123    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 14 22:56:55.129815 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.129055    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 14 22:56:55.129815 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.129152    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da7d977f-1dd0-42e6-b3a7-455490097670-system-cni-dir\") pod \"multus-additional-cni-plugins-r5s76\" (UID: \"da7d977f-1dd0-42e6-b3a7-455490097670\") " pod="openshift-multus/multus-additional-cni-plugins-r5s76"
Jul 14 22:56:55.130230 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.130205    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-c8c42"
Jul 14 22:56:55.131075 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.131055    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 14 22:56:55.131075 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.131071    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nzpml\""
Jul 14 22:56:55.131233 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.131216    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 14 22:56:55.131608 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.131592    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 14 22:56:55.132068 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.132051    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 14 22:56:55.132765 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.132743    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 14 22:56:55.132983 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.132965    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 14 22:56:55.133670 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.133154    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.133670 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.133185    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-clmww\""
Jul 14 22:56:55.133670 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.133299    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 14 22:56:55.136476 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.135704    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 14 22:56:55.136476 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.135904    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 14 22:56:55.136476 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.135919    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 14 22:56:55.136476 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.136114    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ff7p7\""
Jul 14 22:56:55.137907 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.137756    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 14 22:56:55.138003 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.137940    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 14 22:56:55.138141 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.138125    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 14 22:56:55.149713 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.149688    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kx655"
Jul 14 22:56:55.159410 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.159385    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kx655"
Jul 14 22:56:55.217745 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.217716    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 14 22:56:55.228261 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.228208    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-8.ec2.internal" event={"ID":"e408b8465ac9f8b60353a9a16a836248","Type":"ContainerStarted","Data":"0bd3d732e657e45ea5a8adc3cad85677acbb2b7c070b7dde3e203534c8346f03"}
Jul 14 22:56:55.229397 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.229370    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-multus-daemon-config\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.229538 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.229414    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-host-cni-bin\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.229538 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.229443    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-host-run-multus-certs\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.229538 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.229466    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/186005d6-1e47-4ea0-816c-469b21cc77d4-konnectivity-ca\") pod \"konnectivity-agent-pthrz\" (UID: \"186005d6-1e47-4ea0-816c-469b21cc77d4\") " pod="kube-system/konnectivity-agent-pthrz"
Jul 14 22:56:55.229538 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.229488    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/214ee792-0c28-4bcb-b213-4e55d0a69810-ovnkube-config\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.229538 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.229509    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/39e135b7-a5a9-4b22-a87b-d21a0f6657a6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lwzlt\" (UID: \"39e135b7-a5a9-4b22-a87b-d21a0f6657a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwzlt"
Jul 14 22:56:55.229538 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.229533    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/39e135b7-a5a9-4b22-a87b-d21a0f6657a6-device-dir\") pod \"aws-ebs-csi-driver-node-lwzlt\" (UID: \"39e135b7-a5a9-4b22-a87b-d21a0f6657a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwzlt"
Jul 14 22:56:55.229824 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.229561    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/da7d977f-1dd0-42e6-b3a7-455490097670-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r5s76\" (UID: \"da7d977f-1dd0-42e6-b3a7-455490097670\") " pod="openshift-multus/multus-additional-cni-plugins-r5s76"
Jul 14 22:56:55.229824 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.229589    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vg7c\" (UniqueName: \"kubernetes.io/projected/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-kube-api-access-8vg7c\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.229824 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.229609    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eda746ab-d0ce-4007-9db1-f6ee9cbdc317-etc-tuned\") pod \"tuned-7wnl4\" (UID: \"eda746ab-d0ce-4007-9db1-f6ee9cbdc317\") " pod="openshift-cluster-node-tuning-operator/tuned-7wnl4"
Jul 14 22:56:55.229824 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.229633    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/39e135b7-a5a9-4b22-a87b-d21a0f6657a6-registration-dir\") pod \"aws-ebs-csi-driver-node-lwzlt\" (UID: \"39e135b7-a5a9-4b22-a87b-d21a0f6657a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwzlt"
Jul 14 22:56:55.229824 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.229655    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzm4d\" (UniqueName: \"kubernetes.io/projected/3b4a8b8e-6c69-4a3d-9168-a217d09d53d0-kube-api-access-zzm4d\") pod \"iptables-alerter-c8c42\" (UID: \"3b4a8b8e-6c69-4a3d-9168-a217d09d53d0\") " pod="openshift-network-operator/iptables-alerter-c8c42"
Jul 14 22:56:55.229824 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.229678    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/da7d977f-1dd0-42e6-b3a7-455490097670-cnibin\") pod \"multus-additional-cni-plugins-r5s76\" (UID: \"da7d977f-1dd0-42e6-b3a7-455490097670\") " pod="openshift-multus/multus-additional-cni-plugins-r5s76"
Jul 14 22:56:55.229824 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.229699    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/186005d6-1e47-4ea0-816c-469b21cc77d4-agent-certs\") pod \"konnectivity-agent-pthrz\" (UID: \"186005d6-1e47-4ea0-816c-469b21cc77d4\") " pod="kube-system/konnectivity-agent-pthrz"
Jul 14 22:56:55.229824 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.229721    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594-metrics-certs\") pod \"network-metrics-daemon-wjpcm\" (UID: \"8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594\") " pod="openshift-multus/network-metrics-daemon-wjpcm"
Jul 14 22:56:55.229824 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.229745    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-etc-kubernetes\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.229824 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.229771    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eda746ab-d0ce-4007-9db1-f6ee9cbdc317-etc-kubernetes\") pod \"tuned-7wnl4\" (UID: \"eda746ab-d0ce-4007-9db1-f6ee9cbdc317\") " pod="openshift-cluster-node-tuning-operator/tuned-7wnl4"
Jul 14 22:56:55.229824 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.229797    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-systemd-units\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.229824 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.229820    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-host-slash\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.230426 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.229845    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-log-socket\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.230426 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.229867    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/39e135b7-a5a9-4b22-a87b-d21a0f6657a6-socket-dir\") pod \"aws-ebs-csi-driver-node-lwzlt\" (UID: \"39e135b7-a5a9-4b22-a87b-d21a0f6657a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwzlt"
Jul 14 22:56:55.230426 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.229893    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/da7d977f-1dd0-42e6-b3a7-455490097670-os-release\") pod \"multus-additional-cni-plugins-r5s76\" (UID: \"da7d977f-1dd0-42e6-b3a7-455490097670\") " pod="openshift-multus/multus-additional-cni-plugins-r5s76"
Jul 14 22:56:55.230426 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.229916    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1ef7f81d-618b-4d9a-b018-9e3346c27bd1-host\") pod \"node-ca-lzt25\" (UID: \"1ef7f81d-618b-4d9a-b018-9e3346c27bd1\") " pod="openshift-image-registry/node-ca-lzt25"
Jul 14 22:56:55.230426 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.229938    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlckb\" (UniqueName: \"kubernetes.io/projected/eda746ab-d0ce-4007-9db1-f6ee9cbdc317-kube-api-access-qlckb\") pod \"tuned-7wnl4\" (UID: \"eda746ab-d0ce-4007-9db1-f6ee9cbdc317\") " pod="openshift-cluster-node-tuning-operator/tuned-7wnl4"
Jul 14 22:56:55.230426 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.229962    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-host-cni-netd\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.230426 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.230012    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1ef7f81d-618b-4d9a-b018-9e3346c27bd1-serviceca\") pod \"node-ca-lzt25\" (UID: \"1ef7f81d-618b-4d9a-b018-9e3346c27bd1\") " pod="openshift-image-registry/node-ca-lzt25"
Jul 14 22:56:55.230426 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.230038    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eda746ab-d0ce-4007-9db1-f6ee9cbdc317-etc-sysctl-d\") pod \"tuned-7wnl4\" (UID: \"eda746ab-d0ce-4007-9db1-f6ee9cbdc317\") " pod="openshift-cluster-node-tuning-operator/tuned-7wnl4"
Jul 14 22:56:55.230426 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.230062    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eda746ab-d0ce-4007-9db1-f6ee9cbdc317-etc-sysctl-conf\") pod \"tuned-7wnl4\" (UID: \"eda746ab-d0ce-4007-9db1-f6ee9cbdc317\") " pod="openshift-cluster-node-tuning-operator/tuned-7wnl4"
Jul 14 22:56:55.230426 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.230085    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eda746ab-d0ce-4007-9db1-f6ee9cbdc317-sys\") pod \"tuned-7wnl4\" (UID: \"eda746ab-d0ce-4007-9db1-f6ee9cbdc317\") " pod="openshift-cluster-node-tuning-operator/tuned-7wnl4"
Jul 14 22:56:55.230426 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.230110    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eda746ab-d0ce-4007-9db1-f6ee9cbdc317-host\") pod \"tuned-7wnl4\" (UID: \"eda746ab-d0ce-4007-9db1-f6ee9cbdc317\") " pod="openshift-cluster-node-tuning-operator/tuned-7wnl4"
Jul 14 22:56:55.230426 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.230120    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-multus-daemon-config\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.230426 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.230132    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-run-ovn\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.230426 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.230169    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.230426 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.230197    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/39e135b7-a5a9-4b22-a87b-d21a0f6657a6-sys-fs\") pod \"aws-ebs-csi-driver-node-lwzlt\" (UID: \"39e135b7-a5a9-4b22-a87b-d21a0f6657a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwzlt"
Jul 14 22:56:55.230426 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.230216    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1ef7f81d-618b-4d9a-b018-9e3346c27bd1-host\") pod \"node-ca-lzt25\" (UID: \"1ef7f81d-618b-4d9a-b018-9e3346c27bd1\") " pod="openshift-image-registry/node-ca-lzt25"
Jul 14 22:56:55.230426 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.230225    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/da7d977f-1dd0-42e6-b3a7-455490097670-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r5s76\" (UID: \"da7d977f-1dd0-42e6-b3a7-455490097670\") " pod="openshift-multus/multus-additional-cni-plugins-r5s76"
Jul 14 22:56:55.231243 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.230269    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-cnibin\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.231243 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.230300    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-etc-kubernetes\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.231243 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.230314    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-multus-socket-dir-parent\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.231243 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.230338    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eda746ab-d0ce-4007-9db1-f6ee9cbdc317-var-lib-kubelet\") pod \"tuned-7wnl4\" (UID: \"eda746ab-d0ce-4007-9db1-f6ee9cbdc317\") " pod="openshift-cluster-node-tuning-operator/tuned-7wnl4"
Jul 14 22:56:55.231243 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.230356    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eda746ab-d0ce-4007-9db1-f6ee9cbdc317-etc-kubernetes\") pod \"tuned-7wnl4\" (UID: \"eda746ab-d0ce-4007-9db1-f6ee9cbdc317\") " pod="openshift-cluster-node-tuning-operator/tuned-7wnl4"
Jul 14 22:56:55.231243 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.230363    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm7j5\" (UniqueName: \"kubernetes.io/projected/39e135b7-a5a9-4b22-a87b-d21a0f6657a6-kube-api-access-jm7j5\") pod \"aws-ebs-csi-driver-node-lwzlt\" (UID: \"39e135b7-a5a9-4b22-a87b-d21a0f6657a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwzlt"
Jul 14 22:56:55.231243 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.230393    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7e3e444e-1753-442e-8430-24b0022f9418-kubelet-config\") pod \"global-pull-secret-syncer-xjw4z\" (UID: \"7e3e444e-1753-442e-8430-24b0022f9418\") " pod="kube-system/global-pull-secret-syncer-xjw4z"
Jul 14 22:56:55.231243 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.230423    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b4a8b8e-6c69-4a3d-9168-a217d09d53d0-host-slash\") pod \"iptables-alerter-c8c42\" (UID: \"3b4a8b8e-6c69-4a3d-9168-a217d09d53d0\") " pod="openshift-network-operator/iptables-alerter-c8c42"
Jul 14 22:56:55.231243 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.230421    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 14 22:56:55.231243 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.230558    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eda746ab-d0ce-4007-9db1-f6ee9cbdc317-etc-sysctl-d\") pod \"tuned-7wnl4\" (UID: \"eda746ab-d0ce-4007-9db1-f6ee9cbdc317\") " pod="openshift-cluster-node-tuning-operator/tuned-7wnl4"
Jul 14 22:56:55.231243 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.230615    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/da7d977f-1dd0-42e6-b3a7-455490097670-os-release\") pod \"multus-additional-cni-plugins-r5s76\" (UID: \"da7d977f-1dd0-42e6-b3a7-455490097670\") " pod="openshift-multus/multus-additional-cni-plugins-r5s76"
Jul 14 22:56:55.231243 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.230761    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eda746ab-d0ce-4007-9db1-f6ee9cbdc317-etc-sysctl-conf\") pod \"tuned-7wnl4\" (UID: \"eda746ab-d0ce-4007-9db1-f6ee9cbdc317\") " pod="openshift-cluster-node-tuning-operator/tuned-7wnl4"
Jul 14 22:56:55.231243 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.230786    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1ef7f81d-618b-4d9a-b018-9e3346c27bd1-serviceca\") pod \"node-ca-lzt25\" (UID: \"1ef7f81d-618b-4d9a-b018-9e3346c27bd1\") " pod="openshift-image-registry/node-ca-lzt25"
Jul 14 22:56:55.231243 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.230809    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eda746ab-d0ce-4007-9db1-f6ee9cbdc317-sys\") pod \"tuned-7wnl4\" (UID: \"eda746ab-d0ce-4007-9db1-f6ee9cbdc317\") " pod="openshift-cluster-node-tuning-operator/tuned-7wnl4"
Jul 14 22:56:55.231243 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.230982    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eda746ab-d0ce-4007-9db1-f6ee9cbdc317-host\") pod \"tuned-7wnl4\" (UID: \"eda746ab-d0ce-4007-9db1-f6ee9cbdc317\") " pod="openshift-cluster-node-tuning-operator/tuned-7wnl4"
Jul 14 22:56:55.231903 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.231539    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/186005d6-1e47-4ea0-816c-469b21cc77d4-konnectivity-ca\") pod \"konnectivity-agent-pthrz\" (UID: \"186005d6-1e47-4ea0-816c-469b21cc77d4\") " pod="kube-system/konnectivity-agent-pthrz"
Jul 14 22:56:55.231903 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.231547    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/da7d977f-1dd0-42e6-b3a7-455490097670-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r5s76\" (UID: \"da7d977f-1dd0-42e6-b3a7-455490097670\") " pod="openshift-multus/multus-additional-cni-plugins-r5s76"
Jul 14 22:56:55.231903 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.231599    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-cnibin\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.231903 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.231710    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-multus-socket-dir-parent\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.231903 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.231751    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eda746ab-d0ce-4007-9db1-f6ee9cbdc317-var-lib-kubelet\") pod \"tuned-7wnl4\" (UID: \"eda746ab-d0ce-4007-9db1-f6ee9cbdc317\") " pod="openshift-cluster-node-tuning-operator/tuned-7wnl4"
Jul 14 22:56:55.231903 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.231789    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/da7d977f-1dd0-42e6-b3a7-455490097670-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r5s76\" (UID: \"da7d977f-1dd0-42e6-b3a7-455490097670\") " pod="openshift-multus/multus-additional-cni-plugins-r5s76"
Jul 14 22:56:55.231903 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.231817    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eda746ab-d0ce-4007-9db1-f6ee9cbdc317-etc-modprobe-d\") pod \"tuned-7wnl4\" (UID: \"eda746ab-d0ce-4007-9db1-f6ee9cbdc317\") " pod="openshift-cluster-node-tuning-operator/tuned-7wnl4"
Jul 14 22:56:55.231903 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.231842    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-host-run-ovn-kubernetes\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.231903 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.231860    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/214ee792-0c28-4bcb-b213-4e55d0a69810-ovn-node-metrics-cert\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.231903 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.231887    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtcjm\" (UniqueName: \"kubernetes.io/projected/214ee792-0c28-4bcb-b213-4e55d0a69810-kube-api-access-dtcjm\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.231903 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.231599    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-host-run-multus-certs\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.232365 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.231922    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59km6\" (UniqueName: \"kubernetes.io/projected/da7d977f-1dd0-42e6-b3a7-455490097670-kube-api-access-59km6\") pod \"multus-additional-cni-plugins-r5s76\" (UID: \"da7d977f-1dd0-42e6-b3a7-455490097670\") " pod="openshift-multus/multus-additional-cni-plugins-r5s76"
Jul 14 22:56:55.232365 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.231942    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-os-release\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.232365 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.231957    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eda746ab-d0ce-4007-9db1-f6ee9cbdc317-run\") pod \"tuned-7wnl4\" (UID: \"eda746ab-d0ce-4007-9db1-f6ee9cbdc317\") " pod="openshift-cluster-node-tuning-operator/tuned-7wnl4"
Jul 14 22:56:55.232365 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.231972    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-host-kubelet\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.232365 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.231987    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-var-lib-openvswitch\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.232365 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.232002    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-run-systemd\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.232365 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.232016    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-node-log\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.232365 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.232032    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/39e135b7-a5a9-4b22-a87b-d21a0f6657a6-etc-selinux\") pod \"aws-ebs-csi-driver-node-lwzlt\" (UID: \"39e135b7-a5a9-4b22-a87b-d21a0f6657a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwzlt"
Jul 14 22:56:55.232365 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.232049    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-system-cni-dir\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.232365 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.232066    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-multus-cni-dir\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.232365 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.232130    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6bbh\" (UniqueName: \"kubernetes.io/projected/39492113-cc47-4511-bc7d-d0785b892ed9-kube-api-access-p6bbh\") pod \"network-check-target-pbtpg\" (UID: \"39492113-cc47-4511-bc7d-d0785b892ed9\") " pod="openshift-network-diagnostics/network-check-target-pbtpg"
Jul 14 22:56:55.232365 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.232145    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eda746ab-d0ce-4007-9db1-f6ee9cbdc317-etc-sysconfig\") pod \"tuned-7wnl4\" (UID: \"eda746ab-d0ce-4007-9db1-f6ee9cbdc317\") " pod="openshift-cluster-node-tuning-operator/tuned-7wnl4"
Jul 14 22:56:55.232365 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.232162    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eda746ab-d0ce-4007-9db1-f6ee9cbdc317-tmp\") pod \"tuned-7wnl4\" (UID: \"eda746ab-d0ce-4007-9db1-f6ee9cbdc317\") " pod="openshift-cluster-node-tuning-operator/tuned-7wnl4"
Jul 14 22:56:55.232365 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.232178    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59f4f5d3-9650-40e0-969a-e912429a2481-hosts-file\") pod \"node-resolver-s4zgk\" (UID: \"59f4f5d3-9650-40e0-969a-e912429a2481\") " pod="openshift-dns/node-resolver-s4zgk"
Jul 14 22:56:55.232365 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.232193    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7e3e444e-1753-442e-8430-24b0022f9418-dbus\") pod \"global-pull-secret-syncer-xjw4z\" (UID: \"7e3e444e-1753-442e-8430-24b0022f9418\") " pod="kube-system/global-pull-secret-syncer-xjw4z"
Jul 14 22:56:55.232365 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.232209    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-host-run-k8s-cni-cncf-io\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.232365 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.232226    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-host-run-netns\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.233149 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.232241    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-host-var-lib-kubelet\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.233149 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.232257    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf4h6\" (UniqueName: \"kubernetes.io/projected/1ef7f81d-618b-4d9a-b018-9e3346c27bd1-kube-api-access-tf4h6\") pod \"node-ca-lzt25\" (UID: \"1ef7f81d-618b-4d9a-b018-9e3346c27bd1\") " pod="openshift-image-registry/node-ca-lzt25"
Jul 14 22:56:55.233149 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.232271    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eda746ab-d0ce-4007-9db1-f6ee9cbdc317-etc-systemd\") pod \"tuned-7wnl4\" (UID: \"eda746ab-d0ce-4007-9db1-f6ee9cbdc317\") " pod="openshift-cluster-node-tuning-operator/tuned-7wnl4"
Jul 14 22:56:55.233149 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.232311    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-host-run-netns\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.233149 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.232338    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-etc-openvswitch\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.233149 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.232363    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/214ee792-0c28-4bcb-b213-4e55d0a69810-ovnkube-script-lib\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.233149 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.232388    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-host-var-lib-cni-multus\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.233149 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.232410    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eda746ab-d0ce-4007-9db1-f6ee9cbdc317-lib-modules\") pod \"tuned-7wnl4\" (UID: \"eda746ab-d0ce-4007-9db1-f6ee9cbdc317\") " pod="openshift-cluster-node-tuning-operator/tuned-7wnl4"
Jul 14 22:56:55.233149 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.232431    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-run-openvswitch\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.233149 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.232483    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-multus-cni-dir\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.233149 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.232360    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/da7d977f-1dd0-42e6-b3a7-455490097670-cnibin\") pod \"multus-additional-cni-plugins-r5s76\" (UID: \"da7d977f-1dd0-42e6-b3a7-455490097670\") " pod="openshift-multus/multus-additional-cni-plugins-r5s76"
Jul 14 22:56:55.233149 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.232566    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59f4f5d3-9650-40e0-969a-e912429a2481-tmp-dir\") pod \"node-resolver-s4zgk\" (UID: \"59f4f5d3-9650-40e0-969a-e912429a2481\") " pod="openshift-dns/node-resolver-s4zgk"
Jul 14 22:56:55.233149 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.232594    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbm2b\" (UniqueName: \"kubernetes.io/projected/8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594-kube-api-access-hbm2b\") pod \"network-metrics-daemon-wjpcm\" (UID: \"8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594\") " pod="openshift-multus/network-metrics-daemon-wjpcm"
Jul 14 22:56:55.233149 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.232619    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-system-cni-dir\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.233149 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.232619    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e3e444e-1753-442e-8430-24b0022f9418-original-pull-secret\") pod \"global-pull-secret-syncer-xjw4z\" (UID: \"7e3e444e-1753-442e-8430-24b0022f9418\") " pod="kube-system/global-pull-secret-syncer-xjw4z"
Jul 14 22:56:55.233149 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.232734    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eda746ab-d0ce-4007-9db1-f6ee9cbdc317-etc-sysconfig\") pod \"tuned-7wnl4\" (UID: \"eda746ab-d0ce-4007-9db1-f6ee9cbdc317\") " pod="openshift-cluster-node-tuning-operator/tuned-7wnl4"
Jul 14 22:56:55.233149 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.232940    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-os-release\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.233910 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.232988    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eda746ab-d0ce-4007-9db1-f6ee9cbdc317-run\") pod \"tuned-7wnl4\" (UID: \"eda746ab-d0ce-4007-9db1-f6ee9cbdc317\") " pod="openshift-cluster-node-tuning-operator/tuned-7wnl4"
Jul 14 22:56:55.233910 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.232569    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eda746ab-d0ce-4007-9db1-f6ee9cbdc317-etc-systemd\") pod \"tuned-7wnl4\" (UID: \"eda746ab-d0ce-4007-9db1-f6ee9cbdc317\") " pod="openshift-cluster-node-tuning-operator/tuned-7wnl4"
Jul 14 22:56:55.233910 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.233022    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-host-run-netns\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.233910 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.233039    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-host-var-lib-cni-multus\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.233910 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.233101    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-host-var-lib-kubelet\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.233910 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.233048    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3b4a8b8e-6c69-4a3d-9168-a217d09d53d0-iptables-alerter-script\") pod \"iptables-alerter-c8c42\" (UID: \"3b4a8b8e-6c69-4a3d-9168-a217d09d53d0\") " pod="openshift-network-operator/iptables-alerter-c8c42"
Jul 14 22:56:55.233910 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.233114    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eda746ab-d0ce-4007-9db1-f6ee9cbdc317-lib-modules\") pod \"tuned-7wnl4\" (UID: \"eda746ab-d0ce-4007-9db1-f6ee9cbdc317\") " pod="openshift-cluster-node-tuning-operator/tuned-7wnl4"
Jul 14 22:56:55.233910 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.233146    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da7d977f-1dd0-42e6-b3a7-455490097670-system-cni-dir\") pod \"multus-additional-cni-plugins-r5s76\" (UID: \"da7d977f-1dd0-42e6-b3a7-455490097670\") " pod="openshift-multus/multus-additional-cni-plugins-r5s76"
Jul 14 22:56:55.233910 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.233195    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eda746ab-d0ce-4007-9db1-f6ee9cbdc317-etc-modprobe-d\") pod \"tuned-7wnl4\" (UID: \"eda746ab-d0ce-4007-9db1-f6ee9cbdc317\") " pod="openshift-cluster-node-tuning-operator/tuned-7wnl4"
Jul 14 22:56:55.233910 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.233234    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/da7d977f-1dd0-42e6-b3a7-455490097670-cni-binary-copy\") pod \"multus-additional-cni-plugins-r5s76\" (UID: \"da7d977f-1dd0-42e6-b3a7-455490097670\") " pod="openshift-multus/multus-additional-cni-plugins-r5s76"
Jul 14 22:56:55.233910 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.233261    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-multus-conf-dir\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.233910 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.233316    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/214ee792-0c28-4bcb-b213-4e55d0a69810-env-overrides\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.233910 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.233177    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/da7d977f-1dd0-42e6-b3a7-455490097670-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r5s76\" (UID: \"da7d977f-1dd0-42e6-b3a7-455490097670\") " pod="openshift-multus/multus-additional-cni-plugins-r5s76"
Jul 14 22:56:55.233910 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.233363    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da7d977f-1dd0-42e6-b3a7-455490097670-system-cni-dir\") pod \"multus-additional-cni-plugins-r5s76\" (UID: \"da7d977f-1dd0-42e6-b3a7-455490097670\") " pod="openshift-multus/multus-additional-cni-plugins-r5s76"
Jul 14 22:56:55.233910 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.233391    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-host-run-k8s-cni-cncf-io\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.233910 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.233417    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-multus-conf-dir\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.233910 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.233448    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmgxf\" (UniqueName: \"kubernetes.io/projected/59f4f5d3-9650-40e0-969a-e912429a2481-kube-api-access-bmgxf\") pod \"node-resolver-s4zgk\" (UID: \"59f4f5d3-9650-40e0-969a-e912429a2481\") " pod="openshift-dns/node-resolver-s4zgk"
Jul 14 22:56:55.234594 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.233494    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-cni-binary-copy\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.234594 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.233535    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-host-var-lib-cni-bin\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.234594 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.233556    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/da7d977f-1dd0-42e6-b3a7-455490097670-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r5s76\" (UID: \"da7d977f-1dd0-42e6-b3a7-455490097670\") " pod="openshift-multus/multus-additional-cni-plugins-r5s76"
Jul 14 22:56:55.234594 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.233616    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-hostroot\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.234594 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.233684    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-host-var-lib-cni-bin\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.234594 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.233709    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-hostroot\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.234594 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.233814    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/da7d977f-1dd0-42e6-b3a7-455490097670-cni-binary-copy\") pod \"multus-additional-cni-plugins-r5s76\" (UID: \"da7d977f-1dd0-42e6-b3a7-455490097670\") " pod="openshift-multus/multus-additional-cni-plugins-r5s76"
Jul 14 22:56:55.234934 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.234642    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-cni-binary-copy\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.236334 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.236042    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-8.ec2.internal" event={"ID":"3aa82a5123efb70e24ea524a0aeee7ed","Type":"ContainerStarted","Data":"742a2b1bca7df551206d4aa117bbd1dde57f15c7b042aea6b9131ef70352b6c2"}
Jul 14 22:56:55.236827 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.236799    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/186005d6-1e47-4ea0-816c-469b21cc77d4-agent-certs\") pod \"konnectivity-agent-pthrz\" (UID: \"186005d6-1e47-4ea0-816c-469b21cc77d4\") " pod="kube-system/konnectivity-agent-pthrz"
Jul 14 22:56:55.237167 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.237144    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eda746ab-d0ce-4007-9db1-f6ee9cbdc317-etc-tuned\") pod \"tuned-7wnl4\" (UID: \"eda746ab-d0ce-4007-9db1-f6ee9cbdc317\") " pod="openshift-cluster-node-tuning-operator/tuned-7wnl4"
Jul 14 22:56:55.237447 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.237328    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eda746ab-d0ce-4007-9db1-f6ee9cbdc317-tmp\") pod \"tuned-7wnl4\" (UID: \"eda746ab-d0ce-4007-9db1-f6ee9cbdc317\") " pod="openshift-cluster-node-tuning-operator/tuned-7wnl4"
Jul 14 22:56:55.239313 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:55.238863    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 22:56:55.239313 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:55.238888    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 22:56:55.239313 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:55.238905    2563 projected.go:196] Error preparing data for projected volume kube-api-access-p6bbh for pod openshift-network-diagnostics/network-check-target-pbtpg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 22:56:55.239313 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:55.238998    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39492113-cc47-4511-bc7d-d0785b892ed9-kube-api-access-p6bbh podName:39492113-cc47-4511-bc7d-d0785b892ed9 nodeName:}" failed. No retries permitted until 2026-07-14 22:56:55.738958272 +0000 UTC m=+3.041949059 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-p6bbh" (UniqueName: "kubernetes.io/projected/39492113-cc47-4511-bc7d-d0785b892ed9-kube-api-access-p6bbh") pod "network-check-target-pbtpg" (UID: "39492113-cc47-4511-bc7d-d0785b892ed9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 22:56:55.240069 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.240047    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlckb\" (UniqueName: \"kubernetes.io/projected/eda746ab-d0ce-4007-9db1-f6ee9cbdc317-kube-api-access-qlckb\") pod \"tuned-7wnl4\" (UID: \"eda746ab-d0ce-4007-9db1-f6ee9cbdc317\") " pod="openshift-cluster-node-tuning-operator/tuned-7wnl4"
Jul 14 22:56:55.242056 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.242031    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vg7c\" (UniqueName: \"kubernetes.io/projected/cb9705e3-b1b7-44a7-b4b8-49db0e928da9-kube-api-access-8vg7c\") pod \"multus-vlbvv\" (UID: \"cb9705e3-b1b7-44a7-b4b8-49db0e928da9\") " pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.242901 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.242829    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59km6\" (UniqueName: \"kubernetes.io/projected/da7d977f-1dd0-42e6-b3a7-455490097670-kube-api-access-59km6\") pod \"multus-additional-cni-plugins-r5s76\" (UID: \"da7d977f-1dd0-42e6-b3a7-455490097670\") " pod="openshift-multus/multus-additional-cni-plugins-r5s76"
Jul 14 22:56:55.244093 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.244020    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf4h6\" (UniqueName: \"kubernetes.io/projected/1ef7f81d-618b-4d9a-b018-9e3346c27bd1-kube-api-access-tf4h6\") pod \"node-ca-lzt25\" (UID: \"1ef7f81d-618b-4d9a-b018-9e3346c27bd1\") " pod="openshift-image-registry/node-ca-lzt25"
Jul 14 22:56:55.334081 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.334037    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-host-run-netns\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.334081 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.334084    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-etc-openvswitch\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.334332 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.334109    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/214ee792-0c28-4bcb-b213-4e55d0a69810-ovnkube-script-lib\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.334332 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.334131    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-run-openvswitch\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.334332 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.334153    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59f4f5d3-9650-40e0-969a-e912429a2481-tmp-dir\") pod \"node-resolver-s4zgk\" (UID: \"59f4f5d3-9650-40e0-969a-e912429a2481\") " pod="openshift-dns/node-resolver-s4zgk"
Jul 14 22:56:55.334332 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.334254    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-host-run-netns\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.334582 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.334419    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-run-openvswitch\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.334582 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.334564    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-etc-openvswitch\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.334669 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.334614    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbm2b\" (UniqueName: \"kubernetes.io/projected/8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594-kube-api-access-hbm2b\") pod \"network-metrics-daemon-wjpcm\" (UID: \"8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594\") " pod="openshift-multus/network-metrics-daemon-wjpcm"
Jul 14 22:56:55.334669 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.334643    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e3e444e-1753-442e-8430-24b0022f9418-original-pull-secret\") pod \"global-pull-secret-syncer-xjw4z\" (UID: \"7e3e444e-1753-442e-8430-24b0022f9418\") " pod="kube-system/global-pull-secret-syncer-xjw4z"
Jul 14 22:56:55.334669 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.334665    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3b4a8b8e-6c69-4a3d-9168-a217d09d53d0-iptables-alerter-script\") pod \"iptables-alerter-c8c42\" (UID: \"3b4a8b8e-6c69-4a3d-9168-a217d09d53d0\") " pod="openshift-network-operator/iptables-alerter-c8c42"
Jul 14 22:56:55.334768 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.334691    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/214ee792-0c28-4bcb-b213-4e55d0a69810-env-overrides\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.334768 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.334707    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmgxf\" (UniqueName: \"kubernetes.io/projected/59f4f5d3-9650-40e0-969a-e912429a2481-kube-api-access-bmgxf\") pod \"node-resolver-s4zgk\" (UID: \"59f4f5d3-9650-40e0-969a-e912429a2481\") " pod="openshift-dns/node-resolver-s4zgk"
Jul 14 22:56:55.334768 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.334730    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-host-cni-bin\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.334768 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.334753    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/214ee792-0c28-4bcb-b213-4e55d0a69810-ovnkube-config\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.334905 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.334768    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/39e135b7-a5a9-4b22-a87b-d21a0f6657a6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lwzlt\" (UID: \"39e135b7-a5a9-4b22-a87b-d21a0f6657a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwzlt"
Jul 14 22:56:55.334905 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.334778    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59f4f5d3-9650-40e0-969a-e912429a2481-tmp-dir\") pod \"node-resolver-s4zgk\" (UID: \"59f4f5d3-9650-40e0-969a-e912429a2481\") " pod="openshift-dns/node-resolver-s4zgk"
Jul 14 22:56:55.334905 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.334783    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/39e135b7-a5a9-4b22-a87b-d21a0f6657a6-device-dir\") pod \"aws-ebs-csi-driver-node-lwzlt\" (UID: \"39e135b7-a5a9-4b22-a87b-d21a0f6657a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwzlt"
Jul 14 22:56:55.334905 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.334827    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/39e135b7-a5a9-4b22-a87b-d21a0f6657a6-device-dir\") pod \"aws-ebs-csi-driver-node-lwzlt\" (UID: \"39e135b7-a5a9-4b22-a87b-d21a0f6657a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwzlt"
Jul 14 22:56:55.334905 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.334842    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/39e135b7-a5a9-4b22-a87b-d21a0f6657a6-registration-dir\") pod \"aws-ebs-csi-driver-node-lwzlt\" (UID: \"39e135b7-a5a9-4b22-a87b-d21a0f6657a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwzlt"
Jul 14 22:56:55.334905 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.334873    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzm4d\" (UniqueName: \"kubernetes.io/projected/3b4a8b8e-6c69-4a3d-9168-a217d09d53d0-kube-api-access-zzm4d\") pod \"iptables-alerter-c8c42\" (UID: \"3b4a8b8e-6c69-4a3d-9168-a217d09d53d0\") " pod="openshift-network-operator/iptables-alerter-c8c42"
Jul 14 22:56:55.334905 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.334903    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594-metrics-certs\") pod \"network-metrics-daemon-wjpcm\" (UID: \"8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594\") " pod="openshift-multus/network-metrics-daemon-wjpcm"
Jul 14 22:56:55.335197 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.334931    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-systemd-units\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.335197 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.334958    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-host-slash\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.335197 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.334976    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/214ee792-0c28-4bcb-b213-4e55d0a69810-ovnkube-script-lib\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.335197 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.335029    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-log-socket\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.335197 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.335036    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/39e135b7-a5a9-4b22-a87b-d21a0f6657a6-registration-dir\") pod \"aws-ebs-csi-driver-node-lwzlt\" (UID: \"39e135b7-a5a9-4b22-a87b-d21a0f6657a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwzlt"
Jul 14 22:56:55.335197 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:55.335095    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 22:56:55.335197 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:55.335146    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594-metrics-certs podName:8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594 nodeName:}" failed. No retries permitted until 2026-07-14 22:56:55.835131614 +0000 UTC m=+3.138122386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594-metrics-certs") pod "network-metrics-daemon-wjpcm" (UID: "8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 22:56:55.335465 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.335200    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-host-cni-bin\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.335465 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:55.335342    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 22:56:55.335465 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.335375    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/214ee792-0c28-4bcb-b213-4e55d0a69810-env-overrides\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.335465 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:55.335408    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e3e444e-1753-442e-8430-24b0022f9418-original-pull-secret podName:7e3e444e-1753-442e-8430-24b0022f9418 nodeName:}" failed. No retries permitted until 2026-07-14 22:56:55.83538971 +0000 UTC m=+3.138380483 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e3e444e-1753-442e-8430-24b0022f9418-original-pull-secret") pod "global-pull-secret-syncer-xjw4z" (UID: "7e3e444e-1753-442e-8430-24b0022f9418") : object "kube-system"/"original-pull-secret" not registered
Jul 14 22:56:55.335465 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.335425    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-systemd-units\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.335651 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.335470    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-host-slash\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.335651 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.334983    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-log-socket\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.335651 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.335513    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/39e135b7-a5a9-4b22-a87b-d21a0f6657a6-socket-dir\") pod \"aws-ebs-csi-driver-node-lwzlt\" (UID: \"39e135b7-a5a9-4b22-a87b-d21a0f6657a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwzlt"
Jul 14 22:56:55.335651 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.335542    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-host-cni-netd\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.335651 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.335571    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-run-ovn\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.335651 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.335602    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.335651 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.335622    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/214ee792-0c28-4bcb-b213-4e55d0a69810-ovnkube-config\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.335855 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.335680    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/39e135b7-a5a9-4b22-a87b-d21a0f6657a6-sys-fs\") pod \"aws-ebs-csi-driver-node-lwzlt\" (UID: \"39e135b7-a5a9-4b22-a87b-d21a0f6657a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwzlt"
Jul 14 22:56:55.335855 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.335711    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/39e135b7-a5a9-4b22-a87b-d21a0f6657a6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lwzlt\" (UID: \"39e135b7-a5a9-4b22-a87b-d21a0f6657a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwzlt"
Jul 14 22:56:55.335855 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.335629    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/39e135b7-a5a9-4b22-a87b-d21a0f6657a6-sys-fs\") pod \"aws-ebs-csi-driver-node-lwzlt\" (UID: \"39e135b7-a5a9-4b22-a87b-d21a0f6657a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwzlt"
Jul 14 22:56:55.335855 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.335741    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/39e135b7-a5a9-4b22-a87b-d21a0f6657a6-socket-dir\") pod \"aws-ebs-csi-driver-node-lwzlt\" (UID: \"39e135b7-a5a9-4b22-a87b-d21a0f6657a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwzlt"
Jul 14 22:56:55.335855 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.335750    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm7j5\" (UniqueName: \"kubernetes.io/projected/39e135b7-a5a9-4b22-a87b-d21a0f6657a6-kube-api-access-jm7j5\") pod \"aws-ebs-csi-driver-node-lwzlt\" (UID: \"39e135b7-a5a9-4b22-a87b-d21a0f6657a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwzlt"
Jul 14 22:56:55.335855 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.335785    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-host-cni-netd\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.335855 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.335811    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7e3e444e-1753-442e-8430-24b0022f9418-kubelet-config\") pod \"global-pull-secret-syncer-xjw4z\" (UID: \"7e3e444e-1753-442e-8430-24b0022f9418\") " pod="kube-system/global-pull-secret-syncer-xjw4z"
Jul 14 22:56:55.335855 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.335839    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b4a8b8e-6c69-4a3d-9168-a217d09d53d0-host-slash\") pod \"iptables-alerter-c8c42\" (UID: \"3b4a8b8e-6c69-4a3d-9168-a217d09d53d0\") " pod="openshift-network-operator/iptables-alerter-c8c42"
Jul 14 22:56:55.336140 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.335867    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-host-run-ovn-kubernetes\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.336140 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.335893    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/214ee792-0c28-4bcb-b213-4e55d0a69810-ovn-node-metrics-cert\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.336140 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.335918    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtcjm\" (UniqueName: \"kubernetes.io/projected/214ee792-0c28-4bcb-b213-4e55d0a69810-kube-api-access-dtcjm\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.336140 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.335942    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7e3e444e-1753-442e-8430-24b0022f9418-kubelet-config\") pod \"global-pull-secret-syncer-xjw4z\" (UID: \"7e3e444e-1753-442e-8430-24b0022f9418\") " pod="kube-system/global-pull-secret-syncer-xjw4z"
Jul 14 22:56:55.336140 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.335947    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-host-kubelet\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.336140 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.335972    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-run-ovn\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.336140 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.335973    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-var-lib-openvswitch\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.336140 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.336003    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3b4a8b8e-6c69-4a3d-9168-a217d09d53d0-iptables-alerter-script\") pod \"iptables-alerter-c8c42\" (UID: \"3b4a8b8e-6c69-4a3d-9168-a217d09d53d0\") " pod="openshift-network-operator/iptables-alerter-c8c42"
Jul 14 22:56:55.336140 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.336020    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.336140 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.336067    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b4a8b8e-6c69-4a3d-9168-a217d09d53d0-host-slash\") pod \"iptables-alerter-c8c42\" (UID: \"3b4a8b8e-6c69-4a3d-9168-a217d09d53d0\") " pod="openshift-network-operator/iptables-alerter-c8c42"
Jul 14 22:56:55.336140 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.336077    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-host-kubelet\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.336140 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.336104    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-var-lib-openvswitch\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.336140 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.336118    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-host-run-ovn-kubernetes\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.337029 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.336150    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-run-systemd\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.337029 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.336176    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-node-log\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.337029 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.336202    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/39e135b7-a5a9-4b22-a87b-d21a0f6657a6-etc-selinux\") pod \"aws-ebs-csi-driver-node-lwzlt\" (UID: \"39e135b7-a5a9-4b22-a87b-d21a0f6657a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwzlt"
Jul 14 22:56:55.337029 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.336207    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-run-systemd\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.337029 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.336245    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/214ee792-0c28-4bcb-b213-4e55d0a69810-node-log\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.337029 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.336248    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59f4f5d3-9650-40e0-969a-e912429a2481-hosts-file\") pod \"node-resolver-s4zgk\" (UID: \"59f4f5d3-9650-40e0-969a-e912429a2481\") " pod="openshift-dns/node-resolver-s4zgk"
Jul 14 22:56:55.337029 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.336291    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/59f4f5d3-9650-40e0-969a-e912429a2481-hosts-file\") pod \"node-resolver-s4zgk\" (UID: \"59f4f5d3-9650-40e0-969a-e912429a2481\") " pod="openshift-dns/node-resolver-s4zgk"
Jul 14 22:56:55.337029 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.336291    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7e3e444e-1753-442e-8430-24b0022f9418-dbus\") pod \"global-pull-secret-syncer-xjw4z\" (UID: \"7e3e444e-1753-442e-8430-24b0022f9418\") " pod="kube-system/global-pull-secret-syncer-xjw4z"
Jul 14 22:56:55.337029 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.336344    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7e3e444e-1753-442e-8430-24b0022f9418-dbus\") pod \"global-pull-secret-syncer-xjw4z\" (UID: \"7e3e444e-1753-442e-8430-24b0022f9418\") " pod="kube-system/global-pull-secret-syncer-xjw4z"
Jul 14 22:56:55.337029 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.336345    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/39e135b7-a5a9-4b22-a87b-d21a0f6657a6-etc-selinux\") pod \"aws-ebs-csi-driver-node-lwzlt\" (UID: \"39e135b7-a5a9-4b22-a87b-d21a0f6657a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwzlt"
Jul 14 22:56:55.339068 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.339042    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/214ee792-0c28-4bcb-b213-4e55d0a69810-ovn-node-metrics-cert\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.345549 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.345482    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm7j5\" (UniqueName: \"kubernetes.io/projected/39e135b7-a5a9-4b22-a87b-d21a0f6657a6-kube-api-access-jm7j5\") pod \"aws-ebs-csi-driver-node-lwzlt\" (UID: \"39e135b7-a5a9-4b22-a87b-d21a0f6657a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwzlt"
Jul 14 22:56:55.345703 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.345634    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzm4d\" (UniqueName: \"kubernetes.io/projected/3b4a8b8e-6c69-4a3d-9168-a217d09d53d0-kube-api-access-zzm4d\") pod \"iptables-alerter-c8c42\" (UID: \"3b4a8b8e-6c69-4a3d-9168-a217d09d53d0\") " pod="openshift-network-operator/iptables-alerter-c8c42"
Jul 14 22:56:55.347028 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.347002    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbm2b\" (UniqueName: \"kubernetes.io/projected/8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594-kube-api-access-hbm2b\") pod \"network-metrics-daemon-wjpcm\" (UID: \"8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594\") " pod="openshift-multus/network-metrics-daemon-wjpcm"
Jul 14 22:56:55.347832 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.347809    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtcjm\" (UniqueName: \"kubernetes.io/projected/214ee792-0c28-4bcb-b213-4e55d0a69810-kube-api-access-dtcjm\") pod \"ovnkube-node-j2z4s\" (UID: \"214ee792-0c28-4bcb-b213-4e55d0a69810\") " pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.347936 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.347839    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmgxf\" (UniqueName: \"kubernetes.io/projected/59f4f5d3-9650-40e0-969a-e912429a2481-kube-api-access-bmgxf\") pod \"node-resolver-s4zgk\" (UID: \"59f4f5d3-9650-40e0-969a-e912429a2481\") " pod="openshift-dns/node-resolver-s4zgk"
Jul 14 22:56:55.423187 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.423147    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pthrz"
Jul 14 22:56:55.431133 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.431107    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lzt25"
Jul 14 22:56:55.441790 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.441767    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7wnl4"
Jul 14 22:56:55.451850 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.451824    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-r5s76"
Jul 14 22:56:55.459062 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.459038    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vlbvv"
Jul 14 22:56:55.466993 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.466973    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s4zgk"
Jul 14 22:56:55.474838 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.474817    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwzlt"
Jul 14 22:56:55.482684 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.482665    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-c8c42"
Jul 14 22:56:55.489583 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.489564    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:56:55.579790 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.579758    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 22:56:55.740061 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.740020    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6bbh\" (UniqueName: \"kubernetes.io/projected/39492113-cc47-4511-bc7d-d0785b892ed9-kube-api-access-p6bbh\") pod \"network-check-target-pbtpg\" (UID: \"39492113-cc47-4511-bc7d-d0785b892ed9\") " pod="openshift-network-diagnostics/network-check-target-pbtpg"
Jul 14 22:56:55.740267 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:55.740214    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 22:56:55.740267 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:55.740243    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 22:56:55.740267 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:55.740257    2563 projected.go:196] Error preparing data for projected volume kube-api-access-p6bbh for pod openshift-network-diagnostics/network-check-target-pbtpg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 22:56:55.740474 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:55.740350    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39492113-cc47-4511-bc7d-d0785b892ed9-kube-api-access-p6bbh podName:39492113-cc47-4511-bc7d-d0785b892ed9 nodeName:}" failed. No retries permitted until 2026-07-14 22:56:56.74033337 +0000 UTC m=+4.043324142 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-p6bbh" (UniqueName: "kubernetes.io/projected/39492113-cc47-4511-bc7d-d0785b892ed9-kube-api-access-p6bbh") pod "network-check-target-pbtpg" (UID: "39492113-cc47-4511-bc7d-d0785b892ed9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 22:56:55.841170 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.841131    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594-metrics-certs\") pod \"network-metrics-daemon-wjpcm\" (UID: \"8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594\") " pod="openshift-multus/network-metrics-daemon-wjpcm"
Jul 14 22:56:55.841411 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:55.841204    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e3e444e-1753-442e-8430-24b0022f9418-original-pull-secret\") pod \"global-pull-secret-syncer-xjw4z\" (UID: \"7e3e444e-1753-442e-8430-24b0022f9418\") " pod="kube-system/global-pull-secret-syncer-xjw4z"
Jul 14 22:56:55.841411 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:55.841302    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 22:56:55.841411 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:55.841330    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 22:56:55.841411 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:55.841382    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e3e444e-1753-442e-8430-24b0022f9418-original-pull-secret podName:7e3e444e-1753-442e-8430-24b0022f9418 nodeName:}" failed. No retries permitted until 2026-07-14 22:56:56.841366857 +0000 UTC m=+4.144357632 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e3e444e-1753-442e-8430-24b0022f9418-original-pull-secret") pod "global-pull-secret-syncer-xjw4z" (UID: "7e3e444e-1753-442e-8430-24b0022f9418") : object "kube-system"/"original-pull-secret" not registered
Jul 14 22:56:55.841411 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:55.841396    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594-metrics-certs podName:8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594 nodeName:}" failed. No retries permitted until 2026-07-14 22:56:56.841389842 +0000 UTC m=+4.144380615 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594-metrics-certs") pod "network-metrics-daemon-wjpcm" (UID: "8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 22:56:56.163443 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:56.163348    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 22:51:56 +0000 UTC" deadline="2028-04-26 03:53:37.84642797 +0000 UTC"
Jul 14 22:56:56.163443 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:56.163380    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15628h56m41.68305169s"
Jul 14 22:56:56.373432 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:56.373400    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod214ee792_0c28_4bcb_b213_4e55d0a69810.slice/crio-3c1d90503ddb88080fab82f6f6b8465a8ec9bf0da1f3e1cca8710cda9f571d0e WatchSource:0}: Error finding container 3c1d90503ddb88080fab82f6f6b8465a8ec9bf0da1f3e1cca8710cda9f571d0e: Status 404 returned error can't find the container with id 3c1d90503ddb88080fab82f6f6b8465a8ec9bf0da1f3e1cca8710cda9f571d0e
Jul 14 22:56:56.374741 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:56.374716    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb9705e3_b1b7_44a7_b4b8_49db0e928da9.slice/crio-edf18c8fe9ce7e080f55016dfcfb1d085f9da26eee2022f31f72a1541f1badbc WatchSource:0}: Error finding container edf18c8fe9ce7e080f55016dfcfb1d085f9da26eee2022f31f72a1541f1badbc: Status 404 returned error can't find the container with id edf18c8fe9ce7e080f55016dfcfb1d085f9da26eee2022f31f72a1541f1badbc
Jul 14 22:56:56.376203 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:56.376183    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ef7f81d_618b_4d9a_b018_9e3346c27bd1.slice/crio-49b3a9a0c3a85bc26076c836298fc4afa177e3962ff5e3c956e349f13570aa51 WatchSource:0}: Error finding container 49b3a9a0c3a85bc26076c836298fc4afa177e3962ff5e3c956e349f13570aa51: Status 404 returned error can't find the container with id 49b3a9a0c3a85bc26076c836298fc4afa177e3962ff5e3c956e349f13570aa51
Jul 14 22:56:56.381137 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:56.381107    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b4a8b8e_6c69_4a3d_9168_a217d09d53d0.slice/crio-2b242cd10181b12fc062629ed046c4b66c855a9a224dfa62e6e60e70ba74c8b9 WatchSource:0}: Error finding container 2b242cd10181b12fc062629ed046c4b66c855a9a224dfa62e6e60e70ba74c8b9: Status 404 returned error can't find the container with id 2b242cd10181b12fc062629ed046c4b66c855a9a224dfa62e6e60e70ba74c8b9
Jul 14 22:56:56.385208 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:56.385170    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda7d977f_1dd0_42e6_b3a7_455490097670.slice/crio-22e4b111d10b3a4de43953222ea1843861b813738765d981a6d8ca442ee2e9d8 WatchSource:0}: Error finding container 22e4b111d10b3a4de43953222ea1843861b813738765d981a6d8ca442ee2e9d8: Status 404 returned error can't find the container with id 22e4b111d10b3a4de43953222ea1843861b813738765d981a6d8ca442ee2e9d8
Jul 14 22:56:56.388272 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:56:56.387917    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59f4f5d3_9650_40e0_969a_e912429a2481.slice/crio-a8e747e819db6a2d206a792327320650b5fa5ab7188d923d9300ef5cf56aed02 WatchSource:0}: Error finding container a8e747e819db6a2d206a792327320650b5fa5ab7188d923d9300ef5cf56aed02: Status 404 returned error can't find the container with id a8e747e819db6a2d206a792327320650b5fa5ab7188d923d9300ef5cf56aed02
Jul 14 22:56:56.748033 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:56.747620    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6bbh\" (UniqueName: \"kubernetes.io/projected/39492113-cc47-4511-bc7d-d0785b892ed9-kube-api-access-p6bbh\") pod \"network-check-target-pbtpg\" (UID: \"39492113-cc47-4511-bc7d-d0785b892ed9\") " pod="openshift-network-diagnostics/network-check-target-pbtpg"
Jul 14 22:56:56.748033 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:56.747958    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 22:56:56.748383 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:56.748075    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 22:56:56.748383 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:56.748094    2563 projected.go:196] Error preparing data for projected volume kube-api-access-p6bbh for pod openshift-network-diagnostics/network-check-target-pbtpg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 22:56:56.748383 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:56.748176    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39492113-cc47-4511-bc7d-d0785b892ed9-kube-api-access-p6bbh podName:39492113-cc47-4511-bc7d-d0785b892ed9 nodeName:}" failed. No retries permitted until 2026-07-14 22:56:58.748138463 +0000 UTC m=+6.051129234 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-p6bbh" (UniqueName: "kubernetes.io/projected/39492113-cc47-4511-bc7d-d0785b892ed9-kube-api-access-p6bbh") pod "network-check-target-pbtpg" (UID: "39492113-cc47-4511-bc7d-d0785b892ed9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 22:56:56.849077 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:56.849042    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e3e444e-1753-442e-8430-24b0022f9418-original-pull-secret\") pod \"global-pull-secret-syncer-xjw4z\" (UID: \"7e3e444e-1753-442e-8430-24b0022f9418\") " pod="kube-system/global-pull-secret-syncer-xjw4z"
Jul 14 22:56:56.849077 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:56.849092    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594-metrics-certs\") pod \"network-metrics-daemon-wjpcm\" (UID: \"8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594\") " pod="openshift-multus/network-metrics-daemon-wjpcm"
Jul 14 22:56:56.849357 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:56.849210    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 22:56:56.849357 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:56.849259    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594-metrics-certs podName:8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594 nodeName:}" failed. No retries permitted until 2026-07-14 22:56:58.849244964 +0000 UTC m=+6.152235739 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594-metrics-certs") pod "network-metrics-daemon-wjpcm" (UID: "8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 22:56:56.849731 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:56.849609    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 22:56:56.849731 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:56.849683    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e3e444e-1753-442e-8430-24b0022f9418-original-pull-secret podName:7e3e444e-1753-442e-8430-24b0022f9418 nodeName:}" failed. No retries permitted until 2026-07-14 22:56:58.849663327 +0000 UTC m=+6.152654114 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e3e444e-1753-442e-8430-24b0022f9418-original-pull-secret") pod "global-pull-secret-syncer-xjw4z" (UID: "7e3e444e-1753-442e-8430-24b0022f9418") : object "kube-system"/"original-pull-secret" not registered
Jul 14 22:56:57.163699 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:57.163575    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 22:51:56 +0000 UTC" deadline="2028-02-08 10:50:29.069562472 +0000 UTC"
Jul 14 22:56:57.163699 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:57.163646    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13763h53m31.905921313s"
Jul 14 22:56:57.224832 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:57.224795    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wjpcm"
Jul 14 22:56:57.225018 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:57.224963    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wjpcm" podUID="8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594"
Jul 14 22:56:57.225436 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:57.225414    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xjw4z"
Jul 14 22:56:57.225547 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:57.225522    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xjw4z" podUID="7e3e444e-1753-442e-8430-24b0022f9418"
Jul 14 22:56:57.225698 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:57.225679    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pbtpg"
Jul 14 22:56:57.225789 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:57.225768    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pbtpg" podUID="39492113-cc47-4511-bc7d-d0785b892ed9"
Jul 14 22:56:57.255760 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:57.255709    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s4zgk" event={"ID":"59f4f5d3-9650-40e0-969a-e912429a2481","Type":"ContainerStarted","Data":"a8e747e819db6a2d206a792327320650b5fa5ab7188d923d9300ef5cf56aed02"}
Jul 14 22:56:57.257422 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:57.257339    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pthrz" event={"ID":"186005d6-1e47-4ea0-816c-469b21cc77d4","Type":"ContainerStarted","Data":"fb8f20d8db55fe3a6736ced2e3cc3050768f37396d4003c72edc1c7689130ed3"}
Jul 14 22:56:57.260535 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:57.260411    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7wnl4" event={"ID":"eda746ab-d0ce-4007-9db1-f6ee9cbdc317","Type":"ContainerStarted","Data":"0a0b1c295d1079c39a0529201c455e5b2a5f84a95aae23c79dc327aa085736f7"}
Jul 14 22:56:57.266980 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:57.266945    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r5s76" event={"ID":"da7d977f-1dd0-42e6-b3a7-455490097670","Type":"ContainerStarted","Data":"22e4b111d10b3a4de43953222ea1843861b813738765d981a6d8ca442ee2e9d8"}
Jul 14 22:56:57.269965 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:57.269881    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lzt25" event={"ID":"1ef7f81d-618b-4d9a-b018-9e3346c27bd1","Type":"ContainerStarted","Data":"49b3a9a0c3a85bc26076c836298fc4afa177e3962ff5e3c956e349f13570aa51"}
Jul 14 22:56:57.274058 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:57.273945    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s" event={"ID":"214ee792-0c28-4bcb-b213-4e55d0a69810","Type":"ContainerStarted","Data":"3c1d90503ddb88080fab82f6f6b8465a8ec9bf0da1f3e1cca8710cda9f571d0e"}
Jul 14 22:56:57.281414 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:57.280894    2563 generic.go:356] "Generic (PLEG): container finished" podID="e408b8465ac9f8b60353a9a16a836248" containerID="0600ce6be6b36df96bf0823e781d9cd9fe15fdb21de1bcb4c0344519db98113e" exitCode=0
Jul 14 22:56:57.281414 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:57.281189    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-8.ec2.internal" event={"ID":"e408b8465ac9f8b60353a9a16a836248","Type":"ContainerDied","Data":"0600ce6be6b36df96bf0823e781d9cd9fe15fdb21de1bcb4c0344519db98113e"}
Jul 14 22:56:57.298536 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:57.298445    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwzlt" event={"ID":"39e135b7-a5a9-4b22-a87b-d21a0f6657a6","Type":"ContainerStarted","Data":"5526680bf337d356c5277c7c266cc46c62ed728b365466632311310f23d384c1"}
Jul 14 22:56:57.334100 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:57.325597    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-c8c42" event={"ID":"3b4a8b8e-6c69-4a3d-9168-a217d09d53d0","Type":"ContainerStarted","Data":"2b242cd10181b12fc062629ed046c4b66c855a9a224dfa62e6e60e70ba74c8b9"}
Jul 14 22:56:57.341803 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:57.341654    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vlbvv" event={"ID":"cb9705e3-b1b7-44a7-b4b8-49db0e928da9","Type":"ContainerStarted","Data":"edf18c8fe9ce7e080f55016dfcfb1d085f9da26eee2022f31f72a1541f1badbc"}
Jul 14 22:56:57.355344 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:57.355304    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-8.ec2.internal" event={"ID":"3aa82a5123efb70e24ea524a0aeee7ed","Type":"ContainerStarted","Data":"11950572c6a96c2aebd107d6285b66835b1c7d81526ea4d18ab89ef8e0df26ea"}
Jul 14 22:56:58.380942 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:58.380892    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-8.ec2.internal" event={"ID":"e408b8465ac9f8b60353a9a16a836248","Type":"ContainerStarted","Data":"68fd253750643e4a378e9713bbbce9018dfcb635c97a8c0575712b82e8f2d78d"}
Jul 14 22:56:58.399831 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:58.399763    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-8.ec2.internal" podStartSLOduration=3.399738513 podStartE2EDuration="3.399738513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 22:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:56:57.373351819 +0000 UTC m=+4.676342616" watchObservedRunningTime="2026-07-14 22:56:58.399738513 +0000 UTC m=+5.702729310"
Jul 14 22:56:58.775784 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:58.774959    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6bbh\" (UniqueName: \"kubernetes.io/projected/39492113-cc47-4511-bc7d-d0785b892ed9-kube-api-access-p6bbh\") pod \"network-check-target-pbtpg\" (UID: \"39492113-cc47-4511-bc7d-d0785b892ed9\") " pod="openshift-network-diagnostics/network-check-target-pbtpg"
Jul 14 22:56:58.775784 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:58.775214    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 22:56:58.775784 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:58.775249    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 22:56:58.775784 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:58.775266    2563 projected.go:196] Error preparing data for projected volume kube-api-access-p6bbh for pod openshift-network-diagnostics/network-check-target-pbtpg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 22:56:58.775784 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:58.775367    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39492113-cc47-4511-bc7d-d0785b892ed9-kube-api-access-p6bbh podName:39492113-cc47-4511-bc7d-d0785b892ed9 nodeName:}" failed. No retries permitted until 2026-07-14 22:57:02.775324737 +0000 UTC m=+10.078315509 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-p6bbh" (UniqueName: "kubernetes.io/projected/39492113-cc47-4511-bc7d-d0785b892ed9-kube-api-access-p6bbh") pod "network-check-target-pbtpg" (UID: "39492113-cc47-4511-bc7d-d0785b892ed9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 22:56:58.876643 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:58.875774    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e3e444e-1753-442e-8430-24b0022f9418-original-pull-secret\") pod \"global-pull-secret-syncer-xjw4z\" (UID: \"7e3e444e-1753-442e-8430-24b0022f9418\") " pod="kube-system/global-pull-secret-syncer-xjw4z"
Jul 14 22:56:58.876643 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:58.875841    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594-metrics-certs\") pod \"network-metrics-daemon-wjpcm\" (UID: \"8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594\") " pod="openshift-multus/network-metrics-daemon-wjpcm"
Jul 14 22:56:58.876643 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:58.876043    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 22:56:58.876643 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:58.876108    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594-metrics-certs podName:8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594 nodeName:}" failed. No retries permitted until 2026-07-14 22:57:02.876087132 +0000 UTC m=+10.179077907 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594-metrics-certs") pod "network-metrics-daemon-wjpcm" (UID: "8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 22:56:58.876643 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:58.876549    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 22:56:58.876643 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:58.876604    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e3e444e-1753-442e-8430-24b0022f9418-original-pull-secret podName:7e3e444e-1753-442e-8430-24b0022f9418 nodeName:}" failed. No retries permitted until 2026-07-14 22:57:02.876585649 +0000 UTC m=+10.179576426 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e3e444e-1753-442e-8430-24b0022f9418-original-pull-secret") pod "global-pull-secret-syncer-xjw4z" (UID: "7e3e444e-1753-442e-8430-24b0022f9418") : object "kube-system"/"original-pull-secret" not registered
Jul 14 22:56:59.225412 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:59.224476    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wjpcm"
Jul 14 22:56:59.225412 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:59.224627    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wjpcm" podUID="8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594"
Jul 14 22:56:59.225412 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:59.225071    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xjw4z"
Jul 14 22:56:59.225412 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:59.225169    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xjw4z" podUID="7e3e444e-1753-442e-8430-24b0022f9418"
Jul 14 22:56:59.226176 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:56:59.225887    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pbtpg"
Jul 14 22:56:59.226176 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:56:59.226010    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pbtpg" podUID="39492113-cc47-4511-bc7d-d0785b892ed9"
Jul 14 22:57:01.226151 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:01.226106    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wjpcm"
Jul 14 22:57:01.226622 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:01.226259    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wjpcm" podUID="8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594"
Jul 14 22:57:01.226688 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:01.226660    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xjw4z"
Jul 14 22:57:01.226776 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:01.226755    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xjw4z" podUID="7e3e444e-1753-442e-8430-24b0022f9418"
Jul 14 22:57:01.226833 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:01.226804    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pbtpg"
Jul 14 22:57:01.226894 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:01.226876    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pbtpg" podUID="39492113-cc47-4511-bc7d-d0785b892ed9"
Jul 14 22:57:02.812189 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:02.812152    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6bbh\" (UniqueName: \"kubernetes.io/projected/39492113-cc47-4511-bc7d-d0785b892ed9-kube-api-access-p6bbh\") pod \"network-check-target-pbtpg\" (UID: \"39492113-cc47-4511-bc7d-d0785b892ed9\") " pod="openshift-network-diagnostics/network-check-target-pbtpg"
Jul 14 22:57:02.812690 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:02.812360    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 22:57:02.812690 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:02.812386    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 22:57:02.812690 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:02.812399    2563 projected.go:196] Error preparing data for projected volume kube-api-access-p6bbh for pod openshift-network-diagnostics/network-check-target-pbtpg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 22:57:02.812690 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:02.812466    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39492113-cc47-4511-bc7d-d0785b892ed9-kube-api-access-p6bbh podName:39492113-cc47-4511-bc7d-d0785b892ed9 nodeName:}" failed. No retries permitted until 2026-07-14 22:57:10.812444035 +0000 UTC m=+18.115434825 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-p6bbh" (UniqueName: "kubernetes.io/projected/39492113-cc47-4511-bc7d-d0785b892ed9-kube-api-access-p6bbh") pod "network-check-target-pbtpg" (UID: "39492113-cc47-4511-bc7d-d0785b892ed9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 22:57:02.913448 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:02.913408    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e3e444e-1753-442e-8430-24b0022f9418-original-pull-secret\") pod \"global-pull-secret-syncer-xjw4z\" (UID: \"7e3e444e-1753-442e-8430-24b0022f9418\") " pod="kube-system/global-pull-secret-syncer-xjw4z"
Jul 14 22:57:02.913659 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:02.913473    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594-metrics-certs\") pod \"network-metrics-daemon-wjpcm\" (UID: \"8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594\") " pod="openshift-multus/network-metrics-daemon-wjpcm"
Jul 14 22:57:02.913659 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:02.913614    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 22:57:02.913782 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:02.913680    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594-metrics-certs podName:8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594 nodeName:}" failed. No retries permitted until 2026-07-14 22:57:10.913660858 +0000 UTC m=+18.216651646 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594-metrics-certs") pod "network-metrics-daemon-wjpcm" (UID: "8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 22:57:02.914156 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:02.914136    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 22:57:02.914250 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:02.914195    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e3e444e-1753-442e-8430-24b0022f9418-original-pull-secret podName:7e3e444e-1753-442e-8430-24b0022f9418 nodeName:}" failed. No retries permitted until 2026-07-14 22:57:10.914178737 +0000 UTC m=+18.217169518 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e3e444e-1753-442e-8430-24b0022f9418-original-pull-secret") pod "global-pull-secret-syncer-xjw4z" (UID: "7e3e444e-1753-442e-8430-24b0022f9418") : object "kube-system"/"original-pull-secret" not registered
Jul 14 22:57:03.228492 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:03.227770    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wjpcm"
Jul 14 22:57:03.228492 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:03.227918    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wjpcm" podUID="8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594"
Jul 14 22:57:03.228492 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:03.228328    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xjw4z"
Jul 14 22:57:03.229112 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:03.228936    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xjw4z" podUID="7e3e444e-1753-442e-8430-24b0022f9418"
Jul 14 22:57:03.229112 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:03.229004    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pbtpg"
Jul 14 22:57:03.229112 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:03.229081    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pbtpg" podUID="39492113-cc47-4511-bc7d-d0785b892ed9"
Jul 14 22:57:05.224888 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:05.224851    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pbtpg"
Jul 14 22:57:05.225320 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:05.224996    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pbtpg" podUID="39492113-cc47-4511-bc7d-d0785b892ed9"
Jul 14 22:57:05.225416 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:05.225402    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wjpcm"
Jul 14 22:57:05.225456 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:05.225422    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xjw4z"
Jul 14 22:57:05.225537 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:05.225514    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wjpcm" podUID="8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594"
Jul 14 22:57:05.225632 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:05.225614    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xjw4z" podUID="7e3e444e-1753-442e-8430-24b0022f9418"
Jul 14 22:57:07.223712 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:07.223676    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pbtpg"
Jul 14 22:57:07.224240 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:07.223686    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wjpcm"
Jul 14 22:57:07.224240 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:07.223801    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pbtpg" podUID="39492113-cc47-4511-bc7d-d0785b892ed9"
Jul 14 22:57:07.224240 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:07.223914    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wjpcm" podUID="8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594"
Jul 14 22:57:07.224240 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:07.223984    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xjw4z"
Jul 14 22:57:07.224240 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:07.224062    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xjw4z" podUID="7e3e444e-1753-442e-8430-24b0022f9418"
Jul 14 22:57:09.224033 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:09.223994    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pbtpg"
Jul 14 22:57:09.224495 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:09.223994    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wjpcm"
Jul 14 22:57:09.224495 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:09.224126    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pbtpg" podUID="39492113-cc47-4511-bc7d-d0785b892ed9"
Jul 14 22:57:09.224495 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:09.223994    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xjw4z"
Jul 14 22:57:09.224495 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:09.224214    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wjpcm" podUID="8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594"
Jul 14 22:57:09.224495 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:09.224298    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xjw4z" podUID="7e3e444e-1753-442e-8430-24b0022f9418"
Jul 14 22:57:10.879342 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:10.879302    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6bbh\" (UniqueName: \"kubernetes.io/projected/39492113-cc47-4511-bc7d-d0785b892ed9-kube-api-access-p6bbh\") pod \"network-check-target-pbtpg\" (UID: \"39492113-cc47-4511-bc7d-d0785b892ed9\") " pod="openshift-network-diagnostics/network-check-target-pbtpg"
Jul 14 22:57:10.879793 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:10.879516    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 22:57:10.879793 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:10.879542    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 22:57:10.879793 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:10.879557    2563 projected.go:196] Error preparing data for projected volume kube-api-access-p6bbh for pod openshift-network-diagnostics/network-check-target-pbtpg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 22:57:10.879793 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:10.879628    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39492113-cc47-4511-bc7d-d0785b892ed9-kube-api-access-p6bbh podName:39492113-cc47-4511-bc7d-d0785b892ed9 nodeName:}" failed. No retries permitted until 2026-07-14 22:57:26.879605391 +0000 UTC m=+34.182596163 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-p6bbh" (UniqueName: "kubernetes.io/projected/39492113-cc47-4511-bc7d-d0785b892ed9-kube-api-access-p6bbh") pod "network-check-target-pbtpg" (UID: "39492113-cc47-4511-bc7d-d0785b892ed9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 22:57:10.979833 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:10.979796    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e3e444e-1753-442e-8430-24b0022f9418-original-pull-secret\") pod \"global-pull-secret-syncer-xjw4z\" (UID: \"7e3e444e-1753-442e-8430-24b0022f9418\") " pod="kube-system/global-pull-secret-syncer-xjw4z"
Jul 14 22:57:10.979987 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:10.979855    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594-metrics-certs\") pod \"network-metrics-daemon-wjpcm\" (UID: \"8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594\") " pod="openshift-multus/network-metrics-daemon-wjpcm"
Jul 14 22:57:10.980042 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:10.979981    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 22:57:10.980042 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:10.979994    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 22:57:10.980132 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:10.980061    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e3e444e-1753-442e-8430-24b0022f9418-original-pull-secret podName:7e3e444e-1753-442e-8430-24b0022f9418 nodeName:}" failed. No retries permitted until 2026-07-14 22:57:26.980041149 +0000 UTC m=+34.283031931 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e3e444e-1753-442e-8430-24b0022f9418-original-pull-secret") pod "global-pull-secret-syncer-xjw4z" (UID: "7e3e444e-1753-442e-8430-24b0022f9418") : object "kube-system"/"original-pull-secret" not registered
Jul 14 22:57:10.980132 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:10.980081    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594-metrics-certs podName:8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594 nodeName:}" failed. No retries permitted until 2026-07-14 22:57:26.980071677 +0000 UTC m=+34.283062450 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594-metrics-certs") pod "network-metrics-daemon-wjpcm" (UID: "8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 22:57:11.224017 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:11.223935    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xjw4z"
Jul 14 22:57:11.224181 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:11.224083    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xjw4z" podUID="7e3e444e-1753-442e-8430-24b0022f9418"
Jul 14 22:57:11.224403 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:11.224369    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pbtpg"
Jul 14 22:57:11.224530 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:11.224500    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pbtpg" podUID="39492113-cc47-4511-bc7d-d0785b892ed9"
Jul 14 22:57:11.224642 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:11.224613    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wjpcm"
Jul 14 22:57:11.224835 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:11.224791    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wjpcm" podUID="8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594"
Jul 14 22:57:13.224650 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:13.224616    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pbtpg"
Jul 14 22:57:13.224650 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:13.224654    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xjw4z"
Jul 14 22:57:13.225084 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:13.224727    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pbtpg" podUID="39492113-cc47-4511-bc7d-d0785b892ed9"
Jul 14 22:57:13.225084 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:13.224756    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xjw4z" podUID="7e3e444e-1753-442e-8430-24b0022f9418"
Jul 14 22:57:13.225084 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:13.224801    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wjpcm"
Jul 14 22:57:13.225084 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:13.224876    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wjpcm" podUID="8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594"
Jul 14 22:57:15.224391 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:15.224355    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wjpcm"
Jul 14 22:57:15.224391 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:15.224385    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xjw4z"
Jul 14 22:57:15.224895 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:15.224354    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pbtpg"
Jul 14 22:57:15.224895 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:15.224499    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wjpcm" podUID="8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594"
Jul 14 22:57:15.224895 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:15.224567    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pbtpg" podUID="39492113-cc47-4511-bc7d-d0785b892ed9"
Jul 14 22:57:15.224895 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:15.224639    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xjw4z" podUID="7e3e444e-1753-442e-8430-24b0022f9418"
Jul 14 22:57:17.224842 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:17.224481    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xjw4z"
Jul 14 22:57:17.225543 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:17.224481    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wjpcm"
Jul 14 22:57:17.225543 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:17.224919    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xjw4z" podUID="7e3e444e-1753-442e-8430-24b0022f9418"
Jul 14 22:57:17.225543 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:17.224482    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pbtpg"
Jul 14 22:57:17.225543 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:17.225030    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wjpcm" podUID="8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594"
Jul 14 22:57:17.225543 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:17.225110    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pbtpg" podUID="39492113-cc47-4511-bc7d-d0785b892ed9"
Jul 14 22:57:17.424063 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:17.424021    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s4zgk" event={"ID":"59f4f5d3-9650-40e0-969a-e912429a2481","Type":"ContainerStarted","Data":"e7834ce07e6707d9858265a1f19ab5ce5d691a0127b10b8258af7a914c98e648"}
Jul 14 22:57:17.425492 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:17.425463    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pthrz" event={"ID":"186005d6-1e47-4ea0-816c-469b21cc77d4","Type":"ContainerStarted","Data":"2775c5a93b7d35266df75223ef096a53866ab8ef6cebcda689de2a1f536efd13"}
Jul 14 22:57:17.426785 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:17.426753    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7wnl4" event={"ID":"eda746ab-d0ce-4007-9db1-f6ee9cbdc317","Type":"ContainerStarted","Data":"04eb127fe9df8c062619fe588dcaf8bb3f89ca3e66ec65c2741bbeea1f2ec075"}
Jul 14 22:57:17.428330 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:17.428305    2563 generic.go:356] "Generic (PLEG): container finished" podID="da7d977f-1dd0-42e6-b3a7-455490097670" containerID="dd7f1863dee9f94d9f954b2d529daf8a2db40ad14ec11d877c0341fcd0c7b2f0" exitCode=0
Jul 14 22:57:17.428422 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:17.428384    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r5s76" event={"ID":"da7d977f-1dd0-42e6-b3a7-455490097670","Type":"ContainerDied","Data":"dd7f1863dee9f94d9f954b2d529daf8a2db40ad14ec11d877c0341fcd0c7b2f0"}
Jul 14 22:57:17.429803 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:17.429747    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lzt25" event={"ID":"1ef7f81d-618b-4d9a-b018-9e3346c27bd1","Type":"ContainerStarted","Data":"7138872d5c503dda57032df3eb23032d625750fad4e5da616b5c8bda22ecae15"}
Jul 14 22:57:17.432940 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:17.432921    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s" event={"ID":"214ee792-0c28-4bcb-b213-4e55d0a69810","Type":"ContainerStarted","Data":"d046834d79e2f657a236411d35be7076735d0007d090a8cd36072e87569f8b23"}
Jul 14 22:57:17.433125 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:17.432972    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s" event={"ID":"214ee792-0c28-4bcb-b213-4e55d0a69810","Type":"ContainerStarted","Data":"e21ba9cbb578bfd884d4708c2f780ac8caf63273c4466894f6e4ac63e7c08b8a"}
Jul 14 22:57:17.433125 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:17.433020    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s" event={"ID":"214ee792-0c28-4bcb-b213-4e55d0a69810","Type":"ContainerStarted","Data":"6f3d9254eaf07808d3ff94774ca17bfd2f21b9d2e1c0f7493c783820e1147b3d"}
Jul 14 22:57:17.433125 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:17.433049    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s" event={"ID":"214ee792-0c28-4bcb-b213-4e55d0a69810","Type":"ContainerStarted","Data":"a3c4f3513d46cab0f0ec94e61d8e8404ccf7517be6f282ee33cc6ff5bfd82ab2"}
Jul 14 22:57:17.433125 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:17.433084    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s" event={"ID":"214ee792-0c28-4bcb-b213-4e55d0a69810","Type":"ContainerStarted","Data":"9cc8e6c067fb7730c49a500efd8e43932e8ef9e6cacf74080dd80b54d4bc14c5"}
Jul 14 22:57:17.433125 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:17.433113    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s" event={"ID":"214ee792-0c28-4bcb-b213-4e55d0a69810","Type":"ContainerStarted","Data":"cb9b3571990ee6d3aaa8e449284a4464ad0f0006931107b3090ec9d6f19a35d6"}
Jul 14 22:57:17.434425 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:17.434395    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwzlt" event={"ID":"39e135b7-a5a9-4b22-a87b-d21a0f6657a6","Type":"ContainerStarted","Data":"5b6d00be182b6eb4531859c14a31cafc7d78b0a747c04fe550315e5523fa0cb8"}
Jul 14 22:57:17.435978 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:17.435962    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vlbvv" event={"ID":"cb9705e3-b1b7-44a7-b4b8-49db0e928da9","Type":"ContainerStarted","Data":"467d9b3240ad55042df7d1338b0c4f97a65520429cd47ebc60655f359553b929"}
Jul 14 22:57:17.439941 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:17.439909    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-8.ec2.internal" podStartSLOduration=22.439882042 podStartE2EDuration="22.439882042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 22:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:56:58.399612225 +0000 UTC m=+5.702603017" watchObservedRunningTime="2026-07-14 22:57:17.439882042 +0000 UTC m=+24.742872824"
Jul 14 22:57:17.440582 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:17.440555    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-s4zgk" podStartSLOduration=3.335018751 podStartE2EDuration="23.440545794s" totalImagesPullingTime="20.105527043s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 22:56:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:57:17.439830754 +0000 UTC m=+24.742821547" watchObservedRunningTime="2026-07-14 22:57:17.440545794 +0000 UTC m=+24.743536585"
Jul 14 22:57:17.457014 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:17.456958    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pthrz" podStartSLOduration=3.378362791 podStartE2EDuration="23.456945407s" totalImagesPullingTime="20.078582616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:56:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:57:17.456500786 +0000 UTC m=+24.759491559" watchObservedRunningTime="2026-07-14 22:57:17.456945407 +0000 UTC m=+24.759936199"
Jul 14 22:57:17.471529 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:17.471486    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lzt25" podStartSLOduration=3.378956563 podStartE2EDuration="23.47146952s" totalImagesPullingTime="20.092512957s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 22:56:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:57:17.471114149 +0000 UTC m=+24.774104943" watchObservedRunningTime="2026-07-14 22:57:17.47146952 +0000 UTC m=+24.774460312"
Jul 14 22:57:17.489078 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:17.489034    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7wnl4" podStartSLOduration=3.407949893 podStartE2EDuration="23.489019221s" totalImagesPullingTime="20.081069328s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 22:56:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:57:17.488482202 +0000 UTC m=+24.791472993" watchObservedRunningTime="2026-07-14 22:57:17.489019221 +0000 UTC m=+24.792010013"
Jul 14 22:57:17.506634 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:17.506444    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vlbvv" podStartSLOduration=3.364889413 podStartE2EDuration="23.506431304s" totalImagesPullingTime="20.141541891s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 22:56:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:57:17.505838357 +0000 UTC m=+24.808829149" watchObservedRunningTime="2026-07-14 22:57:17.506431304 +0000 UTC m=+24.809422128"
Jul 14 22:57:17.984842 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:17.984818    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 14 22:57:18.187423 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:18.187250    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-14T22:57:17.984838483Z","UUID":"eb583ce5-e863-4b67-939c-3c27e16ea730","Handler":null,"Name":"","Endpoint":""}
Jul 14 22:57:18.188887 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:18.188858    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 14 22:57:18.189010 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:18.188909    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 14 22:57:18.283362 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:18.283326    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pthrz"
Jul 14 22:57:18.284570 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:18.284550    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pthrz"
Jul 14 22:57:18.441240 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:18.441158    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwzlt" event={"ID":"39e135b7-a5a9-4b22-a87b-d21a0f6657a6","Type":"ContainerStarted","Data":"b3f6b0d49f52f1863a2df323953deec86f218a2026ecddd9ebaddd75bffb1a3c"}
Jul 14 22:57:18.443075 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:18.443046    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-c8c42" event={"ID":"3b4a8b8e-6c69-4a3d-9168-a217d09d53d0","Type":"ContainerStarted","Data":"d897fa6de465c1e1a7d2c06f370375cd5df2d8b224c55bb6a675089eb61eebdc"}
Jul 14 22:57:18.444325 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:18.444273    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pthrz"
Jul 14 22:57:18.444703 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:18.444677    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pthrz"
Jul 14 22:57:18.475170 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:18.475088    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-c8c42" podStartSLOduration=4.390299847 podStartE2EDuration="24.475054135s" totalImagesPullingTime="20.084754288s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 22:56:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:57:18.460660464 +0000 UTC m=+25.763651257" watchObservedRunningTime="2026-07-14 22:57:18.475054135 +0000 UTC m=+25.778044928"
Jul 14 22:57:19.223569 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:19.223480    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wjpcm"
Jul 14 22:57:19.223768 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:19.223606    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wjpcm" podUID="8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594"
Jul 14 22:57:19.223768 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:19.223642    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xjw4z"
Jul 14 22:57:19.223768 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:19.223711    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xjw4z" podUID="7e3e444e-1753-442e-8430-24b0022f9418"
Jul 14 22:57:19.223768 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:19.223767    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pbtpg"
Jul 14 22:57:19.223947 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:19.223831    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pbtpg" podUID="39492113-cc47-4511-bc7d-d0785b892ed9"
Jul 14 22:57:19.449788 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:19.449511    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s" event={"ID":"214ee792-0c28-4bcb-b213-4e55d0a69810","Type":"ContainerStarted","Data":"80ae8e20c05f9db2f7b8fc511596e8e2b40ba50f66ee7b6f5a64596b6b55fd97"}
Jul 14 22:57:19.451807 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:19.451723    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwzlt" event={"ID":"39e135b7-a5a9-4b22-a87b-d21a0f6657a6","Type":"ContainerStarted","Data":"510b071429377d80172416dad223a2454163c1b960a75f9537aa9fa12464545d"}
Jul 14 22:57:21.224604 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:21.224565    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wjpcm"
Jul 14 22:57:21.224604 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:21.224598    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pbtpg"
Jul 14 22:57:21.225313 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:21.224569    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xjw4z"
Jul 14 22:57:21.225313 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:21.224732    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wjpcm" podUID="8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594"
Jul 14 22:57:21.225313 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:21.224862    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xjw4z" podUID="7e3e444e-1753-442e-8430-24b0022f9418"
Jul 14 22:57:21.225313 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:21.224940    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pbtpg" podUID="39492113-cc47-4511-bc7d-d0785b892ed9"
Jul 14 22:57:22.461160 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:22.460950    2563 generic.go:356] "Generic (PLEG): container finished" podID="da7d977f-1dd0-42e6-b3a7-455490097670" containerID="9b64d8926bfb181d417020d714871631a84975685fb007cee225d47d9fdfab63" exitCode=0
Jul 14 22:57:22.461881 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:22.461035    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r5s76" event={"ID":"da7d977f-1dd0-42e6-b3a7-455490097670","Type":"ContainerDied","Data":"9b64d8926bfb181d417020d714871631a84975685fb007cee225d47d9fdfab63"}
Jul 14 22:57:22.465048 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:22.465023    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s" event={"ID":"214ee792-0c28-4bcb-b213-4e55d0a69810","Type":"ContainerStarted","Data":"95e22d19e389d0ffb9920c47e6bea2ba4912eef8a15e48421deb96461bba4c95"}
Jul 14 22:57:22.465319 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:22.465299    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:57:22.465425 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:22.465364    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:57:22.465425 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:22.465414    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:57:22.481137 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:22.481112    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:57:22.481258 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:22.481212    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:57:22.488583 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:22.488540    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwzlt" podStartSLOduration=6.086925169 podStartE2EDuration="28.488525304s" totalImagesPullingTime="22.401600135s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 22:56:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:57:19.471665609 +0000 UTC m=+26.774656401" watchObservedRunningTime="2026-07-14 22:57:22.488525304 +0000 UTC m=+29.791516145"
Jul 14 22:57:22.515901 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:22.515849    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s" podStartSLOduration=8.341533542 podStartE2EDuration="28.515834384s" totalImagesPullingTime="20.174300842s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 22:56:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:57:22.514174296 +0000 UTC m=+29.817165128" watchObservedRunningTime="2026-07-14 22:57:22.515834384 +0000 UTC m=+29.818825176"
Jul 14 22:57:23.224525 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:23.224498    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wjpcm"
Jul 14 22:57:23.224752 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:23.224578    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xjw4z"
Jul 14 22:57:23.224752 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:23.224583    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wjpcm" podUID="8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594"
Jul 14 22:57:23.224752 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:23.224691    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xjw4z" podUID="7e3e444e-1753-442e-8430-24b0022f9418"
Jul 14 22:57:23.224752 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:23.224699    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pbtpg"
Jul 14 22:57:23.224878 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:23.224800    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pbtpg" podUID="39492113-cc47-4511-bc7d-d0785b892ed9"
Jul 14 22:57:23.767052 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:23.766826    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xjw4z"]
Jul 14 22:57:23.767518 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:23.767200    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xjw4z"
Jul 14 22:57:23.767518 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:23.767336    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xjw4z" podUID="7e3e444e-1753-442e-8430-24b0022f9418"
Jul 14 22:57:23.767518 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:23.767375    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wjpcm"]
Jul 14 22:57:23.767518 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:23.767469    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wjpcm"
Jul 14 22:57:23.767704 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:23.767546    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wjpcm" podUID="8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594"
Jul 14 22:57:23.769652 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:23.769632    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pbtpg"]
Jul 14 22:57:23.769764 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:23.769728    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pbtpg"
Jul 14 22:57:23.769849 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:23.769827    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pbtpg" podUID="39492113-cc47-4511-bc7d-d0785b892ed9"
Jul 14 22:57:24.476552 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:24.476518    2563 generic.go:356] "Generic (PLEG): container finished" podID="da7d977f-1dd0-42e6-b3a7-455490097670" containerID="18519f079d035df71f537be71abbd4be50f63e84c1e3528ab455136cf9ffb9c9" exitCode=0
Jul 14 22:57:24.476751 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:24.476624    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r5s76" event={"ID":"da7d977f-1dd0-42e6-b3a7-455490097670","Type":"ContainerDied","Data":"18519f079d035df71f537be71abbd4be50f63e84c1e3528ab455136cf9ffb9c9"}
Jul 14 22:57:25.223476 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:25.223440    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pbtpg"
Jul 14 22:57:25.223476 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:25.223468    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xjw4z"
Jul 14 22:57:25.223960 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:25.223448    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wjpcm"
Jul 14 22:57:25.223960 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:25.223572    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pbtpg" podUID="39492113-cc47-4511-bc7d-d0785b892ed9"
Jul 14 22:57:25.223960 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:25.223708    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wjpcm" podUID="8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594"
Jul 14 22:57:25.223960 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:25.223793    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xjw4z" podUID="7e3e444e-1753-442e-8430-24b0022f9418"
Jul 14 22:57:26.483517 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:26.483480    2563 generic.go:356] "Generic (PLEG): container finished" podID="da7d977f-1dd0-42e6-b3a7-455490097670" containerID="9d22bb685030c265fcc3d8040979ade349390ba0d85497008bd0ff4628ee2fbc" exitCode=0
Jul 14 22:57:26.484161 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:26.483547    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r5s76" event={"ID":"da7d977f-1dd0-42e6-b3a7-455490097670","Type":"ContainerDied","Data":"9d22bb685030c265fcc3d8040979ade349390ba0d85497008bd0ff4628ee2fbc"}
Jul 14 22:57:26.892783 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:26.892730    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6bbh\" (UniqueName: \"kubernetes.io/projected/39492113-cc47-4511-bc7d-d0785b892ed9-kube-api-access-p6bbh\") pod \"network-check-target-pbtpg\" (UID: \"39492113-cc47-4511-bc7d-d0785b892ed9\") " pod="openshift-network-diagnostics/network-check-target-pbtpg"
Jul 14 22:57:26.892949 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:26.892893    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 22:57:26.892949 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:26.892912    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 22:57:26.892949 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:26.892923    2563 projected.go:196] Error preparing data for projected volume kube-api-access-p6bbh for pod openshift-network-diagnostics/network-check-target-pbtpg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 22:57:26.893051 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:26.892975    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39492113-cc47-4511-bc7d-d0785b892ed9-kube-api-access-p6bbh podName:39492113-cc47-4511-bc7d-d0785b892ed9 nodeName:}" failed. No retries permitted until 2026-07-14 22:57:58.892961457 +0000 UTC m=+66.195952231 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-p6bbh" (UniqueName: "kubernetes.io/projected/39492113-cc47-4511-bc7d-d0785b892ed9-kube-api-access-p6bbh") pod "network-check-target-pbtpg" (UID: "39492113-cc47-4511-bc7d-d0785b892ed9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 22:57:26.993424 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:26.993380    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e3e444e-1753-442e-8430-24b0022f9418-original-pull-secret\") pod \"global-pull-secret-syncer-xjw4z\" (UID: \"7e3e444e-1753-442e-8430-24b0022f9418\") " pod="kube-system/global-pull-secret-syncer-xjw4z"
Jul 14 22:57:26.993601 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:26.993438    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594-metrics-certs\") pod \"network-metrics-daemon-wjpcm\" (UID: \"8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594\") " pod="openshift-multus/network-metrics-daemon-wjpcm"
Jul 14 22:57:26.993601 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:26.993572    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 22:57:26.993703 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:26.993633    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594-metrics-certs podName:8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594 nodeName:}" failed. No retries permitted until 2026-07-14 22:57:58.993614012 +0000 UTC m=+66.296604788 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594-metrics-certs") pod "network-metrics-daemon-wjpcm" (UID: "8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 22:57:26.993840 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:26.993793    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 22:57:26.993923 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:26.993867    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e3e444e-1753-442e-8430-24b0022f9418-original-pull-secret podName:7e3e444e-1753-442e-8430-24b0022f9418 nodeName:}" failed. No retries permitted until 2026-07-14 22:57:58.993849 +0000 UTC m=+66.296839789 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e3e444e-1753-442e-8430-24b0022f9418-original-pull-secret") pod "global-pull-secret-syncer-xjw4z" (UID: "7e3e444e-1753-442e-8430-24b0022f9418") : object "kube-system"/"original-pull-secret" not registered
Jul 14 22:57:27.223645 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:27.223556    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pbtpg"
Jul 14 22:57:27.223645 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:27.223586    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wjpcm"
Jul 14 22:57:27.223645 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:27.223564    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xjw4z"
Jul 14 22:57:27.223896 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:27.223692    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pbtpg" podUID="39492113-cc47-4511-bc7d-d0785b892ed9"
Jul 14 22:57:27.223896 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:27.223815    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xjw4z" podUID="7e3e444e-1753-442e-8430-24b0022f9418"
Jul 14 22:57:27.224020 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:27.223912    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wjpcm" podUID="8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594"
Jul 14 22:57:29.224372 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.224336    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wjpcm"
Jul 14 22:57:29.224372 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.224370    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pbtpg"
Jul 14 22:57:29.224933 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.224336    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xjw4z"
Jul 14 22:57:29.224933 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:29.224474    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wjpcm" podUID="8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594"
Jul 14 22:57:29.224933 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:29.224566    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pbtpg" podUID="39492113-cc47-4511-bc7d-d0785b892ed9"
Jul 14 22:57:29.224933 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:29.224653    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xjw4z" podUID="7e3e444e-1753-442e-8430-24b0022f9418"
Jul 14 22:57:29.463027 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.462997    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-8.ec2.internal" event="NodeReady"
Jul 14 22:57:29.463208 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.463176    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 14 22:57:29.501113 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.501080    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7955774756-rvmg5"]
Jul 14 22:57:29.503769 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.503746    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7955774756-rvmg5"
Jul 14 22:57:29.506723 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.506698    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 14 22:57:29.506854 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.506807    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 14 22:57:29.506920 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.506880    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gm9qp\""
Jul 14 22:57:29.507318 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.507301    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 14 22:57:29.519442 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.518917    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 14 22:57:29.520035 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.519983    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-q4982"]
Jul 14 22:57:29.522068 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.522050    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vdd85"]
Jul 14 22:57:29.522213 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.522194    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q4982"
Jul 14 22:57:29.523805 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.523785    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7955774756-rvmg5"]
Jul 14 22:57:29.523903 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.523873    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vdd85"
Jul 14 22:57:29.525206 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.525185    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 14 22:57:29.525340 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.525221    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wdwsb\""
Jul 14 22:57:29.525340 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.525221    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 14 22:57:29.526748 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.526728    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 14 22:57:29.526848 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.526725    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 14 22:57:29.526848 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.526761    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 14 22:57:29.526848 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.526778    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-nt85j\""
Jul 14 22:57:29.531190 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.531166    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q4982"]
Jul 14 22:57:29.534332 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.534266    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vdd85"]
Jul 14 22:57:29.615768 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.615733    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-ca-trust-extracted\") pod \"image-registry-7955774756-rvmg5\" (UID: \"3a13b9a0-c0ff-4a8e-bfac-711ee2137471\") " pod="openshift-image-registry/image-registry-7955774756-rvmg5"
Jul 14 22:57:29.615921 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.615795    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-registry-certificates\") pod \"image-registry-7955774756-rvmg5\" (UID: \"3a13b9a0-c0ff-4a8e-bfac-711ee2137471\") " pod="openshift-image-registry/image-registry-7955774756-rvmg5"
Jul 14 22:57:29.615921 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.615883    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4be0471d-f724-4147-b1b0-c5ff7a82b7d1-cert\") pod \"ingress-canary-vdd85\" (UID: \"4be0471d-f724-4147-b1b0-c5ff7a82b7d1\") " pod="openshift-ingress-canary/ingress-canary-vdd85"
Jul 14 22:57:29.615921 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.615916    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ffb5cdb7-2bde-418f-884d-0962fc26414e-config-volume\") pod \"dns-default-q4982\" (UID: \"ffb5cdb7-2bde-418f-884d-0962fc26414e\") " pod="openshift-dns/dns-default-q4982"
Jul 14 22:57:29.616061 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.615940    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-registry-tls\") pod \"image-registry-7955774756-rvmg5\" (UID: \"3a13b9a0-c0ff-4a8e-bfac-711ee2137471\") " pod="openshift-image-registry/image-registry-7955774756-rvmg5"
Jul 14 22:57:29.616061 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.615959    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffb5cdb7-2bde-418f-884d-0962fc26414e-metrics-tls\") pod \"dns-default-q4982\" (UID: \"ffb5cdb7-2bde-418f-884d-0962fc26414e\") " pod="openshift-dns/dns-default-q4982"
Jul 14 22:57:29.616061 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.615982    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pnlf\" (UniqueName: \"kubernetes.io/projected/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-kube-api-access-2pnlf\") pod \"image-registry-7955774756-rvmg5\" (UID: \"3a13b9a0-c0ff-4a8e-bfac-711ee2137471\") " pod="openshift-image-registry/image-registry-7955774756-rvmg5"
Jul 14 22:57:29.616061 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.616021    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-trusted-ca\") pod \"image-registry-7955774756-rvmg5\" (UID: \"3a13b9a0-c0ff-4a8e-bfac-711ee2137471\") " pod="openshift-image-registry/image-registry-7955774756-rvmg5"
Jul 14 22:57:29.616061 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.616043    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffb5cdb7-2bde-418f-884d-0962fc26414e-tmp-dir\") pod \"dns-default-q4982\" (UID: \"ffb5cdb7-2bde-418f-884d-0962fc26414e\") " pod="openshift-dns/dns-default-q4982"
Jul 14 22:57:29.616266 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.616089    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm8w7\" (UniqueName: \"kubernetes.io/projected/ffb5cdb7-2bde-418f-884d-0962fc26414e-kube-api-access-vm8w7\") pod \"dns-default-q4982\" (UID: \"ffb5cdb7-2bde-418f-884d-0962fc26414e\") " pod="openshift-dns/dns-default-q4982"
Jul 14 22:57:29.616266 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.616136    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-bound-sa-token\") pod \"image-registry-7955774756-rvmg5\" (UID: \"3a13b9a0-c0ff-4a8e-bfac-711ee2137471\") " pod="openshift-image-registry/image-registry-7955774756-rvmg5"
Jul 14 22:57:29.616266 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.616196    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-installation-pull-secrets\") pod \"image-registry-7955774756-rvmg5\" (UID: \"3a13b9a0-c0ff-4a8e-bfac-711ee2137471\") " pod="openshift-image-registry/image-registry-7955774756-rvmg5"
Jul 14 22:57:29.616266 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.616235    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-image-registry-private-configuration\") pod \"image-registry-7955774756-rvmg5\" (UID: \"3a13b9a0-c0ff-4a8e-bfac-711ee2137471\") " pod="openshift-image-registry/image-registry-7955774756-rvmg5"
Jul 14 22:57:29.616449 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.616264    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-498w4\" (UniqueName: \"kubernetes.io/projected/4be0471d-f724-4147-b1b0-c5ff7a82b7d1-kube-api-access-498w4\") pod \"ingress-canary-vdd85\" (UID: \"4be0471d-f724-4147-b1b0-c5ff7a82b7d1\") " pod="openshift-ingress-canary/ingress-canary-vdd85"
Jul 14 22:57:29.717059 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.716975    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-installation-pull-secrets\") pod \"image-registry-7955774756-rvmg5\" (UID: \"3a13b9a0-c0ff-4a8e-bfac-711ee2137471\") " pod="openshift-image-registry/image-registry-7955774756-rvmg5"
Jul 14 22:57:29.717059 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.717017    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-image-registry-private-configuration\") pod \"image-registry-7955774756-rvmg5\" (UID: \"3a13b9a0-c0ff-4a8e-bfac-711ee2137471\") " pod="openshift-image-registry/image-registry-7955774756-rvmg5"
Jul 14 22:57:29.717059 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.717044    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-498w4\" (UniqueName: \"kubernetes.io/projected/4be0471d-f724-4147-b1b0-c5ff7a82b7d1-kube-api-access-498w4\") pod \"ingress-canary-vdd85\" (UID: \"4be0471d-f724-4147-b1b0-c5ff7a82b7d1\") " pod="openshift-ingress-canary/ingress-canary-vdd85"
Jul 14 22:57:29.717343 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.717085    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-ca-trust-extracted\") pod \"image-registry-7955774756-rvmg5\" (UID: \"3a13b9a0-c0ff-4a8e-bfac-711ee2137471\") " pod="openshift-image-registry/image-registry-7955774756-rvmg5"
Jul 14 22:57:29.717343 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.717152    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-registry-certificates\") pod \"image-registry-7955774756-rvmg5\" (UID: \"3a13b9a0-c0ff-4a8e-bfac-711ee2137471\") " pod="openshift-image-registry/image-registry-7955774756-rvmg5"
Jul 14 22:57:29.717343 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.717234    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4be0471d-f724-4147-b1b0-c5ff7a82b7d1-cert\") pod \"ingress-canary-vdd85\" (UID: \"4be0471d-f724-4147-b1b0-c5ff7a82b7d1\") " pod="openshift-ingress-canary/ingress-canary-vdd85"
Jul 14 22:57:29.717343 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.717272    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ffb5cdb7-2bde-418f-884d-0962fc26414e-config-volume\") pod \"dns-default-q4982\" (UID: \"ffb5cdb7-2bde-418f-884d-0962fc26414e\") " pod="openshift-dns/dns-default-q4982"
Jul 14 22:57:29.717343 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.717326    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-registry-tls\") pod \"image-registry-7955774756-rvmg5\" (UID: \"3a13b9a0-c0ff-4a8e-bfac-711ee2137471\") " pod="openshift-image-registry/image-registry-7955774756-rvmg5"
Jul 14 22:57:29.717498 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.717348    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffb5cdb7-2bde-418f-884d-0962fc26414e-metrics-tls\") pod \"dns-default-q4982\" (UID: \"ffb5cdb7-2bde-418f-884d-0962fc26414e\") " pod="openshift-dns/dns-default-q4982"
Jul 14 22:57:29.717498 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.717375    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pnlf\" (UniqueName: \"kubernetes.io/projected/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-kube-api-access-2pnlf\") pod \"image-registry-7955774756-rvmg5\" (UID: \"3a13b9a0-c0ff-4a8e-bfac-711ee2137471\") " pod="openshift-image-registry/image-registry-7955774756-rvmg5"
Jul 14 22:57:29.717498 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.717414    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-trusted-ca\") pod \"image-registry-7955774756-rvmg5\" (UID: \"3a13b9a0-c0ff-4a8e-bfac-711ee2137471\") " pod="openshift-image-registry/image-registry-7955774756-rvmg5"
Jul 14 22:57:29.717498 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.717438    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffb5cdb7-2bde-418f-884d-0962fc26414e-tmp-dir\") pod \"dns-default-q4982\" (UID: \"ffb5cdb7-2bde-418f-884d-0962fc26414e\") " pod="openshift-dns/dns-default-q4982"
Jul 14 22:57:29.717498 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.717458    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm8w7\" (UniqueName: \"kubernetes.io/projected/ffb5cdb7-2bde-418f-884d-0962fc26414e-kube-api-access-vm8w7\") pod \"dns-default-q4982\" (UID: \"ffb5cdb7-2bde-418f-884d-0962fc26414e\") " pod="openshift-dns/dns-default-q4982"
Jul 14 22:57:29.717651 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.717502    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-bound-sa-token\") pod \"image-registry-7955774756-rvmg5\" (UID: \"3a13b9a0-c0ff-4a8e-bfac-711ee2137471\") " pod="openshift-image-registry/image-registry-7955774756-rvmg5"
Jul 14 22:57:29.717651 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:29.717541    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 22:57:29.717651 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:29.717608    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffb5cdb7-2bde-418f-884d-0962fc26414e-metrics-tls podName:ffb5cdb7-2bde-418f-884d-0962fc26414e nodeName:}" failed. No retries permitted until 2026-07-14 22:57:30.217588526 +0000 UTC m=+37.520579297 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ffb5cdb7-2bde-418f-884d-0962fc26414e-metrics-tls") pod "dns-default-q4982" (UID: "ffb5cdb7-2bde-418f-884d-0962fc26414e") : secret "dns-default-metrics-tls" not found
Jul 14 22:57:29.717812 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:29.717767    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 14 22:57:29.717812 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:29.717781    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7955774756-rvmg5: secret "image-registry-tls" not found
Jul 14 22:57:29.717892 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:29.717822    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-registry-tls podName:3a13b9a0-c0ff-4a8e-bfac-711ee2137471 nodeName:}" failed. No retries permitted until 2026-07-14 22:57:30.217808503 +0000 UTC m=+37.520799296 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-registry-tls") pod "image-registry-7955774756-rvmg5" (UID: "3a13b9a0-c0ff-4a8e-bfac-711ee2137471") : secret "image-registry-tls" not found
Jul 14 22:57:29.717968 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.717949    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-ca-trust-extracted\") pod \"image-registry-7955774756-rvmg5\" (UID: \"3a13b9a0-c0ff-4a8e-bfac-711ee2137471\") " pod="openshift-image-registry/image-registry-7955774756-rvmg5"
Jul 14 22:57:29.718316 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.718187    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ffb5cdb7-2bde-418f-884d-0962fc26414e-config-volume\") pod \"dns-default-q4982\" (UID: \"ffb5cdb7-2bde-418f-884d-0962fc26414e\") " pod="openshift-dns/dns-default-q4982"
Jul 14 22:57:29.718316 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.718199    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffb5cdb7-2bde-418f-884d-0962fc26414e-tmp-dir\") pod \"dns-default-q4982\" (UID: \"ffb5cdb7-2bde-418f-884d-0962fc26414e\") " pod="openshift-dns/dns-default-q4982"
Jul 14 22:57:29.719001 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.718796    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-registry-certificates\") pod \"image-registry-7955774756-rvmg5\" (UID: \"3a13b9a0-c0ff-4a8e-bfac-711ee2137471\") " pod="openshift-image-registry/image-registry-7955774756-rvmg5"
Jul 14 22:57:29.719001 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.718839    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-trusted-ca\") pod \"image-registry-7955774756-rvmg5\" (UID: \"3a13b9a0-c0ff-4a8e-bfac-711ee2137471\") " pod="openshift-image-registry/image-registry-7955774756-rvmg5"
Jul 14 22:57:29.721734 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.721618    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4be0471d-f724-4147-b1b0-c5ff7a82b7d1-cert\") pod \"ingress-canary-vdd85\" (UID: \"4be0471d-f724-4147-b1b0-c5ff7a82b7d1\") " pod="openshift-ingress-canary/ingress-canary-vdd85"
Jul 14 22:57:29.721842 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.721628    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-image-registry-private-configuration\") pod \"image-registry-7955774756-rvmg5\" (UID: \"3a13b9a0-c0ff-4a8e-bfac-711ee2137471\") " pod="openshift-image-registry/image-registry-7955774756-rvmg5"
Jul 14 22:57:29.721842 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.721641    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-installation-pull-secrets\") pod \"image-registry-7955774756-rvmg5\" (UID: \"3a13b9a0-c0ff-4a8e-bfac-711ee2137471\") " pod="openshift-image-registry/image-registry-7955774756-rvmg5"
Jul 14 22:57:29.728204 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.728164    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-498w4\" (UniqueName: \"kubernetes.io/projected/4be0471d-f724-4147-b1b0-c5ff7a82b7d1-kube-api-access-498w4\") pod \"ingress-canary-vdd85\" (UID: \"4be0471d-f724-4147-b1b0-c5ff7a82b7d1\") " pod="openshift-ingress-canary/ingress-canary-vdd85"
Jul 14 22:57:29.728367 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.728342    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-bound-sa-token\") pod \"image-registry-7955774756-rvmg5\" (UID: \"3a13b9a0-c0ff-4a8e-bfac-711ee2137471\") " pod="openshift-image-registry/image-registry-7955774756-rvmg5"
Jul 14 22:57:29.728450 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.728375    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm8w7\" (UniqueName: \"kubernetes.io/projected/ffb5cdb7-2bde-418f-884d-0962fc26414e-kube-api-access-vm8w7\") pod \"dns-default-q4982\" (UID: \"ffb5cdb7-2bde-418f-884d-0962fc26414e\") " pod="openshift-dns/dns-default-q4982"
Jul 14 22:57:29.728503 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.728478    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pnlf\" (UniqueName: \"kubernetes.io/projected/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-kube-api-access-2pnlf\") pod \"image-registry-7955774756-rvmg5\" (UID: \"3a13b9a0-c0ff-4a8e-bfac-711ee2137471\") " pod="openshift-image-registry/image-registry-7955774756-rvmg5"
Jul 14 22:57:29.840438 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:29.840403    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vdd85"
Jul 14 22:57:30.222053 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:30.222015    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-registry-tls\") pod \"image-registry-7955774756-rvmg5\" (UID: \"3a13b9a0-c0ff-4a8e-bfac-711ee2137471\") " pod="openshift-image-registry/image-registry-7955774756-rvmg5"
Jul 14 22:57:30.222261 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:30.222065    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffb5cdb7-2bde-418f-884d-0962fc26414e-metrics-tls\") pod \"dns-default-q4982\" (UID: \"ffb5cdb7-2bde-418f-884d-0962fc26414e\") " pod="openshift-dns/dns-default-q4982"
Jul 14 22:57:30.222261 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:30.222185    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 14 22:57:30.222261 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:30.222206    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 22:57:30.222261 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:30.222212    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7955774756-rvmg5: secret "image-registry-tls" not found
Jul 14 22:57:30.222481 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:30.222274    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffb5cdb7-2bde-418f-884d-0962fc26414e-metrics-tls podName:ffb5cdb7-2bde-418f-884d-0962fc26414e nodeName:}" failed. No retries permitted until 2026-07-14 22:57:31.22225323 +0000 UTC m=+38.525244004 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ffb5cdb7-2bde-418f-884d-0962fc26414e-metrics-tls") pod "dns-default-q4982" (UID: "ffb5cdb7-2bde-418f-884d-0962fc26414e") : secret "dns-default-metrics-tls" not found
Jul 14 22:57:30.222481 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:30.222317    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-registry-tls podName:3a13b9a0-c0ff-4a8e-bfac-711ee2137471 nodeName:}" failed. No retries permitted until 2026-07-14 22:57:31.222306042 +0000 UTC m=+38.525296827 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-registry-tls") pod "image-registry-7955774756-rvmg5" (UID: "3a13b9a0-c0ff-4a8e-bfac-711ee2137471") : secret "image-registry-tls" not found
Jul 14 22:57:30.562799 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:30.562776    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vdd85"]
Jul 14 22:57:30.567063 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:57:30.567039    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4be0471d_f724_4147_b1b0_c5ff7a82b7d1.slice/crio-47cc3b6ad430e6c6ce72fd36aa12d94591bafe10b4a5f6d7c4d832577abb9a2c WatchSource:0}: Error finding container 47cc3b6ad430e6c6ce72fd36aa12d94591bafe10b4a5f6d7c4d832577abb9a2c: Status 404 returned error can't find the container with id 47cc3b6ad430e6c6ce72fd36aa12d94591bafe10b4a5f6d7c4d832577abb9a2c
Jul 14 22:57:31.223614 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:31.223576    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xjw4z"
Jul 14 22:57:31.223804 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:31.223576    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pbtpg"
Jul 14 22:57:31.223804 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:31.223580    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wjpcm"
Jul 14 22:57:31.228423 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:31.228399    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wx244\""
Jul 14 22:57:31.228689 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:31.228672    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 14 22:57:31.228894 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:31.228879    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 14 22:57:31.229110 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:31.229091    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 14 22:57:31.229376 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:31.229361    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d24rt\""
Jul 14 22:57:31.229561 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:31.229545    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 14 22:57:31.230046 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:31.230024    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-registry-tls\") pod \"image-registry-7955774756-rvmg5\" (UID: \"3a13b9a0-c0ff-4a8e-bfac-711ee2137471\") " pod="openshift-image-registry/image-registry-7955774756-rvmg5"
Jul 14 22:57:31.230128 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:31.230063    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffb5cdb7-2bde-418f-884d-0962fc26414e-metrics-tls\") pod \"dns-default-q4982\" (UID: \"ffb5cdb7-2bde-418f-884d-0962fc26414e\") " pod="openshift-dns/dns-default-q4982"
Jul 14 22:57:31.230213 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:31.230196    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 14 22:57:31.230265 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:31.230210    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 22:57:31.230265 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:31.230218    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7955774756-rvmg5: secret "image-registry-tls" not found
Jul 14 22:57:31.230401 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:31.230276    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-registry-tls podName:3a13b9a0-c0ff-4a8e-bfac-711ee2137471 nodeName:}" failed. No retries permitted until 2026-07-14 22:57:33.23025501 +0000 UTC m=+40.533245797 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-registry-tls") pod "image-registry-7955774756-rvmg5" (UID: "3a13b9a0-c0ff-4a8e-bfac-711ee2137471") : secret "image-registry-tls" not found
Jul 14 22:57:31.230401 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:31.230323    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffb5cdb7-2bde-418f-884d-0962fc26414e-metrics-tls podName:ffb5cdb7-2bde-418f-884d-0962fc26414e nodeName:}" failed. No retries permitted until 2026-07-14 22:57:33.230311909 +0000 UTC m=+40.533302683 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ffb5cdb7-2bde-418f-884d-0962fc26414e-metrics-tls") pod "dns-default-q4982" (UID: "ffb5cdb7-2bde-418f-884d-0962fc26414e") : secret "dns-default-metrics-tls" not found
Jul 14 22:57:31.501742 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:31.501700    2563 generic.go:356] "Generic (PLEG): container finished" podID="da7d977f-1dd0-42e6-b3a7-455490097670" containerID="d2758d843460296ee9e8d24aaae73c6d3c8d09952a359bb6d2f49a028f243d62" exitCode=0
Jul 14 22:57:31.501923 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:31.501760    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r5s76" event={"ID":"da7d977f-1dd0-42e6-b3a7-455490097670","Type":"ContainerDied","Data":"d2758d843460296ee9e8d24aaae73c6d3c8d09952a359bb6d2f49a028f243d62"}
Jul 14 22:57:31.503619 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:31.503581    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vdd85" event={"ID":"4be0471d-f724-4147-b1b0-c5ff7a82b7d1","Type":"ContainerStarted","Data":"47cc3b6ad430e6c6ce72fd36aa12d94591bafe10b4a5f6d7c4d832577abb9a2c"}
Jul 14 22:57:32.511134 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:32.510854    2563 generic.go:356] "Generic (PLEG): container finished" podID="da7d977f-1dd0-42e6-b3a7-455490097670" containerID="477cf46c5074edd392d155e80b17f9b12ec65ad81a2c67c0400308ca860d47ad" exitCode=0
Jul 14 22:57:32.511576 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:32.510901    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r5s76" event={"ID":"da7d977f-1dd0-42e6-b3a7-455490097670","Type":"ContainerDied","Data":"477cf46c5074edd392d155e80b17f9b12ec65ad81a2c67c0400308ca860d47ad"}
Jul 14 22:57:33.247509 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:33.247481    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-registry-tls\") pod \"image-registry-7955774756-rvmg5\" (UID: \"3a13b9a0-c0ff-4a8e-bfac-711ee2137471\") " pod="openshift-image-registry/image-registry-7955774756-rvmg5"
Jul 14 22:57:33.247509 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:33.247514    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffb5cdb7-2bde-418f-884d-0962fc26414e-metrics-tls\") pod \"dns-default-q4982\" (UID: \"ffb5cdb7-2bde-418f-884d-0962fc26414e\") " pod="openshift-dns/dns-default-q4982"
Jul 14 22:57:33.247705 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:33.247621    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 22:57:33.247705 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:33.247629    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 14 22:57:33.247705 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:33.247647    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7955774756-rvmg5: secret "image-registry-tls" not found
Jul 14 22:57:33.247705 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:33.247670    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffb5cdb7-2bde-418f-884d-0962fc26414e-metrics-tls podName:ffb5cdb7-2bde-418f-884d-0962fc26414e nodeName:}" failed. No retries permitted until 2026-07-14 22:57:37.247657679 +0000 UTC m=+44.550648452 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ffb5cdb7-2bde-418f-884d-0962fc26414e-metrics-tls") pod "dns-default-q4982" (UID: "ffb5cdb7-2bde-418f-884d-0962fc26414e") : secret "dns-default-metrics-tls" not found
Jul 14 22:57:33.247705 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:33.247690    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-registry-tls podName:3a13b9a0-c0ff-4a8e-bfac-711ee2137471 nodeName:}" failed. No retries permitted until 2026-07-14 22:57:37.2476769 +0000 UTC m=+44.550667672 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-registry-tls") pod "image-registry-7955774756-rvmg5" (UID: "3a13b9a0-c0ff-4a8e-bfac-711ee2137471") : secret "image-registry-tls" not found
Jul 14 22:57:33.515778 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:33.515744    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vdd85" event={"ID":"4be0471d-f724-4147-b1b0-c5ff7a82b7d1","Type":"ContainerStarted","Data":"6960bfdae5a98cd99e899652b090d876938a9fd865acff58748416fcb1e1b434"}
Jul 14 22:57:33.518897 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:33.518874    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r5s76" event={"ID":"da7d977f-1dd0-42e6-b3a7-455490097670","Type":"ContainerStarted","Data":"3ec6801031a4b40ea91d1e383dd7b7ccf77b234bf09b2ebc7952e58e098ee920"}
Jul 14 22:57:33.534181 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:33.534133    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vdd85" podStartSLOduration=2.353340186 podStartE2EDuration="4.534118842s" totalImagesPullingTime="2.180778656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:57:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:57:33.533234891 +0000 UTC m=+40.836225684" watchObservedRunningTime="2026-07-14 22:57:33.534118842 +0000 UTC m=+40.837109633"
Jul 14 22:57:33.562542 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:33.561507    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-r5s76" podStartSLOduration=9.439770694 podStartE2EDuration="39.561486631s" totalImagesPullingTime="30.121715937s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 22:56:54 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:57:33.560153643 +0000 UTC m=+40.863144435" watchObservedRunningTime="2026-07-14 22:57:33.561486631 +0000 UTC m=+40.864477426"
Jul 14 22:57:34.818226 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:34.818189    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-gzz8f"]
Jul 14 22:57:34.831738 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:34.831703    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-gzz8f"]
Jul 14 22:57:34.831880 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:34.831814    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-gzz8f"
Jul 14 22:57:34.834707 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:34.834676    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 14 22:57:34.834707 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:34.834694    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 14 22:57:34.834902 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:34.834756    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-hcpg9\""
Jul 14 22:57:34.960527 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:34.960487    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-gzz8f\" (UID: \"c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-gzz8f"
Jul 14 22:57:34.960527 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:34.960527    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5-nginx-conf\") pod \"networking-console-plugin-6cf856758c-gzz8f\" (UID: \"c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-gzz8f"
Jul 14 22:57:35.061036 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:35.060999    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-gzz8f\" (UID: \"c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-gzz8f"
Jul 14 22:57:35.061138 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:35.061043    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5-nginx-conf\") pod \"networking-console-plugin-6cf856758c-gzz8f\" (UID: \"c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-gzz8f"
Jul 14 22:57:35.061179 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:35.061146    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 14 22:57:35.061228 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:35.061213    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5-networking-console-plugin-cert podName:c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5 nodeName:}" failed. No retries permitted until 2026-07-14 22:57:35.561198037 +0000 UTC m=+42.864188807 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-gzz8f" (UID: "c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5") : secret "networking-console-plugin-cert" not found
Jul 14 22:57:35.061648 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:35.061631    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5-nginx-conf\") pod \"networking-console-plugin-6cf856758c-gzz8f\" (UID: \"c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-gzz8f"
Jul 14 22:57:35.566779 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:35.566741    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-gzz8f\" (UID: \"c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-gzz8f"
Jul 14 22:57:35.566955 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:35.566886    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 14 22:57:35.566955 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:35.566949    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5-networking-console-plugin-cert podName:c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5 nodeName:}" failed. No retries permitted until 2026-07-14 22:57:36.566932482 +0000 UTC m=+43.869923258 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-gzz8f" (UID: "c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5") : secret "networking-console-plugin-cert" not found
Jul 14 22:57:36.575461 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:36.575425    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-gzz8f\" (UID: \"c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-gzz8f"
Jul 14 22:57:36.575940 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:36.575569    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 14 22:57:36.575940 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:36.575644    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5-networking-console-plugin-cert podName:c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5 nodeName:}" failed. No retries permitted until 2026-07-14 22:57:38.575623009 +0000 UTC m=+45.878613789 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-gzz8f" (UID: "c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5") : secret "networking-console-plugin-cert" not found
Jul 14 22:57:36.705483 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:36.705451    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-28rrp"]
Jul 14 22:57:36.737060 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:36.737032    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-28rrp"]
Jul 14 22:57:36.737229 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:36.737151    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-28rrp"
Jul 14 22:57:36.740239 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:36.740204    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 14 22:57:36.740239 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:36.740214    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 14 22:57:36.740431 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:36.740214    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 14 22:57:36.741763 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:36.741741    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 14 22:57:36.741763 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:36.741757    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9vm8v\""
Jul 14 22:57:36.776815 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:36.776475    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/341a042f-b19e-418e-958a-f42c097be024-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-28rrp\" (UID: \"341a042f-b19e-418e-958a-f42c097be024\") " pod="openshift-insights/insights-runtime-extractor-28rrp"
Jul 14 22:57:36.777138 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:36.777117    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/341a042f-b19e-418e-958a-f42c097be024-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-28rrp\" (UID: \"341a042f-b19e-418e-958a-f42c097be024\") " pod="openshift-insights/insights-runtime-extractor-28rrp"
Jul 14 22:57:36.777299 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:36.777252    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzgbw\" (UniqueName: \"kubernetes.io/projected/341a042f-b19e-418e-958a-f42c097be024-kube-api-access-pzgbw\") pod \"insights-runtime-extractor-28rrp\" (UID: \"341a042f-b19e-418e-958a-f42c097be024\") " pod="openshift-insights/insights-runtime-extractor-28rrp"
Jul 14 22:57:36.777458 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:36.777443    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/341a042f-b19e-418e-958a-f42c097be024-data-volume\") pod \"insights-runtime-extractor-28rrp\" (UID: \"341a042f-b19e-418e-958a-f42c097be024\") " pod="openshift-insights/insights-runtime-extractor-28rrp"
Jul 14 22:57:36.777575 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:36.777562    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/341a042f-b19e-418e-958a-f42c097be024-crio-socket\") pod \"insights-runtime-extractor-28rrp\" (UID: \"341a042f-b19e-418e-958a-f42c097be024\") " pod="openshift-insights/insights-runtime-extractor-28rrp"
Jul 14 22:57:36.878898 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:36.878816    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/341a042f-b19e-418e-958a-f42c097be024-data-volume\") pod \"insights-runtime-extractor-28rrp\" (UID: \"341a042f-b19e-418e-958a-f42c097be024\") " pod="openshift-insights/insights-runtime-extractor-28rrp"
Jul 14 22:57:36.878898 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:36.878857    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/341a042f-b19e-418e-958a-f42c097be024-crio-socket\") pod \"insights-runtime-extractor-28rrp\" (UID: \"341a042f-b19e-418e-958a-f42c097be024\") " pod="openshift-insights/insights-runtime-extractor-28rrp"
Jul 14 22:57:36.879058 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:36.878909    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/341a042f-b19e-418e-958a-f42c097be024-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-28rrp\" (UID: \"341a042f-b19e-418e-958a-f42c097be024\") " pod="openshift-insights/insights-runtime-extractor-28rrp"
Jul 14 22:57:36.879058 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:36.878939    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/341a042f-b19e-418e-958a-f42c097be024-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-28rrp\" (UID: \"341a042f-b19e-418e-958a-f42c097be024\") " pod="openshift-insights/insights-runtime-extractor-28rrp"
Jul 14 22:57:36.879058 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:36.878957    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzgbw\" (UniqueName: \"kubernetes.io/projected/341a042f-b19e-418e-958a-f42c097be024-kube-api-access-pzgbw\") pod \"insights-runtime-extractor-28rrp\" (UID: \"341a042f-b19e-418e-958a-f42c097be024\") " pod="openshift-insights/insights-runtime-extractor-28rrp"
Jul 14 22:57:36.879161 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:36.879066    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 14 22:57:36.879161 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:36.879064    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 14 22:57:36.879161 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:36.879145    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/341a042f-b19e-418e-958a-f42c097be024-insights-runtime-extractor-tls podName:341a042f-b19e-418e-958a-f42c097be024 nodeName:}" failed. No retries permitted until 2026-07-14 22:57:37.379126907 +0000 UTC m=+44.682117680 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/341a042f-b19e-418e-958a-f42c097be024-insights-runtime-extractor-tls") pod "insights-runtime-extractor-28rrp" (UID: "341a042f-b19e-418e-958a-f42c097be024") : secret "insights-runtime-extractor-tls" not found
Jul 14 22:57:36.879161 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:36.879154    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/341a042f-b19e-418e-958a-f42c097be024-data-volume\") pod \"insights-runtime-extractor-28rrp\" (UID: \"341a042f-b19e-418e-958a-f42c097be024\") " pod="openshift-insights/insights-runtime-extractor-28rrp"
Jul 14 22:57:36.879357 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:36.879138    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/341a042f-b19e-418e-958a-f42c097be024-crio-socket\") pod \"insights-runtime-extractor-28rrp\" (UID: \"341a042f-b19e-418e-958a-f42c097be024\") " pod="openshift-insights/insights-runtime-extractor-28rrp"
Jul 14 22:57:36.879357 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:36.879164    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/341a042f-b19e-418e-958a-f42c097be024-kube-rbac-proxy-cm podName:341a042f-b19e-418e-958a-f42c097be024 nodeName:}" failed. No retries permitted until 2026-07-14 22:57:37.379154737 +0000 UTC m=+44.682145508 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/341a042f-b19e-418e-958a-f42c097be024-kube-rbac-proxy-cm") pod "insights-runtime-extractor-28rrp" (UID: "341a042f-b19e-418e-958a-f42c097be024") : configmap "kube-rbac-proxy" not found
Jul 14 22:57:36.893311 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:36.893273    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzgbw\" (UniqueName: \"kubernetes.io/projected/341a042f-b19e-418e-958a-f42c097be024-kube-api-access-pzgbw\") pod \"insights-runtime-extractor-28rrp\" (UID: \"341a042f-b19e-418e-958a-f42c097be024\") " pod="openshift-insights/insights-runtime-extractor-28rrp"
Jul 14 22:57:36.941049 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:36.941019    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nn4qg"]
Jul 14 22:57:36.957326 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:36.957307    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nn4qg"]
Jul 14 22:57:36.957423 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:36.957407    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nn4qg"
Jul 14 22:57:36.960730 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:36.960713    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 14 22:57:36.960841 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:36.960793    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 14 22:57:36.960909 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:36.960862    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-bh8c8\""
Jul 14 22:57:36.980219 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:36.980194    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4gsw\" (UniqueName: \"kubernetes.io/projected/acd21154-5243-4aeb-bd93-acd9da0ca28c-kube-api-access-j4gsw\") pod \"migrator-7f59b54f6c-nn4qg\" (UID: \"acd21154-5243-4aeb-bd93-acd9da0ca28c\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nn4qg"
Jul 14 22:57:37.081133 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:37.081095    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4gsw\" (UniqueName: \"kubernetes.io/projected/acd21154-5243-4aeb-bd93-acd9da0ca28c-kube-api-access-j4gsw\") pod \"migrator-7f59b54f6c-nn4qg\" (UID: \"acd21154-5243-4aeb-bd93-acd9da0ca28c\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nn4qg"
Jul 14 22:57:37.093133 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:37.093105    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4gsw\" (UniqueName: \"kubernetes.io/projected/acd21154-5243-4aeb-bd93-acd9da0ca28c-kube-api-access-j4gsw\") pod \"migrator-7f59b54f6c-nn4qg\" (UID: \"acd21154-5243-4aeb-bd93-acd9da0ca28c\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nn4qg"
Jul 14 22:57:37.266214 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:37.266189    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nn4qg"
Jul 14 22:57:37.283700 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:37.283671    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-registry-tls\") pod \"image-registry-7955774756-rvmg5\" (UID: \"3a13b9a0-c0ff-4a8e-bfac-711ee2137471\") " pod="openshift-image-registry/image-registry-7955774756-rvmg5"
Jul 14 22:57:37.283843 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:37.283704    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffb5cdb7-2bde-418f-884d-0962fc26414e-metrics-tls\") pod \"dns-default-q4982\" (UID: \"ffb5cdb7-2bde-418f-884d-0962fc26414e\") " pod="openshift-dns/dns-default-q4982"
Jul 14 22:57:37.283843 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:37.283811    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 22:57:37.283843 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:37.283820    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 14 22:57:37.283843 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:37.283839    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7955774756-rvmg5: secret "image-registry-tls" not found
Jul 14 22:57:37.284262 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:37.283875    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffb5cdb7-2bde-418f-884d-0962fc26414e-metrics-tls podName:ffb5cdb7-2bde-418f-884d-0962fc26414e nodeName:}" failed. No retries permitted until 2026-07-14 22:57:45.28386069 +0000 UTC m=+52.586851472 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ffb5cdb7-2bde-418f-884d-0962fc26414e-metrics-tls") pod "dns-default-q4982" (UID: "ffb5cdb7-2bde-418f-884d-0962fc26414e") : secret "dns-default-metrics-tls" not found
Jul 14 22:57:37.284262 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:37.283901    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-registry-tls podName:3a13b9a0-c0ff-4a8e-bfac-711ee2137471 nodeName:}" failed. No retries permitted until 2026-07-14 22:57:45.28388251 +0000 UTC m=+52.586873300 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-registry-tls") pod "image-registry-7955774756-rvmg5" (UID: "3a13b9a0-c0ff-4a8e-bfac-711ee2137471") : secret "image-registry-tls" not found
Jul 14 22:57:37.384818 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:37.384792    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/341a042f-b19e-418e-958a-f42c097be024-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-28rrp\" (UID: \"341a042f-b19e-418e-958a-f42c097be024\") " pod="openshift-insights/insights-runtime-extractor-28rrp"
Jul 14 22:57:37.384950 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:37.384841    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/341a042f-b19e-418e-958a-f42c097be024-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-28rrp\" (UID: \"341a042f-b19e-418e-958a-f42c097be024\") " pod="openshift-insights/insights-runtime-extractor-28rrp"
Jul 14 22:57:37.384950 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:37.384923    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 14 22:57:37.385063 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:37.384964    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 14 22:57:37.385063 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:37.384974    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/341a042f-b19e-418e-958a-f42c097be024-kube-rbac-proxy-cm podName:341a042f-b19e-418e-958a-f42c097be024 nodeName:}" failed. No retries permitted until 2026-07-14 22:57:38.384960494 +0000 UTC m=+45.687951264 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/341a042f-b19e-418e-958a-f42c097be024-kube-rbac-proxy-cm") pod "insights-runtime-extractor-28rrp" (UID: "341a042f-b19e-418e-958a-f42c097be024") : configmap "kube-rbac-proxy" not found
Jul 14 22:57:37.385063 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:37.385043    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/341a042f-b19e-418e-958a-f42c097be024-insights-runtime-extractor-tls podName:341a042f-b19e-418e-958a-f42c097be024 nodeName:}" failed. No retries permitted until 2026-07-14 22:57:38.385022841 +0000 UTC m=+45.688013624 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/341a042f-b19e-418e-958a-f42c097be024-insights-runtime-extractor-tls") pod "insights-runtime-extractor-28rrp" (UID: "341a042f-b19e-418e-958a-f42c097be024") : secret "insights-runtime-extractor-tls" not found
Jul 14 22:57:37.440962 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:37.440919    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nn4qg"]
Jul 14 22:57:37.442783 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:57:37.442752    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacd21154_5243_4aeb_bd93_acd9da0ca28c.slice/crio-7725e560e6c6abcc572715beda26d6917bf88b829574a9cf8a2ad10b8e0644ee WatchSource:0}: Error finding container 7725e560e6c6abcc572715beda26d6917bf88b829574a9cf8a2ad10b8e0644ee: Status 404 returned error can't find the container with id 7725e560e6c6abcc572715beda26d6917bf88b829574a9cf8a2ad10b8e0644ee
Jul 14 22:57:37.529889 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:37.529799    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nn4qg" event={"ID":"acd21154-5243-4aeb-bd93-acd9da0ca28c","Type":"ContainerStarted","Data":"7725e560e6c6abcc572715beda26d6917bf88b829574a9cf8a2ad10b8e0644ee"}
Jul 14 22:57:38.294429 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:38.294399    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-s4zgk_59f4f5d3-9650-40e0-969a-e912429a2481/dns-node-resolver/0.log"
Jul 14 22:57:38.392959 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:38.392916    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/341a042f-b19e-418e-958a-f42c097be024-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-28rrp\" (UID: \"341a042f-b19e-418e-958a-f42c097be024\") " pod="openshift-insights/insights-runtime-extractor-28rrp"
Jul 14 22:57:38.393115 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:38.393050    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 14 22:57:38.393115 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:38.393059    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/341a042f-b19e-418e-958a-f42c097be024-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-28rrp\" (UID: \"341a042f-b19e-418e-958a-f42c097be024\") " pod="openshift-insights/insights-runtime-extractor-28rrp"
Jul 14 22:57:38.393223 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:38.393126    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/341a042f-b19e-418e-958a-f42c097be024-kube-rbac-proxy-cm podName:341a042f-b19e-418e-958a-f42c097be024 nodeName:}" failed. No retries permitted until 2026-07-14 22:57:40.39310697 +0000 UTC m=+47.696097743 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/341a042f-b19e-418e-958a-f42c097be024-kube-rbac-proxy-cm") pod "insights-runtime-extractor-28rrp" (UID: "341a042f-b19e-418e-958a-f42c097be024") : configmap "kube-rbac-proxy" not found
Jul 14 22:57:38.393223 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:38.393174    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 14 22:57:38.393334 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:38.393226    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/341a042f-b19e-418e-958a-f42c097be024-insights-runtime-extractor-tls podName:341a042f-b19e-418e-958a-f42c097be024 nodeName:}" failed. No retries permitted until 2026-07-14 22:57:40.393210215 +0000 UTC m=+47.696200989 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/341a042f-b19e-418e-958a-f42c097be024-insights-runtime-extractor-tls") pod "insights-runtime-extractor-28rrp" (UID: "341a042f-b19e-418e-958a-f42c097be024") : secret "insights-runtime-extractor-tls" not found
Jul 14 22:57:38.595121 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:38.595035    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-gzz8f\" (UID: \"c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-gzz8f"
Jul 14 22:57:38.595368 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:38.595185    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 14 22:57:38.595368 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:38.595254    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5-networking-console-plugin-cert podName:c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5 nodeName:}" failed. No retries permitted until 2026-07-14 22:57:42.595236004 +0000 UTC m=+49.898226781 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-gzz8f" (UID: "c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5") : secret "networking-console-plugin-cert" not found
Jul 14 22:57:38.639563 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:38.639528    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-7d6w9"]
Jul 14 22:57:38.647412 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:38.647390    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-7d6w9"
Jul 14 22:57:38.650212 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:38.650182    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-7d6w9"]
Jul 14 22:57:38.650738 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:38.650714    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 14 22:57:38.651994 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:38.651972    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-bpbhx\""
Jul 14 22:57:38.651994 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:38.651991    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 14 22:57:38.652149 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:38.652039    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 14 22:57:38.652149 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:38.651991    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 14 22:57:38.696314 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:38.696271    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/674701b4-5708-4e0e-858a-553e458a7351-signing-cabundle\") pod \"service-ca-79f68b9f4f-7d6w9\" (UID: \"674701b4-5708-4e0e-858a-553e458a7351\") " pod="openshift-service-ca/service-ca-79f68b9f4f-7d6w9"
Jul 14 22:57:38.696440 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:38.696339    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/674701b4-5708-4e0e-858a-553e458a7351-signing-key\") pod \"service-ca-79f68b9f4f-7d6w9\" (UID: \"674701b4-5708-4e0e-858a-553e458a7351\") " pod="openshift-service-ca/service-ca-79f68b9f4f-7d6w9"
Jul 14 22:57:38.696440 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:38.696385    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvzxt\" (UniqueName: \"kubernetes.io/projected/674701b4-5708-4e0e-858a-553e458a7351-kube-api-access-pvzxt\") pod \"service-ca-79f68b9f4f-7d6w9\" (UID: \"674701b4-5708-4e0e-858a-553e458a7351\") " pod="openshift-service-ca/service-ca-79f68b9f4f-7d6w9"
Jul 14 22:57:38.797700 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:38.797671    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/674701b4-5708-4e0e-858a-553e458a7351-signing-cabundle\") pod \"service-ca-79f68b9f4f-7d6w9\" (UID: \"674701b4-5708-4e0e-858a-553e458a7351\") " pod="openshift-service-ca/service-ca-79f68b9f4f-7d6w9"
Jul 14 22:57:38.797825 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:38.797713    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/674701b4-5708-4e0e-858a-553e458a7351-signing-key\") pod \"service-ca-79f68b9f4f-7d6w9\" (UID: \"674701b4-5708-4e0e-858a-553e458a7351\") " pod="openshift-service-ca/service-ca-79f68b9f4f-7d6w9"
Jul 14 22:57:38.797825 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:38.797744    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvzxt\" (UniqueName: \"kubernetes.io/projected/674701b4-5708-4e0e-858a-553e458a7351-kube-api-access-pvzxt\") pod \"service-ca-79f68b9f4f-7d6w9\" (UID: \"674701b4-5708-4e0e-858a-553e458a7351\") " pod="openshift-service-ca/service-ca-79f68b9f4f-7d6w9"
Jul 14 22:57:38.798309 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:38.798258    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/674701b4-5708-4e0e-858a-553e458a7351-signing-cabundle\") pod \"service-ca-79f68b9f4f-7d6w9\" (UID: \"674701b4-5708-4e0e-858a-553e458a7351\") " pod="openshift-service-ca/service-ca-79f68b9f4f-7d6w9"
Jul 14 22:57:38.800132 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:38.800113    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/674701b4-5708-4e0e-858a-553e458a7351-signing-key\") pod \"service-ca-79f68b9f4f-7d6w9\" (UID: \"674701b4-5708-4e0e-858a-553e458a7351\") " pod="openshift-service-ca/service-ca-79f68b9f4f-7d6w9"
Jul 14 22:57:38.807390 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:38.807372    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvzxt\" (UniqueName: \"kubernetes.io/projected/674701b4-5708-4e0e-858a-553e458a7351-kube-api-access-pvzxt\") pod \"service-ca-79f68b9f4f-7d6w9\" (UID: \"674701b4-5708-4e0e-858a-553e458a7351\") " pod="openshift-service-ca/service-ca-79f68b9f4f-7d6w9"
Jul 14 22:57:38.956796 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:38.956775    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-7d6w9"
Jul 14 22:57:39.104951 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:39.104924    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-7d6w9"]
Jul 14 22:57:39.106048 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:57:39.106023    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod674701b4_5708_4e0e_858a_553e458a7351.slice/crio-1e42e3567be9363f3370371ce87d7df99d83043979ee9e96432feac04f07fb82 WatchSource:0}: Error finding container 1e42e3567be9363f3370371ce87d7df99d83043979ee9e96432feac04f07fb82: Status 404 returned error can't find the container with id 1e42e3567be9363f3370371ce87d7df99d83043979ee9e96432feac04f07fb82
Jul 14 22:57:39.294080 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:39.294047    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lzt25_1ef7f81d-618b-4d9a-b018-9e3346c27bd1/node-ca/0.log"
Jul 14 22:57:39.536123 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:39.536085    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nn4qg" event={"ID":"acd21154-5243-4aeb-bd93-acd9da0ca28c","Type":"ContainerStarted","Data":"493a371bf4cfdaa569f1a0ec9745c3218c2da72faca115b09b35654d0a2c0419"}
Jul 14 22:57:39.536567 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:39.536144    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nn4qg" event={"ID":"acd21154-5243-4aeb-bd93-acd9da0ca28c","Type":"ContainerStarted","Data":"1d3334c1dcefc1c9bcf8dc2dbd90db364ac8313ccc9f105732c8013b3e19705d"}
Jul 14 22:57:39.537202 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:39.537179    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-7d6w9" event={"ID":"674701b4-5708-4e0e-858a-553e458a7351","Type":"ContainerStarted","Data":"1e42e3567be9363f3370371ce87d7df99d83043979ee9e96432feac04f07fb82"}
Jul 14 22:57:39.555581 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:39.555533    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nn4qg" podStartSLOduration=2.19419834 podStartE2EDuration="3.555519787s" totalImagesPullingTime="1.361321447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:57:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:57:39.554763761 +0000 UTC m=+46.857754554" watchObservedRunningTime="2026-07-14 22:57:39.555519787 +0000 UTC m=+46.858510579"
Jul 14 22:57:40.094345 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:40.094320    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vdd85_4be0471d-f724-4147-b1b0-c5ff7a82b7d1/serve-healthcheck-canary/0.log"
Jul 14 22:57:40.412267 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:40.412191    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/341a042f-b19e-418e-958a-f42c097be024-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-28rrp\" (UID: \"341a042f-b19e-418e-958a-f42c097be024\") " pod="openshift-insights/insights-runtime-extractor-28rrp"
Jul 14 22:57:40.412267 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:40.412239    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/341a042f-b19e-418e-958a-f42c097be024-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-28rrp\" (UID: \"341a042f-b19e-418e-958a-f42c097be024\") " pod="openshift-insights/insights-runtime-extractor-28rrp"
Jul 14 22:57:40.412458 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:40.412329    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 14 22:57:40.412458 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:40.412375    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 14 22:57:40.412458 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:40.412383    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/341a042f-b19e-418e-958a-f42c097be024-kube-rbac-proxy-cm podName:341a042f-b19e-418e-958a-f42c097be024 nodeName:}" failed. No retries permitted until 2026-07-14 22:57:44.412368997 +0000 UTC m=+51.715359767 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/341a042f-b19e-418e-958a-f42c097be024-kube-rbac-proxy-cm") pod "insights-runtime-extractor-28rrp" (UID: "341a042f-b19e-418e-958a-f42c097be024") : configmap "kube-rbac-proxy" not found
Jul 14 22:57:40.412458 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:40.412443    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/341a042f-b19e-418e-958a-f42c097be024-insights-runtime-extractor-tls podName:341a042f-b19e-418e-958a-f42c097be024 nodeName:}" failed. No retries permitted until 2026-07-14 22:57:44.412425757 +0000 UTC m=+51.715416529 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/341a042f-b19e-418e-958a-f42c097be024-insights-runtime-extractor-tls") pod "insights-runtime-extractor-28rrp" (UID: "341a042f-b19e-418e-958a-f42c097be024") : secret "insights-runtime-extractor-tls" not found
Jul 14 22:57:42.545002 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:42.544968    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-7d6w9" event={"ID":"674701b4-5708-4e0e-858a-553e458a7351","Type":"ContainerStarted","Data":"a75393ea57da71c8121009c422abcfb6e3c7686c594f2cbc898e13ca720ebfaf"}
Jul 14 22:57:42.563866 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:42.563816    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-7d6w9" podStartSLOduration=1.983425845 podStartE2EDuration="4.563799918s" totalImagesPullingTime="2.580374073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:57:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:57:42.563463424 +0000 UTC m=+49.866454217" watchObservedRunningTime="2026-07-14 22:57:42.563799918 +0000 UTC m=+49.866790709"
Jul 14 22:57:42.630868 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:42.630831    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-gzz8f\" (UID: \"c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-gzz8f"
Jul 14 22:57:42.631073 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:42.631007    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 14 22:57:42.631142 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:42.631088    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5-networking-console-plugin-cert podName:c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5 nodeName:}" failed. No retries permitted until 2026-07-14 22:57:50.631068228 +0000 UTC m=+57.934059010 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-gzz8f" (UID: "c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5") : secret "networking-console-plugin-cert" not found
Jul 14 22:57:44.447968 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:44.447919    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/341a042f-b19e-418e-958a-f42c097be024-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-28rrp\" (UID: \"341a042f-b19e-418e-958a-f42c097be024\") " pod="openshift-insights/insights-runtime-extractor-28rrp"
Jul 14 22:57:44.448475 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:44.447989    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/341a042f-b19e-418e-958a-f42c097be024-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-28rrp\" (UID: \"341a042f-b19e-418e-958a-f42c097be024\") " pod="openshift-insights/insights-runtime-extractor-28rrp"
Jul 14 22:57:44.448475 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:44.448078    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 14 22:57:44.448475 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:44.448148    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/341a042f-b19e-418e-958a-f42c097be024-insights-runtime-extractor-tls podName:341a042f-b19e-418e-958a-f42c097be024 nodeName:}" failed. No retries permitted until 2026-07-14 22:57:52.448128933 +0000 UTC m=+59.751119709 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/341a042f-b19e-418e-958a-f42c097be024-insights-runtime-extractor-tls") pod "insights-runtime-extractor-28rrp" (UID: "341a042f-b19e-418e-958a-f42c097be024") : secret "insights-runtime-extractor-tls" not found
Jul 14 22:57:44.448705 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:44.448680    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/341a042f-b19e-418e-958a-f42c097be024-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-28rrp\" (UID: \"341a042f-b19e-418e-958a-f42c097be024\") " pod="openshift-insights/insights-runtime-extractor-28rrp"
Jul 14 22:57:45.354769 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:45.354736    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-registry-tls\") pod \"image-registry-7955774756-rvmg5\" (UID: \"3a13b9a0-c0ff-4a8e-bfac-711ee2137471\") " pod="openshift-image-registry/image-registry-7955774756-rvmg5"
Jul 14 22:57:45.354769 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:45.354770    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffb5cdb7-2bde-418f-884d-0962fc26414e-metrics-tls\") pod \"dns-default-q4982\" (UID: \"ffb5cdb7-2bde-418f-884d-0962fc26414e\") " pod="openshift-dns/dns-default-q4982"
Jul 14 22:57:45.354982 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:45.354877    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 22:57:45.354982 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:45.354906    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 14 22:57:45.354982 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:45.354926    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7955774756-rvmg5: secret "image-registry-tls" not found
Jul 14 22:57:45.354982 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:45.354940    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffb5cdb7-2bde-418f-884d-0962fc26414e-metrics-tls podName:ffb5cdb7-2bde-418f-884d-0962fc26414e nodeName:}" failed. No retries permitted until 2026-07-14 22:58:01.354926089 +0000 UTC m=+68.657916860 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ffb5cdb7-2bde-418f-884d-0962fc26414e-metrics-tls") pod "dns-default-q4982" (UID: "ffb5cdb7-2bde-418f-884d-0962fc26414e") : secret "dns-default-metrics-tls" not found
Jul 14 22:57:45.354982 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:45.354967    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-registry-tls podName:3a13b9a0-c0ff-4a8e-bfac-711ee2137471 nodeName:}" failed. No retries permitted until 2026-07-14 22:58:01.354954451 +0000 UTC m=+68.657945225 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-registry-tls") pod "image-registry-7955774756-rvmg5" (UID: "3a13b9a0-c0ff-4a8e-bfac-711ee2137471") : secret "image-registry-tls" not found
Jul 14 22:57:50.702595 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:50.702553    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-gzz8f\" (UID: \"c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-gzz8f"
Jul 14 22:57:50.703121 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:50.702707    2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 14 22:57:50.703121 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:50.702775    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5-networking-console-plugin-cert podName:c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5 nodeName:}" failed. No retries permitted until 2026-07-14 22:58:06.702758169 +0000 UTC m=+74.005748942 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-gzz8f" (UID: "c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5") : secret "networking-console-plugin-cert" not found
Jul 14 22:57:52.517299 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:52.517255    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/341a042f-b19e-418e-958a-f42c097be024-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-28rrp\" (UID: \"341a042f-b19e-418e-958a-f42c097be024\") " pod="openshift-insights/insights-runtime-extractor-28rrp"
Jul 14 22:57:52.519631 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:52.519605    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/341a042f-b19e-418e-958a-f42c097be024-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-28rrp\" (UID: \"341a042f-b19e-418e-958a-f42c097be024\") " pod="openshift-insights/insights-runtime-extractor-28rrp"
Jul 14 22:57:52.646214 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:52.646182    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-28rrp"
Jul 14 22:57:52.773899 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:52.773813    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-28rrp"]
Jul 14 22:57:52.776473 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:57:52.776438    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod341a042f_b19e_418e_958a_f42c097be024.slice/crio-d720d21b8c34b4c2d40ec9b00d47901b864b0e70d770435059f9b6b0ade4144f WatchSource:0}: Error finding container d720d21b8c34b4c2d40ec9b00d47901b864b0e70d770435059f9b6b0ade4144f: Status 404 returned error can't find the container with id d720d21b8c34b4c2d40ec9b00d47901b864b0e70d770435059f9b6b0ade4144f
Jul 14 22:57:53.575484 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:53.575436    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-28rrp" event={"ID":"341a042f-b19e-418e-958a-f42c097be024","Type":"ContainerStarted","Data":"d720d21b8c34b4c2d40ec9b00d47901b864b0e70d770435059f9b6b0ade4144f"}
Jul 14 22:57:54.491527 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:54.491495    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j2z4s"
Jul 14 22:57:57.591672 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:57.591633    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-28rrp" event={"ID":"341a042f-b19e-418e-958a-f42c097be024","Type":"ContainerStarted","Data":"93dd3d6128f2858130bd2e605ef1a0a9bd4468e4a7c7dab982083a91699bbf65"}
Jul 14 22:57:57.592037 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:57.591689    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-28rrp" event={"ID":"341a042f-b19e-418e-958a-f42c097be024","Type":"ContainerStarted","Data":"59ec57e0038369fbe41fe103730a435edf16d4eb257c256b6a78676e2f7bc6eb"}
Jul 14 22:57:58.831494 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:58.830536    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-58gwx"]
Jul 14 22:57:58.833645 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:58.833621    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-58gwx"
Jul 14 22:57:58.836748 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:58.836724    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 14 22:57:58.837207 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:58.837187    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-rbks5\""
Jul 14 22:57:58.837341 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:58.837229    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 14 22:57:58.848967 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:58.848945    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-58gwx"]
Jul 14 22:57:58.853416 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:58.853398    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7955774756-rvmg5"]
Jul 14 22:57:58.853584 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:57:58.853568    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-7955774756-rvmg5" podUID="3a13b9a0-c0ff-4a8e-bfac-711ee2137471"
Jul 14 22:57:58.929745 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:58.929712    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-548c9cffcc-2qm89"]
Jul 14 22:57:58.932311 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:58.932271    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-548c9cffcc-2qm89"
Jul 14 22:57:58.950865 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:58.950840    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-548c9cffcc-2qm89"]
Jul 14 22:57:58.976916 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:58.976880    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr6g2\" (UniqueName: \"kubernetes.io/projected/f136e031-4413-4c00-ade9-a01658d10755-kube-api-access-pr6g2\") pod \"downloads-68c9cbb497-58gwx\" (UID: \"f136e031-4413-4c00-ade9-a01658d10755\") " pod="openshift-console/downloads-68c9cbb497-58gwx"
Jul 14 22:57:58.977060 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:58.976969    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6bbh\" (UniqueName: \"kubernetes.io/projected/39492113-cc47-4511-bc7d-d0785b892ed9-kube-api-access-p6bbh\") pod \"network-check-target-pbtpg\" (UID: \"39492113-cc47-4511-bc7d-d0785b892ed9\") " pod="openshift-network-diagnostics/network-check-target-pbtpg"
Jul 14 22:57:58.980646 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:58.980625    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 14 22:57:58.990327 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:58.990304    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 14 22:57:59.000545 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.000515    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6bbh\" (UniqueName: \"kubernetes.io/projected/39492113-cc47-4511-bc7d-d0785b892ed9-kube-api-access-p6bbh\") pod \"network-check-target-pbtpg\" (UID: \"39492113-cc47-4511-bc7d-d0785b892ed9\") " pod="openshift-network-diagnostics/network-check-target-pbtpg"
Jul 14 22:57:59.077986 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.077953    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/13b8983f-4071-41c4-9946-7a48c932d4ca-registry-certificates\") pod \"image-registry-548c9cffcc-2qm89\" (UID: \"13b8983f-4071-41c4-9946-7a48c932d4ca\") " pod="openshift-image-registry/image-registry-548c9cffcc-2qm89"
Jul 14 22:57:59.078140 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.077995    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/13b8983f-4071-41c4-9946-7a48c932d4ca-ca-trust-extracted\") pod \"image-registry-548c9cffcc-2qm89\" (UID: \"13b8983f-4071-41c4-9946-7a48c932d4ca\") " pod="openshift-image-registry/image-registry-548c9cffcc-2qm89"
Jul 14 22:57:59.078140 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.078018    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/13b8983f-4071-41c4-9946-7a48c932d4ca-image-registry-private-configuration\") pod \"image-registry-548c9cffcc-2qm89\" (UID: \"13b8983f-4071-41c4-9946-7a48c932d4ca\") " pod="openshift-image-registry/image-registry-548c9cffcc-2qm89"
Jul 14 22:57:59.078347 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.078131    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/13b8983f-4071-41c4-9946-7a48c932d4ca-trusted-ca\") pod \"image-registry-548c9cffcc-2qm89\" (UID: \"13b8983f-4071-41c4-9946-7a48c932d4ca\") " pod="openshift-image-registry/image-registry-548c9cffcc-2qm89"
Jul 14 22:57:59.078347 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.078186    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594-metrics-certs\") pod \"network-metrics-daemon-wjpcm\" (UID: \"8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594\") " pod="openshift-multus/network-metrics-daemon-wjpcm"
Jul 14 22:57:59.078347 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.078219    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13b8983f-4071-41c4-9946-7a48c932d4ca-registry-tls\") pod \"image-registry-548c9cffcc-2qm89\" (UID: \"13b8983f-4071-41c4-9946-7a48c932d4ca\") " pod="openshift-image-registry/image-registry-548c9cffcc-2qm89"
Jul 14 22:57:59.078347 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.078250    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13b8983f-4071-41c4-9946-7a48c932d4ca-bound-sa-token\") pod \"image-registry-548c9cffcc-2qm89\" (UID: \"13b8983f-4071-41c4-9946-7a48c932d4ca\") " pod="openshift-image-registry/image-registry-548c9cffcc-2qm89"
Jul 14 22:57:59.078347 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.078324    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e3e444e-1753-442e-8430-24b0022f9418-original-pull-secret\") pod \"global-pull-secret-syncer-xjw4z\" (UID: \"7e3e444e-1753-442e-8430-24b0022f9418\") " pod="kube-system/global-pull-secret-syncer-xjw4z"
Jul 14 22:57:59.078581 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.078362    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/13b8983f-4071-41c4-9946-7a48c932d4ca-installation-pull-secrets\") pod \"image-registry-548c9cffcc-2qm89\" (UID: \"13b8983f-4071-41c4-9946-7a48c932d4ca\") " pod="openshift-image-registry/image-registry-548c9cffcc-2qm89"
Jul 14 22:57:59.078581 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.078409    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkldt\" (UniqueName: \"kubernetes.io/projected/13b8983f-4071-41c4-9946-7a48c932d4ca-kube-api-access-lkldt\") pod \"image-registry-548c9cffcc-2qm89\" (UID: \"13b8983f-4071-41c4-9946-7a48c932d4ca\") " pod="openshift-image-registry/image-registry-548c9cffcc-2qm89"
Jul 14 22:57:59.078581 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.078438    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr6g2\" (UniqueName: \"kubernetes.io/projected/f136e031-4413-4c00-ade9-a01658d10755-kube-api-access-pr6g2\") pod \"downloads-68c9cbb497-58gwx\" (UID: \"f136e031-4413-4c00-ade9-a01658d10755\") " pod="openshift-console/downloads-68c9cbb497-58gwx"
Jul 14 22:57:59.081607 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.081535    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 14 22:57:59.081714 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.081613    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 14 22:57:59.089844 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.089808    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr6g2\" (UniqueName: \"kubernetes.io/projected/f136e031-4413-4c00-ade9-a01658d10755-kube-api-access-pr6g2\") pod \"downloads-68c9cbb497-58gwx\" (UID: \"f136e031-4413-4c00-ade9-a01658d10755\") " pod="openshift-console/downloads-68c9cbb497-58gwx"
Jul 14 22:57:59.091614 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.091590    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e3e444e-1753-442e-8430-24b0022f9418-original-pull-secret\") pod \"global-pull-secret-syncer-xjw4z\" (UID: \"7e3e444e-1753-442e-8430-24b0022f9418\") " pod="kube-system/global-pull-secret-syncer-xjw4z"
Jul 14 22:57:59.091899 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.091874    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594-metrics-certs\") pod \"network-metrics-daemon-wjpcm\" (UID: \"8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594\") " pod="openshift-multus/network-metrics-daemon-wjpcm"
Jul 14 22:57:59.136012 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.135969    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xjw4z"
Jul 14 22:57:59.143634 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.143609    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-58gwx"
Jul 14 22:57:59.146307 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.146272    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d24rt\""
Jul 14 22:57:59.152448 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.152429    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wx244\""
Jul 14 22:57:59.153449 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.153433    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wjpcm"
Jul 14 22:57:59.160202 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.160183    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pbtpg"
Jul 14 22:57:59.179545 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.179518    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/13b8983f-4071-41c4-9946-7a48c932d4ca-trusted-ca\") pod \"image-registry-548c9cffcc-2qm89\" (UID: \"13b8983f-4071-41c4-9946-7a48c932d4ca\") " pod="openshift-image-registry/image-registry-548c9cffcc-2qm89"
Jul 14 22:57:59.179663 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.179575    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13b8983f-4071-41c4-9946-7a48c932d4ca-registry-tls\") pod \"image-registry-548c9cffcc-2qm89\" (UID: \"13b8983f-4071-41c4-9946-7a48c932d4ca\") " pod="openshift-image-registry/image-registry-548c9cffcc-2qm89"
Jul 14 22:57:59.179663 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.179607    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13b8983f-4071-41c4-9946-7a48c932d4ca-bound-sa-token\") pod \"image-registry-548c9cffcc-2qm89\" (UID: \"13b8983f-4071-41c4-9946-7a48c932d4ca\") " pod="openshift-image-registry/image-registry-548c9cffcc-2qm89"
Jul 14 22:57:59.179663 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.179653    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/13b8983f-4071-41c4-9946-7a48c932d4ca-installation-pull-secrets\") pod \"image-registry-548c9cffcc-2qm89\" (UID: \"13b8983f-4071-41c4-9946-7a48c932d4ca\") " pod="openshift-image-registry/image-registry-548c9cffcc-2qm89"
Jul 14 22:57:59.179797 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.179696    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkldt\" (UniqueName: \"kubernetes.io/projected/13b8983f-4071-41c4-9946-7a48c932d4ca-kube-api-access-lkldt\") pod \"image-registry-548c9cffcc-2qm89\" (UID: \"13b8983f-4071-41c4-9946-7a48c932d4ca\") " pod="openshift-image-registry/image-registry-548c9cffcc-2qm89"
Jul 14 22:57:59.179797 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.179731    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/13b8983f-4071-41c4-9946-7a48c932d4ca-registry-certificates\") pod \"image-registry-548c9cffcc-2qm89\" (UID: \"13b8983f-4071-41c4-9946-7a48c932d4ca\") " pod="openshift-image-registry/image-registry-548c9cffcc-2qm89"
Jul 14 22:57:59.179797 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.179756    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/13b8983f-4071-41c4-9946-7a48c932d4ca-ca-trust-extracted\") pod \"image-registry-548c9cffcc-2qm89\" (UID: \"13b8983f-4071-41c4-9946-7a48c932d4ca\") " pod="openshift-image-registry/image-registry-548c9cffcc-2qm89"
Jul 14 22:57:59.179797 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.179781    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/13b8983f-4071-41c4-9946-7a48c932d4ca-image-registry-private-configuration\") pod \"image-registry-548c9cffcc-2qm89\" (UID: \"13b8983f-4071-41c4-9946-7a48c932d4ca\") " pod="openshift-image-registry/image-registry-548c9cffcc-2qm89"
Jul 14 22:57:59.180357 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.180331    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/13b8983f-4071-41c4-9946-7a48c932d4ca-ca-trust-extracted\") pod \"image-registry-548c9cffcc-2qm89\" (UID: \"13b8983f-4071-41c4-9946-7a48c932d4ca\") " pod="openshift-image-registry/image-registry-548c9cffcc-2qm89"
Jul 14 22:57:59.180597 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.180572    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/13b8983f-4071-41c4-9946-7a48c932d4ca-registry-certificates\") pod \"image-registry-548c9cffcc-2qm89\" (UID: \"13b8983f-4071-41c4-9946-7a48c932d4ca\") " pod="openshift-image-registry/image-registry-548c9cffcc-2qm89"
Jul 14 22:57:59.180766 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.180745    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/13b8983f-4071-41c4-9946-7a48c932d4ca-trusted-ca\") pod \"image-registry-548c9cffcc-2qm89\" (UID: \"13b8983f-4071-41c4-9946-7a48c932d4ca\") " pod="openshift-image-registry/image-registry-548c9cffcc-2qm89"
Jul 14 22:57:59.182470 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.182441    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13b8983f-4071-41c4-9946-7a48c932d4ca-registry-tls\") pod \"image-registry-548c9cffcc-2qm89\" (UID: \"13b8983f-4071-41c4-9946-7a48c932d4ca\") " pod="openshift-image-registry/image-registry-548c9cffcc-2qm89"
Jul 14 22:57:59.182470 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.182448    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/13b8983f-4071-41c4-9946-7a48c932d4ca-installation-pull-secrets\") pod \"image-registry-548c9cffcc-2qm89\" (UID: \"13b8983f-4071-41c4-9946-7a48c932d4ca\") " pod="openshift-image-registry/image-registry-548c9cffcc-2qm89"
Jul 14 22:57:59.184650 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.184631    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/13b8983f-4071-41c4-9946-7a48c932d4ca-image-registry-private-configuration\") pod \"image-registry-548c9cffcc-2qm89\" (UID: \"13b8983f-4071-41c4-9946-7a48c932d4ca\") " pod="openshift-image-registry/image-registry-548c9cffcc-2qm89"
Jul 14 22:57:59.193209 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.193187    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkldt\" (UniqueName: \"kubernetes.io/projected/13b8983f-4071-41c4-9946-7a48c932d4ca-kube-api-access-lkldt\") pod \"image-registry-548c9cffcc-2qm89\" (UID: \"13b8983f-4071-41c4-9946-7a48c932d4ca\") " pod="openshift-image-registry/image-registry-548c9cffcc-2qm89"
Jul 14 22:57:59.193209 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.193201    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13b8983f-4071-41c4-9946-7a48c932d4ca-bound-sa-token\") pod \"image-registry-548c9cffcc-2qm89\" (UID: \"13b8983f-4071-41c4-9946-7a48c932d4ca\") " pod="openshift-image-registry/image-registry-548c9cffcc-2qm89"
Jul 14 22:57:59.245297 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.245249    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gm9qp\""
Jul 14 22:57:59.252774 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.252750    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-548c9cffcc-2qm89"
Jul 14 22:57:59.598745 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.598701    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7955774756-rvmg5"
Jul 14 22:57:59.610637 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.610607    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7955774756-rvmg5"
Jul 14 22:57:59.684419 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.684387    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-bound-sa-token\") pod \"3a13b9a0-c0ff-4a8e-bfac-711ee2137471\" (UID: \"3a13b9a0-c0ff-4a8e-bfac-711ee2137471\") "
Jul 14 22:57:59.684599 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.684484    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-trusted-ca\") pod \"3a13b9a0-c0ff-4a8e-bfac-711ee2137471\" (UID: \"3a13b9a0-c0ff-4a8e-bfac-711ee2137471\") "
Jul 14 22:57:59.684599 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.684511    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-ca-trust-extracted\") pod \"3a13b9a0-c0ff-4a8e-bfac-711ee2137471\" (UID: \"3a13b9a0-c0ff-4a8e-bfac-711ee2137471\") "
Jul 14 22:57:59.684599 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.684527    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2pnlf\" (UniqueName: \"kubernetes.io/projected/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-kube-api-access-2pnlf\") pod \"3a13b9a0-c0ff-4a8e-bfac-711ee2137471\" (UID: \"3a13b9a0-c0ff-4a8e-bfac-711ee2137471\") "
Jul 14 22:57:59.684599 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.684589    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-installation-pull-secrets\") pod \"3a13b9a0-c0ff-4a8e-bfac-711ee2137471\" (UID: \"3a13b9a0-c0ff-4a8e-bfac-711ee2137471\") "
Jul 14 22:57:59.684801 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.684623    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-image-registry-private-configuration\") pod \"3a13b9a0-c0ff-4a8e-bfac-711ee2137471\" (UID: \"3a13b9a0-c0ff-4a8e-bfac-711ee2137471\") "
Jul 14 22:57:59.684801 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.684670    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-registry-certificates\") pod \"3a13b9a0-c0ff-4a8e-bfac-711ee2137471\" (UID: \"3a13b9a0-c0ff-4a8e-bfac-711ee2137471\") "
Jul 14 22:57:59.684895 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.684833    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "3a13b9a0-c0ff-4a8e-bfac-711ee2137471" (UID: "3a13b9a0-c0ff-4a8e-bfac-711ee2137471"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 22:57:59.684944 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.684927    2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-ca-trust-extracted\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 22:57:59.685230 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.685203    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "3a13b9a0-c0ff-4a8e-bfac-711ee2137471" (UID: "3a13b9a0-c0ff-4a8e-bfac-711ee2137471"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 22:57:59.685519 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.685410    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "3a13b9a0-c0ff-4a8e-bfac-711ee2137471" (UID: "3a13b9a0-c0ff-4a8e-bfac-711ee2137471"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 22:57:59.687256 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.687195    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "3a13b9a0-c0ff-4a8e-bfac-711ee2137471" (UID: "3a13b9a0-c0ff-4a8e-bfac-711ee2137471"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 22:57:59.687256 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.687220    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "3a13b9a0-c0ff-4a8e-bfac-711ee2137471" (UID: "3a13b9a0-c0ff-4a8e-bfac-711ee2137471"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 22:57:59.687419 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.687313    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-kube-api-access-2pnlf" (OuterVolumeSpecName: "kube-api-access-2pnlf") pod "3a13b9a0-c0ff-4a8e-bfac-711ee2137471" (UID: "3a13b9a0-c0ff-4a8e-bfac-711ee2137471"). InnerVolumeSpecName "kube-api-access-2pnlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 22:57:59.687676 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.687654    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "3a13b9a0-c0ff-4a8e-bfac-711ee2137471" (UID: "3a13b9a0-c0ff-4a8e-bfac-711ee2137471"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 22:57:59.785890 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.785858    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-trusted-ca\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 22:57:59.786005 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.785897    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pnlf\" (UniqueName: \"kubernetes.io/projected/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-kube-api-access-2pnlf\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 22:57:59.786005 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.785940    2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-installation-pull-secrets\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 22:57:59.786005 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.785955    2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-image-registry-private-configuration\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 22:57:59.786005 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.785971    2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-registry-certificates\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 22:57:59.786005 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.785985    2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-bound-sa-token\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 22:57:59.937805 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:57:59.937772    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pbtpg"]
Jul 14 22:57:59.942483 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:57:59.942415    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39492113_cc47_4511_bc7d_d0785b892ed9.slice/crio-0b0e6f3fbd83adf48fee39fddbbb86c593e4883ae4b0043730383a77261670ce WatchSource:0}: Error finding container 0b0e6f3fbd83adf48fee39fddbbb86c593e4883ae4b0043730383a77261670ce: Status 404 returned error can't find the container with id 0b0e6f3fbd83adf48fee39fddbbb86c593e4883ae4b0043730383a77261670ce
Jul 14 22:58:00.170937 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:00.170850    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-548c9cffcc-2qm89"]
Jul 14 22:58:00.174760 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:58:00.174734    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13b8983f_4071_41c4_9946_7a48c932d4ca.slice/crio-84980a0c08f16a98b2ac974bd0b19d8810cb4101d59f7e8c1cab203699c6cf0a WatchSource:0}: Error finding container 84980a0c08f16a98b2ac974bd0b19d8810cb4101d59f7e8c1cab203699c6cf0a: Status 404 returned error can't find the container with id 84980a0c08f16a98b2ac974bd0b19d8810cb4101d59f7e8c1cab203699c6cf0a
Jul 14 22:58:00.183432 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:00.183406    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xjw4z"]
Jul 14 22:58:00.186210 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:58:00.186186    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e3e444e_1753_442e_8430_24b0022f9418.slice/crio-39864b20585de0a7bbd79c785fe3320db7a10e3750d0827f0a5f0f1e7b59a834 WatchSource:0}: Error finding container 39864b20585de0a7bbd79c785fe3320db7a10e3750d0827f0a5f0f1e7b59a834: Status 404 returned error can't find the container with id 39864b20585de0a7bbd79c785fe3320db7a10e3750d0827f0a5f0f1e7b59a834
Jul 14 22:58:00.189912 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:00.189872    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-58gwx"]
Jul 14 22:58:00.193562 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:00.193543    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wjpcm"]
Jul 14 22:58:00.195923 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:58:00.195901    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ba0cabb_9c0f_46e2_a6e9_a90ff9c35594.slice/crio-38c4a4c4a9c87d844fac98218557e9fa4ba38735e21fe908c494beb5d0bba6d1 WatchSource:0}: Error finding container 38c4a4c4a9c87d844fac98218557e9fa4ba38735e21fe908c494beb5d0bba6d1: Status 404 returned error can't find the container with id 38c4a4c4a9c87d844fac98218557e9fa4ba38735e21fe908c494beb5d0bba6d1
Jul 14 22:58:00.198097 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:58:00.197973    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf136e031_4413_4c00_ade9_a01658d10755.slice/crio-b206b377f6c5cd9b640696c6ab29902b3f9f689020ff421c24835c0494982598 WatchSource:0}: Error finding container b206b377f6c5cd9b640696c6ab29902b3f9f689020ff421c24835c0494982598: Status 404 returned error can't find the container with id b206b377f6c5cd9b640696c6ab29902b3f9f689020ff421c24835c0494982598
Jul 14 22:58:00.608238 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:00.608199    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-28rrp" event={"ID":"341a042f-b19e-418e-958a-f42c097be024","Type":"ContainerStarted","Data":"458538ff36aad55a95eac0043769d49e60ac0eea0084f244a23319f34fadcd38"}
Jul 14 22:58:00.611552 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:00.611497    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xjw4z" event={"ID":"7e3e444e-1753-442e-8430-24b0022f9418","Type":"ContainerStarted","Data":"39864b20585de0a7bbd79c785fe3320db7a10e3750d0827f0a5f0f1e7b59a834"}
Jul 14 22:58:00.615766 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:00.615707    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-548c9cffcc-2qm89" event={"ID":"13b8983f-4071-41c4-9946-7a48c932d4ca","Type":"ContainerStarted","Data":"1036b8072709d0fb2bafa3b1e79705b8d2d4fe8d4d22c4270fb1681bf3318a57"}
Jul 14 22:58:00.615766 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:00.615756    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-548c9cffcc-2qm89" event={"ID":"13b8983f-4071-41c4-9946-7a48c932d4ca","Type":"ContainerStarted","Data":"84980a0c08f16a98b2ac974bd0b19d8810cb4101d59f7e8c1cab203699c6cf0a"}
Jul 14 22:58:00.615943 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:00.615905    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-548c9cffcc-2qm89"
Jul 14 22:58:00.618210 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:00.618145    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pbtpg" event={"ID":"39492113-cc47-4511-bc7d-d0785b892ed9","Type":"ContainerStarted","Data":"0b0e6f3fbd83adf48fee39fddbbb86c593e4883ae4b0043730383a77261670ce"}
Jul 14 22:58:00.619828 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:00.619573    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wjpcm" event={"ID":"8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594","Type":"ContainerStarted","Data":"38c4a4c4a9c87d844fac98218557e9fa4ba38735e21fe908c494beb5d0bba6d1"}
Jul 14 22:58:00.626967 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:00.621883    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7955774756-rvmg5"
Jul 14 22:58:00.626967 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:00.623347    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-58gwx" event={"ID":"f136e031-4413-4c00-ade9-a01658d10755","Type":"ContainerStarted","Data":"b206b377f6c5cd9b640696c6ab29902b3f9f689020ff421c24835c0494982598"}
Jul 14 22:58:00.629102 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:00.629055    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-28rrp" podStartSLOduration=17.720065885 podStartE2EDuration="24.629041328s" totalImagesPullingTime="6.908975443s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 22:57:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:58:00.628349129 +0000 UTC m=+67.931339925" watchObservedRunningTime="2026-07-14 22:58:00.629041328 +0000 UTC m=+67.932032121"
Jul 14 22:58:00.664496 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:00.664438    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-548c9cffcc-2qm89" podStartSLOduration=2.664415367 podStartE2EDuration="2.664415367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:57:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:58:00.663146922 +0000 UTC m=+67.966137727" watchObservedRunningTime="2026-07-14 22:58:00.664415367 +0000 UTC m=+67.967406160"
Jul 14 22:58:00.698450 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:00.698421    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7955774756-rvmg5"]
Jul 14 22:58:00.701905 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:00.701875    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7955774756-rvmg5"]
Jul 14 22:58:00.794052 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:00.793999    2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a13b9a0-c0ff-4a8e-bfac-711ee2137471-registry-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 22:58:01.230193 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:01.230019    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a13b9a0-c0ff-4a8e-bfac-711ee2137471" path="/var/lib/kubelet/pods/3a13b9a0-c0ff-4a8e-bfac-711ee2137471/volumes"
Jul 14 22:58:01.399446 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:01.399314    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffb5cdb7-2bde-418f-884d-0962fc26414e-metrics-tls\") pod \"dns-default-q4982\" (UID: \"ffb5cdb7-2bde-418f-884d-0962fc26414e\") " pod="openshift-dns/dns-default-q4982"
Jul 14 22:58:01.402414 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:01.402387    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffb5cdb7-2bde-418f-884d-0962fc26414e-metrics-tls\") pod \"dns-default-q4982\" (UID: \"ffb5cdb7-2bde-418f-884d-0962fc26414e\") " pod="openshift-dns/dns-default-q4982"
Jul 14 22:58:01.639177 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:01.638918    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wdwsb\""
Jul 14 22:58:01.644384 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:01.644352    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q4982"
Jul 14 22:58:01.834396 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:01.833862    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q4982"]
Jul 14 22:58:02.645726 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:02.645488    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wjpcm" event={"ID":"8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594","Type":"ContainerStarted","Data":"1ff5bb55d785dd96f647ee2f832acf6177fa9316e7b0e5b7f3dc55950170ba1a"}
Jul 14 22:58:02.645726 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:02.645555    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wjpcm" event={"ID":"8ba0cabb-9c0f-46e2-a6e9-a90ff9c35594","Type":"ContainerStarted","Data":"9aa3929afb489e86eafc6916a5be0f5d43a67f3b6412810e0216ae593f77482c"}
Jul 14 22:58:02.649172 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:02.648962    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q4982" event={"ID":"ffb5cdb7-2bde-418f-884d-0962fc26414e","Type":"ContainerStarted","Data":"1e7f763ce92b7b5b1d96212421509de17eb24f9397cdd06373f63dfa0337e69b"}
Jul 14 22:58:02.664194 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:02.664135    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wjpcm" podStartSLOduration=67.221268028 podStartE2EDuration="1m8.664118567s" totalImagesPullingTime="1.442850539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:56:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:58:02.663890285 +0000 UTC m=+69.966881070" watchObservedRunningTime="2026-07-14 22:58:02.664118567 +0000 UTC m=+69.967109355"
Jul 14 22:58:03.920330 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:03.920298    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-586dd6b845-wzwbr"]
Jul 14 22:58:03.938694 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:03.938661    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-586dd6b845-wzwbr"]
Jul 14 22:58:03.938862 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:03.938813    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-586dd6b845-wzwbr"
Jul 14 22:58:03.942473 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:03.942449    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 14 22:58:03.943839 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:03.943759    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-q5xp8\""
Jul 14 22:58:03.943839 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:03.943771    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 14 22:58:03.943839 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:03.943778    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 14 22:58:03.943839 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:03.943835    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 14 22:58:03.944061 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:03.943766    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 14 22:58:04.022012 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:04.021976    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d97dcd3-d994-45f9-a7cf-15d1ead626a0-console-serving-cert\") pod \"console-586dd6b845-wzwbr\" (UID: \"9d97dcd3-d994-45f9-a7cf-15d1ead626a0\") " pod="openshift-console/console-586dd6b845-wzwbr"
Jul 14 22:58:04.022210 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:04.022042    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d97dcd3-d994-45f9-a7cf-15d1ead626a0-console-oauth-config\") pod \"console-586dd6b845-wzwbr\" (UID: \"9d97dcd3-d994-45f9-a7cf-15d1ead626a0\") " pod="openshift-console/console-586dd6b845-wzwbr"
Jul 14 22:58:04.022210 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:04.022112    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d97dcd3-d994-45f9-a7cf-15d1ead626a0-service-ca\") pod \"console-586dd6b845-wzwbr\" (UID: \"9d97dcd3-d994-45f9-a7cf-15d1ead626a0\") " pod="openshift-console/console-586dd6b845-wzwbr"
Jul 14 22:58:04.022382 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:04.022207    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d97dcd3-d994-45f9-a7cf-15d1ead626a0-console-config\") pod \"console-586dd6b845-wzwbr\" (UID: \"9d97dcd3-d994-45f9-a7cf-15d1ead626a0\") " pod="openshift-console/console-586dd6b845-wzwbr"
Jul 14 22:58:04.022382 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:04.022239    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d97dcd3-d994-45f9-a7cf-15d1ead626a0-oauth-serving-cert\") pod \"console-586dd6b845-wzwbr\" (UID: \"9d97dcd3-d994-45f9-a7cf-15d1ead626a0\") " pod="openshift-console/console-586dd6b845-wzwbr"
Jul 14 22:58:04.022382 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:04.022262    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brh7s\" (UniqueName: \"kubernetes.io/projected/9d97dcd3-d994-45f9-a7cf-15d1ead626a0-kube-api-access-brh7s\") pod \"console-586dd6b845-wzwbr\" (UID: \"9d97dcd3-d994-45f9-a7cf-15d1ead626a0\") " pod="openshift-console/console-586dd6b845-wzwbr"
Jul 14 22:58:04.123336 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:04.123276    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d97dcd3-d994-45f9-a7cf-15d1ead626a0-console-oauth-config\") pod \"console-586dd6b845-wzwbr\" (UID: \"9d97dcd3-d994-45f9-a7cf-15d1ead626a0\") " pod="openshift-console/console-586dd6b845-wzwbr"
Jul 14 22:58:04.123509 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:04.123358    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d97dcd3-d994-45f9-a7cf-15d1ead626a0-service-ca\") pod \"console-586dd6b845-wzwbr\" (UID: \"9d97dcd3-d994-45f9-a7cf-15d1ead626a0\") " pod="openshift-console/console-586dd6b845-wzwbr"
Jul 14 22:58:04.123509 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:04.123428    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d97dcd3-d994-45f9-a7cf-15d1ead626a0-console-config\") pod \"console-586dd6b845-wzwbr\" (UID: \"9d97dcd3-d994-45f9-a7cf-15d1ead626a0\") " pod="openshift-console/console-586dd6b845-wzwbr"
Jul 14 22:58:04.123509 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:04.123456    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d97dcd3-d994-45f9-a7cf-15d1ead626a0-oauth-serving-cert\") pod \"console-586dd6b845-wzwbr\" (UID: \"9d97dcd3-d994-45f9-a7cf-15d1ead626a0\") " pod="openshift-console/console-586dd6b845-wzwbr"
Jul 14 22:58:04.123509 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:04.123480    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brh7s\" (UniqueName: \"kubernetes.io/projected/9d97dcd3-d994-45f9-a7cf-15d1ead626a0-kube-api-access-brh7s\") pod \"console-586dd6b845-wzwbr\" (UID: \"9d97dcd3-d994-45f9-a7cf-15d1ead626a0\") " pod="openshift-console/console-586dd6b845-wzwbr"
Jul 14 22:58:04.123658 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:04.123511    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d97dcd3-d994-45f9-a7cf-15d1ead626a0-console-serving-cert\") pod \"console-586dd6b845-wzwbr\" (UID: \"9d97dcd3-d994-45f9-a7cf-15d1ead626a0\") " pod="openshift-console/console-586dd6b845-wzwbr"
Jul 14 22:58:04.124222 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:04.124183    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d97dcd3-d994-45f9-a7cf-15d1ead626a0-oauth-serving-cert\") pod \"console-586dd6b845-wzwbr\" (UID: \"9d97dcd3-d994-45f9-a7cf-15d1ead626a0\") " pod="openshift-console/console-586dd6b845-wzwbr"
Jul 14 22:58:04.124222 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:04.124193    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d97dcd3-d994-45f9-a7cf-15d1ead626a0-console-config\") pod \"console-586dd6b845-wzwbr\" (UID: \"9d97dcd3-d994-45f9-a7cf-15d1ead626a0\") " pod="openshift-console/console-586dd6b845-wzwbr"
Jul 14 22:58:04.124411 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:04.124356    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d97dcd3-d994-45f9-a7cf-15d1ead626a0-service-ca\") pod \"console-586dd6b845-wzwbr\" (UID: \"9d97dcd3-d994-45f9-a7cf-15d1ead626a0\") " pod="openshift-console/console-586dd6b845-wzwbr"
Jul 14 22:58:04.126650 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:04.126605    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d97dcd3-d994-45f9-a7cf-15d1ead626a0-console-oauth-config\") pod \"console-586dd6b845-wzwbr\" (UID: \"9d97dcd3-d994-45f9-a7cf-15d1ead626a0\") " pod="openshift-console/console-586dd6b845-wzwbr"
Jul 14 22:58:04.126852 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:04.126827    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d97dcd3-d994-45f9-a7cf-15d1ead626a0-console-serving-cert\") pod \"console-586dd6b845-wzwbr\" (UID: \"9d97dcd3-d994-45f9-a7cf-15d1ead626a0\") " pod="openshift-console/console-586dd6b845-wzwbr"
Jul 14 22:58:04.136131 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:04.136096    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brh7s\" (UniqueName: \"kubernetes.io/projected/9d97dcd3-d994-45f9-a7cf-15d1ead626a0-kube-api-access-brh7s\") pod \"console-586dd6b845-wzwbr\" (UID: \"9d97dcd3-d994-45f9-a7cf-15d1ead626a0\") " pod="openshift-console/console-586dd6b845-wzwbr"
Jul 14 22:58:04.251321 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:04.251268    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-586dd6b845-wzwbr"
Jul 14 22:58:06.590833 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:06.590807    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-586dd6b845-wzwbr"]
Jul 14 22:58:06.596528 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:58:06.596474    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d97dcd3_d994_45f9_a7cf_15d1ead626a0.slice/crio-9a8380204d058f0b52a7284ffd4fbfc7f844f0926458f445d023bb2d4c220121 WatchSource:0}: Error finding container 9a8380204d058f0b52a7284ffd4fbfc7f844f0926458f445d023bb2d4c220121: Status 404 returned error can't find the container with id 9a8380204d058f0b52a7284ffd4fbfc7f844f0926458f445d023bb2d4c220121
Jul 14 22:58:06.670016 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:06.669867    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xjw4z" event={"ID":"7e3e444e-1753-442e-8430-24b0022f9418","Type":"ContainerStarted","Data":"3545cc39a0451a50db39f6542de89f54788da736ea0e9a5cb296f3627212be96"}
Jul 14 22:58:06.672592 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:06.672527    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q4982" event={"ID":"ffb5cdb7-2bde-418f-884d-0962fc26414e","Type":"ContainerStarted","Data":"b83cf3cc7bcc06e526daf675b72170e23af81f13b0fdabf120a3d4afdbcf1660"}
Jul 14 22:58:06.674932 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:06.674389    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pbtpg" event={"ID":"39492113-cc47-4511-bc7d-d0785b892ed9","Type":"ContainerStarted","Data":"a2b7a4ae9162cd3a3c99c8d6b8e7e2407549fcb021f74b361964d85b6d69a25c"}
Jul 14 22:58:06.674932 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:06.674873    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pbtpg"
Jul 14 22:58:06.676960 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:06.676881    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-586dd6b845-wzwbr" event={"ID":"9d97dcd3-d994-45f9-a7cf-15d1ead626a0","Type":"ContainerStarted","Data":"9a8380204d058f0b52a7284ffd4fbfc7f844f0926458f445d023bb2d4c220121"}
Jul 14 22:58:06.690595 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:06.690328    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xjw4z" podStartSLOduration=66.433752116 podStartE2EDuration="1m12.690307534s" totalImagesPullingTime="6.256555418s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 22:56:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:58:06.689975855 +0000 UTC m=+73.992966648" watchObservedRunningTime="2026-07-14 22:58:06.690307534 +0000 UTC m=+73.993298325"
Jul 14 22:58:06.707548 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:06.706896    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pbtpg" podStartSLOduration=66.215317272 podStartE2EDuration="1m12.706875902s" totalImagesPullingTime="6.49155863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:56:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:58:06.706182122 +0000 UTC m=+74.009172915" watchObservedRunningTime="2026-07-14 22:58:06.706875902 +0000 UTC m=+74.009866696"
Jul 14 22:58:06.745960 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:06.745929    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-gzz8f\" (UID: \"c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-gzz8f"
Jul 14 22:58:06.748911 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:06.748889    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-gzz8f\" (UID: \"c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-gzz8f"
Jul 14 22:58:06.944932 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:06.944839    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-hcpg9\""
Jul 14 22:58:06.952668 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:06.952641    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-gzz8f"
Jul 14 22:58:07.108714 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:07.108682    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-gzz8f"]
Jul 14 22:58:07.113805 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:58:07.113776    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8c0fd62_6e13_4d36_b83e_c2a7542cf8b5.slice/crio-315a1df7bef2bef65ba461a538e009554097c1852c603c3f3cbc476b1a6e95e1 WatchSource:0}: Error finding container 315a1df7bef2bef65ba461a538e009554097c1852c603c3f3cbc476b1a6e95e1: Status 404 returned error can't find the container with id 315a1df7bef2bef65ba461a538e009554097c1852c603c3f3cbc476b1a6e95e1
Jul 14 22:58:07.682602 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:07.682561    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-gzz8f" event={"ID":"c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5","Type":"ContainerStarted","Data":"315a1df7bef2bef65ba461a538e009554097c1852c603c3f3cbc476b1a6e95e1"}
Jul 14 22:58:07.684873 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:07.684845    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q4982" event={"ID":"ffb5cdb7-2bde-418f-884d-0962fc26414e","Type":"ContainerStarted","Data":"4a5cde8c43494461c16bc7ec35fff209f1a019fe168390e5475e0ef05c324a71"}
Jul 14 22:58:07.685218 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:07.685198    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-q4982"
Jul 14 22:58:07.702388 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:07.702330    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-q4982" podStartSLOduration=34.103019842 podStartE2EDuration="38.702316215s" totalImagesPullingTime="4.599296373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:57:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:58:07.701602064 +0000 UTC m=+75.004592861" watchObservedRunningTime="2026-07-14 22:58:07.702316215 +0000 UTC m=+75.005307000"
Jul 14 22:58:10.698221 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:10.698183    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-gzz8f" event={"ID":"c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5","Type":"ContainerStarted","Data":"e000eba537da4cadc4518a072d9257a9cb09778931ce728a0acce9ecc0e44212"}
Jul 14 22:58:10.699950 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:10.699921    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-586dd6b845-wzwbr" event={"ID":"9d97dcd3-d994-45f9-a7cf-15d1ead626a0","Type":"ContainerStarted","Data":"1e2533fad042edc70acbfb60af3420aba3956f4c19ccaf0c68185c0c607a855b"}
Jul 14 22:58:10.716004 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:10.715713    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-gzz8f" podStartSLOduration=33.862180577 podStartE2EDuration="36.715696096s" totalImagesPullingTime="2.853515519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:57:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:58:10.715609966 +0000 UTC m=+78.018600759" watchObservedRunningTime="2026-07-14 22:58:10.715696096 +0000 UTC m=+78.018686888"
Jul 14 22:58:10.736374 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:10.736307    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-586dd6b845-wzwbr" podStartSLOduration=4.358385315 podStartE2EDuration="7.736263233s" totalImagesPullingTime="3.377877918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:58:10.73525077 +0000 UTC m=+78.038241569" watchObservedRunningTime="2026-07-14 22:58:10.736263233 +0000 UTC m=+78.039254026"
Jul 14 22:58:14.252348 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:14.252302    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-586dd6b845-wzwbr"
Jul 14 22:58:14.252829 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:14.252379    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-586dd6b845-wzwbr"
Jul 14 22:58:14.254173 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:14.254146    2563 patch_prober.go:28] interesting pod/console-586dd6b845-wzwbr container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.15:8443/health\": dial tcp 10.133.0.15:8443: connect: connection refused" start-of-body=
Jul 14 22:58:14.254308 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:14.254196    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-586dd6b845-wzwbr" podUID="9d97dcd3-d994-45f9-a7cf-15d1ead626a0" containerName="console" probeResult="failure" output="Get \"https://10.133.0.15:8443/health\": dial tcp 10.133.0.15:8443: connect: connection refused"
Jul 14 22:58:18.694845 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:18.694808    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-q4982"
Jul 14 22:58:19.257875 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:19.257832    2563 patch_prober.go:28] interesting pod/image-registry-548c9cffcc-2qm89 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 14 22:58:19.258045 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:19.257883    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-548c9cffcc-2qm89" podUID="13b8983f-4071-41c4-9946-7a48c932d4ca" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 14 22:58:19.733341 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:19.733295    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-58gwx" event={"ID":"f136e031-4413-4c00-ade9-a01658d10755","Type":"ContainerStarted","Data":"4f487c09d693f209ad9e32e583c670845b99db686081319b9a5b76b5c6713f9d"}
Jul 14 22:58:19.733878 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:19.733810    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-58gwx"
Jul 14 22:58:19.750421 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:19.750390    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-58gwx"
Jul 14 22:58:19.761126 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:19.761043    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-58gwx" podStartSLOduration=2.929325167 podStartE2EDuration="21.761026798s" totalImagesPullingTime="18.831701631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:57:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:58:19.7583074 +0000 UTC m=+87.061298192" watchObservedRunningTime="2026-07-14 22:58:19.761026798 +0000 UTC m=+87.064017591"
Jul 14 22:58:21.158342 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:21.158304    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-74qb8"]
Jul 14 22:58:21.195098 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:21.195064    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-74qb8"
Jul 14 22:58:21.199070 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:21.199050    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 14 22:58:21.199519 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:21.199501    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 14 22:58:21.199915 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:21.199896    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ztnh6\""
Jul 14 22:58:21.200133 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:21.200107    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 14 22:58:21.200632 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:21.200607    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 14 22:58:21.200943 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:21.200926    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 14 22:58:21.201404 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:21.201367    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 14 22:58:21.361995 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:21.361966    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a998b1ac-c2d1-4c50-bfcd-4a700894beaa-node-exporter-accelerators-collector-config\") pod \"node-exporter-74qb8\" (UID: \"a998b1ac-c2d1-4c50-bfcd-4a700894beaa\") " pod="openshift-monitoring/node-exporter-74qb8"
Jul 14 22:58:21.362320 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:21.362270    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a998b1ac-c2d1-4c50-bfcd-4a700894beaa-node-exporter-wtmp\") pod \"node-exporter-74qb8\" (UID: \"a998b1ac-c2d1-4c50-bfcd-4a700894beaa\") " pod="openshift-monitoring/node-exporter-74qb8"
Jul 14 22:58:21.362442 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:21.362359    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a998b1ac-c2d1-4c50-bfcd-4a700894beaa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-74qb8\" (UID: \"a998b1ac-c2d1-4c50-bfcd-4a700894beaa\") " pod="openshift-monitoring/node-exporter-74qb8"
Jul 14 22:58:21.362442 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:21.362393    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkd4d\" (UniqueName: \"kubernetes.io/projected/a998b1ac-c2d1-4c50-bfcd-4a700894beaa-kube-api-access-lkd4d\") pod \"node-exporter-74qb8\" (UID: \"a998b1ac-c2d1-4c50-bfcd-4a700894beaa\") " pod="openshift-monitoring/node-exporter-74qb8"
Jul 14 22:58:21.362442 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:21.362432    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a998b1ac-c2d1-4c50-bfcd-4a700894beaa-metrics-client-ca\") pod \"node-exporter-74qb8\" (UID: \"a998b1ac-c2d1-4c50-bfcd-4a700894beaa\") " pod="openshift-monitoring/node-exporter-74qb8"
Jul 14 22:58:21.362593 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:21.362473    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a998b1ac-c2d1-4c50-bfcd-4a700894beaa-node-exporter-textfile\") pod \"node-exporter-74qb8\" (UID: \"a998b1ac-c2d1-4c50-bfcd-4a700894beaa\") " pod="openshift-monitoring/node-exporter-74qb8"
Jul 14 22:58:21.362593 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:21.362507    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a998b1ac-c2d1-4c50-bfcd-4a700894beaa-node-exporter-tls\") pod \"node-exporter-74qb8\" (UID: \"a998b1ac-c2d1-4c50-bfcd-4a700894beaa\") " pod="openshift-monitoring/node-exporter-74qb8"
Jul 14 22:58:21.362593 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:21.362537    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a998b1ac-c2d1-4c50-bfcd-4a700894beaa-sys\") pod \"node-exporter-74qb8\" (UID: \"a998b1ac-c2d1-4c50-bfcd-4a700894beaa\") " pod="openshift-monitoring/node-exporter-74qb8"
Jul 14 22:58:21.362749 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:21.362601    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a998b1ac-c2d1-4c50-bfcd-4a700894beaa-root\") pod \"node-exporter-74qb8\" (UID: \"a998b1ac-c2d1-4c50-bfcd-4a700894beaa\") " pod="openshift-monitoring/node-exporter-74qb8"
Jul 14 22:58:21.463089 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:21.462992    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a998b1ac-c2d1-4c50-bfcd-4a700894beaa-metrics-client-ca\") pod \"node-exporter-74qb8\" (UID: \"a998b1ac-c2d1-4c50-bfcd-4a700894beaa\") " pod="openshift-monitoring/node-exporter-74qb8"
Jul 14 22:58:21.463815 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:21.463774    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a998b1ac-c2d1-4c50-bfcd-4a700894beaa-node-exporter-textfile\") pod \"node-exporter-74qb8\" (UID: \"a998b1ac-c2d1-4c50-bfcd-4a700894beaa\") " pod="openshift-monitoring/node-exporter-74qb8"
Jul 14 22:58:21.463815 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:21.463613    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a998b1ac-c2d1-4c50-bfcd-4a700894beaa-metrics-client-ca\") pod \"node-exporter-74qb8\" (UID: \"a998b1ac-c2d1-4c50-bfcd-4a700894beaa\") " pod="openshift-monitoring/node-exporter-74qb8"
Jul 14 22:58:21.464010 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:21.463837    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a998b1ac-c2d1-4c50-bfcd-4a700894beaa-node-exporter-tls\") pod \"node-exporter-74qb8\" (UID: \"a998b1ac-c2d1-4c50-bfcd-4a700894beaa\") " pod="openshift-monitoring/node-exporter-74qb8"
Jul 14 22:58:21.464010 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:21.463868    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a998b1ac-c2d1-4c50-bfcd-4a700894beaa-sys\") pod \"node-exporter-74qb8\" (UID: \"a998b1ac-c2d1-4c50-bfcd-4a700894beaa\") " pod="openshift-monitoring/node-exporter-74qb8"
Jul 14 22:58:21.464010 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:21.463912    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a998b1ac-c2d1-4c50-bfcd-4a700894beaa-root\") pod \"node-exporter-74qb8\" (UID: \"a998b1ac-c2d1-4c50-bfcd-4a700894beaa\") " pod="openshift-monitoring/node-exporter-74qb8"
Jul 14 22:58:21.464010 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:21.463961    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a998b1ac-c2d1-4c50-bfcd-4a700894beaa-node-exporter-accelerators-collector-config\") pod \"node-exporter-74qb8\" (UID: \"a998b1ac-c2d1-4c50-bfcd-4a700894beaa\") " pod="openshift-monitoring/node-exporter-74qb8"
Jul 14 22:58:21.464010 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:21.463993    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a998b1ac-c2d1-4c50-bfcd-4a700894beaa-node-exporter-wtmp\") pod \"node-exporter-74qb8\" (UID: \"a998b1ac-c2d1-4c50-bfcd-4a700894beaa\") " pod="openshift-monitoring/node-exporter-74qb8"
Jul 14 22:58:21.464255 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:21.464018    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a998b1ac-c2d1-4c50-bfcd-4a700894beaa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-74qb8\" (UID: \"a998b1ac-c2d1-4c50-bfcd-4a700894beaa\") " pod="openshift-monitoring/node-exporter-74qb8"
Jul 14 22:58:21.464255 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:21.464047    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkd4d\" (UniqueName: \"kubernetes.io/projected/a998b1ac-c2d1-4c50-bfcd-4a700894beaa-kube-api-access-lkd4d\") pod \"node-exporter-74qb8\" (UID: \"a998b1ac-c2d1-4c50-bfcd-4a700894beaa\") " pod="openshift-monitoring/node-exporter-74qb8"
Jul 14 22:58:21.464255 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:21.464086    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a998b1ac-c2d1-4c50-bfcd-4a700894beaa-node-exporter-textfile\") pod \"node-exporter-74qb8\" (UID: \"a998b1ac-c2d1-4c50-bfcd-4a700894beaa\") " pod="openshift-monitoring/node-exporter-74qb8"
Jul 14 22:58:21.464255 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:21.464144    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a998b1ac-c2d1-4c50-bfcd-4a700894beaa-root\") pod \"node-exporter-74qb8\" (UID: \"a998b1ac-c2d1-4c50-bfcd-4a700894beaa\") " pod="openshift-monitoring/node-exporter-74qb8"
Jul 14 22:58:21.465306 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:21.464896    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a998b1ac-c2d1-4c50-bfcd-4a700894beaa-sys\") pod \"node-exporter-74qb8\" (UID: \"a998b1ac-c2d1-4c50-bfcd-4a700894beaa\") " pod="openshift-monitoring/node-exporter-74qb8"
Jul 14 22:58:21.465306 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:21.465075    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a998b1ac-c2d1-4c50-bfcd-4a700894beaa-node-exporter-accelerators-collector-config\") pod \"node-exporter-74qb8\" (UID: \"a998b1ac-c2d1-4c50-bfcd-4a700894beaa\") " pod="openshift-monitoring/node-exporter-74qb8"
Jul 14 22:58:21.465306 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:21.465089    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a998b1ac-c2d1-4c50-bfcd-4a700894beaa-node-exporter-wtmp\") pod \"node-exporter-74qb8\" (UID: \"a998b1ac-c2d1-4c50-bfcd-4a700894beaa\") " pod="openshift-monitoring/node-exporter-74qb8"
Jul 14 22:58:21.467210 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:21.467188    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a998b1ac-c2d1-4c50-bfcd-4a700894beaa-node-exporter-tls\") pod \"node-exporter-74qb8\" (UID: \"a998b1ac-c2d1-4c50-bfcd-4a700894beaa\") " pod="openshift-monitoring/node-exporter-74qb8"
Jul 14 22:58:21.467908 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:21.467887    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a998b1ac-c2d1-4c50-bfcd-4a700894beaa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-74qb8\" (UID: \"a998b1ac-c2d1-4c50-bfcd-4a700894beaa\") " pod="openshift-monitoring/node-exporter-74qb8"
Jul 14 22:58:21.472837 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:21.472812    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkd4d\" (UniqueName: \"kubernetes.io/projected/a998b1ac-c2d1-4c50-bfcd-4a700894beaa-kube-api-access-lkd4d\") pod \"node-exporter-74qb8\" (UID: \"a998b1ac-c2d1-4c50-bfcd-4a700894beaa\") " pod="openshift-monitoring/node-exporter-74qb8"
Jul 14 22:58:21.510702 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:21.510668    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-74qb8"
Jul 14 22:58:21.545423 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:58:21.545394    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda998b1ac_c2d1_4c50_bfcd_4a700894beaa.slice/crio-473c746d513097c2f52ec6871e5f0b99e70d2dce333e35a6e91538385bf61a96 WatchSource:0}: Error finding container 473c746d513097c2f52ec6871e5f0b99e70d2dce333e35a6e91538385bf61a96: Status 404 returned error can't find the container with id 473c746d513097c2f52ec6871e5f0b99e70d2dce333e35a6e91538385bf61a96
Jul 14 22:58:21.644436 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:21.644404    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-548c9cffcc-2qm89"
Jul 14 22:58:21.743546 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:21.743507    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-74qb8" event={"ID":"a998b1ac-c2d1-4c50-bfcd-4a700894beaa","Type":"ContainerStarted","Data":"473c746d513097c2f52ec6871e5f0b99e70d2dce333e35a6e91538385bf61a96"}
Jul 14 22:58:23.753975 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:23.753933    2563 generic.go:356] "Generic (PLEG): container finished" podID="a998b1ac-c2d1-4c50-bfcd-4a700894beaa" containerID="feed3b5f11022d5ce57f8b56b4f70f9fcad43cdd761f00aa230e3a3f3bd41133" exitCode=0
Jul 14 22:58:23.754430 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:23.753998    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-74qb8" event={"ID":"a998b1ac-c2d1-4c50-bfcd-4a700894beaa","Type":"ContainerDied","Data":"feed3b5f11022d5ce57f8b56b4f70f9fcad43cdd761f00aa230e3a3f3bd41133"}
Jul 14 22:58:24.256960 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:24.256927    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-586dd6b845-wzwbr"
Jul 14 22:58:24.261526 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:24.261491    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-586dd6b845-wzwbr"
Jul 14 22:58:24.760617 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:24.760582    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-74qb8" event={"ID":"a998b1ac-c2d1-4c50-bfcd-4a700894beaa","Type":"ContainerStarted","Data":"a01425fa159bb435b15358039381c1474faac36898ae460993a213f79ea887d4"}
Jul 14 22:58:24.761179 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:24.761149    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-74qb8" event={"ID":"a998b1ac-c2d1-4c50-bfcd-4a700894beaa","Type":"ContainerStarted","Data":"d8d4437f1d93aaddd8a1632e541afd9b687f80461f4420ed7936167a86709505"}
Jul 14 22:58:24.784878 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:24.784820    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-74qb8" podStartSLOduration=2.002220518 podStartE2EDuration="3.784802s" totalImagesPullingTime="1.679063831s" totalInitContainerRuntime="103.517651ms" isStatefulPod=true podCreationTimestamp="2026-07-14 22:58:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:58:24.782853617 +0000 UTC m=+92.085844423" watchObservedRunningTime="2026-07-14 22:58:24.784802 +0000 UTC m=+92.087792792"
Jul 14 22:58:26.751079 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:26.751029    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-69c5f6d6cb-qc4nn"]
Jul 14 22:58:26.765415 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:26.765373    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69c5f6d6cb-qc4nn"
Jul 14 22:58:26.781113 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:26.781087    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69c5f6d6cb-qc4nn"]
Jul 14 22:58:26.786333 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:26.786276    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 14 22:58:26.807564 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:26.807528    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f66f937-853a-438f-9bb2-c11f7fc46a31-oauth-serving-cert\") pod \"console-69c5f6d6cb-qc4nn\" (UID: \"8f66f937-853a-438f-9bb2-c11f7fc46a31\") " pod="openshift-console/console-69c5f6d6cb-qc4nn"
Jul 14 22:58:26.807696 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:26.807570    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f66f937-853a-438f-9bb2-c11f7fc46a31-console-config\") pod \"console-69c5f6d6cb-qc4nn\" (UID: \"8f66f937-853a-438f-9bb2-c11f7fc46a31\") " pod="openshift-console/console-69c5f6d6cb-qc4nn"
Jul 14 22:58:26.807696 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:26.807590    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptght\" (UniqueName: \"kubernetes.io/projected/8f66f937-853a-438f-9bb2-c11f7fc46a31-kube-api-access-ptght\") pod \"console-69c5f6d6cb-qc4nn\" (UID: \"8f66f937-853a-438f-9bb2-c11f7fc46a31\") " pod="openshift-console/console-69c5f6d6cb-qc4nn"
Jul 14 22:58:26.807696 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:26.807646    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f66f937-853a-438f-9bb2-c11f7fc46a31-console-oauth-config\") pod \"console-69c5f6d6cb-qc4nn\" (UID: \"8f66f937-853a-438f-9bb2-c11f7fc46a31\") " pod="openshift-console/console-69c5f6d6cb-qc4nn"
Jul 14 22:58:26.807860 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:26.807699    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f66f937-853a-438f-9bb2-c11f7fc46a31-trusted-ca-bundle\") pod \"console-69c5f6d6cb-qc4nn\" (UID: \"8f66f937-853a-438f-9bb2-c11f7fc46a31\") " pod="openshift-console/console-69c5f6d6cb-qc4nn"
Jul 14 22:58:26.807860 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:26.807742    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f66f937-853a-438f-9bb2-c11f7fc46a31-console-serving-cert\") pod \"console-69c5f6d6cb-qc4nn\" (UID: \"8f66f937-853a-438f-9bb2-c11f7fc46a31\") " pod="openshift-console/console-69c5f6d6cb-qc4nn"
Jul 14 22:58:26.807860 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:26.807778    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f66f937-853a-438f-9bb2-c11f7fc46a31-service-ca\") pod \"console-69c5f6d6cb-qc4nn\" (UID: \"8f66f937-853a-438f-9bb2-c11f7fc46a31\") " pod="openshift-console/console-69c5f6d6cb-qc4nn"
Jul 14 22:58:26.908242 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:26.908188    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f66f937-853a-438f-9bb2-c11f7fc46a31-service-ca\") pod \"console-69c5f6d6cb-qc4nn\" (UID: \"8f66f937-853a-438f-9bb2-c11f7fc46a31\") " pod="openshift-console/console-69c5f6d6cb-qc4nn"
Jul 14 22:58:26.908440 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:26.908273    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f66f937-853a-438f-9bb2-c11f7fc46a31-oauth-serving-cert\") pod \"console-69c5f6d6cb-qc4nn\" (UID: \"8f66f937-853a-438f-9bb2-c11f7fc46a31\") " pod="openshift-console/console-69c5f6d6cb-qc4nn"
Jul 14 22:58:26.908440 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:26.908321    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f66f937-853a-438f-9bb2-c11f7fc46a31-console-config\") pod \"console-69c5f6d6cb-qc4nn\" (UID: \"8f66f937-853a-438f-9bb2-c11f7fc46a31\") " pod="openshift-console/console-69c5f6d6cb-qc4nn"
Jul 14 22:58:26.908440 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:26.908348    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptght\" (UniqueName: \"kubernetes.io/projected/8f66f937-853a-438f-9bb2-c11f7fc46a31-kube-api-access-ptght\") pod \"console-69c5f6d6cb-qc4nn\" (UID: \"8f66f937-853a-438f-9bb2-c11f7fc46a31\") " pod="openshift-console/console-69c5f6d6cb-qc4nn"
Jul 14 22:58:26.908440 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:26.908374    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f66f937-853a-438f-9bb2-c11f7fc46a31-console-oauth-config\") pod \"console-69c5f6d6cb-qc4nn\" (UID: \"8f66f937-853a-438f-9bb2-c11f7fc46a31\") " pod="openshift-console/console-69c5f6d6cb-qc4nn"
Jul 14 22:58:26.908440 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:26.908411    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f66f937-853a-438f-9bb2-c11f7fc46a31-trusted-ca-bundle\") pod \"console-69c5f6d6cb-qc4nn\" (UID: \"8f66f937-853a-438f-9bb2-c11f7fc46a31\") " pod="openshift-console/console-69c5f6d6cb-qc4nn"
Jul 14 22:58:26.908675 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:26.908451    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f66f937-853a-438f-9bb2-c11f7fc46a31-console-serving-cert\") pod \"console-69c5f6d6cb-qc4nn\" (UID: \"8f66f937-853a-438f-9bb2-c11f7fc46a31\") " pod="openshift-console/console-69c5f6d6cb-qc4nn"
Jul 14 22:58:26.909097 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:26.909066    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f66f937-853a-438f-9bb2-c11f7fc46a31-service-ca\") pod \"console-69c5f6d6cb-qc4nn\" (UID: \"8f66f937-853a-438f-9bb2-c11f7fc46a31\") " pod="openshift-console/console-69c5f6d6cb-qc4nn"
Jul 14 22:58:26.909305 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:26.909205    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f66f937-853a-438f-9bb2-c11f7fc46a31-console-config\") pod \"console-69c5f6d6cb-qc4nn\" (UID: \"8f66f937-853a-438f-9bb2-c11f7fc46a31\") " pod="openshift-console/console-69c5f6d6cb-qc4nn"
Jul 14 22:58:26.909687 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:26.909662    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f66f937-853a-438f-9bb2-c11f7fc46a31-oauth-serving-cert\") pod \"console-69c5f6d6cb-qc4nn\" (UID: \"8f66f937-853a-438f-9bb2-c11f7fc46a31\") " pod="openshift-console/console-69c5f6d6cb-qc4nn"
Jul 14 22:58:26.909791 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:26.909748    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f66f937-853a-438f-9bb2-c11f7fc46a31-trusted-ca-bundle\") pod \"console-69c5f6d6cb-qc4nn\" (UID: \"8f66f937-853a-438f-9bb2-c11f7fc46a31\") " pod="openshift-console/console-69c5f6d6cb-qc4nn"
Jul 14 22:58:26.911816 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:26.911794    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f66f937-853a-438f-9bb2-c11f7fc46a31-console-serving-cert\") pod \"console-69c5f6d6cb-qc4nn\" (UID: \"8f66f937-853a-438f-9bb2-c11f7fc46a31\") " pod="openshift-console/console-69c5f6d6cb-qc4nn"
Jul 14 22:58:26.911916 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:26.911878    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f66f937-853a-438f-9bb2-c11f7fc46a31-console-oauth-config\") pod \"console-69c5f6d6cb-qc4nn\" (UID: \"8f66f937-853a-438f-9bb2-c11f7fc46a31\") " pod="openshift-console/console-69c5f6d6cb-qc4nn"
Jul 14 22:58:26.917266 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:26.917244    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptght\" (UniqueName: \"kubernetes.io/projected/8f66f937-853a-438f-9bb2-c11f7fc46a31-kube-api-access-ptght\") pod \"console-69c5f6d6cb-qc4nn\" (UID: \"8f66f937-853a-438f-9bb2-c11f7fc46a31\") " pod="openshift-console/console-69c5f6d6cb-qc4nn"
Jul 14 22:58:27.076676 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:27.076588    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69c5f6d6cb-qc4nn"
Jul 14 22:58:27.261750 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:27.261715    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69c5f6d6cb-qc4nn"]
Jul 14 22:58:27.266101 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:58:27.266067    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f66f937_853a_438f_9bb2_c11f7fc46a31.slice/crio-02ea4c9b33d34f893d2a95a45f172961203396139c59266610ab23ccc9c73c77 WatchSource:0}: Error finding container 02ea4c9b33d34f893d2a95a45f172961203396139c59266610ab23ccc9c73c77: Status 404 returned error can't find the container with id 02ea4c9b33d34f893d2a95a45f172961203396139c59266610ab23ccc9c73c77
Jul 14 22:58:27.773945 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:27.773905    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69c5f6d6cb-qc4nn" event={"ID":"8f66f937-853a-438f-9bb2-c11f7fc46a31","Type":"ContainerStarted","Data":"486ea1fcf5e94cbeb01f89db6f878162ba4d242139179ff327dcb8e414062117"}
Jul 14 22:58:27.774418 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:27.773971    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69c5f6d6cb-qc4nn" event={"ID":"8f66f937-853a-438f-9bb2-c11f7fc46a31","Type":"ContainerStarted","Data":"02ea4c9b33d34f893d2a95a45f172961203396139c59266610ab23ccc9c73c77"}
Jul 14 22:58:27.794459 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:27.794392    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-69c5f6d6cb-qc4nn" podStartSLOduration=1.794369909 podStartE2EDuration="1.794369909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:58:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:58:27.792559984 +0000 UTC m=+95.095550794" watchObservedRunningTime="2026-07-14 22:58:27.794369909 +0000 UTC m=+95.097360700"
Jul 14 22:58:37.076940 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:37.076903    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-69c5f6d6cb-qc4nn"
Jul 14 22:58:37.077463 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:37.076969    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-69c5f6d6cb-qc4nn"
Jul 14 22:58:37.081549 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:37.081529    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-69c5f6d6cb-qc4nn"
Jul 14 22:58:37.812756 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:37.812729    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-69c5f6d6cb-qc4nn"
Jul 14 22:58:37.875790 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:37.875754    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-586dd6b845-wzwbr"]
Jul 14 22:58:38.691162 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:58:38.691127    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pbtpg"
Jul 14 22:59:02.896927 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:02.896867    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-586dd6b845-wzwbr" podUID="9d97dcd3-d994-45f9-a7cf-15d1ead626a0" containerName="console" containerID="cri-o://1e2533fad042edc70acbfb60af3420aba3956f4c19ccaf0c68185c0c607a855b" gracePeriod=15
Jul 14 22:59:03.166196 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:03.166119    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-586dd6b845-wzwbr"
Jul 14 22:59:03.297336 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:03.297299    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d97dcd3-d994-45f9-a7cf-15d1ead626a0-console-config\") pod \"9d97dcd3-d994-45f9-a7cf-15d1ead626a0\" (UID: \"9d97dcd3-d994-45f9-a7cf-15d1ead626a0\") "
Jul 14 22:59:03.297539 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:03.297347    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d97dcd3-d994-45f9-a7cf-15d1ead626a0-oauth-serving-cert\") pod \"9d97dcd3-d994-45f9-a7cf-15d1ead626a0\" (UID: \"9d97dcd3-d994-45f9-a7cf-15d1ead626a0\") "
Jul 14 22:59:03.297539 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:03.297374    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brh7s\" (UniqueName: \"kubernetes.io/projected/9d97dcd3-d994-45f9-a7cf-15d1ead626a0-kube-api-access-brh7s\") pod \"9d97dcd3-d994-45f9-a7cf-15d1ead626a0\" (UID: \"9d97dcd3-d994-45f9-a7cf-15d1ead626a0\") "
Jul 14 22:59:03.297539 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:03.297392    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d97dcd3-d994-45f9-a7cf-15d1ead626a0-console-serving-cert\") pod \"9d97dcd3-d994-45f9-a7cf-15d1ead626a0\" (UID: \"9d97dcd3-d994-45f9-a7cf-15d1ead626a0\") "
Jul 14 22:59:03.297539 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:03.297416    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d97dcd3-d994-45f9-a7cf-15d1ead626a0-console-oauth-config\") pod \"9d97dcd3-d994-45f9-a7cf-15d1ead626a0\" (UID: \"9d97dcd3-d994-45f9-a7cf-15d1ead626a0\") "
Jul 14 22:59:03.297539 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:03.297449    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d97dcd3-d994-45f9-a7cf-15d1ead626a0-service-ca\") pod \"9d97dcd3-d994-45f9-a7cf-15d1ead626a0\" (UID: \"9d97dcd3-d994-45f9-a7cf-15d1ead626a0\") "
Jul 14 22:59:03.297791 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:03.297689    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d97dcd3-d994-45f9-a7cf-15d1ead626a0-console-config" (OuterVolumeSpecName: "console-config") pod "9d97dcd3-d994-45f9-a7cf-15d1ead626a0" (UID: "9d97dcd3-d994-45f9-a7cf-15d1ead626a0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 22:59:03.297848 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:03.297807    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d97dcd3-d994-45f9-a7cf-15d1ead626a0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9d97dcd3-d994-45f9-a7cf-15d1ead626a0" (UID: "9d97dcd3-d994-45f9-a7cf-15d1ead626a0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 22:59:03.297930 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:03.297904    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d97dcd3-d994-45f9-a7cf-15d1ead626a0-service-ca" (OuterVolumeSpecName: "service-ca") pod "9d97dcd3-d994-45f9-a7cf-15d1ead626a0" (UID: "9d97dcd3-d994-45f9-a7cf-15d1ead626a0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 22:59:03.299909 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:03.299885    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d97dcd3-d994-45f9-a7cf-15d1ead626a0-kube-api-access-brh7s" (OuterVolumeSpecName: "kube-api-access-brh7s") pod "9d97dcd3-d994-45f9-a7cf-15d1ead626a0" (UID: "9d97dcd3-d994-45f9-a7cf-15d1ead626a0"). InnerVolumeSpecName "kube-api-access-brh7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 22:59:03.299991 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:03.299907    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d97dcd3-d994-45f9-a7cf-15d1ead626a0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9d97dcd3-d994-45f9-a7cf-15d1ead626a0" (UID: "9d97dcd3-d994-45f9-a7cf-15d1ead626a0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 22:59:03.299991 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:03.299960    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d97dcd3-d994-45f9-a7cf-15d1ead626a0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9d97dcd3-d994-45f9-a7cf-15d1ead626a0" (UID: "9d97dcd3-d994-45f9-a7cf-15d1ead626a0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 22:59:03.398371 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:03.398329    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d97dcd3-d994-45f9-a7cf-15d1ead626a0-console-config\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 22:59:03.398371 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:03.398362    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d97dcd3-d994-45f9-a7cf-15d1ead626a0-oauth-serving-cert\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 22:59:03.398371 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:03.398373    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brh7s\" (UniqueName: \"kubernetes.io/projected/9d97dcd3-d994-45f9-a7cf-15d1ead626a0-kube-api-access-brh7s\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 22:59:03.398371 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:03.398382    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d97dcd3-d994-45f9-a7cf-15d1ead626a0-console-serving-cert\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 22:59:03.398662 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:03.398391    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d97dcd3-d994-45f9-a7cf-15d1ead626a0-console-oauth-config\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 22:59:03.398662 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:03.398399    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d97dcd3-d994-45f9-a7cf-15d1ead626a0-service-ca\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 22:59:03.895923 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:03.895887    2563 generic.go:356] "Generic (PLEG): container finished" podID="9d97dcd3-d994-45f9-a7cf-15d1ead626a0" containerID="1e2533fad042edc70acbfb60af3420aba3956f4c19ccaf0c68185c0c607a855b" exitCode=0
Jul 14 22:59:03.896096 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:03.895935    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-586dd6b845-wzwbr"
Jul 14 22:59:03.896096 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:03.895944    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-586dd6b845-wzwbr" event={"ID":"9d97dcd3-d994-45f9-a7cf-15d1ead626a0","Type":"ContainerDied","Data":"1e2533fad042edc70acbfb60af3420aba3956f4c19ccaf0c68185c0c607a855b"}
Jul 14 22:59:03.896096 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:03.895985    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-586dd6b845-wzwbr" event={"ID":"9d97dcd3-d994-45f9-a7cf-15d1ead626a0","Type":"ContainerDied","Data":"9a8380204d058f0b52a7284ffd4fbfc7f844f0926458f445d023bb2d4c220121"}
Jul 14 22:59:03.896096 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:03.896004    2563 scope.go:117] "RemoveContainer" containerID="1e2533fad042edc70acbfb60af3420aba3956f4c19ccaf0c68185c0c607a855b"
Jul 14 22:59:03.904235 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:03.904183    2563 scope.go:117] "RemoveContainer" containerID="1e2533fad042edc70acbfb60af3420aba3956f4c19ccaf0c68185c0c607a855b"
Jul 14 22:59:03.904525 ip-10-0-134-8 kubenswrapper[2563]: E0714 22:59:03.904504    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e2533fad042edc70acbfb60af3420aba3956f4c19ccaf0c68185c0c607a855b\": container with ID starting with 1e2533fad042edc70acbfb60af3420aba3956f4c19ccaf0c68185c0c607a855b not found: ID does not exist" containerID="1e2533fad042edc70acbfb60af3420aba3956f4c19ccaf0c68185c0c607a855b"
Jul 14 22:59:03.904574 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:03.904532    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e2533fad042edc70acbfb60af3420aba3956f4c19ccaf0c68185c0c607a855b"} err="failed to get container status \"1e2533fad042edc70acbfb60af3420aba3956f4c19ccaf0c68185c0c607a855b\": rpc error: code = NotFound desc = could not find container \"1e2533fad042edc70acbfb60af3420aba3956f4c19ccaf0c68185c0c607a855b\": container with ID starting with 1e2533fad042edc70acbfb60af3420aba3956f4c19ccaf0c68185c0c607a855b not found: ID does not exist"
Jul 14 22:59:03.921121 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:03.921099    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-586dd6b845-wzwbr"]
Jul 14 22:59:03.925419 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:03.925398    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-586dd6b845-wzwbr"]
Jul 14 22:59:05.227652 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:05.227614    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d97dcd3-d994-45f9-a7cf-15d1ead626a0" path="/var/lib/kubelet/pods/9d97dcd3-d994-45f9-a7cf-15d1ead626a0/volumes"
Jul 14 22:59:50.552326 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:50.552222    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6678556dd-mfvps"]
Jul 14 22:59:50.552788 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:50.552551    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d97dcd3-d994-45f9-a7cf-15d1ead626a0" containerName="console"
Jul 14 22:59:50.552788 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:50.552571    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d97dcd3-d994-45f9-a7cf-15d1ead626a0" containerName="console"
Jul 14 22:59:50.552788 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:50.552635    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d97dcd3-d994-45f9-a7cf-15d1ead626a0" containerName="console"
Jul 14 22:59:50.554778 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:50.554759    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6678556dd-mfvps"
Jul 14 22:59:50.566931 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:50.566908    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6678556dd-mfvps"]
Jul 14 22:59:50.624398 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:50.624355    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncklh\" (UniqueName: \"kubernetes.io/projected/2704e48c-843e-4f28-88ce-1af16a0f8a5e-kube-api-access-ncklh\") pod \"console-6678556dd-mfvps\" (UID: \"2704e48c-843e-4f28-88ce-1af16a0f8a5e\") " pod="openshift-console/console-6678556dd-mfvps"
Jul 14 22:59:50.624398 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:50.624399    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2704e48c-843e-4f28-88ce-1af16a0f8a5e-service-ca\") pod \"console-6678556dd-mfvps\" (UID: \"2704e48c-843e-4f28-88ce-1af16a0f8a5e\") " pod="openshift-console/console-6678556dd-mfvps"
Jul 14 22:59:50.624605 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:50.624470    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2704e48c-843e-4f28-88ce-1af16a0f8a5e-console-serving-cert\") pod \"console-6678556dd-mfvps\" (UID: \"2704e48c-843e-4f28-88ce-1af16a0f8a5e\") " pod="openshift-console/console-6678556dd-mfvps"
Jul 14 22:59:50.624605 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:50.624488    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2704e48c-843e-4f28-88ce-1af16a0f8a5e-oauth-serving-cert\") pod \"console-6678556dd-mfvps\" (UID: \"2704e48c-843e-4f28-88ce-1af16a0f8a5e\") " pod="openshift-console/console-6678556dd-mfvps"
Jul 14 22:59:50.624605 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:50.624539    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2704e48c-843e-4f28-88ce-1af16a0f8a5e-console-oauth-config\") pod \"console-6678556dd-mfvps\" (UID: \"2704e48c-843e-4f28-88ce-1af16a0f8a5e\") " pod="openshift-console/console-6678556dd-mfvps"
Jul 14 22:59:50.624605 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:50.624574    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2704e48c-843e-4f28-88ce-1af16a0f8a5e-console-config\") pod \"console-6678556dd-mfvps\" (UID: \"2704e48c-843e-4f28-88ce-1af16a0f8a5e\") " pod="openshift-console/console-6678556dd-mfvps"
Jul 14 22:59:50.624724 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:50.624601    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2704e48c-843e-4f28-88ce-1af16a0f8a5e-trusted-ca-bundle\") pod \"console-6678556dd-mfvps\" (UID: \"2704e48c-843e-4f28-88ce-1af16a0f8a5e\") " pod="openshift-console/console-6678556dd-mfvps"
Jul 14 22:59:50.725203 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:50.725160    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2704e48c-843e-4f28-88ce-1af16a0f8a5e-console-config\") pod \"console-6678556dd-mfvps\" (UID: \"2704e48c-843e-4f28-88ce-1af16a0f8a5e\") " pod="openshift-console/console-6678556dd-mfvps"
Jul 14 22:59:50.725203 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:50.725203    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2704e48c-843e-4f28-88ce-1af16a0f8a5e-trusted-ca-bundle\") pod \"console-6678556dd-mfvps\" (UID: \"2704e48c-843e-4f28-88ce-1af16a0f8a5e\") " pod="openshift-console/console-6678556dd-mfvps"
Jul 14 22:59:50.725487 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:50.725241    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncklh\" (UniqueName: \"kubernetes.io/projected/2704e48c-843e-4f28-88ce-1af16a0f8a5e-kube-api-access-ncklh\") pod \"console-6678556dd-mfvps\" (UID: \"2704e48c-843e-4f28-88ce-1af16a0f8a5e\") " pod="openshift-console/console-6678556dd-mfvps"
Jul 14 22:59:50.725487 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:50.725266    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2704e48c-843e-4f28-88ce-1af16a0f8a5e-service-ca\") pod \"console-6678556dd-mfvps\" (UID: \"2704e48c-843e-4f28-88ce-1af16a0f8a5e\") " pod="openshift-console/console-6678556dd-mfvps"
Jul 14 22:59:50.725487 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:50.725335    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2704e48c-843e-4f28-88ce-1af16a0f8a5e-console-serving-cert\") pod \"console-6678556dd-mfvps\" (UID: \"2704e48c-843e-4f28-88ce-1af16a0f8a5e\") " pod="openshift-console/console-6678556dd-mfvps"
Jul 14 22:59:50.725640 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:50.725478    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2704e48c-843e-4f28-88ce-1af16a0f8a5e-oauth-serving-cert\") pod \"console-6678556dd-mfvps\" (UID: \"2704e48c-843e-4f28-88ce-1af16a0f8a5e\") " pod="openshift-console/console-6678556dd-mfvps"
Jul 14 22:59:50.725640 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:50.725536    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2704e48c-843e-4f28-88ce-1af16a0f8a5e-console-oauth-config\") pod \"console-6678556dd-mfvps\" (UID: \"2704e48c-843e-4f28-88ce-1af16a0f8a5e\") " pod="openshift-console/console-6678556dd-mfvps"
Jul 14 22:59:50.726050 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:50.725991    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2704e48c-843e-4f28-88ce-1af16a0f8a5e-console-config\") pod \"console-6678556dd-mfvps\" (UID: \"2704e48c-843e-4f28-88ce-1af16a0f8a5e\") " pod="openshift-console/console-6678556dd-mfvps"
Jul 14 22:59:50.726050 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:50.726073    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2704e48c-843e-4f28-88ce-1af16a0f8a5e-service-ca\") pod \"console-6678556dd-mfvps\" (UID: \"2704e48c-843e-4f28-88ce-1af16a0f8a5e\") " pod="openshift-console/console-6678556dd-mfvps"
Jul 14 22:59:50.726355 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:50.726124    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2704e48c-843e-4f28-88ce-1af16a0f8a5e-oauth-serving-cert\") pod \"console-6678556dd-mfvps\" (UID: \"2704e48c-843e-4f28-88ce-1af16a0f8a5e\") " pod="openshift-console/console-6678556dd-mfvps"
Jul 14 22:59:50.726355 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:50.726240    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2704e48c-843e-4f28-88ce-1af16a0f8a5e-trusted-ca-bundle\") pod \"console-6678556dd-mfvps\" (UID: \"2704e48c-843e-4f28-88ce-1af16a0f8a5e\") " pod="openshift-console/console-6678556dd-mfvps"
Jul 14 22:59:50.727868 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:50.727846    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2704e48c-843e-4f28-88ce-1af16a0f8a5e-console-oauth-config\") pod \"console-6678556dd-mfvps\" (UID: \"2704e48c-843e-4f28-88ce-1af16a0f8a5e\") " pod="openshift-console/console-6678556dd-mfvps"
Jul 14 22:59:50.728005 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:50.727987    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2704e48c-843e-4f28-88ce-1af16a0f8a5e-console-serving-cert\") pod \"console-6678556dd-mfvps\" (UID: \"2704e48c-843e-4f28-88ce-1af16a0f8a5e\") " pod="openshift-console/console-6678556dd-mfvps"
Jul 14 22:59:50.734322 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:50.734299    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncklh\" (UniqueName: \"kubernetes.io/projected/2704e48c-843e-4f28-88ce-1af16a0f8a5e-kube-api-access-ncklh\") pod \"console-6678556dd-mfvps\" (UID: \"2704e48c-843e-4f28-88ce-1af16a0f8a5e\") " pod="openshift-console/console-6678556dd-mfvps"
Jul 14 22:59:50.863494 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:50.863391    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6678556dd-mfvps"
Jul 14 22:59:50.988022 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:50.987991    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6678556dd-mfvps"]
Jul 14 22:59:50.991128 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:59:50.991102    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2704e48c_843e_4f28_88ce_1af16a0f8a5e.slice/crio-500c4651745a25ad7b8c2e771aacdd8dafd4bd42815c8bdcd521ad1a7fd1ffb0 WatchSource:0}: Error finding container 500c4651745a25ad7b8c2e771aacdd8dafd4bd42815c8bdcd521ad1a7fd1ffb0: Status 404 returned error can't find the container with id 500c4651745a25ad7b8c2e771aacdd8dafd4bd42815c8bdcd521ad1a7fd1ffb0
Jul 14 22:59:51.061249 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:51.061218    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6678556dd-mfvps" event={"ID":"2704e48c-843e-4f28-88ce-1af16a0f8a5e","Type":"ContainerStarted","Data":"89914327996b7fb9081cd938ca5e65160fb0d19d3b2e0f2c7c7340f2f64f8f77"}
Jul 14 22:59:51.061424 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:51.061315    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6678556dd-mfvps" event={"ID":"2704e48c-843e-4f28-88ce-1af16a0f8a5e","Type":"ContainerStarted","Data":"500c4651745a25ad7b8c2e771aacdd8dafd4bd42815c8bdcd521ad1a7fd1ffb0"}
Jul 14 22:59:51.080749 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:51.080703    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6678556dd-mfvps" podStartSLOduration=1.080686743 podStartE2EDuration="1.080686743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:59:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:59:51.078932034 +0000 UTC m=+178.381922836" watchObservedRunningTime="2026-07-14 22:59:51.080686743 +0000 UTC m=+178.383677602"
Jul 14 22:59:56.372188 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:56.372152    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6678556dd-mfvps"]
Jul 14 22:59:56.402099 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:56.402069    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-c59575677-kj6sb"]
Jul 14 22:59:56.405348 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:56.405332    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c59575677-kj6sb"
Jul 14 22:59:56.415333 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:56.415312    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c59575677-kj6sb"]
Jul 14 22:59:56.466505 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:56.466481    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4512e88b-99f9-440f-b7d4-78a8f5dfed1a-service-ca\") pod \"console-c59575677-kj6sb\" (UID: \"4512e88b-99f9-440f-b7d4-78a8f5dfed1a\") " pod="openshift-console/console-c59575677-kj6sb"
Jul 14 22:59:56.466608 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:56.466511    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4512e88b-99f9-440f-b7d4-78a8f5dfed1a-console-config\") pod \"console-c59575677-kj6sb\" (UID: \"4512e88b-99f9-440f-b7d4-78a8f5dfed1a\") " pod="openshift-console/console-c59575677-kj6sb"
Jul 14 22:59:56.466608 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:56.466534    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4512e88b-99f9-440f-b7d4-78a8f5dfed1a-trusted-ca-bundle\") pod \"console-c59575677-kj6sb\" (UID: \"4512e88b-99f9-440f-b7d4-78a8f5dfed1a\") " pod="openshift-console/console-c59575677-kj6sb"
Jul 14 22:59:56.466681 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:56.466605    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4512e88b-99f9-440f-b7d4-78a8f5dfed1a-console-oauth-config\") pod \"console-c59575677-kj6sb\" (UID: \"4512e88b-99f9-440f-b7d4-78a8f5dfed1a\") " pod="openshift-console/console-c59575677-kj6sb"
Jul 14 22:59:56.466681 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:56.466667    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4512e88b-99f9-440f-b7d4-78a8f5dfed1a-console-serving-cert\") pod \"console-c59575677-kj6sb\" (UID: \"4512e88b-99f9-440f-b7d4-78a8f5dfed1a\") " pod="openshift-console/console-c59575677-kj6sb"
Jul 14 22:59:56.466744 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:56.466707    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sw4q\" (UniqueName: \"kubernetes.io/projected/4512e88b-99f9-440f-b7d4-78a8f5dfed1a-kube-api-access-8sw4q\") pod \"console-c59575677-kj6sb\" (UID: \"4512e88b-99f9-440f-b7d4-78a8f5dfed1a\") " pod="openshift-console/console-c59575677-kj6sb"
Jul 14 22:59:56.466744 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:56.466734    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4512e88b-99f9-440f-b7d4-78a8f5dfed1a-oauth-serving-cert\") pod \"console-c59575677-kj6sb\" (UID: \"4512e88b-99f9-440f-b7d4-78a8f5dfed1a\") " pod="openshift-console/console-c59575677-kj6sb"
Jul 14 22:59:56.567885 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:56.567852    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4512e88b-99f9-440f-b7d4-78a8f5dfed1a-oauth-serving-cert\") pod \"console-c59575677-kj6sb\" (UID: \"4512e88b-99f9-440f-b7d4-78a8f5dfed1a\") " pod="openshift-console/console-c59575677-kj6sb"
Jul 14 22:59:56.567998 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:56.567892    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4512e88b-99f9-440f-b7d4-78a8f5dfed1a-service-ca\") pod \"console-c59575677-kj6sb\" (UID: \"4512e88b-99f9-440f-b7d4-78a8f5dfed1a\") " pod="openshift-console/console-c59575677-kj6sb"
Jul 14 22:59:56.567998 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:56.567970    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4512e88b-99f9-440f-b7d4-78a8f5dfed1a-console-config\") pod \"console-c59575677-kj6sb\" (UID: \"4512e88b-99f9-440f-b7d4-78a8f5dfed1a\") " pod="openshift-console/console-c59575677-kj6sb"
Jul 14 22:59:56.568078 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:56.568007    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4512e88b-99f9-440f-b7d4-78a8f5dfed1a-trusted-ca-bundle\") pod \"console-c59575677-kj6sb\" (UID: \"4512e88b-99f9-440f-b7d4-78a8f5dfed1a\") " pod="openshift-console/console-c59575677-kj6sb"
Jul 14 22:59:56.568078 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:56.568040    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4512e88b-99f9-440f-b7d4-78a8f5dfed1a-console-oauth-config\") pod \"console-c59575677-kj6sb\" (UID: \"4512e88b-99f9-440f-b7d4-78a8f5dfed1a\") " pod="openshift-console/console-c59575677-kj6sb"
Jul 14 22:59:56.568137 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:56.568074    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4512e88b-99f9-440f-b7d4-78a8f5dfed1a-console-serving-cert\") pod \"console-c59575677-kj6sb\" (UID: \"4512e88b-99f9-440f-b7d4-78a8f5dfed1a\") " pod="openshift-console/console-c59575677-kj6sb"
Jul 14 22:59:56.568182 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:56.568136    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sw4q\" (UniqueName: \"kubernetes.io/projected/4512e88b-99f9-440f-b7d4-78a8f5dfed1a-kube-api-access-8sw4q\") pod \"console-c59575677-kj6sb\" (UID: \"4512e88b-99f9-440f-b7d4-78a8f5dfed1a\") " pod="openshift-console/console-c59575677-kj6sb"
Jul 14 22:59:56.568684 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:56.568663    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4512e88b-99f9-440f-b7d4-78a8f5dfed1a-service-ca\") pod \"console-c59575677-kj6sb\" (UID: \"4512e88b-99f9-440f-b7d4-78a8f5dfed1a\") " pod="openshift-console/console-c59575677-kj6sb"
Jul 14 22:59:56.568888 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:56.568663    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4512e88b-99f9-440f-b7d4-78a8f5dfed1a-oauth-serving-cert\") pod \"console-c59575677-kj6sb\" (UID: \"4512e88b-99f9-440f-b7d4-78a8f5dfed1a\") " pod="openshift-console/console-c59575677-kj6sb"
Jul 14 22:59:56.568888 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:56.568689    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4512e88b-99f9-440f-b7d4-78a8f5dfed1a-console-config\") pod \"console-c59575677-kj6sb\" (UID: \"4512e88b-99f9-440f-b7d4-78a8f5dfed1a\") " pod="openshift-console/console-c59575677-kj6sb"
Jul 14 22:59:56.569101 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:56.569080    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4512e88b-99f9-440f-b7d4-78a8f5dfed1a-trusted-ca-bundle\") pod \"console-c59575677-kj6sb\" (UID: \"4512e88b-99f9-440f-b7d4-78a8f5dfed1a\") " pod="openshift-console/console-c59575677-kj6sb"
Jul 14 22:59:56.570692 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:56.570671    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4512e88b-99f9-440f-b7d4-78a8f5dfed1a-console-oauth-config\") pod \"console-c59575677-kj6sb\" (UID: \"4512e88b-99f9-440f-b7d4-78a8f5dfed1a\") " pod="openshift-console/console-c59575677-kj6sb"
Jul 14 22:59:56.570812 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:56.570733    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4512e88b-99f9-440f-b7d4-78a8f5dfed1a-console-serving-cert\") pod \"console-c59575677-kj6sb\" (UID: \"4512e88b-99f9-440f-b7d4-78a8f5dfed1a\") " pod="openshift-console/console-c59575677-kj6sb"
Jul 14 22:59:56.577250 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:56.577221    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sw4q\" (UniqueName: \"kubernetes.io/projected/4512e88b-99f9-440f-b7d4-78a8f5dfed1a-kube-api-access-8sw4q\") pod \"console-c59575677-kj6sb\" (UID: \"4512e88b-99f9-440f-b7d4-78a8f5dfed1a\") " pod="openshift-console/console-c59575677-kj6sb"
Jul 14 22:59:56.713262 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:56.713188    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c59575677-kj6sb"
Jul 14 22:59:56.857131 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:56.857107    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c59575677-kj6sb"]
Jul 14 22:59:56.859625 ip-10-0-134-8 kubenswrapper[2563]: W0714 22:59:56.859599    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4512e88b_99f9_440f_b7d4_78a8f5dfed1a.slice/crio-931fcf7c05284e67e065747cf9161b0fe9555246adac1341eb0f54a61c19c11a WatchSource:0}: Error finding container 931fcf7c05284e67e065747cf9161b0fe9555246adac1341eb0f54a61c19c11a: Status 404 returned error can't find the container with id 931fcf7c05284e67e065747cf9161b0fe9555246adac1341eb0f54a61c19c11a
Jul 14 22:59:57.084660 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:57.084624    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c59575677-kj6sb" event={"ID":"4512e88b-99f9-440f-b7d4-78a8f5dfed1a","Type":"ContainerStarted","Data":"6b61a72733a9d6cf16c13b95e85375b634b981e0dbc1667d9f4fff5bf21e29a4"}
Jul 14 22:59:57.084864 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:57.084684    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c59575677-kj6sb" event={"ID":"4512e88b-99f9-440f-b7d4-78a8f5dfed1a","Type":"ContainerStarted","Data":"931fcf7c05284e67e065747cf9161b0fe9555246adac1341eb0f54a61c19c11a"}
Jul 14 22:59:57.104215 ip-10-0-134-8 kubenswrapper[2563]: I0714 22:59:57.104167    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-c59575677-kj6sb" podStartSLOduration=1.104152903 podStartE2EDuration="1.104152903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 22:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 22:59:57.103726154 +0000 UTC m=+184.406716946" watchObservedRunningTime="2026-07-14 22:59:57.104152903 +0000 UTC m=+184.407143753"
Jul 14 23:00:00.864260 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:00.864230    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6678556dd-mfvps"
Jul 14 23:00:06.714194 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:06.714158    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c59575677-kj6sb"
Jul 14 23:00:06.714714 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:06.714231    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c59575677-kj6sb"
Jul 14 23:00:06.718884 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:06.718861    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c59575677-kj6sb"
Jul 14 23:00:07.121923 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:07.121894    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c59575677-kj6sb"
Jul 14 23:00:07.171841 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:07.171796    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69c5f6d6cb-qc4nn"]
Jul 14 23:00:21.391508 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:21.391447    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6678556dd-mfvps" podUID="2704e48c-843e-4f28-88ce-1af16a0f8a5e" containerName="console" containerID="cri-o://89914327996b7fb9081cd938ca5e65160fb0d19d3b2e0f2c7c7340f2f64f8f77" gracePeriod=15
Jul 14 23:00:21.626004 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:21.625967    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6678556dd-mfvps"
Jul 14 23:00:21.751465 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:21.751440    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2704e48c-843e-4f28-88ce-1af16a0f8a5e-service-ca\") pod \"2704e48c-843e-4f28-88ce-1af16a0f8a5e\" (UID: \"2704e48c-843e-4f28-88ce-1af16a0f8a5e\") "
Jul 14 23:00:21.751683 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:21.751478    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ncklh\" (UniqueName: \"kubernetes.io/projected/2704e48c-843e-4f28-88ce-1af16a0f8a5e-kube-api-access-ncklh\") pod \"2704e48c-843e-4f28-88ce-1af16a0f8a5e\" (UID: \"2704e48c-843e-4f28-88ce-1af16a0f8a5e\") "
Jul 14 23:00:21.751683 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:21.751498    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2704e48c-843e-4f28-88ce-1af16a0f8a5e-oauth-serving-cert\") pod \"2704e48c-843e-4f28-88ce-1af16a0f8a5e\" (UID: \"2704e48c-843e-4f28-88ce-1af16a0f8a5e\") "
Jul 14 23:00:21.751683 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:21.751514    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2704e48c-843e-4f28-88ce-1af16a0f8a5e-console-serving-cert\") pod \"2704e48c-843e-4f28-88ce-1af16a0f8a5e\" (UID: \"2704e48c-843e-4f28-88ce-1af16a0f8a5e\") "
Jul 14 23:00:21.751683 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:21.751550    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2704e48c-843e-4f28-88ce-1af16a0f8a5e-console-oauth-config\") pod \"2704e48c-843e-4f28-88ce-1af16a0f8a5e\" (UID: \"2704e48c-843e-4f28-88ce-1af16a0f8a5e\") "
Jul 14 23:00:21.751683 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:21.751599    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2704e48c-843e-4f28-88ce-1af16a0f8a5e-console-config\") pod \"2704e48c-843e-4f28-88ce-1af16a0f8a5e\" (UID: \"2704e48c-843e-4f28-88ce-1af16a0f8a5e\") "
Jul 14 23:00:21.751683 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:21.751653    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2704e48c-843e-4f28-88ce-1af16a0f8a5e-trusted-ca-bundle\") pod \"2704e48c-843e-4f28-88ce-1af16a0f8a5e\" (UID: \"2704e48c-843e-4f28-88ce-1af16a0f8a5e\") "
Jul 14 23:00:21.751963 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:21.751925    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2704e48c-843e-4f28-88ce-1af16a0f8a5e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2704e48c-843e-4f28-88ce-1af16a0f8a5e" (UID: "2704e48c-843e-4f28-88ce-1af16a0f8a5e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 23:00:21.752013 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:21.751941    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2704e48c-843e-4f28-88ce-1af16a0f8a5e-service-ca" (OuterVolumeSpecName: "service-ca") pod "2704e48c-843e-4f28-88ce-1af16a0f8a5e" (UID: "2704e48c-843e-4f28-88ce-1af16a0f8a5e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 23:00:21.752067 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:21.752015    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2704e48c-843e-4f28-88ce-1af16a0f8a5e-console-config" (OuterVolumeSpecName: "console-config") pod "2704e48c-843e-4f28-88ce-1af16a0f8a5e" (UID: "2704e48c-843e-4f28-88ce-1af16a0f8a5e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 23:00:21.752142 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:21.752126    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2704e48c-843e-4f28-88ce-1af16a0f8a5e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2704e48c-843e-4f28-88ce-1af16a0f8a5e" (UID: "2704e48c-843e-4f28-88ce-1af16a0f8a5e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 23:00:21.753884 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:21.753861    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2704e48c-843e-4f28-88ce-1af16a0f8a5e-kube-api-access-ncklh" (OuterVolumeSpecName: "kube-api-access-ncklh") pod "2704e48c-843e-4f28-88ce-1af16a0f8a5e" (UID: "2704e48c-843e-4f28-88ce-1af16a0f8a5e"). InnerVolumeSpecName "kube-api-access-ncklh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:00:21.754318 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:21.754275    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2704e48c-843e-4f28-88ce-1af16a0f8a5e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2704e48c-843e-4f28-88ce-1af16a0f8a5e" (UID: "2704e48c-843e-4f28-88ce-1af16a0f8a5e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 23:00:21.754384 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:21.754310    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2704e48c-843e-4f28-88ce-1af16a0f8a5e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2704e48c-843e-4f28-88ce-1af16a0f8a5e" (UID: "2704e48c-843e-4f28-88ce-1af16a0f8a5e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 23:00:21.852753 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:21.852703    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2704e48c-843e-4f28-88ce-1af16a0f8a5e-console-config\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:00:21.852753 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:21.852745    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2704e48c-843e-4f28-88ce-1af16a0f8a5e-trusted-ca-bundle\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:00:21.852753 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:21.852757    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2704e48c-843e-4f28-88ce-1af16a0f8a5e-service-ca\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:00:21.852962 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:21.852770    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncklh\" (UniqueName: \"kubernetes.io/projected/2704e48c-843e-4f28-88ce-1af16a0f8a5e-kube-api-access-ncklh\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:00:21.852962 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:21.852783    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2704e48c-843e-4f28-88ce-1af16a0f8a5e-oauth-serving-cert\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:00:21.852962 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:21.852794    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2704e48c-843e-4f28-88ce-1af16a0f8a5e-console-serving-cert\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:00:21.852962 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:21.852805    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2704e48c-843e-4f28-88ce-1af16a0f8a5e-console-oauth-config\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:00:22.167522 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:22.167433    2563 generic.go:356] "Generic (PLEG): container finished" podID="2704e48c-843e-4f28-88ce-1af16a0f8a5e" containerID="89914327996b7fb9081cd938ca5e65160fb0d19d3b2e0f2c7c7340f2f64f8f77" exitCode=0
Jul 14 23:00:22.167522 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:22.167480    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6678556dd-mfvps"
Jul 14 23:00:22.167742 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:22.167523    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6678556dd-mfvps" event={"ID":"2704e48c-843e-4f28-88ce-1af16a0f8a5e","Type":"ContainerDied","Data":"89914327996b7fb9081cd938ca5e65160fb0d19d3b2e0f2c7c7340f2f64f8f77"}
Jul 14 23:00:22.167742 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:22.167617    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6678556dd-mfvps" event={"ID":"2704e48c-843e-4f28-88ce-1af16a0f8a5e","Type":"ContainerDied","Data":"500c4651745a25ad7b8c2e771aacdd8dafd4bd42815c8bdcd521ad1a7fd1ffb0"}
Jul 14 23:00:22.167742 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:22.167646    2563 scope.go:117] "RemoveContainer" containerID="89914327996b7fb9081cd938ca5e65160fb0d19d3b2e0f2c7c7340f2f64f8f77"
Jul 14 23:00:22.175584 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:22.175565    2563 scope.go:117] "RemoveContainer" containerID="89914327996b7fb9081cd938ca5e65160fb0d19d3b2e0f2c7c7340f2f64f8f77"
Jul 14 23:00:22.175851 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:00:22.175834    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89914327996b7fb9081cd938ca5e65160fb0d19d3b2e0f2c7c7340f2f64f8f77\": container with ID starting with 89914327996b7fb9081cd938ca5e65160fb0d19d3b2e0f2c7c7340f2f64f8f77 not found: ID does not exist" containerID="89914327996b7fb9081cd938ca5e65160fb0d19d3b2e0f2c7c7340f2f64f8f77"
Jul 14 23:00:22.175928 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:22.175867    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89914327996b7fb9081cd938ca5e65160fb0d19d3b2e0f2c7c7340f2f64f8f77"} err="failed to get container status \"89914327996b7fb9081cd938ca5e65160fb0d19d3b2e0f2c7c7340f2f64f8f77\": rpc error: code = NotFound desc = could not find container \"89914327996b7fb9081cd938ca5e65160fb0d19d3b2e0f2c7c7340f2f64f8f77\": container with ID starting with 89914327996b7fb9081cd938ca5e65160fb0d19d3b2e0f2c7c7340f2f64f8f77 not found: ID does not exist"
Jul 14 23:00:22.188476 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:22.188451    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6678556dd-mfvps"]
Jul 14 23:00:22.194331 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:22.194308    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6678556dd-mfvps"]
Jul 14 23:00:23.228640 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:23.228601    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2704e48c-843e-4f28-88ce-1af16a0f8a5e" path="/var/lib/kubelet/pods/2704e48c-843e-4f28-88ce-1af16a0f8a5e/volumes"
Jul 14 23:00:32.191880 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:32.191817    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-69c5f6d6cb-qc4nn" podUID="8f66f937-853a-438f-9bb2-c11f7fc46a31" containerName="console" containerID="cri-o://486ea1fcf5e94cbeb01f89db6f878162ba4d242139179ff327dcb8e414062117" gracePeriod=15
Jul 14 23:00:32.428331 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:32.428300    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69c5f6d6cb-qc4nn"
Jul 14 23:00:32.532950 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:32.532913    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptght\" (UniqueName: \"kubernetes.io/projected/8f66f937-853a-438f-9bb2-c11f7fc46a31-kube-api-access-ptght\") pod \"8f66f937-853a-438f-9bb2-c11f7fc46a31\" (UID: \"8f66f937-853a-438f-9bb2-c11f7fc46a31\") "
Jul 14 23:00:32.533138 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:32.532966    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f66f937-853a-438f-9bb2-c11f7fc46a31-service-ca\") pod \"8f66f937-853a-438f-9bb2-c11f7fc46a31\" (UID: \"8f66f937-853a-438f-9bb2-c11f7fc46a31\") "
Jul 14 23:00:32.533138 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:32.532992    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f66f937-853a-438f-9bb2-c11f7fc46a31-console-config\") pod \"8f66f937-853a-438f-9bb2-c11f7fc46a31\" (UID: \"8f66f937-853a-438f-9bb2-c11f7fc46a31\") "
Jul 14 23:00:32.533138 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:32.533017    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f66f937-853a-438f-9bb2-c11f7fc46a31-trusted-ca-bundle\") pod \"8f66f937-853a-438f-9bb2-c11f7fc46a31\" (UID: \"8f66f937-853a-438f-9bb2-c11f7fc46a31\") "
Jul 14 23:00:32.533138 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:32.533038    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f66f937-853a-438f-9bb2-c11f7fc46a31-console-serving-cert\") pod \"8f66f937-853a-438f-9bb2-c11f7fc46a31\" (UID: \"8f66f937-853a-438f-9bb2-c11f7fc46a31\") "
Jul 14 23:00:32.533138 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:32.533072    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f66f937-853a-438f-9bb2-c11f7fc46a31-console-oauth-config\") pod \"8f66f937-853a-438f-9bb2-c11f7fc46a31\" (UID: \"8f66f937-853a-438f-9bb2-c11f7fc46a31\") "
Jul 14 23:00:32.533402 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:32.533155    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f66f937-853a-438f-9bb2-c11f7fc46a31-oauth-serving-cert\") pod \"8f66f937-853a-438f-9bb2-c11f7fc46a31\" (UID: \"8f66f937-853a-438f-9bb2-c11f7fc46a31\") "
Jul 14 23:00:32.533481 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:32.533400    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f66f937-853a-438f-9bb2-c11f7fc46a31-service-ca" (OuterVolumeSpecName: "service-ca") pod "8f66f937-853a-438f-9bb2-c11f7fc46a31" (UID: "8f66f937-853a-438f-9bb2-c11f7fc46a31"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 23:00:32.533539 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:32.533506    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f66f937-853a-438f-9bb2-c11f7fc46a31-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8f66f937-853a-438f-9bb2-c11f7fc46a31" (UID: "8f66f937-853a-438f-9bb2-c11f7fc46a31"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 23:00:32.533648 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:32.533613    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f66f937-853a-438f-9bb2-c11f7fc46a31-console-config" (OuterVolumeSpecName: "console-config") pod "8f66f937-853a-438f-9bb2-c11f7fc46a31" (UID: "8f66f937-853a-438f-9bb2-c11f7fc46a31"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 23:00:32.533766 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:32.533654    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f66f937-853a-438f-9bb2-c11f7fc46a31-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8f66f937-853a-438f-9bb2-c11f7fc46a31" (UID: "8f66f937-853a-438f-9bb2-c11f7fc46a31"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 23:00:32.535299 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:32.535253    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f66f937-853a-438f-9bb2-c11f7fc46a31-kube-api-access-ptght" (OuterVolumeSpecName: "kube-api-access-ptght") pod "8f66f937-853a-438f-9bb2-c11f7fc46a31" (UID: "8f66f937-853a-438f-9bb2-c11f7fc46a31"). InnerVolumeSpecName "kube-api-access-ptght". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:00:32.535396 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:32.535321    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f66f937-853a-438f-9bb2-c11f7fc46a31-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8f66f937-853a-438f-9bb2-c11f7fc46a31" (UID: "8f66f937-853a-438f-9bb2-c11f7fc46a31"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 23:00:32.535396 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:32.535340    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f66f937-853a-438f-9bb2-c11f7fc46a31-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8f66f937-853a-438f-9bb2-c11f7fc46a31" (UID: "8f66f937-853a-438f-9bb2-c11f7fc46a31"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 23:00:32.634052 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:32.634025    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f66f937-853a-438f-9bb2-c11f7fc46a31-trusted-ca-bundle\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:00:32.634052 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:32.634047    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f66f937-853a-438f-9bb2-c11f7fc46a31-console-serving-cert\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:00:32.634052 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:32.634058    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f66f937-853a-438f-9bb2-c11f7fc46a31-console-oauth-config\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:00:32.634219 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:32.634066    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f66f937-853a-438f-9bb2-c11f7fc46a31-oauth-serving-cert\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:00:32.634219 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:32.634075    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptght\" (UniqueName: \"kubernetes.io/projected/8f66f937-853a-438f-9bb2-c11f7fc46a31-kube-api-access-ptght\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:00:32.634219 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:32.634084    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f66f937-853a-438f-9bb2-c11f7fc46a31-service-ca\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:00:32.634219 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:32.634092    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f66f937-853a-438f-9bb2-c11f7fc46a31-console-config\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:00:33.207001 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:33.206969    2563 generic.go:356] "Generic (PLEG): container finished" podID="8f66f937-853a-438f-9bb2-c11f7fc46a31" containerID="486ea1fcf5e94cbeb01f89db6f878162ba4d242139179ff327dcb8e414062117" exitCode=0
Jul 14 23:00:33.207426 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:33.207016    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69c5f6d6cb-qc4nn"
Jul 14 23:00:33.207426 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:33.207041    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69c5f6d6cb-qc4nn" event={"ID":"8f66f937-853a-438f-9bb2-c11f7fc46a31","Type":"ContainerDied","Data":"486ea1fcf5e94cbeb01f89db6f878162ba4d242139179ff327dcb8e414062117"}
Jul 14 23:00:33.207426 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:33.207080    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69c5f6d6cb-qc4nn" event={"ID":"8f66f937-853a-438f-9bb2-c11f7fc46a31","Type":"ContainerDied","Data":"02ea4c9b33d34f893d2a95a45f172961203396139c59266610ab23ccc9c73c77"}
Jul 14 23:00:33.207426 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:33.207099    2563 scope.go:117] "RemoveContainer" containerID="486ea1fcf5e94cbeb01f89db6f878162ba4d242139179ff327dcb8e414062117"
Jul 14 23:00:33.214989 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:33.214969    2563 scope.go:117] "RemoveContainer" containerID="486ea1fcf5e94cbeb01f89db6f878162ba4d242139179ff327dcb8e414062117"
Jul 14 23:00:33.215221 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:00:33.215200    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"486ea1fcf5e94cbeb01f89db6f878162ba4d242139179ff327dcb8e414062117\": container with ID starting with 486ea1fcf5e94cbeb01f89db6f878162ba4d242139179ff327dcb8e414062117 not found: ID does not exist" containerID="486ea1fcf5e94cbeb01f89db6f878162ba4d242139179ff327dcb8e414062117"
Jul 14 23:00:33.215267 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:33.215230    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"486ea1fcf5e94cbeb01f89db6f878162ba4d242139179ff327dcb8e414062117"} err="failed to get container status \"486ea1fcf5e94cbeb01f89db6f878162ba4d242139179ff327dcb8e414062117\": rpc error: code = NotFound desc = could not find container \"486ea1fcf5e94cbeb01f89db6f878162ba4d242139179ff327dcb8e414062117\": container with ID starting with 486ea1fcf5e94cbeb01f89db6f878162ba4d242139179ff327dcb8e414062117 not found: ID does not exist"
Jul 14 23:00:33.228401 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:33.228378    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69c5f6d6cb-qc4nn"]
Jul 14 23:00:33.231807 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:33.231788    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-69c5f6d6cb-qc4nn"]
Jul 14 23:00:35.227770 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:35.227736    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f66f937-853a-438f-9bb2-c11f7fc46a31" path="/var/lib/kubelet/pods/8f66f937-853a-438f-9bb2-c11f7fc46a31/volumes"
Jul 14 23:00:59.286105 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:59.286070    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb8f65c67-9c5lh"]
Jul 14 23:00:59.286595 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:59.286339    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2704e48c-843e-4f28-88ce-1af16a0f8a5e" containerName="console"
Jul 14 23:00:59.286595 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:59.286350    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2704e48c-843e-4f28-88ce-1af16a0f8a5e" containerName="console"
Jul 14 23:00:59.286595 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:59.286363    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f66f937-853a-438f-9bb2-c11f7fc46a31" containerName="console"
Jul 14 23:00:59.286595 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:59.286369    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f66f937-853a-438f-9bb2-c11f7fc46a31" containerName="console"
Jul 14 23:00:59.286595 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:59.286417    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2704e48c-843e-4f28-88ce-1af16a0f8a5e" containerName="console"
Jul 14 23:00:59.286595 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:59.286426    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f66f937-853a-438f-9bb2-c11f7fc46a31" containerName="console"
Jul 14 23:00:59.289152 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:59.289135    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb8f65c67-9c5lh"
Jul 14 23:00:59.293878 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:59.293848    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 14 23:00:59.294020 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:59.293898    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 14 23:00:59.294020 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:59.293915    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 14 23:00:59.294020 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:59.293906    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 14 23:00:59.302034 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:59.302010    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb8f65c67-9c5lh"]
Jul 14 23:00:59.420199 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:59.420157    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3cf879ca-1865-404d-b0b0-63bab4b193cb-klusterlet-config\") pod \"klusterlet-addon-workmgr-7fb8f65c67-9c5lh\" (UID: \"3cf879ca-1865-404d-b0b0-63bab4b193cb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb8f65c67-9c5lh"
Jul 14 23:00:59.420199 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:59.420200    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h479n\" (UniqueName: \"kubernetes.io/projected/3cf879ca-1865-404d-b0b0-63bab4b193cb-kube-api-access-h479n\") pod \"klusterlet-addon-workmgr-7fb8f65c67-9c5lh\" (UID: \"3cf879ca-1865-404d-b0b0-63bab4b193cb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb8f65c67-9c5lh"
Jul 14 23:00:59.420446 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:59.420229    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3cf879ca-1865-404d-b0b0-63bab4b193cb-tmp\") pod \"klusterlet-addon-workmgr-7fb8f65c67-9c5lh\" (UID: \"3cf879ca-1865-404d-b0b0-63bab4b193cb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb8f65c67-9c5lh"
Jul 14 23:00:59.521166 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:59.521127    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3cf879ca-1865-404d-b0b0-63bab4b193cb-klusterlet-config\") pod \"klusterlet-addon-workmgr-7fb8f65c67-9c5lh\" (UID: \"3cf879ca-1865-404d-b0b0-63bab4b193cb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb8f65c67-9c5lh"
Jul 14 23:00:59.521166 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:59.521167    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h479n\" (UniqueName: \"kubernetes.io/projected/3cf879ca-1865-404d-b0b0-63bab4b193cb-kube-api-access-h479n\") pod \"klusterlet-addon-workmgr-7fb8f65c67-9c5lh\" (UID: \"3cf879ca-1865-404d-b0b0-63bab4b193cb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb8f65c67-9c5lh"
Jul 14 23:00:59.521471 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:59.521194    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3cf879ca-1865-404d-b0b0-63bab4b193cb-tmp\") pod \"klusterlet-addon-workmgr-7fb8f65c67-9c5lh\" (UID: \"3cf879ca-1865-404d-b0b0-63bab4b193cb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb8f65c67-9c5lh"
Jul 14 23:00:59.521594 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:59.521573    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3cf879ca-1865-404d-b0b0-63bab4b193cb-tmp\") pod \"klusterlet-addon-workmgr-7fb8f65c67-9c5lh\" (UID: \"3cf879ca-1865-404d-b0b0-63bab4b193cb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb8f65c67-9c5lh"
Jul 14 23:00:59.524008 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:59.523980    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3cf879ca-1865-404d-b0b0-63bab4b193cb-klusterlet-config\") pod \"klusterlet-addon-workmgr-7fb8f65c67-9c5lh\" (UID: \"3cf879ca-1865-404d-b0b0-63bab4b193cb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb8f65c67-9c5lh"
Jul 14 23:00:59.557847 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:59.557772    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h479n\" (UniqueName: \"kubernetes.io/projected/3cf879ca-1865-404d-b0b0-63bab4b193cb-kube-api-access-h479n\") pod \"klusterlet-addon-workmgr-7fb8f65c67-9c5lh\" (UID: \"3cf879ca-1865-404d-b0b0-63bab4b193cb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb8f65c67-9c5lh"
Jul 14 23:00:59.598698 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:59.598642    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb8f65c67-9c5lh"
Jul 14 23:00:59.731737 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:00:59.731694    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb8f65c67-9c5lh"]
Jul 14 23:00:59.734626 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:00:59.734598    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cf879ca_1865_404d_b0b0_63bab4b193cb.slice/crio-71882d230b1a546066e4ffe5c9d7a8b09fb3cb3c9f19964a871b8b07ef80af89 WatchSource:0}: Error finding container 71882d230b1a546066e4ffe5c9d7a8b09fb3cb3c9f19964a871b8b07ef80af89: Status 404 returned error can't find the container with id 71882d230b1a546066e4ffe5c9d7a8b09fb3cb3c9f19964a871b8b07ef80af89
Jul 14 23:01:00.296432 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:00.296393    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb8f65c67-9c5lh" event={"ID":"3cf879ca-1865-404d-b0b0-63bab4b193cb","Type":"ContainerStarted","Data":"71882d230b1a546066e4ffe5c9d7a8b09fb3cb3c9f19964a871b8b07ef80af89"}
Jul 14 23:01:02.032196 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:02.032159    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vsw5p"]
Jul 14 23:01:02.034565 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:02.034544    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vsw5p"
Jul 14 23:01:02.037612 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:02.037519    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 14 23:01:02.037612 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:02.037561    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-897js\""
Jul 14 23:01:02.037774 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:02.037617    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 14 23:01:02.047395 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:02.047371    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vsw5p"]
Jul 14 23:01:02.143664 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:02.143623    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/027f6bfe-0f0a-49c2-af9f-0acdb00b6a56-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-vsw5p\" (UID: \"027f6bfe-0f0a-49c2-af9f-0acdb00b6a56\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vsw5p"
Jul 14 23:01:02.143828 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:02.143704    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm5rz\" (UniqueName: \"kubernetes.io/projected/027f6bfe-0f0a-49c2-af9f-0acdb00b6a56-kube-api-access-mm5rz\") pod \"cert-manager-operator-controller-manager-55c444bf86-vsw5p\" (UID: \"027f6bfe-0f0a-49c2-af9f-0acdb00b6a56\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vsw5p"
Jul 14 23:01:02.244098 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:02.244063    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm5rz\" (UniqueName: \"kubernetes.io/projected/027f6bfe-0f0a-49c2-af9f-0acdb00b6a56-kube-api-access-mm5rz\") pod \"cert-manager-operator-controller-manager-55c444bf86-vsw5p\" (UID: \"027f6bfe-0f0a-49c2-af9f-0acdb00b6a56\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vsw5p"
Jul 14 23:01:02.244251 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:02.244114    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/027f6bfe-0f0a-49c2-af9f-0acdb00b6a56-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-vsw5p\" (UID: \"027f6bfe-0f0a-49c2-af9f-0acdb00b6a56\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vsw5p"
Jul 14 23:01:02.244450 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:02.244435    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/027f6bfe-0f0a-49c2-af9f-0acdb00b6a56-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-vsw5p\" (UID: \"027f6bfe-0f0a-49c2-af9f-0acdb00b6a56\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vsw5p"
Jul 14 23:01:02.253872 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:02.253851    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm5rz\" (UniqueName: \"kubernetes.io/projected/027f6bfe-0f0a-49c2-af9f-0acdb00b6a56-kube-api-access-mm5rz\") pod \"cert-manager-operator-controller-manager-55c444bf86-vsw5p\" (UID: \"027f6bfe-0f0a-49c2-af9f-0acdb00b6a56\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vsw5p"
Jul 14 23:01:02.346970 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:02.346889    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vsw5p"
Jul 14 23:01:03.505800 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:03.505773    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vsw5p"]
Jul 14 23:01:03.507542 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:01:03.507514    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod027f6bfe_0f0a_49c2_af9f_0acdb00b6a56.slice/crio-8cbe946a2faa1c577a3b90dd0efa8152601a3de4bf62b0420d09e533a68750c4 WatchSource:0}: Error finding container 8cbe946a2faa1c577a3b90dd0efa8152601a3de4bf62b0420d09e533a68750c4: Status 404 returned error can't find the container with id 8cbe946a2faa1c577a3b90dd0efa8152601a3de4bf62b0420d09e533a68750c4
Jul 14 23:01:04.315716 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:04.315681    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb8f65c67-9c5lh" event={"ID":"3cf879ca-1865-404d-b0b0-63bab4b193cb","Type":"ContainerStarted","Data":"3d80cd98769229a19ae626f3f97d134aa49727b8ddfd91fcc96dc54194c27c5c"}
Jul 14 23:01:04.315930 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:04.315910    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb8f65c67-9c5lh"
Jul 14 23:01:04.317182 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:04.317156    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vsw5p" event={"ID":"027f6bfe-0f0a-49c2-af9f-0acdb00b6a56","Type":"ContainerStarted","Data":"8cbe946a2faa1c577a3b90dd0efa8152601a3de4bf62b0420d09e533a68750c4"}
Jul 14 23:01:04.317868 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:04.317847    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb8f65c67-9c5lh"
Jul 14 23:01:04.333890 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:04.333826    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb8f65c67-9c5lh" podStartSLOduration=1.6337504809999999 podStartE2EDuration="5.333807587s" totalImagesPullingTime="3.700057106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:00:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:01:04.332487306 +0000 UTC m=+251.635478099" watchObservedRunningTime="2026-07-14 23:01:04.333807587 +0000 UTC m=+251.636798371"
Jul 14 23:01:06.326404 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:06.326358    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vsw5p" event={"ID":"027f6bfe-0f0a-49c2-af9f-0acdb00b6a56","Type":"ContainerStarted","Data":"51ae2a442d917121a69d372c779265d0b053154c2556a18630890e1387d8895a"}
Jul 14 23:01:06.349343 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:06.349273    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vsw5p" podStartSLOduration=1.7546888470000002 podStartE2EDuration="4.349256318s" totalImagesPullingTime="2.594567471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:01:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:01:06.348638378 +0000 UTC m=+253.651629171" watchObservedRunningTime="2026-07-14 23:01:06.349256318 +0000 UTC m=+253.652247109"
Jul 14 23:01:07.219587 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:07.219554    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-btzhx"]
Jul 14 23:01:07.221872 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:07.221852    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-btzhx"
Jul 14 23:01:07.240830 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:07.240808    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 14 23:01:07.241133 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:07.240833    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-82tpc\""
Jul 14 23:01:07.241319 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:07.240863    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 14 23:01:07.242247 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:07.242225    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-btzhx"]
Jul 14 23:01:07.283835 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:07.283797    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2dfc2e0f-226d-45d8-bc53-fb99eb46d9d2-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-btzhx\" (UID: \"2dfc2e0f-226d-45d8-bc53-fb99eb46d9d2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-btzhx"
Jul 14 23:01:07.284008 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:07.283849    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwwsx\" (UniqueName: \"kubernetes.io/projected/2dfc2e0f-226d-45d8-bc53-fb99eb46d9d2-kube-api-access-lwwsx\") pod \"cert-manager-webhook-6486ffd997-btzhx\" (UID: \"2dfc2e0f-226d-45d8-bc53-fb99eb46d9d2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-btzhx"
Jul 14 23:01:07.384761 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:07.384722    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2dfc2e0f-226d-45d8-bc53-fb99eb46d9d2-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-btzhx\" (UID: \"2dfc2e0f-226d-45d8-bc53-fb99eb46d9d2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-btzhx"
Jul 14 23:01:07.384761 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:07.384769    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwwsx\" (UniqueName: \"kubernetes.io/projected/2dfc2e0f-226d-45d8-bc53-fb99eb46d9d2-kube-api-access-lwwsx\") pod \"cert-manager-webhook-6486ffd997-btzhx\" (UID: \"2dfc2e0f-226d-45d8-bc53-fb99eb46d9d2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-btzhx"
Jul 14 23:01:07.395604 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:07.395579    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2dfc2e0f-226d-45d8-bc53-fb99eb46d9d2-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-btzhx\" (UID: \"2dfc2e0f-226d-45d8-bc53-fb99eb46d9d2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-btzhx"
Jul 14 23:01:07.395885 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:07.395860    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwwsx\" (UniqueName: \"kubernetes.io/projected/2dfc2e0f-226d-45d8-bc53-fb99eb46d9d2-kube-api-access-lwwsx\") pod \"cert-manager-webhook-6486ffd997-btzhx\" (UID: \"2dfc2e0f-226d-45d8-bc53-fb99eb46d9d2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-btzhx"
Jul 14 23:01:07.544848 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:07.544805    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-btzhx"
Jul 14 23:01:07.671251 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:07.671229    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-btzhx"]
Jul 14 23:01:07.673429 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:01:07.673401    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2dfc2e0f_226d_45d8_bc53_fb99eb46d9d2.slice/crio-28c18e476264f5e1b467ca47cf084cdcdbdfb216b3ef6e1775594f48280b3799 WatchSource:0}: Error finding container 28c18e476264f5e1b467ca47cf084cdcdbdfb216b3ef6e1775594f48280b3799: Status 404 returned error can't find the container with id 28c18e476264f5e1b467ca47cf084cdcdbdfb216b3ef6e1775594f48280b3799
Jul 14 23:01:08.334851 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:08.334816    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-btzhx" event={"ID":"2dfc2e0f-226d-45d8-bc53-fb99eb46d9d2","Type":"ContainerStarted","Data":"28c18e476264f5e1b467ca47cf084cdcdbdfb216b3ef6e1775594f48280b3799"}
Jul 14 23:01:11.348786 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:11.348749    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-btzhx" event={"ID":"2dfc2e0f-226d-45d8-bc53-fb99eb46d9d2","Type":"ContainerStarted","Data":"255fd1a69d4a3b457433a4ad69e69064e099d86f5f67afda6ec3a9cec46c74cf"}
Jul 14 23:01:11.349169 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:11.348861    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-btzhx"
Jul 14 23:01:11.367933 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:11.367886    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-btzhx" podStartSLOduration=1.537727326 podStartE2EDuration="4.367870995s" totalImagesPullingTime="2.830143669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:01:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:01:11.366001677 +0000 UTC m=+258.668992495" watchObservedRunningTime="2026-07-14 23:01:11.367870995 +0000 UTC m=+258.670861787"
Jul 14 23:01:17.355096 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:17.355020    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-btzhx"
Jul 14 23:01:19.747751 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:19.747715    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-qb7j9"]
Jul 14 23:01:19.749943 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:19.749927    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-qb7j9"
Jul 14 23:01:19.756929 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:19.756914    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-s6988\""
Jul 14 23:01:19.765909 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:19.765888    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-qb7j9"]
Jul 14 23:01:19.775012 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:19.774988    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m8bg\" (UniqueName: \"kubernetes.io/projected/6d8fb9b7-5cc9-498b-bc60-af7f52ec1d04-kube-api-access-2m8bg\") pod \"cert-manager-55b4d954d5-qb7j9\" (UID: \"6d8fb9b7-5cc9-498b-bc60-af7f52ec1d04\") " pod="cert-manager/cert-manager-55b4d954d5-qb7j9"
Jul 14 23:01:19.775115 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:19.775028    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d8fb9b7-5cc9-498b-bc60-af7f52ec1d04-bound-sa-token\") pod \"cert-manager-55b4d954d5-qb7j9\" (UID: \"6d8fb9b7-5cc9-498b-bc60-af7f52ec1d04\") " pod="cert-manager/cert-manager-55b4d954d5-qb7j9"
Jul 14 23:01:19.875647 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:19.875612    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m8bg\" (UniqueName: \"kubernetes.io/projected/6d8fb9b7-5cc9-498b-bc60-af7f52ec1d04-kube-api-access-2m8bg\") pod \"cert-manager-55b4d954d5-qb7j9\" (UID: \"6d8fb9b7-5cc9-498b-bc60-af7f52ec1d04\") " pod="cert-manager/cert-manager-55b4d954d5-qb7j9"
Jul 14 23:01:19.875814 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:19.875660    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d8fb9b7-5cc9-498b-bc60-af7f52ec1d04-bound-sa-token\") pod \"cert-manager-55b4d954d5-qb7j9\" (UID: \"6d8fb9b7-5cc9-498b-bc60-af7f52ec1d04\") " pod="cert-manager/cert-manager-55b4d954d5-qb7j9"
Jul 14 23:01:19.885101 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:19.885069    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m8bg\" (UniqueName: \"kubernetes.io/projected/6d8fb9b7-5cc9-498b-bc60-af7f52ec1d04-kube-api-access-2m8bg\") pod \"cert-manager-55b4d954d5-qb7j9\" (UID: \"6d8fb9b7-5cc9-498b-bc60-af7f52ec1d04\") " pod="cert-manager/cert-manager-55b4d954d5-qb7j9"
Jul 14 23:01:19.888404 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:19.888382    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d8fb9b7-5cc9-498b-bc60-af7f52ec1d04-bound-sa-token\") pod \"cert-manager-55b4d954d5-qb7j9\" (UID: \"6d8fb9b7-5cc9-498b-bc60-af7f52ec1d04\") " pod="cert-manager/cert-manager-55b4d954d5-qb7j9"
Jul 14 23:01:20.058030 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:20.057945    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-qb7j9"
Jul 14 23:01:20.196600 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:20.196440    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-qb7j9"]
Jul 14 23:01:20.198459 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:01:20.198425    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d8fb9b7_5cc9_498b_bc60_af7f52ec1d04.slice/crio-4b24c3eeeaa904d43ca815d1c86bf8321f51dbf2276dd13eedfbe25e96b128e2 WatchSource:0}: Error finding container 4b24c3eeeaa904d43ca815d1c86bf8321f51dbf2276dd13eedfbe25e96b128e2: Status 404 returned error can't find the container with id 4b24c3eeeaa904d43ca815d1c86bf8321f51dbf2276dd13eedfbe25e96b128e2
Jul 14 23:01:20.381878 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:20.381789    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-qb7j9" event={"ID":"6d8fb9b7-5cc9-498b-bc60-af7f52ec1d04","Type":"ContainerStarted","Data":"80a827fa74afe51af245840f4e0c01f07b4550d8825edd9f3a361801e7666263"}
Jul 14 23:01:20.381878 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:20.381829    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-qb7j9" event={"ID":"6d8fb9b7-5cc9-498b-bc60-af7f52ec1d04","Type":"ContainerStarted","Data":"4b24c3eeeaa904d43ca815d1c86bf8321f51dbf2276dd13eedfbe25e96b128e2"}
Jul 14 23:01:20.420109 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:20.420057    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-qb7j9" podStartSLOduration=1.420037782 podStartE2EDuration="1.420037782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:01:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:01:20.419630378 +0000 UTC m=+267.722621183" watchObservedRunningTime="2026-07-14 23:01:20.420037782 +0000 UTC m=+267.723028568"
Jul 14 23:01:44.272970 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:44.272937    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d756c8cff-xt4p6"]
Jul 14 23:01:44.275370 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:44.275353    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7d756c8cff-xt4p6"
Jul 14 23:01:44.278808 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:44.278787    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 14 23:01:44.278967 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:44.278905    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-j4t6n\""
Jul 14 23:01:44.279080 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:44.278998    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 14 23:01:44.279080 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:44.279022    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 14 23:01:44.279354 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:44.279337    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 14 23:01:44.303523 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:44.303489    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d756c8cff-xt4p6"]
Jul 14 23:01:44.361414 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:44.361388    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc1f564c-8810-45db-9764-2cce76d67e29-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d756c8cff-xt4p6\" (UID: \"fc1f564c-8810-45db-9764-2cce76d67e29\") " pod="opendatahub/opendatahub-operator-controller-manager-7d756c8cff-xt4p6"
Jul 14 23:01:44.361554 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:44.361418    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fc1f564c-8810-45db-9764-2cce76d67e29-manifests\") pod \"opendatahub-operator-controller-manager-7d756c8cff-xt4p6\" (UID: \"fc1f564c-8810-45db-9764-2cce76d67e29\") " pod="opendatahub/opendatahub-operator-controller-manager-7d756c8cff-xt4p6"
Jul 14 23:01:44.361554 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:44.361438    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc1f564c-8810-45db-9764-2cce76d67e29-tmp\") pod \"opendatahub-operator-controller-manager-7d756c8cff-xt4p6\" (UID: \"fc1f564c-8810-45db-9764-2cce76d67e29\") " pod="opendatahub/opendatahub-operator-controller-manager-7d756c8cff-xt4p6"
Jul 14 23:01:44.361554 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:44.361453    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzp2c\" (UniqueName: \"kubernetes.io/projected/fc1f564c-8810-45db-9764-2cce76d67e29-kube-api-access-nzp2c\") pod \"opendatahub-operator-controller-manager-7d756c8cff-xt4p6\" (UID: \"fc1f564c-8810-45db-9764-2cce76d67e29\") " pod="opendatahub/opendatahub-operator-controller-manager-7d756c8cff-xt4p6"
Jul 14 23:01:44.361554 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:44.361499    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fc1f564c-8810-45db-9764-2cce76d67e29-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d756c8cff-xt4p6\" (UID: \"fc1f564c-8810-45db-9764-2cce76d67e29\") " pod="opendatahub/opendatahub-operator-controller-manager-7d756c8cff-xt4p6"
Jul 14 23:01:44.462873 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:44.462825    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc1f564c-8810-45db-9764-2cce76d67e29-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d756c8cff-xt4p6\" (UID: \"fc1f564c-8810-45db-9764-2cce76d67e29\") " pod="opendatahub/opendatahub-operator-controller-manager-7d756c8cff-xt4p6"
Jul 14 23:01:44.462873 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:44.462864    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fc1f564c-8810-45db-9764-2cce76d67e29-manifests\") pod \"opendatahub-operator-controller-manager-7d756c8cff-xt4p6\" (UID: \"fc1f564c-8810-45db-9764-2cce76d67e29\") " pod="opendatahub/opendatahub-operator-controller-manager-7d756c8cff-xt4p6"
Jul 14 23:01:44.462873 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:44.462885    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc1f564c-8810-45db-9764-2cce76d67e29-tmp\") pod \"opendatahub-operator-controller-manager-7d756c8cff-xt4p6\" (UID: \"fc1f564c-8810-45db-9764-2cce76d67e29\") " pod="opendatahub/opendatahub-operator-controller-manager-7d756c8cff-xt4p6"
Jul 14 23:01:44.463121 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:44.462904    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzp2c\" (UniqueName: \"kubernetes.io/projected/fc1f564c-8810-45db-9764-2cce76d67e29-kube-api-access-nzp2c\") pod \"opendatahub-operator-controller-manager-7d756c8cff-xt4p6\" (UID: \"fc1f564c-8810-45db-9764-2cce76d67e29\") " pod="opendatahub/opendatahub-operator-controller-manager-7d756c8cff-xt4p6"
Jul 14 23:01:44.463121 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:44.462941    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fc1f564c-8810-45db-9764-2cce76d67e29-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d756c8cff-xt4p6\" (UID: \"fc1f564c-8810-45db-9764-2cce76d67e29\") " pod="opendatahub/opendatahub-operator-controller-manager-7d756c8cff-xt4p6"
Jul 14 23:01:44.463336 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:44.463313    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fc1f564c-8810-45db-9764-2cce76d67e29-manifests\") pod \"opendatahub-operator-controller-manager-7d756c8cff-xt4p6\" (UID: \"fc1f564c-8810-45db-9764-2cce76d67e29\") " pod="opendatahub/opendatahub-operator-controller-manager-7d756c8cff-xt4p6"
Jul 14 23:01:44.465414 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:44.465391    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc1f564c-8810-45db-9764-2cce76d67e29-tmp\") pod \"opendatahub-operator-controller-manager-7d756c8cff-xt4p6\" (UID: \"fc1f564c-8810-45db-9764-2cce76d67e29\") " pod="opendatahub/opendatahub-operator-controller-manager-7d756c8cff-xt4p6"
Jul 14 23:01:44.465636 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:44.465616    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc1f564c-8810-45db-9764-2cce76d67e29-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d756c8cff-xt4p6\" (UID: \"fc1f564c-8810-45db-9764-2cce76d67e29\") " pod="opendatahub/opendatahub-operator-controller-manager-7d756c8cff-xt4p6"
Jul 14 23:01:44.465750 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:44.465731    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fc1f564c-8810-45db-9764-2cce76d67e29-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d756c8cff-xt4p6\" (UID: \"fc1f564c-8810-45db-9764-2cce76d67e29\") " pod="opendatahub/opendatahub-operator-controller-manager-7d756c8cff-xt4p6"
Jul 14 23:01:44.472573 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:44.472542    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzp2c\" (UniqueName: \"kubernetes.io/projected/fc1f564c-8810-45db-9764-2cce76d67e29-kube-api-access-nzp2c\") pod \"opendatahub-operator-controller-manager-7d756c8cff-xt4p6\" (UID: \"fc1f564c-8810-45db-9764-2cce76d67e29\") " pod="opendatahub/opendatahub-operator-controller-manager-7d756c8cff-xt4p6"
Jul 14 23:01:44.588942 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:44.588843    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7d756c8cff-xt4p6"
Jul 14 23:01:44.729023 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:44.728991    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d756c8cff-xt4p6"]
Jul 14 23:01:44.732765 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:01:44.732728    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc1f564c_8810_45db_9764_2cce76d67e29.slice/crio-65097f3f177b3e72e79e3e2510f55f509107df57b87fde60052e9e3f9ed80191 WatchSource:0}: Error finding container 65097f3f177b3e72e79e3e2510f55f509107df57b87fde60052e9e3f9ed80191: Status 404 returned error can't find the container with id 65097f3f177b3e72e79e3e2510f55f509107df57b87fde60052e9e3f9ed80191
Jul 14 23:01:45.480752 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:45.480706    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d756c8cff-xt4p6" event={"ID":"fc1f564c-8810-45db-9764-2cce76d67e29","Type":"ContainerStarted","Data":"65097f3f177b3e72e79e3e2510f55f509107df57b87fde60052e9e3f9ed80191"}
Jul 14 23:01:48.494734 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:48.494698    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d756c8cff-xt4p6" event={"ID":"fc1f564c-8810-45db-9764-2cce76d67e29","Type":"ContainerStarted","Data":"43c71f2a43e5c247f1e8b26ee32895d99cef64a189463fd543bda79f038b1f9a"}
Jul 14 23:01:49.500370 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:49.500338    2563 generic.go:356] "Generic (PLEG): container finished" podID="fc1f564c-8810-45db-9764-2cce76d67e29" containerID="43c71f2a43e5c247f1e8b26ee32895d99cef64a189463fd543bda79f038b1f9a" exitCode=0
Jul 14 23:01:49.500748 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:49.500385    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d756c8cff-xt4p6" event={"ID":"fc1f564c-8810-45db-9764-2cce76d67e29","Type":"ContainerDied","Data":"43c71f2a43e5c247f1e8b26ee32895d99cef64a189463fd543bda79f038b1f9a"}
Jul 14 23:01:50.505575 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:50.505540    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d756c8cff-xt4p6" event={"ID":"fc1f564c-8810-45db-9764-2cce76d67e29","Type":"ContainerStarted","Data":"07f197512097ef5596ed1c83a8c113236545ff2bd59c9c0a182a7b2c56925683"}
Jul 14 23:01:50.506036 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:50.505654    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7d756c8cff-xt4p6"
Jul 14 23:01:50.539394 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:50.539340    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7d756c8cff-xt4p6" podStartSLOduration=2.018092118 podStartE2EDuration="6.539321424s" totalImagesPullingTime="3.443810597s" totalInitContainerRuntime="1.077418709s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:01:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:01:50.537928346 +0000 UTC m=+297.840919137" watchObservedRunningTime="2026-07-14 23:01:50.539321424 +0000 UTC m=+297.842312217"
Jul 14 23:01:53.133837 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:01:53.133807    2563 kubelet.go:1648] "Image garbage collection succeeded"
Jul 14 23:02:01.512112 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:01.512082    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7d756c8cff-xt4p6"
Jul 14 23:02:04.615268 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:04.615233    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-drvt2"]
Jul 14 23:02:04.617373 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:04.617357    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-drvt2"
Jul 14 23:02:04.620698 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:04.620674    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 14 23:02:04.620813 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:04.620778    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-p9f26\""
Jul 14 23:02:04.629852 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:04.629831    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-drvt2"]
Jul 14 23:02:04.723260 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:04.723224    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vztn\" (UniqueName: \"kubernetes.io/projected/e6034bd0-b905-40c7-8380-eff83fd5522d-kube-api-access-4vztn\") pod \"odh-model-controller-7998c67dc4-drvt2\" (UID: \"e6034bd0-b905-40c7-8380-eff83fd5522d\") " pod="opendatahub/odh-model-controller-7998c67dc4-drvt2"
Jul 14 23:02:04.723438 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:04.723273    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e6034bd0-b905-40c7-8380-eff83fd5522d-cert\") pod \"odh-model-controller-7998c67dc4-drvt2\" (UID: \"e6034bd0-b905-40c7-8380-eff83fd5522d\") " pod="opendatahub/odh-model-controller-7998c67dc4-drvt2"
Jul 14 23:02:04.824835 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:04.824785    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vztn\" (UniqueName: \"kubernetes.io/projected/e6034bd0-b905-40c7-8380-eff83fd5522d-kube-api-access-4vztn\") pod \"odh-model-controller-7998c67dc4-drvt2\" (UID: \"e6034bd0-b905-40c7-8380-eff83fd5522d\") " pod="opendatahub/odh-model-controller-7998c67dc4-drvt2"
Jul 14 23:02:04.825002 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:04.824848    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e6034bd0-b905-40c7-8380-eff83fd5522d-cert\") pod \"odh-model-controller-7998c67dc4-drvt2\" (UID: \"e6034bd0-b905-40c7-8380-eff83fd5522d\") " pod="opendatahub/odh-model-controller-7998c67dc4-drvt2"
Jul 14 23:02:04.825002 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:02:04.824964    2563 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 14 23:02:04.825083 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:02:04.825035    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6034bd0-b905-40c7-8380-eff83fd5522d-cert podName:e6034bd0-b905-40c7-8380-eff83fd5522d nodeName:}" failed. No retries permitted until 2026-07-14 23:02:05.325012 +0000 UTC m=+312.628002778 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e6034bd0-b905-40c7-8380-eff83fd5522d-cert") pod "odh-model-controller-7998c67dc4-drvt2" (UID: "e6034bd0-b905-40c7-8380-eff83fd5522d") : secret "odh-model-controller-webhook-cert" not found
Jul 14 23:02:04.834735 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:04.834700    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vztn\" (UniqueName: \"kubernetes.io/projected/e6034bd0-b905-40c7-8380-eff83fd5522d-kube-api-access-4vztn\") pod \"odh-model-controller-7998c67dc4-drvt2\" (UID: \"e6034bd0-b905-40c7-8380-eff83fd5522d\") " pod="opendatahub/odh-model-controller-7998c67dc4-drvt2"
Jul 14 23:02:05.281173 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:05.281134    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-9w2wb"]
Jul 14 23:02:05.283853 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:05.283836    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-9w2wb"
Jul 14 23:02:05.287016 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:05.286995    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 14 23:02:05.295705 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:05.295685    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-9w2wb"]
Jul 14 23:02:05.327411 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:05.327390    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e6034bd0-b905-40c7-8380-eff83fd5522d-cert\") pod \"odh-model-controller-7998c67dc4-drvt2\" (UID: \"e6034bd0-b905-40c7-8380-eff83fd5522d\") " pod="opendatahub/odh-model-controller-7998c67dc4-drvt2"
Jul 14 23:02:05.329789 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:05.329766    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e6034bd0-b905-40c7-8380-eff83fd5522d-cert\") pod \"odh-model-controller-7998c67dc4-drvt2\" (UID: \"e6034bd0-b905-40c7-8380-eff83fd5522d\") " pod="opendatahub/odh-model-controller-7998c67dc4-drvt2"
Jul 14 23:02:05.428374 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:05.428337    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/435e7622-b867-42ea-a6e6-f7e6f857972b-tls-certs\") pod \"model-serving-api-54d89bd5fc-9w2wb\" (UID: \"435e7622-b867-42ea-a6e6-f7e6f857972b\") " pod="opendatahub/model-serving-api-54d89bd5fc-9w2wb"
Jul 14 23:02:05.428374 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:05.428375    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhppm\" (UniqueName: \"kubernetes.io/projected/435e7622-b867-42ea-a6e6-f7e6f857972b-kube-api-access-jhppm\") pod \"model-serving-api-54d89bd5fc-9w2wb\" (UID: \"435e7622-b867-42ea-a6e6-f7e6f857972b\") " pod="opendatahub/model-serving-api-54d89bd5fc-9w2wb"
Jul 14 23:02:05.527039 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:05.526998    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-drvt2"
Jul 14 23:02:05.529386 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:05.529358    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/435e7622-b867-42ea-a6e6-f7e6f857972b-tls-certs\") pod \"model-serving-api-54d89bd5fc-9w2wb\" (UID: \"435e7622-b867-42ea-a6e6-f7e6f857972b\") " pod="opendatahub/model-serving-api-54d89bd5fc-9w2wb"
Jul 14 23:02:05.529461 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:05.529402    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhppm\" (UniqueName: \"kubernetes.io/projected/435e7622-b867-42ea-a6e6-f7e6f857972b-kube-api-access-jhppm\") pod \"model-serving-api-54d89bd5fc-9w2wb\" (UID: \"435e7622-b867-42ea-a6e6-f7e6f857972b\") " pod="opendatahub/model-serving-api-54d89bd5fc-9w2wb"
Jul 14 23:02:05.532065 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:05.531957    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/435e7622-b867-42ea-a6e6-f7e6f857972b-tls-certs\") pod \"model-serving-api-54d89bd5fc-9w2wb\" (UID: \"435e7622-b867-42ea-a6e6-f7e6f857972b\") " pod="opendatahub/model-serving-api-54d89bd5fc-9w2wb"
Jul 14 23:02:05.541910 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:05.541887    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhppm\" (UniqueName: \"kubernetes.io/projected/435e7622-b867-42ea-a6e6-f7e6f857972b-kube-api-access-jhppm\") pod \"model-serving-api-54d89bd5fc-9w2wb\" (UID: \"435e7622-b867-42ea-a6e6-f7e6f857972b\") " pod="opendatahub/model-serving-api-54d89bd5fc-9w2wb"
Jul 14 23:02:05.594682 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:05.594638    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-9w2wb"
Jul 14 23:02:05.671028 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:05.670996    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-drvt2"]
Jul 14 23:02:05.674567 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:02:05.674538    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6034bd0_b905_40c7_8380_eff83fd5522d.slice/crio-74d0acd94bdfd7a4cd42ea1e3c9de7d3e3168a559b47ebaab75b3a17ffaa7852 WatchSource:0}: Error finding container 74d0acd94bdfd7a4cd42ea1e3c9de7d3e3168a559b47ebaab75b3a17ffaa7852: Status 404 returned error can't find the container with id 74d0acd94bdfd7a4cd42ea1e3c9de7d3e3168a559b47ebaab75b3a17ffaa7852
Jul 14 23:02:05.676022 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:05.676004    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 23:02:05.750737 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:05.750527    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-9w2wb"]
Jul 14 23:02:05.753124 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:02:05.753084    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod435e7622_b867_42ea_a6e6_f7e6f857972b.slice/crio-6cd604cad232a0285914a6fd423f8c65cab913050fb0ab1c21ebfd1e887edb83 WatchSource:0}: Error finding container 6cd604cad232a0285914a6fd423f8c65cab913050fb0ab1c21ebfd1e887edb83: Status 404 returned error can't find the container with id 6cd604cad232a0285914a6fd423f8c65cab913050fb0ab1c21ebfd1e887edb83
Jul 14 23:02:05.896183 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:05.896094    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-946bb8597-h8jmp"]
Jul 14 23:02:05.899137 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:05.899115    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-946bb8597-h8jmp"
Jul 14 23:02:05.902442 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:05.902407    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-gt9kd\""
Jul 14 23:02:05.902580 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:05.902441    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 14 23:02:05.902580 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:05.902484    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 14 23:02:05.902580 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:05.902515    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 14 23:02:05.902580 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:05.902516    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 14 23:02:05.910553 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:05.910530    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-946bb8597-h8jmp"]
Jul 14 23:02:06.033640 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:06.033602    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e061977-cf57-46d7-9804-ed6f91ce7e60-tmp\") pod \"kube-auth-proxy-946bb8597-h8jmp\" (UID: \"4e061977-cf57-46d7-9804-ed6f91ce7e60\") " pod="openshift-ingress/kube-auth-proxy-946bb8597-h8jmp"
Jul 14 23:02:06.033805 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:06.033656    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e061977-cf57-46d7-9804-ed6f91ce7e60-tls-certs\") pod \"kube-auth-proxy-946bb8597-h8jmp\" (UID: \"4e061977-cf57-46d7-9804-ed6f91ce7e60\") " pod="openshift-ingress/kube-auth-proxy-946bb8597-h8jmp"
Jul 14 23:02:06.033805 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:06.033726    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjtfq\" (UniqueName: \"kubernetes.io/projected/4e061977-cf57-46d7-9804-ed6f91ce7e60-kube-api-access-bjtfq\") pod \"kube-auth-proxy-946bb8597-h8jmp\" (UID: \"4e061977-cf57-46d7-9804-ed6f91ce7e60\") " pod="openshift-ingress/kube-auth-proxy-946bb8597-h8jmp"
Jul 14 23:02:06.134228 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:06.134185    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjtfq\" (UniqueName: \"kubernetes.io/projected/4e061977-cf57-46d7-9804-ed6f91ce7e60-kube-api-access-bjtfq\") pod \"kube-auth-proxy-946bb8597-h8jmp\" (UID: \"4e061977-cf57-46d7-9804-ed6f91ce7e60\") " pod="openshift-ingress/kube-auth-proxy-946bb8597-h8jmp"
Jul 14 23:02:06.134426 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:06.134272    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e061977-cf57-46d7-9804-ed6f91ce7e60-tmp\") pod \"kube-auth-proxy-946bb8597-h8jmp\" (UID: \"4e061977-cf57-46d7-9804-ed6f91ce7e60\") " pod="openshift-ingress/kube-auth-proxy-946bb8597-h8jmp"
Jul 14 23:02:06.134426 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:06.134369    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e061977-cf57-46d7-9804-ed6f91ce7e60-tls-certs\") pod \"kube-auth-proxy-946bb8597-h8jmp\" (UID: \"4e061977-cf57-46d7-9804-ed6f91ce7e60\") " pod="openshift-ingress/kube-auth-proxy-946bb8597-h8jmp"
Jul 14 23:02:06.136667 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:06.136644    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e061977-cf57-46d7-9804-ed6f91ce7e60-tmp\") pod \"kube-auth-proxy-946bb8597-h8jmp\" (UID: \"4e061977-cf57-46d7-9804-ed6f91ce7e60\") " pod="openshift-ingress/kube-auth-proxy-946bb8597-h8jmp"
Jul 14 23:02:06.136859 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:06.136842    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e061977-cf57-46d7-9804-ed6f91ce7e60-tls-certs\") pod \"kube-auth-proxy-946bb8597-h8jmp\" (UID: \"4e061977-cf57-46d7-9804-ed6f91ce7e60\") " pod="openshift-ingress/kube-auth-proxy-946bb8597-h8jmp"
Jul 14 23:02:06.143107 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:06.143080    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjtfq\" (UniqueName: \"kubernetes.io/projected/4e061977-cf57-46d7-9804-ed6f91ce7e60-kube-api-access-bjtfq\") pod \"kube-auth-proxy-946bb8597-h8jmp\" (UID: \"4e061977-cf57-46d7-9804-ed6f91ce7e60\") " pod="openshift-ingress/kube-auth-proxy-946bb8597-h8jmp"
Jul 14 23:02:06.210126 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:06.210039    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-946bb8597-h8jmp"
Jul 14 23:02:06.346412 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:06.346377    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-946bb8597-h8jmp"]
Jul 14 23:02:06.349968 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:02:06.349936    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e061977_cf57_46d7_9804_ed6f91ce7e60.slice/crio-53e9500b069952f509d08cc51789abc541bce053df5881079d42cbc898cdb1a3 WatchSource:0}: Error finding container 53e9500b069952f509d08cc51789abc541bce053df5881079d42cbc898cdb1a3: Status 404 returned error can't find the container with id 53e9500b069952f509d08cc51789abc541bce053df5881079d42cbc898cdb1a3
Jul 14 23:02:06.568596 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:06.568545    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-946bb8597-h8jmp" event={"ID":"4e061977-cf57-46d7-9804-ed6f91ce7e60","Type":"ContainerStarted","Data":"53e9500b069952f509d08cc51789abc541bce053df5881079d42cbc898cdb1a3"}
Jul 14 23:02:06.569674 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:06.569642    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-9w2wb" event={"ID":"435e7622-b867-42ea-a6e6-f7e6f857972b","Type":"ContainerStarted","Data":"6cd604cad232a0285914a6fd423f8c65cab913050fb0ab1c21ebfd1e887edb83"}
Jul 14 23:02:06.570698 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:06.570679    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-drvt2" event={"ID":"e6034bd0-b905-40c7-8380-eff83fd5522d","Type":"ContainerStarted","Data":"74d0acd94bdfd7a4cd42ea1e3c9de7d3e3168a559b47ebaab75b3a17ffaa7852"}
Jul 14 23:02:06.972359 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:06.971625    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-wmhhk"]
Jul 14 23:02:06.977777 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:06.977746    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-wmhhk"
Jul 14 23:02:06.982704 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:06.982620    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 14 23:02:06.982957 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:06.982841    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 14 23:02:06.983209 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:06.983189    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 14 23:02:06.983477 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:06.983459    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 14 23:02:06.983551 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:06.983534    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-nw8n9\""
Jul 14 23:02:06.984111 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:06.984072    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-wmhhk"]
Jul 14 23:02:07.142616 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:07.142577    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/892dd988-f1a1-41d2-bd5c-9056aec2c889-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-wmhhk\" (UID: \"892dd988-f1a1-41d2-bd5c-9056aec2c889\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-wmhhk"
Jul 14 23:02:07.142616 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:07.142627    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/892dd988-f1a1-41d2-bd5c-9056aec2c889-cacerts\") pod \"istiod-openshift-gateway-7574459656-wmhhk\" (UID: \"892dd988-f1a1-41d2-bd5c-9056aec2c889\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-wmhhk"
Jul 14 23:02:07.142875 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:07.142693    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/892dd988-f1a1-41d2-bd5c-9056aec2c889-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-wmhhk\" (UID: \"892dd988-f1a1-41d2-bd5c-9056aec2c889\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-wmhhk"
Jul 14 23:02:07.142875 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:07.142746    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/892dd988-f1a1-41d2-bd5c-9056aec2c889-local-certs\") pod \"istiod-openshift-gateway-7574459656-wmhhk\" (UID: \"892dd988-f1a1-41d2-bd5c-9056aec2c889\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-wmhhk"
Jul 14 23:02:07.142875 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:07.142772    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/892dd988-f1a1-41d2-bd5c-9056aec2c889-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-wmhhk\" (UID: \"892dd988-f1a1-41d2-bd5c-9056aec2c889\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-wmhhk"
Jul 14 23:02:07.142875 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:07.142805    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892dd988-f1a1-41d2-bd5c-9056aec2c889-istio-token\") pod \"istiod-openshift-gateway-7574459656-wmhhk\" (UID: \"892dd988-f1a1-41d2-bd5c-9056aec2c889\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-wmhhk"
Jul 14 23:02:07.142875 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:07.142866    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn6xc\" (UniqueName: \"kubernetes.io/projected/892dd988-f1a1-41d2-bd5c-9056aec2c889-kube-api-access-hn6xc\") pod \"istiod-openshift-gateway-7574459656-wmhhk\" (UID: \"892dd988-f1a1-41d2-bd5c-9056aec2c889\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-wmhhk"
Jul 14 23:02:07.243896 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:07.243860    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892dd988-f1a1-41d2-bd5c-9056aec2c889-istio-token\") pod \"istiod-openshift-gateway-7574459656-wmhhk\" (UID: \"892dd988-f1a1-41d2-bd5c-9056aec2c889\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-wmhhk"
Jul 14 23:02:07.244074 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:07.243932    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn6xc\" (UniqueName: \"kubernetes.io/projected/892dd988-f1a1-41d2-bd5c-9056aec2c889-kube-api-access-hn6xc\") pod \"istiod-openshift-gateway-7574459656-wmhhk\" (UID: \"892dd988-f1a1-41d2-bd5c-9056aec2c889\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-wmhhk"
Jul 14 23:02:07.244074 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:07.243967    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/892dd988-f1a1-41d2-bd5c-9056aec2c889-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-wmhhk\" (UID: \"892dd988-f1a1-41d2-bd5c-9056aec2c889\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-wmhhk"
Jul 14 23:02:07.244074 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:07.243998    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/892dd988-f1a1-41d2-bd5c-9056aec2c889-cacerts\") pod \"istiod-openshift-gateway-7574459656-wmhhk\" (UID: \"892dd988-f1a1-41d2-bd5c-9056aec2c889\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-wmhhk"
Jul 14 23:02:07.244074 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:07.244034    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/892dd988-f1a1-41d2-bd5c-9056aec2c889-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-wmhhk\" (UID: \"892dd988-f1a1-41d2-bd5c-9056aec2c889\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-wmhhk"
Jul 14 23:02:07.244336 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:07.244082    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/892dd988-f1a1-41d2-bd5c-9056aec2c889-local-certs\") pod \"istiod-openshift-gateway-7574459656-wmhhk\" (UID: \"892dd988-f1a1-41d2-bd5c-9056aec2c889\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-wmhhk"
Jul 14 23:02:07.244336 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:07.244115    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/892dd988-f1a1-41d2-bd5c-9056aec2c889-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-wmhhk\" (UID: \"892dd988-f1a1-41d2-bd5c-9056aec2c889\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-wmhhk"
Jul 14 23:02:07.245260 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:07.245205    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/892dd988-f1a1-41d2-bd5c-9056aec2c889-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-wmhhk\" (UID: \"892dd988-f1a1-41d2-bd5c-9056aec2c889\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-wmhhk"
Jul 14 23:02:07.249508 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:07.249463    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/892dd988-f1a1-41d2-bd5c-9056aec2c889-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-wmhhk\" (UID: \"892dd988-f1a1-41d2-bd5c-9056aec2c889\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-wmhhk"
Jul 14 23:02:07.250763 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:07.250671    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/892dd988-f1a1-41d2-bd5c-9056aec2c889-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-wmhhk\" (UID: \"892dd988-f1a1-41d2-bd5c-9056aec2c889\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-wmhhk"
Jul 14 23:02:07.254547 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:07.254509    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn6xc\" (UniqueName: \"kubernetes.io/projected/892dd988-f1a1-41d2-bd5c-9056aec2c889-kube-api-access-hn6xc\") pod \"istiod-openshift-gateway-7574459656-wmhhk\" (UID: \"892dd988-f1a1-41d2-bd5c-9056aec2c889\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-wmhhk"
Jul 14 23:02:07.255416 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:07.255379    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/892dd988-f1a1-41d2-bd5c-9056aec2c889-cacerts\") pod \"istiod-openshift-gateway-7574459656-wmhhk\" (UID: \"892dd988-f1a1-41d2-bd5c-9056aec2c889\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-wmhhk"
Jul 14 23:02:07.256831 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:07.256785    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/892dd988-f1a1-41d2-bd5c-9056aec2c889-local-certs\") pod \"istiod-openshift-gateway-7574459656-wmhhk\" (UID: \"892dd988-f1a1-41d2-bd5c-9056aec2c889\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-wmhhk"
Jul 14 23:02:07.257448 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:07.257408    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892dd988-f1a1-41d2-bd5c-9056aec2c889-istio-token\") pod \"istiod-openshift-gateway-7574459656-wmhhk\" (UID: \"892dd988-f1a1-41d2-bd5c-9056aec2c889\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-wmhhk"
Jul 14 23:02:07.292524 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:07.292423    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-wmhhk"
Jul 14 23:02:07.479184 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:07.479113    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-wmhhk"]
Jul 14 23:02:07.844915 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:02:07.844878    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod892dd988_f1a1_41d2_bd5c_9056aec2c889.slice/crio-7932485a853670406cfa7f91215bcc053de9d9d2e64be12fb6e05a540f5894e4 WatchSource:0}: Error finding container 7932485a853670406cfa7f91215bcc053de9d9d2e64be12fb6e05a540f5894e4: Status 404 returned error can't find the container with id 7932485a853670406cfa7f91215bcc053de9d9d2e64be12fb6e05a540f5894e4
Jul 14 23:02:08.581205 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:08.581173    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-wmhhk" event={"ID":"892dd988-f1a1-41d2-bd5c-9056aec2c889","Type":"ContainerStarted","Data":"7932485a853670406cfa7f91215bcc053de9d9d2e64be12fb6e05a540f5894e4"}
Jul 14 23:02:10.592832 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:10.592785    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-946bb8597-h8jmp" event={"ID":"4e061977-cf57-46d7-9804-ed6f91ce7e60","Type":"ContainerStarted","Data":"272f5e52703e4cc7a509b38c90a8236ff011d9d2ebea1debdca43c9b05e59710"}
Jul 14 23:02:10.594559 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:10.594531    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-9w2wb" event={"ID":"435e7622-b867-42ea-a6e6-f7e6f857972b","Type":"ContainerStarted","Data":"82dc37fd4f17204a8ad1eec618f766b12d1d945b512562204494286ac808ab98"}
Jul 14 23:02:10.594674 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:10.594623    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-9w2wb"
Jul 14 23:02:10.596128 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:10.596105    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-drvt2" event={"ID":"e6034bd0-b905-40c7-8380-eff83fd5522d","Type":"ContainerStarted","Data":"4bdc7d512650ef80cbbf0d895c76b036acb83657e66f8bebb91ee900563d513c"}
Jul 14 23:02:10.596271 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:10.596251    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-drvt2"
Jul 14 23:02:10.615229 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:10.615175    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-946bb8597-h8jmp" podStartSLOduration=2.037695237 podStartE2EDuration="5.615154488s" totalImagesPullingTime="3.577459251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:02:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:02:10.613803685 +0000 UTC m=+317.916794479" watchObservedRunningTime="2026-07-14 23:02:10.615154488 +0000 UTC m=+317.918145282"
Jul 14 23:02:10.634578 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:10.634520    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-drvt2" podStartSLOduration=2.350988849 podStartE2EDuration="6.6345006s" totalImagesPullingTime="4.283511751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:02:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:02:10.63448225 +0000 UTC m=+317.937473041" watchObservedRunningTime="2026-07-14 23:02:10.6345006 +0000 UTC m=+317.937491392"
Jul 14 23:02:10.657308 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:10.657227    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-9w2wb" podStartSLOduration=1.488417683 podStartE2EDuration="5.657206371s" totalImagesPullingTime="4.168788688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:02:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:02:10.656211096 +0000 UTC m=+317.959201888" watchObservedRunningTime="2026-07-14 23:02:10.657206371 +0000 UTC m=+317.960197164"
Jul 14 23:02:11.197949 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:11.197913    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:02:11.198010 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:11.197979    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:02:11.602084 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:11.602041    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-wmhhk" event={"ID":"892dd988-f1a1-41d2-bd5c-9056aec2c889","Type":"ContainerStarted","Data":"2d9b993c3308c76bf62c90c7d7308fc1b8a37372a3d113c57ecfd55a309d4879"}
Jul 14 23:02:11.602521 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:11.602444    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-wmhhk"
Jul 14 23:02:11.652609 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:11.652560    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-wmhhk" podStartSLOduration=2.302778664 podStartE2EDuration="5.652546898s" totalImagesPullingTime="3.349768234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:02:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:02:11.652236841 +0000 UTC m=+318.955227633" watchObservedRunningTime="2026-07-14 23:02:11.652546898 +0000 UTC m=+318.955537689"
Jul 14 23:02:12.607853 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:12.607813    2563 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-wmhhk container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 14 23:02:12.608299 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:12.607877    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-wmhhk" podUID="892dd988-f1a1-41d2-bd5c-9056aec2c889" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 14 23:02:13.610173 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:13.610143    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-wmhhk"
Jul 14 23:02:13.858019 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:13.857984    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-86f675f86-94m4p"]
Jul 14 23:02:13.860317 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:13.860257    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-86f675f86-94m4p"
Jul 14 23:02:13.865992 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:13.865962    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 14 23:02:13.866110 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:13.866013    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 14 23:02:13.866110 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:13.866028    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 14 23:02:13.866110 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:13.866058    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 14 23:02:13.866273 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:13.866112    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-fkkfj\""
Jul 14 23:02:13.866273 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:13.866169    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 14 23:02:13.873666 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:13.873644    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-86f675f86-94m4p"]
Jul 14 23:02:14.004797 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:14.004756    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a7c1c58-c882-400e-9d2b-a764c20bbbcb-metrics-cert\") pod \"lws-controller-manager-86f675f86-94m4p\" (UID: \"8a7c1c58-c882-400e-9d2b-a764c20bbbcb\") " pod="openshift-lws-operator/lws-controller-manager-86f675f86-94m4p"
Jul 14 23:02:14.004984 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:14.004816    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8a7c1c58-c882-400e-9d2b-a764c20bbbcb-manager-config\") pod \"lws-controller-manager-86f675f86-94m4p\" (UID: \"8a7c1c58-c882-400e-9d2b-a764c20bbbcb\") " pod="openshift-lws-operator/lws-controller-manager-86f675f86-94m4p"
Jul 14 23:02:14.004984 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:14.004854    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jfxp\" (UniqueName: \"kubernetes.io/projected/8a7c1c58-c882-400e-9d2b-a764c20bbbcb-kube-api-access-5jfxp\") pod \"lws-controller-manager-86f675f86-94m4p\" (UID: \"8a7c1c58-c882-400e-9d2b-a764c20bbbcb\") " pod="openshift-lws-operator/lws-controller-manager-86f675f86-94m4p"
Jul 14 23:02:14.004984 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:14.004969    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a7c1c58-c882-400e-9d2b-a764c20bbbcb-cert\") pod \"lws-controller-manager-86f675f86-94m4p\" (UID: \"8a7c1c58-c882-400e-9d2b-a764c20bbbcb\") " pod="openshift-lws-operator/lws-controller-manager-86f675f86-94m4p"
Jul 14 23:02:14.105588 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:14.105545    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a7c1c58-c882-400e-9d2b-a764c20bbbcb-metrics-cert\") pod \"lws-controller-manager-86f675f86-94m4p\" (UID: \"8a7c1c58-c882-400e-9d2b-a764c20bbbcb\") " pod="openshift-lws-operator/lws-controller-manager-86f675f86-94m4p"
Jul 14 23:02:14.105789 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:14.105594    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8a7c1c58-c882-400e-9d2b-a764c20bbbcb-manager-config\") pod \"lws-controller-manager-86f675f86-94m4p\" (UID: \"8a7c1c58-c882-400e-9d2b-a764c20bbbcb\") " pod="openshift-lws-operator/lws-controller-manager-86f675f86-94m4p"
Jul 14 23:02:14.105789 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:14.105631    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jfxp\" (UniqueName: \"kubernetes.io/projected/8a7c1c58-c882-400e-9d2b-a764c20bbbcb-kube-api-access-5jfxp\") pod \"lws-controller-manager-86f675f86-94m4p\" (UID: \"8a7c1c58-c882-400e-9d2b-a764c20bbbcb\") " pod="openshift-lws-operator/lws-controller-manager-86f675f86-94m4p"
Jul 14 23:02:14.105789 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:14.105709    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a7c1c58-c882-400e-9d2b-a764c20bbbcb-cert\") pod \"lws-controller-manager-86f675f86-94m4p\" (UID: \"8a7c1c58-c882-400e-9d2b-a764c20bbbcb\") " pod="openshift-lws-operator/lws-controller-manager-86f675f86-94m4p"
Jul 14 23:02:14.106393 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:14.106315    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8a7c1c58-c882-400e-9d2b-a764c20bbbcb-manager-config\") pod \"lws-controller-manager-86f675f86-94m4p\" (UID: \"8a7c1c58-c882-400e-9d2b-a764c20bbbcb\") " pod="openshift-lws-operator/lws-controller-manager-86f675f86-94m4p"
Jul 14 23:02:14.108306 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:14.108250    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a7c1c58-c882-400e-9d2b-a764c20bbbcb-cert\") pod \"lws-controller-manager-86f675f86-94m4p\" (UID: \"8a7c1c58-c882-400e-9d2b-a764c20bbbcb\") " pod="openshift-lws-operator/lws-controller-manager-86f675f86-94m4p"
Jul 14 23:02:14.108306 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:14.108267    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a7c1c58-c882-400e-9d2b-a764c20bbbcb-metrics-cert\") pod \"lws-controller-manager-86f675f86-94m4p\" (UID: \"8a7c1c58-c882-400e-9d2b-a764c20bbbcb\") " pod="openshift-lws-operator/lws-controller-manager-86f675f86-94m4p"
Jul 14 23:02:14.127651 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:14.127594    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jfxp\" (UniqueName: \"kubernetes.io/projected/8a7c1c58-c882-400e-9d2b-a764c20bbbcb-kube-api-access-5jfxp\") pod \"lws-controller-manager-86f675f86-94m4p\" (UID: \"8a7c1c58-c882-400e-9d2b-a764c20bbbcb\") " pod="openshift-lws-operator/lws-controller-manager-86f675f86-94m4p"
Jul 14 23:02:14.169492 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:14.169469    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-86f675f86-94m4p"
Jul 14 23:02:14.328970 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:14.328945    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-86f675f86-94m4p"]
Jul 14 23:02:14.329926 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:02:14.329903    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a7c1c58_c882_400e_9d2b_a764c20bbbcb.slice/crio-76b5ec09c2e58d2f0ad9b9b1605e3f66e15abf344fb19bf16d43d20eddbbb279 WatchSource:0}: Error finding container 76b5ec09c2e58d2f0ad9b9b1605e3f66e15abf344fb19bf16d43d20eddbbb279: Status 404 returned error can't find the container with id 76b5ec09c2e58d2f0ad9b9b1605e3f66e15abf344fb19bf16d43d20eddbbb279
Jul 14 23:02:14.614978 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:14.614945    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-86f675f86-94m4p" event={"ID":"8a7c1c58-c882-400e-9d2b-a764c20bbbcb","Type":"ContainerStarted","Data":"76b5ec09c2e58d2f0ad9b9b1605e3f66e15abf344fb19bf16d43d20eddbbb279"}
Jul 14 23:02:16.419430 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:16.419389    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-wmhhk"]
Jul 14 23:02:16.419886 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:16.419614    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-wmhhk" podUID="892dd988-f1a1-41d2-bd5c-9056aec2c889" containerName="discovery" containerID="cri-o://2d9b993c3308c76bf62c90c7d7308fc1b8a37372a3d113c57ecfd55a309d4879" gracePeriod=30
Jul 14 23:02:16.626116 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:16.626062    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-86f675f86-94m4p" event={"ID":"8a7c1c58-c882-400e-9d2b-a764c20bbbcb","Type":"ContainerStarted","Data":"348b754d14af1df917ddf3e3cdfeff5366fdb2f605d7d32c00eb66144745d912"}
Jul 14 23:02:16.628057 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:16.628030    2563 generic.go:356] "Generic (PLEG): container finished" podID="892dd988-f1a1-41d2-bd5c-9056aec2c889" containerID="2d9b993c3308c76bf62c90c7d7308fc1b8a37372a3d113c57ecfd55a309d4879" exitCode=0
Jul 14 23:02:16.628173 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:16.628063    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-wmhhk" event={"ID":"892dd988-f1a1-41d2-bd5c-9056aec2c889","Type":"ContainerDied","Data":"2d9b993c3308c76bf62c90c7d7308fc1b8a37372a3d113c57ecfd55a309d4879"}
Jul 14 23:02:16.650667 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:16.650620    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-86f675f86-94m4p" podStartSLOduration=1.557740142 podStartE2EDuration="3.65060636s" totalImagesPullingTime="2.092866218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:02:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:02:16.649801643 +0000 UTC m=+323.952792434" watchObservedRunningTime="2026-07-14 23:02:16.65060636 +0000 UTC m=+323.953597152"
Jul 14 23:02:16.672172 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:16.672110    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-wmhhk"
Jul 14 23:02:16.731837 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:16.731811    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/892dd988-f1a1-41d2-bd5c-9056aec2c889-istio-csr-ca-configmap\") pod \"892dd988-f1a1-41d2-bd5c-9056aec2c889\" (UID: \"892dd988-f1a1-41d2-bd5c-9056aec2c889\") "
Jul 14 23:02:16.732004 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:16.731844    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/892dd988-f1a1-41d2-bd5c-9056aec2c889-istio-kubeconfig\") pod \"892dd988-f1a1-41d2-bd5c-9056aec2c889\" (UID: \"892dd988-f1a1-41d2-bd5c-9056aec2c889\") "
Jul 14 23:02:16.732004 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:16.731877    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hn6xc\" (UniqueName: \"kubernetes.io/projected/892dd988-f1a1-41d2-bd5c-9056aec2c889-kube-api-access-hn6xc\") pod \"892dd988-f1a1-41d2-bd5c-9056aec2c889\" (UID: \"892dd988-f1a1-41d2-bd5c-9056aec2c889\") "
Jul 14 23:02:16.732004 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:16.731902    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/892dd988-f1a1-41d2-bd5c-9056aec2c889-local-certs\") pod \"892dd988-f1a1-41d2-bd5c-9056aec2c889\" (UID: \"892dd988-f1a1-41d2-bd5c-9056aec2c889\") "
Jul 14 23:02:16.732004 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:16.731916    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/892dd988-f1a1-41d2-bd5c-9056aec2c889-istio-csr-dns-cert\") pod \"892dd988-f1a1-41d2-bd5c-9056aec2c889\" (UID: \"892dd988-f1a1-41d2-bd5c-9056aec2c889\") "
Jul 14 23:02:16.732004 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:16.731984    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/892dd988-f1a1-41d2-bd5c-9056aec2c889-cacerts\") pod \"892dd988-f1a1-41d2-bd5c-9056aec2c889\" (UID: \"892dd988-f1a1-41d2-bd5c-9056aec2c889\") "
Jul 14 23:02:16.732252 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:16.732010    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892dd988-f1a1-41d2-bd5c-9056aec2c889-istio-token\") pod \"892dd988-f1a1-41d2-bd5c-9056aec2c889\" (UID: \"892dd988-f1a1-41d2-bd5c-9056aec2c889\") "
Jul 14 23:02:16.732340 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:16.732276    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/892dd988-f1a1-41d2-bd5c-9056aec2c889-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "892dd988-f1a1-41d2-bd5c-9056aec2c889" (UID: "892dd988-f1a1-41d2-bd5c-9056aec2c889"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 23:02:16.734529 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:16.734503    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/892dd988-f1a1-41d2-bd5c-9056aec2c889-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "892dd988-f1a1-41d2-bd5c-9056aec2c889" (UID: "892dd988-f1a1-41d2-bd5c-9056aec2c889"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 23:02:16.734650 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:16.734550    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/892dd988-f1a1-41d2-bd5c-9056aec2c889-cacerts" (OuterVolumeSpecName: "cacerts") pod "892dd988-f1a1-41d2-bd5c-9056aec2c889" (UID: "892dd988-f1a1-41d2-bd5c-9056aec2c889"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 23:02:16.734650 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:16.734633    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/892dd988-f1a1-41d2-bd5c-9056aec2c889-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "892dd988-f1a1-41d2-bd5c-9056aec2c889" (UID: "892dd988-f1a1-41d2-bd5c-9056aec2c889"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 23:02:16.734730 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:16.734650    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/892dd988-f1a1-41d2-bd5c-9056aec2c889-kube-api-access-hn6xc" (OuterVolumeSpecName: "kube-api-access-hn6xc") pod "892dd988-f1a1-41d2-bd5c-9056aec2c889" (UID: "892dd988-f1a1-41d2-bd5c-9056aec2c889"). InnerVolumeSpecName "kube-api-access-hn6xc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:02:16.734786 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:16.734769    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/892dd988-f1a1-41d2-bd5c-9056aec2c889-istio-token" (OuterVolumeSpecName: "istio-token") pod "892dd988-f1a1-41d2-bd5c-9056aec2c889" (UID: "892dd988-f1a1-41d2-bd5c-9056aec2c889"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:02:16.734831 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:16.734813    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892dd988-f1a1-41d2-bd5c-9056aec2c889-local-certs" (OuterVolumeSpecName: "local-certs") pod "892dd988-f1a1-41d2-bd5c-9056aec2c889" (UID: "892dd988-f1a1-41d2-bd5c-9056aec2c889"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:02:16.832837 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:16.832805    2563 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/892dd988-f1a1-41d2-bd5c-9056aec2c889-cacerts\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:02:16.832837 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:16.832834    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892dd988-f1a1-41d2-bd5c-9056aec2c889-istio-token\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:02:16.833019 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:16.832845    2563 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/892dd988-f1a1-41d2-bd5c-9056aec2c889-istio-csr-ca-configmap\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:02:16.833019 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:16.832873    2563 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/892dd988-f1a1-41d2-bd5c-9056aec2c889-istio-kubeconfig\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:02:16.833019 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:16.832882    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn6xc\" (UniqueName: \"kubernetes.io/projected/892dd988-f1a1-41d2-bd5c-9056aec2c889-kube-api-access-hn6xc\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:02:16.833019 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:16.832891    2563 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/892dd988-f1a1-41d2-bd5c-9056aec2c889-local-certs\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:02:16.833019 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:16.832899    2563 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/892dd988-f1a1-41d2-bd5c-9056aec2c889-istio-csr-dns-cert\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:02:17.633616 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:17.633583    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-wmhhk"
Jul 14 23:02:17.634078 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:17.633621    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-wmhhk" event={"ID":"892dd988-f1a1-41d2-bd5c-9056aec2c889","Type":"ContainerDied","Data":"7932485a853670406cfa7f91215bcc053de9d9d2e64be12fb6e05a540f5894e4"}
Jul 14 23:02:17.634078 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:17.633678    2563 scope.go:117] "RemoveContainer" containerID="2d9b993c3308c76bf62c90c7d7308fc1b8a37372a3d113c57ecfd55a309d4879"
Jul 14 23:02:17.634078 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:17.633993    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-86f675f86-94m4p"
Jul 14 23:02:17.668163 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:17.668132    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-wmhhk"]
Jul 14 23:02:17.674581 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:17.674550    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-wmhhk"]
Jul 14 23:02:19.229130 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:19.229091    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="892dd988-f1a1-41d2-bd5c-9056aec2c889" path="/var/lib/kubelet/pods/892dd988-f1a1-41d2-bd5c-9056aec2c889/volumes"
Jul 14 23:02:21.604465 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:21.604435    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-drvt2"
Jul 14 23:02:21.607168 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:21.607146    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-9w2wb"
Jul 14 23:02:21.609784 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:21.609763    2563 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-wmhhk container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.27:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 14 23:02:21.609838 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:21.609809    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-wmhhk" podUID="892dd988-f1a1-41d2-bd5c-9056aec2c889" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.27:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 14 23:02:28.643225 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:02:28.643192    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-86f675f86-94m4p"
Jul 14 23:03:56.398010 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:03:56.397979    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-8f4gh"]
Jul 14 23:03:56.398474 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:03:56.398310    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="892dd988-f1a1-41d2-bd5c-9056aec2c889" containerName="discovery"
Jul 14 23:03:56.398474 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:03:56.398324    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="892dd988-f1a1-41d2-bd5c-9056aec2c889" containerName="discovery"
Jul 14 23:03:56.398474 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:03:56.398396    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="892dd988-f1a1-41d2-bd5c-9056aec2c889" containerName="discovery"
Jul 14 23:03:56.400544 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:03:56.400528    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8f4gh"
Jul 14 23:03:56.403717 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:03:56.403691    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 14 23:03:56.403881 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:03:56.403717    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 14 23:03:56.403881 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:03:56.403693    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 14 23:03:56.403881 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:03:56.403694    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 14 23:03:56.405380 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:03:56.405362    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-c524q\""
Jul 14 23:03:56.410861 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:03:56.410837    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-8f4gh"]
Jul 14 23:03:56.539485 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:03:56.539446    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cdf2efa0-3d6a-4168-ba16-fb365f021f1c-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-8f4gh\" (UID: \"cdf2efa0-3d6a-4168-ba16-fb365f021f1c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8f4gh"
Jul 14 23:03:56.539676 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:03:56.539515    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7b25\" (UniqueName: \"kubernetes.io/projected/cdf2efa0-3d6a-4168-ba16-fb365f021f1c-kube-api-access-p7b25\") pod \"kuadrant-console-plugin-66b5774985-8f4gh\" (UID: \"cdf2efa0-3d6a-4168-ba16-fb365f021f1c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8f4gh"
Jul 14 23:03:56.539676 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:03:56.539554    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/cdf2efa0-3d6a-4168-ba16-fb365f021f1c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-8f4gh\" (UID: \"cdf2efa0-3d6a-4168-ba16-fb365f021f1c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8f4gh"
Jul 14 23:03:56.639986 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:03:56.639949    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7b25\" (UniqueName: \"kubernetes.io/projected/cdf2efa0-3d6a-4168-ba16-fb365f021f1c-kube-api-access-p7b25\") pod \"kuadrant-console-plugin-66b5774985-8f4gh\" (UID: \"cdf2efa0-3d6a-4168-ba16-fb365f021f1c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8f4gh"
Jul 14 23:03:56.640168 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:03:56.639994    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/cdf2efa0-3d6a-4168-ba16-fb365f021f1c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-8f4gh\" (UID: \"cdf2efa0-3d6a-4168-ba16-fb365f021f1c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8f4gh"
Jul 14 23:03:56.640168 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:03:56.640035    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cdf2efa0-3d6a-4168-ba16-fb365f021f1c-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-8f4gh\" (UID: \"cdf2efa0-3d6a-4168-ba16-fb365f021f1c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8f4gh"
Jul 14 23:03:56.640665 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:03:56.640643    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cdf2efa0-3d6a-4168-ba16-fb365f021f1c-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-8f4gh\" (UID: \"cdf2efa0-3d6a-4168-ba16-fb365f021f1c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8f4gh"
Jul 14 23:03:56.642644 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:03:56.642620    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/cdf2efa0-3d6a-4168-ba16-fb365f021f1c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-8f4gh\" (UID: \"cdf2efa0-3d6a-4168-ba16-fb365f021f1c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8f4gh"
Jul 14 23:03:56.654269 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:03:56.654200    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7b25\" (UniqueName: \"kubernetes.io/projected/cdf2efa0-3d6a-4168-ba16-fb365f021f1c-kube-api-access-p7b25\") pod \"kuadrant-console-plugin-66b5774985-8f4gh\" (UID: \"cdf2efa0-3d6a-4168-ba16-fb365f021f1c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8f4gh"
Jul 14 23:03:56.709543 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:03:56.709502    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8f4gh"
Jul 14 23:03:56.853161 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:03:56.853133    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-8f4gh"]
Jul 14 23:03:56.855808 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:03:56.855778    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcdf2efa0_3d6a_4168_ba16_fb365f021f1c.slice/crio-e841916bf1b49c50903513ad9480a6219df70d222eeaedc0d880ca3a2a48292e WatchSource:0}: Error finding container e841916bf1b49c50903513ad9480a6219df70d222eeaedc0d880ca3a2a48292e: Status 404 returned error can't find the container with id e841916bf1b49c50903513ad9480a6219df70d222eeaedc0d880ca3a2a48292e
Jul 14 23:03:57.034651 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:03:57.034612    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8f4gh" event={"ID":"cdf2efa0-3d6a-4168-ba16-fb365f021f1c","Type":"ContainerStarted","Data":"e841916bf1b49c50903513ad9480a6219df70d222eeaedc0d880ca3a2a48292e"}
Jul 14 23:04:21.149515 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:04:21.149477    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8f4gh" event={"ID":"cdf2efa0-3d6a-4168-ba16-fb365f021f1c","Type":"ContainerStarted","Data":"37d14c7391f22974f6f5ca9b4bee4e05dddc355bedd13f2dc819f01c50c014bb"}
Jul 14 23:04:21.166461 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:04:21.166401    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8f4gh" podStartSLOduration=1.344189098 podStartE2EDuration="25.166385314s" totalImagesPullingTime="23.822196216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:03:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:04:21.165391349 +0000 UTC m=+448.468382154" watchObservedRunningTime="2026-07-14 23:04:21.166385314 +0000 UTC m=+448.469376115"
Jul 14 23:04:53.932293 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:04:53.932246    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-fh827"]
Jul 14 23:04:53.962089 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:04:53.962051    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-fh827"]
Jul 14 23:04:53.962259 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:04:53.962152    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-fh827"
Jul 14 23:04:53.965239 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:04:53.965217    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-8zh2s\""
Jul 14 23:04:54.091019 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:04:54.090986    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-wtg6d"]
Jul 14 23:04:54.093507 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:04:54.093490    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-wtg6d"
Jul 14 23:04:54.100831 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:04:54.100802    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-wtg6d"]
Jul 14 23:04:54.140565 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:04:54.140521    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc6tg\" (UniqueName: \"kubernetes.io/projected/93b7f7a6-d691-44c5-b21a-68c43e385d2e-kube-api-access-sc6tg\") pod \"authorino-58cd955675-fh827\" (UID: \"93b7f7a6-d691-44c5-b21a-68c43e385d2e\") " pod="kuadrant-system/authorino-58cd955675-fh827"
Jul 14 23:04:54.241785 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:04:54.241749    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kgcw\" (UniqueName: \"kubernetes.io/projected/c9fe72a4-eed0-4314-b0f3-4568afc2e120-kube-api-access-4kgcw\") pod \"authorino-75d556bd77-wtg6d\" (UID: \"c9fe72a4-eed0-4314-b0f3-4568afc2e120\") " pod="kuadrant-system/authorino-75d556bd77-wtg6d"
Jul 14 23:04:54.241948 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:04:54.241814    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc6tg\" (UniqueName: \"kubernetes.io/projected/93b7f7a6-d691-44c5-b21a-68c43e385d2e-kube-api-access-sc6tg\") pod \"authorino-58cd955675-fh827\" (UID: \"93b7f7a6-d691-44c5-b21a-68c43e385d2e\") " pod="kuadrant-system/authorino-58cd955675-fh827"
Jul 14 23:04:54.251634 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:04:54.251603    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc6tg\" (UniqueName: \"kubernetes.io/projected/93b7f7a6-d691-44c5-b21a-68c43e385d2e-kube-api-access-sc6tg\") pod \"authorino-58cd955675-fh827\" (UID: \"93b7f7a6-d691-44c5-b21a-68c43e385d2e\") " pod="kuadrant-system/authorino-58cd955675-fh827"
Jul 14 23:04:54.271010 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:04:54.270980    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-fh827"
Jul 14 23:04:54.342452 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:04:54.342415    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kgcw\" (UniqueName: \"kubernetes.io/projected/c9fe72a4-eed0-4314-b0f3-4568afc2e120-kube-api-access-4kgcw\") pod \"authorino-75d556bd77-wtg6d\" (UID: \"c9fe72a4-eed0-4314-b0f3-4568afc2e120\") " pod="kuadrant-system/authorino-75d556bd77-wtg6d"
Jul 14 23:04:54.352085 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:04:54.352015    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kgcw\" (UniqueName: \"kubernetes.io/projected/c9fe72a4-eed0-4314-b0f3-4568afc2e120-kube-api-access-4kgcw\") pod \"authorino-75d556bd77-wtg6d\" (UID: \"c9fe72a4-eed0-4314-b0f3-4568afc2e120\") " pod="kuadrant-system/authorino-75d556bd77-wtg6d"
Jul 14 23:04:54.402624 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:04:54.402589    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-wtg6d"
Jul 14 23:04:54.410581 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:04:54.410561    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-fh827"]
Jul 14 23:04:54.413374 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:04:54.413307    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93b7f7a6_d691_44c5_b21a_68c43e385d2e.slice/crio-6c32c04fdfd8d553e369a3088f2f37be13e4b58ef06430bf5fcad3f6a552b4ad WatchSource:0}: Error finding container 6c32c04fdfd8d553e369a3088f2f37be13e4b58ef06430bf5fcad3f6a552b4ad: Status 404 returned error can't find the container with id 6c32c04fdfd8d553e369a3088f2f37be13e4b58ef06430bf5fcad3f6a552b4ad
Jul 14 23:04:54.539150 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:04:54.539120    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-wtg6d"]
Jul 14 23:04:54.541543 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:04:54.541515    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9fe72a4_eed0_4314_b0f3_4568afc2e120.slice/crio-b74cc9fad9fab480d4816b09fdab163052517b994a0eb7acabafb9ad8b2a4bb5 WatchSource:0}: Error finding container b74cc9fad9fab480d4816b09fdab163052517b994a0eb7acabafb9ad8b2a4bb5: Status 404 returned error can't find the container with id b74cc9fad9fab480d4816b09fdab163052517b994a0eb7acabafb9ad8b2a4bb5
Jul 14 23:04:55.296147 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:04:55.296105    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-wtg6d" event={"ID":"c9fe72a4-eed0-4314-b0f3-4568afc2e120","Type":"ContainerStarted","Data":"b74cc9fad9fab480d4816b09fdab163052517b994a0eb7acabafb9ad8b2a4bb5"}
Jul 14 23:04:55.298825 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:04:55.298757    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-fh827" event={"ID":"93b7f7a6-d691-44c5-b21a-68c43e385d2e","Type":"ContainerStarted","Data":"6c32c04fdfd8d553e369a3088f2f37be13e4b58ef06430bf5fcad3f6a552b4ad"}
Jul 14 23:04:58.316464 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:04:58.316421    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-fh827" event={"ID":"93b7f7a6-d691-44c5-b21a-68c43e385d2e","Type":"ContainerStarted","Data":"47cb1d9666e7be1304903c776176ae05c4aebb2b9851c89e21ef1d5157eb5b15"}
Jul 14 23:04:58.317965 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:04:58.317941    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-wtg6d" event={"ID":"c9fe72a4-eed0-4314-b0f3-4568afc2e120","Type":"ContainerStarted","Data":"87da8d800acef5fcc508effd8121c28b179a3b3d57ec78333598f0f47e0cb71f"}
Jul 14 23:04:58.334209 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:04:58.334160    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-fh827" podStartSLOduration=1.799128317 podStartE2EDuration="5.334146144s" totalImagesPullingTime="3.535017827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:04:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:04:58.333471615 +0000 UTC m=+485.636462407" watchObservedRunningTime="2026-07-14 23:04:58.334146144 +0000 UTC m=+485.637136935"
Jul 14 23:04:58.353395 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:04:58.353338    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-wtg6d" podStartSLOduration=0.934153924 podStartE2EDuration="4.353314096s" totalImagesPullingTime="3.419160172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:04:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:04:58.352446186 +0000 UTC m=+485.655436979" watchObservedRunningTime="2026-07-14 23:04:58.353314096 +0000 UTC m=+485.656304893"
Jul 14 23:04:58.386970 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:04:58.386938    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-fh827"]
Jul 14 23:05:00.325991 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:00.325953    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-fh827" podUID="93b7f7a6-d691-44c5-b21a-68c43e385d2e" containerName="authorino" containerID="cri-o://47cb1d9666e7be1304903c776176ae05c4aebb2b9851c89e21ef1d5157eb5b15" gracePeriod=30
Jul 14 23:05:00.580273 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:00.580212    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-fh827"
Jul 14 23:05:00.589097 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:00.589076    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sc6tg\" (UniqueName: \"kubernetes.io/projected/93b7f7a6-d691-44c5-b21a-68c43e385d2e-kube-api-access-sc6tg\") pod \"93b7f7a6-d691-44c5-b21a-68c43e385d2e\" (UID: \"93b7f7a6-d691-44c5-b21a-68c43e385d2e\") "
Jul 14 23:05:00.591294 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:00.591258    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93b7f7a6-d691-44c5-b21a-68c43e385d2e-kube-api-access-sc6tg" (OuterVolumeSpecName: "kube-api-access-sc6tg") pod "93b7f7a6-d691-44c5-b21a-68c43e385d2e" (UID: "93b7f7a6-d691-44c5-b21a-68c43e385d2e"). InnerVolumeSpecName "kube-api-access-sc6tg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:05:00.689615 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:00.689580    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc6tg\" (UniqueName: \"kubernetes.io/projected/93b7f7a6-d691-44c5-b21a-68c43e385d2e-kube-api-access-sc6tg\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:05:01.331544 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:01.331502    2563 generic.go:356] "Generic (PLEG): container finished" podID="93b7f7a6-d691-44c5-b21a-68c43e385d2e" containerID="47cb1d9666e7be1304903c776176ae05c4aebb2b9851c89e21ef1d5157eb5b15" exitCode=0
Jul 14 23:05:01.331940 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:01.331561    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-fh827"
Jul 14 23:05:01.331940 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:01.331576    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-fh827" event={"ID":"93b7f7a6-d691-44c5-b21a-68c43e385d2e","Type":"ContainerDied","Data":"47cb1d9666e7be1304903c776176ae05c4aebb2b9851c89e21ef1d5157eb5b15"}
Jul 14 23:05:01.331940 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:01.331613    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-fh827" event={"ID":"93b7f7a6-d691-44c5-b21a-68c43e385d2e","Type":"ContainerDied","Data":"6c32c04fdfd8d553e369a3088f2f37be13e4b58ef06430bf5fcad3f6a552b4ad"}
Jul 14 23:05:01.331940 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:01.331632    2563 scope.go:117] "RemoveContainer" containerID="47cb1d9666e7be1304903c776176ae05c4aebb2b9851c89e21ef1d5157eb5b15"
Jul 14 23:05:01.339615 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:01.339596    2563 scope.go:117] "RemoveContainer" containerID="47cb1d9666e7be1304903c776176ae05c4aebb2b9851c89e21ef1d5157eb5b15"
Jul 14 23:05:01.339865 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:05:01.339845    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47cb1d9666e7be1304903c776176ae05c4aebb2b9851c89e21ef1d5157eb5b15\": container with ID starting with 47cb1d9666e7be1304903c776176ae05c4aebb2b9851c89e21ef1d5157eb5b15 not found: ID does not exist" containerID="47cb1d9666e7be1304903c776176ae05c4aebb2b9851c89e21ef1d5157eb5b15"
Jul 14 23:05:01.339925 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:01.339877    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47cb1d9666e7be1304903c776176ae05c4aebb2b9851c89e21ef1d5157eb5b15"} err="failed to get container status \"47cb1d9666e7be1304903c776176ae05c4aebb2b9851c89e21ef1d5157eb5b15\": rpc error: code = NotFound desc = could not find container \"47cb1d9666e7be1304903c776176ae05c4aebb2b9851c89e21ef1d5157eb5b15\": container with ID starting with 47cb1d9666e7be1304903c776176ae05c4aebb2b9851c89e21ef1d5157eb5b15 not found: ID does not exist"
Jul 14 23:05:01.349544 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:01.349521    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-fh827"]
Jul 14 23:05:01.352852 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:01.352829    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-fh827"]
Jul 14 23:05:03.228811 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:03.228777    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93b7f7a6-d691-44c5-b21a-68c43e385d2e" path="/var/lib/kubelet/pods/93b7f7a6-d691-44c5-b21a-68c43e385d2e/volumes"
Jul 14 23:05:25.912910 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:25.912871    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-d7hjq"]
Jul 14 23:05:25.913420 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:25.913150    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93b7f7a6-d691-44c5-b21a-68c43e385d2e" containerName="authorino"
Jul 14 23:05:25.913420 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:25.913162    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="93b7f7a6-d691-44c5-b21a-68c43e385d2e" containerName="authorino"
Jul 14 23:05:25.913420 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:25.913229    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="93b7f7a6-d691-44c5-b21a-68c43e385d2e" containerName="authorino"
Jul 14 23:05:25.916069 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:25.916052    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-d7hjq"
Jul 14 23:05:25.923221 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:25.923198    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-d7hjq"]
Jul 14 23:05:25.977678 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:25.977641    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6lws\" (UniqueName: \"kubernetes.io/projected/39fc194e-5eff-438e-897c-ecfed4775d78-kube-api-access-z6lws\") pod \"authorino-5d4b67c446-d7hjq\" (UID: \"39fc194e-5eff-438e-897c-ecfed4775d78\") " pod="kuadrant-system/authorino-5d4b67c446-d7hjq"
Jul 14 23:05:26.079119 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:26.079083    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6lws\" (UniqueName: \"kubernetes.io/projected/39fc194e-5eff-438e-897c-ecfed4775d78-kube-api-access-z6lws\") pod \"authorino-5d4b67c446-d7hjq\" (UID: \"39fc194e-5eff-438e-897c-ecfed4775d78\") " pod="kuadrant-system/authorino-5d4b67c446-d7hjq"
Jul 14 23:05:26.083308 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:26.083264    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-d7hjq"]
Jul 14 23:05:26.083496 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:05:26.083477    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-z6lws], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-d7hjq" podUID="39fc194e-5eff-438e-897c-ecfed4775d78"
Jul 14 23:05:26.088944 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:26.088922    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6lws\" (UniqueName: \"kubernetes.io/projected/39fc194e-5eff-438e-897c-ecfed4775d78-kube-api-access-z6lws\") pod \"authorino-5d4b67c446-d7hjq\" (UID: \"39fc194e-5eff-438e-897c-ecfed4775d78\") " pod="kuadrant-system/authorino-5d4b67c446-d7hjq"
Jul 14 23:05:26.111486 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:26.111458    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-lkvbw"]
Jul 14 23:05:26.114475 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:26.114460    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-lkvbw"
Jul 14 23:05:26.117915 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:26.117893    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 14 23:05:26.125078 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:26.125056    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-lkvbw"]
Jul 14 23:05:26.179727 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:26.179630    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9527df3f-bb02-4126-83e6-db7c2583662d-tls-cert\") pod \"authorino-76877994-lkvbw\" (UID: \"9527df3f-bb02-4126-83e6-db7c2583662d\") " pod="kuadrant-system/authorino-76877994-lkvbw"
Jul 14 23:05:26.179727 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:26.179677    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmgrk\" (UniqueName: \"kubernetes.io/projected/9527df3f-bb02-4126-83e6-db7c2583662d-kube-api-access-cmgrk\") pod \"authorino-76877994-lkvbw\" (UID: \"9527df3f-bb02-4126-83e6-db7c2583662d\") " pod="kuadrant-system/authorino-76877994-lkvbw"
Jul 14 23:05:26.198057 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:26.198030    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-lkvbw"]
Jul 14 23:05:26.198302 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:05:26.198268    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-cmgrk tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-76877994-lkvbw" podUID="9527df3f-bb02-4126-83e6-db7c2583662d"
Jul 14 23:05:26.264455 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:26.264406    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b778f979c-wzcrq"]
Jul 14 23:05:26.267327 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:26.267303    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b778f979c-wzcrq"
Jul 14 23:05:26.280748 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:26.280721    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9527df3f-bb02-4126-83e6-db7c2583662d-tls-cert\") pod \"authorino-76877994-lkvbw\" (UID: \"9527df3f-bb02-4126-83e6-db7c2583662d\") " pod="kuadrant-system/authorino-76877994-lkvbw"
Jul 14 23:05:26.280866 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:26.280774    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmgrk\" (UniqueName: \"kubernetes.io/projected/9527df3f-bb02-4126-83e6-db7c2583662d-kube-api-access-cmgrk\") pod \"authorino-76877994-lkvbw\" (UID: \"9527df3f-bb02-4126-83e6-db7c2583662d\") " pod="kuadrant-system/authorino-76877994-lkvbw"
Jul 14 23:05:26.282422 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:26.282390    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b778f979c-wzcrq"]
Jul 14 23:05:26.283719 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:26.283698    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9527df3f-bb02-4126-83e6-db7c2583662d-tls-cert\") pod \"authorino-76877994-lkvbw\" (UID: \"9527df3f-bb02-4126-83e6-db7c2583662d\") " pod="kuadrant-system/authorino-76877994-lkvbw"
Jul 14 23:05:26.299034 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:26.299007    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmgrk\" (UniqueName: \"kubernetes.io/projected/9527df3f-bb02-4126-83e6-db7c2583662d-kube-api-access-cmgrk\") pod \"authorino-76877994-lkvbw\" (UID: \"9527df3f-bb02-4126-83e6-db7c2583662d\") " pod="kuadrant-system/authorino-76877994-lkvbw"
Jul 14 23:05:26.381904 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:26.381870    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/83daf71c-20f0-4a07-b209-72bc932a5ab2-tls-cert\") pod \"authorino-b778f979c-wzcrq\" (UID: \"83daf71c-20f0-4a07-b209-72bc932a5ab2\") " pod="kuadrant-system/authorino-b778f979c-wzcrq"
Jul 14 23:05:26.382085 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:26.381934    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqmtm\" (UniqueName: \"kubernetes.io/projected/83daf71c-20f0-4a07-b209-72bc932a5ab2-kube-api-access-dqmtm\") pod \"authorino-b778f979c-wzcrq\" (UID: \"83daf71c-20f0-4a07-b209-72bc932a5ab2\") " pod="kuadrant-system/authorino-b778f979c-wzcrq"
Jul 14 23:05:26.434778 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:26.434702    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-lkvbw"
Jul 14 23:05:26.434906 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:26.434702    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-d7hjq"
Jul 14 23:05:26.441554 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:26.441534    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-lkvbw"
Jul 14 23:05:26.444347 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:26.444330    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-d7hjq"
Jul 14 23:05:26.482915 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:26.482889    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9527df3f-bb02-4126-83e6-db7c2583662d-tls-cert\") pod \"9527df3f-bb02-4126-83e6-db7c2583662d\" (UID: \"9527df3f-bb02-4126-83e6-db7c2583662d\") "
Jul 14 23:05:26.483046 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:26.482963    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6lws\" (UniqueName: \"kubernetes.io/projected/39fc194e-5eff-438e-897c-ecfed4775d78-kube-api-access-z6lws\") pod \"39fc194e-5eff-438e-897c-ecfed4775d78\" (UID: \"39fc194e-5eff-438e-897c-ecfed4775d78\") "
Jul 14 23:05:26.483046 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:26.482997    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmgrk\" (UniqueName: \"kubernetes.io/projected/9527df3f-bb02-4126-83e6-db7c2583662d-kube-api-access-cmgrk\") pod \"9527df3f-bb02-4126-83e6-db7c2583662d\" (UID: \"9527df3f-bb02-4126-83e6-db7c2583662d\") "
Jul 14 23:05:26.483122 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:26.483076    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqmtm\" (UniqueName: \"kubernetes.io/projected/83daf71c-20f0-4a07-b209-72bc932a5ab2-kube-api-access-dqmtm\") pod \"authorino-b778f979c-wzcrq\" (UID: \"83daf71c-20f0-4a07-b209-72bc932a5ab2\") " pod="kuadrant-system/authorino-b778f979c-wzcrq"
Jul 14 23:05:26.483154 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:26.483145    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/83daf71c-20f0-4a07-b209-72bc932a5ab2-tls-cert\") pod \"authorino-b778f979c-wzcrq\" (UID: \"83daf71c-20f0-4a07-b209-72bc932a5ab2\") " pod="kuadrant-system/authorino-b778f979c-wzcrq"
Jul 14 23:05:26.485298 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:26.485248    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9527df3f-bb02-4126-83e6-db7c2583662d-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "9527df3f-bb02-4126-83e6-db7c2583662d" (UID: "9527df3f-bb02-4126-83e6-db7c2583662d"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 23:05:26.485394 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:26.485327    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9527df3f-bb02-4126-83e6-db7c2583662d-kube-api-access-cmgrk" (OuterVolumeSpecName: "kube-api-access-cmgrk") pod "9527df3f-bb02-4126-83e6-db7c2583662d" (UID: "9527df3f-bb02-4126-83e6-db7c2583662d"). InnerVolumeSpecName "kube-api-access-cmgrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:05:26.485720 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:26.485703    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39fc194e-5eff-438e-897c-ecfed4775d78-kube-api-access-z6lws" (OuterVolumeSpecName: "kube-api-access-z6lws") pod "39fc194e-5eff-438e-897c-ecfed4775d78" (UID: "39fc194e-5eff-438e-897c-ecfed4775d78"). InnerVolumeSpecName "kube-api-access-z6lws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:05:26.486164 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:26.486144    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/83daf71c-20f0-4a07-b209-72bc932a5ab2-tls-cert\") pod \"authorino-b778f979c-wzcrq\" (UID: \"83daf71c-20f0-4a07-b209-72bc932a5ab2\") " pod="kuadrant-system/authorino-b778f979c-wzcrq"
Jul 14 23:05:26.491480 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:26.491456    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqmtm\" (UniqueName: \"kubernetes.io/projected/83daf71c-20f0-4a07-b209-72bc932a5ab2-kube-api-access-dqmtm\") pod \"authorino-b778f979c-wzcrq\" (UID: \"83daf71c-20f0-4a07-b209-72bc932a5ab2\") " pod="kuadrant-system/authorino-b778f979c-wzcrq"
Jul 14 23:05:26.575630 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:26.575602    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b778f979c-wzcrq"
Jul 14 23:05:26.584336 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:26.584315    2563 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9527df3f-bb02-4126-83e6-db7c2583662d-tls-cert\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:05:26.584336 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:26.584335    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6lws\" (UniqueName: \"kubernetes.io/projected/39fc194e-5eff-438e-897c-ecfed4775d78-kube-api-access-z6lws\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:05:26.584465 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:26.584346    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmgrk\" (UniqueName: \"kubernetes.io/projected/9527df3f-bb02-4126-83e6-db7c2583662d-kube-api-access-cmgrk\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:05:26.705335 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:26.705228    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b778f979c-wzcrq"]
Jul 14 23:05:26.708121 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:05:26.708091    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83daf71c_20f0_4a07_b209_72bc932a5ab2.slice/crio-6350bbab6e630c273522663398af417fe0d0fbe73ac08789d20ae84b82a8c5da WatchSource:0}: Error finding container 6350bbab6e630c273522663398af417fe0d0fbe73ac08789d20ae84b82a8c5da: Status 404 returned error can't find the container with id 6350bbab6e630c273522663398af417fe0d0fbe73ac08789d20ae84b82a8c5da
Jul 14 23:05:27.442039 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:27.441965    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-lkvbw"
Jul 14 23:05:27.442039 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:27.441980    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-d7hjq"
Jul 14 23:05:27.442039 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:27.441959    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b778f979c-wzcrq" event={"ID":"83daf71c-20f0-4a07-b209-72bc932a5ab2","Type":"ContainerStarted","Data":"8be83167afd10af3e8369bebb1b9e50d1bf7c9c6c1b5326860938d7fc266882f"}
Jul 14 23:05:27.442039 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:27.442027    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b778f979c-wzcrq" event={"ID":"83daf71c-20f0-4a07-b209-72bc932a5ab2","Type":"ContainerStarted","Data":"6350bbab6e630c273522663398af417fe0d0fbe73ac08789d20ae84b82a8c5da"}
Jul 14 23:05:27.461866 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:27.461795    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b778f979c-wzcrq" podStartSLOduration=0.871871607 podStartE2EDuration="1.461780409s" totalImagesPullingTime="589.908802ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:05:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:05:27.460539011 +0000 UTC m=+514.763529804" watchObservedRunningTime="2026-07-14 23:05:27.461780409 +0000 UTC m=+514.764771202"
Jul 14 23:05:27.499760 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:27.499419    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-d7hjq"]
Jul 14 23:05:27.502972 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:27.502941    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-wtg6d"]
Jul 14 23:05:27.503488 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:27.503307    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-wtg6d" podUID="c9fe72a4-eed0-4314-b0f3-4568afc2e120" containerName="authorino" containerID="cri-o://87da8d800acef5fcc508effd8121c28b179a3b3d57ec78333598f0f47e0cb71f" gracePeriod=30
Jul 14 23:05:27.505399 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:27.505379    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-d7hjq"]
Jul 14 23:05:27.529269 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:27.529238    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-lkvbw"]
Jul 14 23:05:27.537109 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:27.537082    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-lkvbw"]
Jul 14 23:05:27.738616 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:27.738591    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-wtg6d"
Jul 14 23:05:27.795081 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:27.794986    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kgcw\" (UniqueName: \"kubernetes.io/projected/c9fe72a4-eed0-4314-b0f3-4568afc2e120-kube-api-access-4kgcw\") pod \"c9fe72a4-eed0-4314-b0f3-4568afc2e120\" (UID: \"c9fe72a4-eed0-4314-b0f3-4568afc2e120\") "
Jul 14 23:05:27.797164 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:27.797133    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9fe72a4-eed0-4314-b0f3-4568afc2e120-kube-api-access-4kgcw" (OuterVolumeSpecName: "kube-api-access-4kgcw") pod "c9fe72a4-eed0-4314-b0f3-4568afc2e120" (UID: "c9fe72a4-eed0-4314-b0f3-4568afc2e120"). InnerVolumeSpecName "kube-api-access-4kgcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:05:27.896469 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:27.896429    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kgcw\" (UniqueName: \"kubernetes.io/projected/c9fe72a4-eed0-4314-b0f3-4568afc2e120-kube-api-access-4kgcw\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:05:28.447809 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:28.447717    2563 generic.go:356] "Generic (PLEG): container finished" podID="c9fe72a4-eed0-4314-b0f3-4568afc2e120" containerID="87da8d800acef5fcc508effd8121c28b179a3b3d57ec78333598f0f47e0cb71f" exitCode=0
Jul 14 23:05:28.447809 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:28.447764    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-wtg6d"
Jul 14 23:05:28.448340 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:28.447806    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-wtg6d" event={"ID":"c9fe72a4-eed0-4314-b0f3-4568afc2e120","Type":"ContainerDied","Data":"87da8d800acef5fcc508effd8121c28b179a3b3d57ec78333598f0f47e0cb71f"}
Jul 14 23:05:28.448340 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:28.447846    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-wtg6d" event={"ID":"c9fe72a4-eed0-4314-b0f3-4568afc2e120","Type":"ContainerDied","Data":"b74cc9fad9fab480d4816b09fdab163052517b994a0eb7acabafb9ad8b2a4bb5"}
Jul 14 23:05:28.448340 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:28.447863    2563 scope.go:117] "RemoveContainer" containerID="87da8d800acef5fcc508effd8121c28b179a3b3d57ec78333598f0f47e0cb71f"
Jul 14 23:05:28.456214 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:28.456196    2563 scope.go:117] "RemoveContainer" containerID="87da8d800acef5fcc508effd8121c28b179a3b3d57ec78333598f0f47e0cb71f"
Jul 14 23:05:28.456500 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:05:28.456478    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87da8d800acef5fcc508effd8121c28b179a3b3d57ec78333598f0f47e0cb71f\": container with ID starting with 87da8d800acef5fcc508effd8121c28b179a3b3d57ec78333598f0f47e0cb71f not found: ID does not exist" containerID="87da8d800acef5fcc508effd8121c28b179a3b3d57ec78333598f0f47e0cb71f"
Jul 14 23:05:28.456570 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:28.456512    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87da8d800acef5fcc508effd8121c28b179a3b3d57ec78333598f0f47e0cb71f"} err="failed to get container status \"87da8d800acef5fcc508effd8121c28b179a3b3d57ec78333598f0f47e0cb71f\": rpc error: code = NotFound desc = could not find container \"87da8d800acef5fcc508effd8121c28b179a3b3d57ec78333598f0f47e0cb71f\": container with ID starting with 87da8d800acef5fcc508effd8121c28b179a3b3d57ec78333598f0f47e0cb71f not found: ID does not exist"
Jul 14 23:05:28.468084 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:28.468049    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-wtg6d"]
Jul 14 23:05:28.470442 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:28.470421    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-wtg6d"]
Jul 14 23:05:29.229598 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:29.229565    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39fc194e-5eff-438e-897c-ecfed4775d78" path="/var/lib/kubelet/pods/39fc194e-5eff-438e-897c-ecfed4775d78/volumes"
Jul 14 23:05:29.229822 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:29.229810    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9527df3f-bb02-4126-83e6-db7c2583662d" path="/var/lib/kubelet/pods/9527df3f-bb02-4126-83e6-db7c2583662d/volumes"
Jul 14 23:05:29.230000 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:05:29.229990    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9fe72a4-eed0-4314-b0f3-4568afc2e120" path="/var/lib/kubelet/pods/c9fe72a4-eed0-4314-b0f3-4568afc2e120/volumes"
Jul 14 23:06:04.853889 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:04.853847    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-x84b2"]
Jul 14 23:06:04.854405 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:04.854201    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9fe72a4-eed0-4314-b0f3-4568afc2e120" containerName="authorino"
Jul 14 23:06:04.854405 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:04.854213    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9fe72a4-eed0-4314-b0f3-4568afc2e120" containerName="authorino"
Jul 14 23:06:04.854405 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:04.854290    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9fe72a4-eed0-4314-b0f3-4568afc2e120" containerName="authorino"
Jul 14 23:06:04.857183 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:04.857161    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-x84b2"
Jul 14 23:06:04.864902 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:04.864877    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 14 23:06:04.864995 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:04.864877    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-glnkb\""
Jul 14 23:06:04.873019 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:04.872995    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-x84b2"]
Jul 14 23:06:04.914545 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:04.914503    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5l6n\" (UniqueName: \"kubernetes.io/projected/b89eeddb-0012-4f87-9ec6-baafc6fb8013-kube-api-access-c5l6n\") pod \"payload-pre-processing-74584bb9f8-x84b2\" (UID: \"b89eeddb-0012-4f87-9ec6-baafc6fb8013\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-x84b2"
Jul 14 23:06:04.914731 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:04.914597    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b89eeddb-0012-4f87-9ec6-baafc6fb8013-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-x84b2\" (UID: \"b89eeddb-0012-4f87-9ec6-baafc6fb8013\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-x84b2"
Jul 14 23:06:05.015525 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:05.015480    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b89eeddb-0012-4f87-9ec6-baafc6fb8013-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-x84b2\" (UID: \"b89eeddb-0012-4f87-9ec6-baafc6fb8013\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-x84b2"
Jul 14 23:06:05.015714 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:05.015536    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5l6n\" (UniqueName: \"kubernetes.io/projected/b89eeddb-0012-4f87-9ec6-baafc6fb8013-kube-api-access-c5l6n\") pod \"payload-pre-processing-74584bb9f8-x84b2\" (UID: \"b89eeddb-0012-4f87-9ec6-baafc6fb8013\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-x84b2"
Jul 14 23:06:05.016202 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:05.016181    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b89eeddb-0012-4f87-9ec6-baafc6fb8013-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-x84b2\" (UID: \"b89eeddb-0012-4f87-9ec6-baafc6fb8013\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-x84b2"
Jul 14 23:06:05.025809 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:05.025786    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5l6n\" (UniqueName: \"kubernetes.io/projected/b89eeddb-0012-4f87-9ec6-baafc6fb8013-kube-api-access-c5l6n\") pod \"payload-pre-processing-74584bb9f8-x84b2\" (UID: \"b89eeddb-0012-4f87-9ec6-baafc6fb8013\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-x84b2"
Jul 14 23:06:05.167911 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:05.167817    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-x84b2"
Jul 14 23:06:05.299326 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:05.299305    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-x84b2"]
Jul 14 23:06:05.301474 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:06:05.301444    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb89eeddb_0012_4f87_9ec6_baafc6fb8013.slice/crio-7eaa71a6b57edbd1592c848c858b179d097dedaf41377d0b281c888da85a44fd WatchSource:0}: Error finding container 7eaa71a6b57edbd1592c848c858b179d097dedaf41377d0b281c888da85a44fd: Status 404 returned error can't find the container with id 7eaa71a6b57edbd1592c848c858b179d097dedaf41377d0b281c888da85a44fd
Jul 14 23:06:05.608078 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:05.608033    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-x84b2" event={"ID":"b89eeddb-0012-4f87-9ec6-baafc6fb8013","Type":"ContainerStarted","Data":"7eaa71a6b57edbd1592c848c858b179d097dedaf41377d0b281c888da85a44fd"}
Jul 14 23:06:07.619815 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:07.619779    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-x84b2" event={"ID":"b89eeddb-0012-4f87-9ec6-baafc6fb8013","Type":"ContainerStarted","Data":"f5f70c004397f01f063e56d30c67ef00bf99c6f7806270ccfdc7234db3a7fad9"}
Jul 14 23:06:07.620276 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:07.619910    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-x84b2"
Jul 14 23:06:07.638268 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:07.638216    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-74584bb9f8-x84b2" podStartSLOduration=2.217326963 podStartE2EDuration="3.638198648s" totalImagesPullingTime="1.420871685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:06:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:06:07.637202911 +0000 UTC m=+554.940193731" watchObservedRunningTime="2026-07-14 23:06:07.638198648 +0000 UTC m=+554.941189439"
Jul 14 23:06:18.626466 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:18.626437    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-x84b2"
Jul 14 23:06:49.186900 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:49.186857    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-4jthh"]
Jul 14 23:06:49.190088 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:49.190069    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4jthh"
Jul 14 23:06:49.194494 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:49.194472    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-9bm26\""
Jul 14 23:06:49.194645 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:49.194476    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 14 23:06:49.194714 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:49.194531    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 14 23:06:49.194778 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:49.194531    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 14 23:06:49.204965 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:49.204941    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-4jthh"]
Jul 14 23:06:49.261005 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:49.260978    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dac78e62-49ce-446f-960b-f1c5ab165acc-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4jthh\" (UID: \"dac78e62-49ce-446f-960b-f1c5ab165acc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4jthh"
Jul 14 23:06:49.261159 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:49.261022    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dac78e62-49ce-446f-960b-f1c5ab165acc-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4jthh\" (UID: \"dac78e62-49ce-446f-960b-f1c5ab165acc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4jthh"
Jul 14 23:06:49.261159 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:49.261042    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcjzz\" (UniqueName: \"kubernetes.io/projected/dac78e62-49ce-446f-960b-f1c5ab165acc-kube-api-access-gcjzz\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4jthh\" (UID: \"dac78e62-49ce-446f-960b-f1c5ab165acc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4jthh"
Jul 14 23:06:49.261159 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:49.261125    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dac78e62-49ce-446f-960b-f1c5ab165acc-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4jthh\" (UID: \"dac78e62-49ce-446f-960b-f1c5ab165acc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4jthh"
Jul 14 23:06:49.261276 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:49.261160    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dac78e62-49ce-446f-960b-f1c5ab165acc-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4jthh\" (UID: \"dac78e62-49ce-446f-960b-f1c5ab165acc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4jthh"
Jul 14 23:06:49.261276 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:49.261176    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dac78e62-49ce-446f-960b-f1c5ab165acc-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4jthh\" (UID: \"dac78e62-49ce-446f-960b-f1c5ab165acc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4jthh"
Jul 14 23:06:49.362330 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:49.362274    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dac78e62-49ce-446f-960b-f1c5ab165acc-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4jthh\" (UID: \"dac78e62-49ce-446f-960b-f1c5ab165acc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4jthh"
Jul 14 23:06:49.362583 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:49.362371    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dac78e62-49ce-446f-960b-f1c5ab165acc-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4jthh\" (UID: \"dac78e62-49ce-446f-960b-f1c5ab165acc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4jthh"
Jul 14 23:06:49.362583 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:49.362402    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcjzz\" (UniqueName: \"kubernetes.io/projected/dac78e62-49ce-446f-960b-f1c5ab165acc-kube-api-access-gcjzz\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4jthh\" (UID: \"dac78e62-49ce-446f-960b-f1c5ab165acc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4jthh"
Jul 14 23:06:49.362698 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:49.362618    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dac78e62-49ce-446f-960b-f1c5ab165acc-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4jthh\" (UID: \"dac78e62-49ce-446f-960b-f1c5ab165acc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4jthh"
Jul 14 23:06:49.362698 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:49.362683    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dac78e62-49ce-446f-960b-f1c5ab165acc-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4jthh\" (UID: \"dac78e62-49ce-446f-960b-f1c5ab165acc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4jthh"
Jul 14 23:06:49.362786 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:49.362710    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dac78e62-49ce-446f-960b-f1c5ab165acc-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4jthh\" (UID: \"dac78e62-49ce-446f-960b-f1c5ab165acc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4jthh"
Jul 14 23:06:49.363013 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:49.362988    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dac78e62-49ce-446f-960b-f1c5ab165acc-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4jthh\" (UID: \"dac78e62-49ce-446f-960b-f1c5ab165acc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4jthh"
Jul 14 23:06:49.363268 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:49.363071    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dac78e62-49ce-446f-960b-f1c5ab165acc-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4jthh\" (UID: \"dac78e62-49ce-446f-960b-f1c5ab165acc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4jthh"
Jul 14 23:06:49.363268 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:49.363071    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dac78e62-49ce-446f-960b-f1c5ab165acc-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4jthh\" (UID: \"dac78e62-49ce-446f-960b-f1c5ab165acc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4jthh"
Jul 14 23:06:49.364673 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:49.364650    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dac78e62-49ce-446f-960b-f1c5ab165acc-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4jthh\" (UID: \"dac78e62-49ce-446f-960b-f1c5ab165acc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4jthh"
Jul 14 23:06:49.364993 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:49.364974    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dac78e62-49ce-446f-960b-f1c5ab165acc-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4jthh\" (UID: \"dac78e62-49ce-446f-960b-f1c5ab165acc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4jthh"
Jul 14 23:06:49.370616 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:49.370599    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcjzz\" (UniqueName: \"kubernetes.io/projected/dac78e62-49ce-446f-960b-f1c5ab165acc-kube-api-access-gcjzz\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4jthh\" (UID: \"dac78e62-49ce-446f-960b-f1c5ab165acc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4jthh"
Jul 14 23:06:49.499972 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:49.499936    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4jthh"
Jul 14 23:06:49.639308 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:49.635552    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-4jthh"]
Jul 14 23:06:49.640863 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:06:49.640820    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddac78e62_49ce_446f_960b_f1c5ab165acc.slice/crio-6e89ba89d5740bce46abc2dc982078aba2278710689c61632c62b9f722561a6b WatchSource:0}: Error finding container 6e89ba89d5740bce46abc2dc982078aba2278710689c61632c62b9f722561a6b: Status 404 returned error can't find the container with id 6e89ba89d5740bce46abc2dc982078aba2278710689c61632c62b9f722561a6b
Jul 14 23:06:49.792597 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:49.792509    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4jthh" event={"ID":"dac78e62-49ce-446f-960b-f1c5ab165acc","Type":"ContainerStarted","Data":"6e89ba89d5740bce46abc2dc982078aba2278710689c61632c62b9f722561a6b"}
Jul 14 23:06:51.802981 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:51.802945    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4jthh" event={"ID":"dac78e62-49ce-446f-960b-f1c5ab165acc","Type":"ContainerStarted","Data":"6aabac1231bd07c92cab6172c8f55ce23044c5be861ee34719809cade3e7b861"}
Jul 14 23:06:51.824147 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:51.824096    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4jthh" podStartSLOduration=1.728343267 podStartE2EDuration="2.824081059s" totalImagesPullingTime="1.095737792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:06:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:06:51.822915871 +0000 UTC m=+599.125906665" watchObservedRunningTime="2026-07-14 23:06:51.824081059 +0000 UTC m=+599.127071851"
Jul 14 23:06:59.500594 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:59.500555    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4jthh"
Jul 14 23:06:59.501046 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:59.500618    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4jthh"
Jul 14 23:06:59.513112 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:59.513091    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4jthh"
Jul 14 23:06:59.848639 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:06:59.848547    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4jthh"
Jul 14 23:09:03.947214 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:09:03.947129    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d685bdbbd-6vx8d"]
Jul 14 23:09:03.950291 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:09:03.950266    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d685bdbbd-6vx8d"
Jul 14 23:09:03.958256 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:09:03.958230    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d685bdbbd-6vx8d"]
Jul 14 23:09:04.080503 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:09:04.080462    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xt2d\" (UniqueName: \"kubernetes.io/projected/eda3da51-cec1-4290-965f-e8162f9079b2-kube-api-access-6xt2d\") pod \"authorino-5d685bdbbd-6vx8d\" (UID: \"eda3da51-cec1-4290-965f-e8162f9079b2\") " pod="kuadrant-system/authorino-5d685bdbbd-6vx8d"
Jul 14 23:09:04.080688 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:09:04.080550    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eda3da51-cec1-4290-965f-e8162f9079b2-tls-cert\") pod \"authorino-5d685bdbbd-6vx8d\" (UID: \"eda3da51-cec1-4290-965f-e8162f9079b2\") " pod="kuadrant-system/authorino-5d685bdbbd-6vx8d"
Jul 14 23:09:04.181535 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:09:04.181488    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xt2d\" (UniqueName: \"kubernetes.io/projected/eda3da51-cec1-4290-965f-e8162f9079b2-kube-api-access-6xt2d\") pod \"authorino-5d685bdbbd-6vx8d\" (UID: \"eda3da51-cec1-4290-965f-e8162f9079b2\") " pod="kuadrant-system/authorino-5d685bdbbd-6vx8d"
Jul 14 23:09:04.181708 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:09:04.181586    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eda3da51-cec1-4290-965f-e8162f9079b2-tls-cert\") pod \"authorino-5d685bdbbd-6vx8d\" (UID: \"eda3da51-cec1-4290-965f-e8162f9079b2\") " pod="kuadrant-system/authorino-5d685bdbbd-6vx8d"
Jul 14 23:09:04.184163 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:09:04.184129    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eda3da51-cec1-4290-965f-e8162f9079b2-tls-cert\") pod \"authorino-5d685bdbbd-6vx8d\" (UID: \"eda3da51-cec1-4290-965f-e8162f9079b2\") " pod="kuadrant-system/authorino-5d685bdbbd-6vx8d"
Jul 14 23:09:04.191182 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:09:04.191161    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xt2d\" (UniqueName: \"kubernetes.io/projected/eda3da51-cec1-4290-965f-e8162f9079b2-kube-api-access-6xt2d\") pod \"authorino-5d685bdbbd-6vx8d\" (UID: \"eda3da51-cec1-4290-965f-e8162f9079b2\") " pod="kuadrant-system/authorino-5d685bdbbd-6vx8d"
Jul 14 23:09:04.259929 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:09:04.259896    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d685bdbbd-6vx8d"
Jul 14 23:09:04.396686 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:09:04.396655    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d685bdbbd-6vx8d"]
Jul 14 23:09:04.399427 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:09:04.399401    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeda3da51_cec1_4290_965f_e8162f9079b2.slice/crio-aa72d9d610f454afd90e10de51c1e05b03bfeb924a355dbd22017c958fbda943 WatchSource:0}: Error finding container aa72d9d610f454afd90e10de51c1e05b03bfeb924a355dbd22017c958fbda943: Status 404 returned error can't find the container with id aa72d9d610f454afd90e10de51c1e05b03bfeb924a355dbd22017c958fbda943
Jul 14 23:09:04.400892 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:09:04.400871    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 23:09:05.372984 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:09:05.372951    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d685bdbbd-6vx8d" event={"ID":"eda3da51-cec1-4290-965f-e8162f9079b2","Type":"ContainerStarted","Data":"aa72d9d610f454afd90e10de51c1e05b03bfeb924a355dbd22017c958fbda943"}
Jul 14 23:09:06.380660 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:09:06.380604    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d685bdbbd-6vx8d" event={"ID":"eda3da51-cec1-4290-965f-e8162f9079b2","Type":"ContainerStarted","Data":"230195e155a17ebc0c9528990c4447cd5fe3000fe11710ddd757345f8e991838"}
Jul 14 23:09:06.398051 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:09:06.397987    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d685bdbbd-6vx8d" podStartSLOduration=2.488649024 podStartE2EDuration="3.397967601s" totalImagesPullingTime="909.318577ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:09:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:09:06.396264046 +0000 UTC m=+733.699254835" watchObservedRunningTime="2026-07-14 23:09:06.397967601 +0000 UTC m=+733.700958396"
Jul 14 23:09:06.422968 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:09:06.422930    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b778f979c-wzcrq"]
Jul 14 23:09:06.423187 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:09:06.423159    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b778f979c-wzcrq" podUID="83daf71c-20f0-4a07-b209-72bc932a5ab2" containerName="authorino" containerID="cri-o://8be83167afd10af3e8369bebb1b9e50d1bf7c9c6c1b5326860938d7fc266882f" gracePeriod=30
Jul 14 23:09:06.700456 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:09:06.700429    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b778f979c-wzcrq"
Jul 14 23:09:06.805546 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:09:06.805506    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqmtm\" (UniqueName: \"kubernetes.io/projected/83daf71c-20f0-4a07-b209-72bc932a5ab2-kube-api-access-dqmtm\") pod \"83daf71c-20f0-4a07-b209-72bc932a5ab2\" (UID: \"83daf71c-20f0-4a07-b209-72bc932a5ab2\") "
Jul 14 23:09:06.805546 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:09:06.805549    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/83daf71c-20f0-4a07-b209-72bc932a5ab2-tls-cert\") pod \"83daf71c-20f0-4a07-b209-72bc932a5ab2\" (UID: \"83daf71c-20f0-4a07-b209-72bc932a5ab2\") "
Jul 14 23:09:06.808672 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:09:06.808636    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83daf71c-20f0-4a07-b209-72bc932a5ab2-kube-api-access-dqmtm" (OuterVolumeSpecName: "kube-api-access-dqmtm") pod "83daf71c-20f0-4a07-b209-72bc932a5ab2" (UID: "83daf71c-20f0-4a07-b209-72bc932a5ab2"). InnerVolumeSpecName "kube-api-access-dqmtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:09:06.819552 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:09:06.819514    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83daf71c-20f0-4a07-b209-72bc932a5ab2-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "83daf71c-20f0-4a07-b209-72bc932a5ab2" (UID: "83daf71c-20f0-4a07-b209-72bc932a5ab2"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 23:09:06.907267 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:09:06.907223    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqmtm\" (UniqueName: \"kubernetes.io/projected/83daf71c-20f0-4a07-b209-72bc932a5ab2-kube-api-access-dqmtm\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:09:06.907267 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:09:06.907264    2563 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/83daf71c-20f0-4a07-b209-72bc932a5ab2-tls-cert\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:09:07.385420 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:09:07.385388    2563 generic.go:356] "Generic (PLEG): container finished" podID="83daf71c-20f0-4a07-b209-72bc932a5ab2" containerID="8be83167afd10af3e8369bebb1b9e50d1bf7c9c6c1b5326860938d7fc266882f" exitCode=0
Jul 14 23:09:07.385930 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:09:07.385438    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b778f979c-wzcrq"
Jul 14 23:09:07.385930 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:09:07.385477    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b778f979c-wzcrq" event={"ID":"83daf71c-20f0-4a07-b209-72bc932a5ab2","Type":"ContainerDied","Data":"8be83167afd10af3e8369bebb1b9e50d1bf7c9c6c1b5326860938d7fc266882f"}
Jul 14 23:09:07.385930 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:09:07.385515    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b778f979c-wzcrq" event={"ID":"83daf71c-20f0-4a07-b209-72bc932a5ab2","Type":"ContainerDied","Data":"6350bbab6e630c273522663398af417fe0d0fbe73ac08789d20ae84b82a8c5da"}
Jul 14 23:09:07.385930 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:09:07.385532    2563 scope.go:117] "RemoveContainer" containerID="8be83167afd10af3e8369bebb1b9e50d1bf7c9c6c1b5326860938d7fc266882f"
Jul 14 23:09:07.393565 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:09:07.393545    2563 scope.go:117] "RemoveContainer" containerID="8be83167afd10af3e8369bebb1b9e50d1bf7c9c6c1b5326860938d7fc266882f"
Jul 14 23:09:07.393829 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:09:07.393807    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8be83167afd10af3e8369bebb1b9e50d1bf7c9c6c1b5326860938d7fc266882f\": container with ID starting with 8be83167afd10af3e8369bebb1b9e50d1bf7c9c6c1b5326860938d7fc266882f not found: ID does not exist" containerID="8be83167afd10af3e8369bebb1b9e50d1bf7c9c6c1b5326860938d7fc266882f"
Jul 14 23:09:07.393875 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:09:07.393837    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8be83167afd10af3e8369bebb1b9e50d1bf7c9c6c1b5326860938d7fc266882f"} err="failed to get container status \"8be83167afd10af3e8369bebb1b9e50d1bf7c9c6c1b5326860938d7fc266882f\": rpc error: code = NotFound desc = could not find container \"8be83167afd10af3e8369bebb1b9e50d1bf7c9c6c1b5326860938d7fc266882f\": container with ID starting with 8be83167afd10af3e8369bebb1b9e50d1bf7c9c6c1b5326860938d7fc266882f not found: ID does not exist"
Jul 14 23:09:07.409609 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:09:07.409583    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b778f979c-wzcrq"]
Jul 14 23:09:07.415508 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:09:07.415479    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b778f979c-wzcrq"]
Jul 14 23:09:09.229276 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:09:09.229243    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83daf71c-20f0-4a07-b209-72bc932a5ab2" path="/var/lib/kubelet/pods/83daf71c-20f0-4a07-b209-72bc932a5ab2/volumes"
Jul 14 23:15:00.148135 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:15:00.148041    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734515-tf6xf"]
Jul 14 23:15:00.150487 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:15:00.148371    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83daf71c-20f0-4a07-b209-72bc932a5ab2" containerName="authorino"
Jul 14 23:15:00.150487 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:15:00.148382    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="83daf71c-20f0-4a07-b209-72bc932a5ab2" containerName="authorino"
Jul 14 23:15:00.150487 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:15:00.148436    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="83daf71c-20f0-4a07-b209-72bc932a5ab2" containerName="authorino"
Jul 14 23:15:00.151486 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:15:00.151468    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734515-tf6xf"
Jul 14 23:15:00.154520 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:15:00.154496    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jw626\""
Jul 14 23:15:00.160744 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:15:00.160725    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734515-tf6xf"
Jul 14 23:15:00.166219 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:15:00.166183    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734515-tf6xf"]
Jul 14 23:15:00.296562 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:15:00.296536    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734515-tf6xf"]
Jul 14 23:15:00.299095 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:15:00.299061    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb91367bb_5189_4278_9e1e_90f33f2343d3.slice/crio-47c5871b7e7a9b4174c62c5ced0a682b3e77adef8fd5086f580afa1a219d22ef WatchSource:0}: Error finding container 47c5871b7e7a9b4174c62c5ced0a682b3e77adef8fd5086f580afa1a219d22ef: Status 404 returned error can't find the container with id 47c5871b7e7a9b4174c62c5ced0a682b3e77adef8fd5086f580afa1a219d22ef
Jul 14 23:15:00.300969 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:15:00.300950    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 23:15:00.880595 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:15:00.880559    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734515-tf6xf" event={"ID":"b91367bb-5189-4278-9e1e-90f33f2343d3","Type":"ContainerStarted","Data":"47c5871b7e7a9b4174c62c5ced0a682b3e77adef8fd5086f580afa1a219d22ef"}
Jul 14 23:15:03.896167 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:15:03.896134    2563 generic.go:356] "Generic (PLEG): container finished" podID="b91367bb-5189-4278-9e1e-90f33f2343d3" containerID="40828f736d27bf79d13bbb3103d39c2507fe4bdd2623ab0eaef4077b238c141d" exitCode=0
Jul 14 23:15:03.896552 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:15:03.896230    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734515-tf6xf" event={"ID":"b91367bb-5189-4278-9e1e-90f33f2343d3","Type":"ContainerDied","Data":"40828f736d27bf79d13bbb3103d39c2507fe4bdd2623ab0eaef4077b238c141d"}
Jul 14 23:15:05.030880 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:15:05.030852    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734515-tf6xf"
Jul 14 23:15:05.904981 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:15:05.904949    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734515-tf6xf"
Jul 14 23:15:05.904981 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:15:05.904962    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734515-tf6xf" event={"ID":"b91367bb-5189-4278-9e1e-90f33f2343d3","Type":"ContainerDied","Data":"47c5871b7e7a9b4174c62c5ced0a682b3e77adef8fd5086f580afa1a219d22ef"}
Jul 14 23:15:05.905194 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:15:05.905005    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47c5871b7e7a9b4174c62c5ced0a682b3e77adef8fd5086f580afa1a219d22ef"
Jul 14 23:22:23.504594 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:23.504557    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-37a8657d-679bcd9946-8qvpb"]
Jul 14 23:22:23.505009 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:23.504924    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b91367bb-5189-4278-9e1e-90f33f2343d3" containerName="cleanup"
Jul 14 23:22:23.505009 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:23.504938    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b91367bb-5189-4278-9e1e-90f33f2343d3" containerName="cleanup"
Jul 14 23:22:23.505009 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:23.505004    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b91367bb-5189-4278-9e1e-90f33f2343d3" containerName="cleanup"
Jul 14 23:22:23.507693 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:23.507678    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-37a8657d-679bcd9946-8qvpb"
Jul 14 23:22:23.512064 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:23.512041    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 23:22:23.513365 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:23.513345    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 23:22:23.513496 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:23.513369    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-37a8657d-serving-cert\""
Jul 14 23:22:23.513496 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:23.513381    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7jvtc\""
Jul 14 23:22:23.523939 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:23.523912    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-37a8657d-679bcd9946-8qvpb"]
Jul 14 23:22:23.572418 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:23.572380    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdphx\" (UniqueName: \"kubernetes.io/projected/654409bc-7e3b-438b-9aea-a4e4407550e9-kube-api-access-qdphx\") pod \"maas-api-e2e-ait-37a8657d-679bcd9946-8qvpb\" (UID: \"654409bc-7e3b-438b-9aea-a4e4407550e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-37a8657d-679bcd9946-8qvpb"
Jul 14 23:22:23.572582 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:23.572427    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/654409bc-7e3b-438b-9aea-a4e4407550e9-openshift-service-ca\") pod \"maas-api-e2e-ait-37a8657d-679bcd9946-8qvpb\" (UID: \"654409bc-7e3b-438b-9aea-a4e4407550e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-37a8657d-679bcd9946-8qvpb"
Jul 14 23:22:23.572582 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:23.572529    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/654409bc-7e3b-438b-9aea-a4e4407550e9-maas-api-tls\") pod \"maas-api-e2e-ait-37a8657d-679bcd9946-8qvpb\" (UID: \"654409bc-7e3b-438b-9aea-a4e4407550e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-37a8657d-679bcd9946-8qvpb"
Jul 14 23:22:23.673013 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:23.672976    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdphx\" (UniqueName: \"kubernetes.io/projected/654409bc-7e3b-438b-9aea-a4e4407550e9-kube-api-access-qdphx\") pod \"maas-api-e2e-ait-37a8657d-679bcd9946-8qvpb\" (UID: \"654409bc-7e3b-438b-9aea-a4e4407550e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-37a8657d-679bcd9946-8qvpb"
Jul 14 23:22:23.673204 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:23.673022    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/654409bc-7e3b-438b-9aea-a4e4407550e9-openshift-service-ca\") pod \"maas-api-e2e-ait-37a8657d-679bcd9946-8qvpb\" (UID: \"654409bc-7e3b-438b-9aea-a4e4407550e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-37a8657d-679bcd9946-8qvpb"
Jul 14 23:22:23.673204 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:23.673075    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/654409bc-7e3b-438b-9aea-a4e4407550e9-maas-api-tls\") pod \"maas-api-e2e-ait-37a8657d-679bcd9946-8qvpb\" (UID: \"654409bc-7e3b-438b-9aea-a4e4407550e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-37a8657d-679bcd9946-8qvpb"
Jul 14 23:22:23.673348 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:22:23.673204    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-37a8657d-serving-cert: secret "maas-api-e2e-ait-37a8657d-serving-cert" not found
Jul 14 23:22:23.673348 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:22:23.673303    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/654409bc-7e3b-438b-9aea-a4e4407550e9-maas-api-tls podName:654409bc-7e3b-438b-9aea-a4e4407550e9 nodeName:}" failed. No retries permitted until 2026-07-14 23:22:24.173261404 +0000 UTC m=+1531.476252191 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/654409bc-7e3b-438b-9aea-a4e4407550e9-maas-api-tls") pod "maas-api-e2e-ait-37a8657d-679bcd9946-8qvpb" (UID: "654409bc-7e3b-438b-9aea-a4e4407550e9") : secret "maas-api-e2e-ait-37a8657d-serving-cert" not found
Jul 14 23:22:23.673699 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:23.673678    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/654409bc-7e3b-438b-9aea-a4e4407550e9-openshift-service-ca\") pod \"maas-api-e2e-ait-37a8657d-679bcd9946-8qvpb\" (UID: \"654409bc-7e3b-438b-9aea-a4e4407550e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-37a8657d-679bcd9946-8qvpb"
Jul 14 23:22:23.686184 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:23.686151    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdphx\" (UniqueName: \"kubernetes.io/projected/654409bc-7e3b-438b-9aea-a4e4407550e9-kube-api-access-qdphx\") pod \"maas-api-e2e-ait-37a8657d-679bcd9946-8qvpb\" (UID: \"654409bc-7e3b-438b-9aea-a4e4407550e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-37a8657d-679bcd9946-8qvpb"
Jul 14 23:22:24.178378 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:24.178332    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/654409bc-7e3b-438b-9aea-a4e4407550e9-maas-api-tls\") pod \"maas-api-e2e-ait-37a8657d-679bcd9946-8qvpb\" (UID: \"654409bc-7e3b-438b-9aea-a4e4407550e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-37a8657d-679bcd9946-8qvpb"
Jul 14 23:22:24.180864 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:24.180843    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/654409bc-7e3b-438b-9aea-a4e4407550e9-maas-api-tls\") pod \"maas-api-e2e-ait-37a8657d-679bcd9946-8qvpb\" (UID: \"654409bc-7e3b-438b-9aea-a4e4407550e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-37a8657d-679bcd9946-8qvpb"
Jul 14 23:22:24.417987 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:24.417949    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-37a8657d-679bcd9946-8qvpb"
Jul 14 23:22:24.555234 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:24.555206    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-37a8657d-679bcd9946-8qvpb"]
Jul 14 23:22:24.558166 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:22:24.558137    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod654409bc_7e3b_438b_9aea_a4e4407550e9.slice/crio-f071057d0035d6047fc8645932c125bcb0a6d7923dc24187a51bbf93d9631fca WatchSource:0}: Error finding container f071057d0035d6047fc8645932c125bcb0a6d7923dc24187a51bbf93d9631fca: Status 404 returned error can't find the container with id f071057d0035d6047fc8645932c125bcb0a6d7923dc24187a51bbf93d9631fca
Jul 14 23:22:24.559469 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:24.559450    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 23:22:24.815360 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:24.815325    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-37a8657d-679bcd9946-8qvpb" event={"ID":"654409bc-7e3b-438b-9aea-a4e4407550e9","Type":"ContainerStarted","Data":"f071057d0035d6047fc8645932c125bcb0a6d7923dc24187a51bbf93d9631fca"}
Jul 14 23:22:26.827548 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:26.827512    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-37a8657d-679bcd9946-8qvpb" event={"ID":"654409bc-7e3b-438b-9aea-a4e4407550e9","Type":"ContainerStarted","Data":"b2131245446dac98e07ffab77d62020997d0a95d1591e2114401aae03910113c"}
Jul 14 23:22:26.827962 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:26.827655    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-37a8657d-679bcd9946-8qvpb"
Jul 14 23:22:26.845868 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:26.845817    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-37a8657d-679bcd9946-8qvpb" podStartSLOduration=2.033804122 podStartE2EDuration="3.845800562s" totalImagesPullingTime="1.81199644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:22:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:22:26.845027271 +0000 UTC m=+1534.148018064" watchObservedRunningTime="2026-07-14 23:22:26.845800562 +0000 UTC m=+1534.148791354"
Jul 14 23:22:28.372781 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:28.372741    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37a8657d-t4d8l"]
Jul 14 23:22:28.376110 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:28.376092    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37a8657d-t4d8l"
Jul 14 23:22:28.379126 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:28.379100    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jw626\""
Jul 14 23:22:28.384391 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:28.384369    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37a8657d-t4d8l"]
Jul 14 23:22:28.416589 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:28.416557    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1e0808e-008d-4b89-9728-6be8762bdb19-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-37a8657d-t4d8l\" (UID: \"b1e0808e-008d-4b89-9728-6be8762bdb19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37a8657d-t4d8l"
Jul 14 23:22:28.517475 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:28.517440    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1e0808e-008d-4b89-9728-6be8762bdb19-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-37a8657d-t4d8l\" (UID: \"b1e0808e-008d-4b89-9728-6be8762bdb19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37a8657d-t4d8l"
Jul 14 23:22:28.518058 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:28.518038    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1e0808e-008d-4b89-9728-6be8762bdb19-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-37a8657d-t4d8l\" (UID: \"b1e0808e-008d-4b89-9728-6be8762bdb19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37a8657d-t4d8l"
Jul 14 23:22:28.687348 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:28.687240    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37a8657d-t4d8l"
Jul 14 23:22:28.823909 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:28.823775    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37a8657d-t4d8l"]
Jul 14 23:22:28.826625 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:22:28.826594    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1e0808e_008d_4b89_9728_6be8762bdb19.slice/crio-403c9429c1d1f31c422dfa3272055c31e9d2ed6ef28ce72bd61860bad6322053 WatchSource:0}: Error finding container 403c9429c1d1f31c422dfa3272055c31e9d2ed6ef28ce72bd61860bad6322053: Status 404 returned error can't find the container with id 403c9429c1d1f31c422dfa3272055c31e9d2ed6ef28ce72bd61860bad6322053
Jul 14 23:22:28.836627 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:28.836596    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37a8657d-t4d8l" event={"ID":"b1e0808e-008d-4b89-9728-6be8762bdb19","Type":"ContainerStarted","Data":"403c9429c1d1f31c422dfa3272055c31e9d2ed6ef28ce72bd61860bad6322053"}
Jul 14 23:22:29.842980 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:29.842942    2563 generic.go:356] "Generic (PLEG): container finished" podID="b1e0808e-008d-4b89-9728-6be8762bdb19" containerID="bd85672f799a4b558228a6b91abdfb929de3b66912fc219d22b06fe303cbfd98" exitCode=7
Jul 14 23:22:29.843390 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:29.843033    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37a8657d-t4d8l" event={"ID":"b1e0808e-008d-4b89-9728-6be8762bdb19","Type":"ContainerDied","Data":"bd85672f799a4b558228a6b91abdfb929de3b66912fc219d22b06fe303cbfd98"}
Jul 14 23:22:29.843390 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:29.843205    2563 scope.go:117] "RemoveContainer" containerID="bd85672f799a4b558228a6b91abdfb929de3b66912fc219d22b06fe303cbfd98"
Jul 14 23:22:30.848780 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:30.848743    2563 generic.go:356] "Generic (PLEG): container finished" podID="b1e0808e-008d-4b89-9728-6be8762bdb19" containerID="3cdea305c98dd7e01a2b42fb6eade004d77ac4d6079df445e85f05651bb66560" exitCode=7
Jul 14 23:22:30.849171 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:30.848835    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37a8657d-t4d8l" event={"ID":"b1e0808e-008d-4b89-9728-6be8762bdb19","Type":"ContainerDied","Data":"3cdea305c98dd7e01a2b42fb6eade004d77ac4d6079df445e85f05651bb66560"}
Jul 14 23:22:30.849171 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:30.848900    2563 scope.go:117] "RemoveContainer" containerID="bd85672f799a4b558228a6b91abdfb929de3b66912fc219d22b06fe303cbfd98"
Jul 14 23:22:30.849171 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:30.849159    2563 scope.go:117] "RemoveContainer" containerID="3cdea305c98dd7e01a2b42fb6eade004d77ac4d6079df445e85f05651bb66560"
Jul 14 23:22:30.849402 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:22:30.849382    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-37a8657d-t4d8l_odh-ai-gateway-infra(b1e0808e-008d-4b89-9728-6be8762bdb19)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37a8657d-t4d8l" podUID="b1e0808e-008d-4b89-9728-6be8762bdb19"
Jul 14 23:22:31.854706 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:31.854679    2563 scope.go:117] "RemoveContainer" containerID="3cdea305c98dd7e01a2b42fb6eade004d77ac4d6079df445e85f05651bb66560"
Jul 14 23:22:31.855080 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:22:31.854855    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-37a8657d-t4d8l_odh-ai-gateway-infra(b1e0808e-008d-4b89-9728-6be8762bdb19)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37a8657d-t4d8l" podUID="b1e0808e-008d-4b89-9728-6be8762bdb19"
Jul 14 23:22:32.837407 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:32.837374    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-37a8657d-679bcd9946-8qvpb"
Jul 14 23:22:45.224025 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:45.223990    2563 scope.go:117] "RemoveContainer" containerID="3cdea305c98dd7e01a2b42fb6eade004d77ac4d6079df445e85f05651bb66560"
Jul 14 23:22:45.917057 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:45.917019    2563 generic.go:356] "Generic (PLEG): container finished" podID="b1e0808e-008d-4b89-9728-6be8762bdb19" containerID="0f3a86f343aa9ca47eeaf3bf68435414bcee416f5e425ed2a7c2dd44d5f9cef7" exitCode=0
Jul 14 23:22:45.917216 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:45.917093    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37a8657d-t4d8l" event={"ID":"b1e0808e-008d-4b89-9728-6be8762bdb19","Type":"ContainerDied","Data":"0f3a86f343aa9ca47eeaf3bf68435414bcee416f5e425ed2a7c2dd44d5f9cef7"}
Jul 14 23:22:45.917216 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:45.917155    2563 scope.go:117] "RemoveContainer" containerID="3cdea305c98dd7e01a2b42fb6eade004d77ac4d6079df445e85f05651bb66560"
Jul 14 23:22:46.945062 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:46.945029    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37a8657d-t4d8l"]
Jul 14 23:22:47.059560 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:47.059531    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37a8657d-t4d8l"
Jul 14 23:22:47.175257 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:47.175161    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1e0808e-008d-4b89-9728-6be8762bdb19-maas-api-service-ca\") pod \"b1e0808e-008d-4b89-9728-6be8762bdb19\" (UID: \"b1e0808e-008d-4b89-9728-6be8762bdb19\") "
Jul 14 23:22:47.175573 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:47.175548    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1e0808e-008d-4b89-9728-6be8762bdb19-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b1e0808e-008d-4b89-9728-6be8762bdb19" (UID: "b1e0808e-008d-4b89-9728-6be8762bdb19"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 23:22:47.276429 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:47.276390    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1e0808e-008d-4b89-9728-6be8762bdb19-maas-api-service-ca\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:22:47.929880 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:47.929846    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37a8657d-t4d8l" event={"ID":"b1e0808e-008d-4b89-9728-6be8762bdb19","Type":"ContainerDied","Data":"403c9429c1d1f31c422dfa3272055c31e9d2ed6ef28ce72bd61860bad6322053"}
Jul 14 23:22:47.929880 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:47.929881    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37a8657d-t4d8l"
Jul 14 23:22:47.930086 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:47.929894    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="403c9429c1d1f31c422dfa3272055c31e9d2ed6ef28ce72bd61860bad6322053"
Jul 14 23:22:47.960194 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:47.960152    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37a8657d-t4d8l"]
Jul 14 23:22:47.963870 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:47.963845    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37a8657d-t4d8l"]
Jul 14 23:22:49.229603 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:22:49.229572    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1e0808e-008d-4b89-9728-6be8762bdb19" path="/var/lib/kubelet/pods/b1e0808e-008d-4b89-9728-6be8762bdb19/volumes"
Jul 14 23:23:18.418520 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:18.418476    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37a8657d-7cm86"]
Jul 14 23:23:18.418993 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:18.418781    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1e0808e-008d-4b89-9728-6be8762bdb19" containerName="revoke-keys"
Jul 14 23:23:18.418993 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:18.418791    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1e0808e-008d-4b89-9728-6be8762bdb19" containerName="revoke-keys"
Jul 14 23:23:18.418993 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:18.418814    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1e0808e-008d-4b89-9728-6be8762bdb19" containerName="revoke-keys"
Jul 14 23:23:18.418993 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:18.418819    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1e0808e-008d-4b89-9728-6be8762bdb19" containerName="revoke-keys"
Jul 14 23:23:18.418993 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:18.418884    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1e0808e-008d-4b89-9728-6be8762bdb19" containerName="revoke-keys"
Jul 14 23:23:18.418993 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:18.418892    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1e0808e-008d-4b89-9728-6be8762bdb19" containerName="revoke-keys"
Jul 14 23:23:18.418993 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:18.418900    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1e0808e-008d-4b89-9728-6be8762bdb19" containerName="revoke-keys"
Jul 14 23:23:18.422076 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:18.422060    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37a8657d-7cm86"
Jul 14 23:23:18.424927 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:18.424908    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jw626\""
Jul 14 23:23:18.430301 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:18.430248    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37a8657d-7cm86"]
Jul 14 23:23:18.547913 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:18.547873    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c82f60e2-39e5-4c61-8fcc-331db1e25e24-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-37a8657d-7cm86\" (UID: \"c82f60e2-39e5-4c61-8fcc-331db1e25e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37a8657d-7cm86"
Jul 14 23:23:18.648909 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:18.648881    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c82f60e2-39e5-4c61-8fcc-331db1e25e24-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-37a8657d-7cm86\" (UID: \"c82f60e2-39e5-4c61-8fcc-331db1e25e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37a8657d-7cm86"
Jul 14 23:23:18.649502 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:18.649482    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c82f60e2-39e5-4c61-8fcc-331db1e25e24-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-37a8657d-7cm86\" (UID: \"c82f60e2-39e5-4c61-8fcc-331db1e25e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37a8657d-7cm86"
Jul 14 23:23:18.732886 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:18.732805    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37a8657d-7cm86"
Jul 14 23:23:18.863206 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:18.863183    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37a8657d-7cm86"]
Jul 14 23:23:18.865234 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:23:18.865205    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc82f60e2_39e5_4c61_8fcc_331db1e25e24.slice/crio-5186ab48de1f974198a203211d25ff96f7acbf2667f5455245705d6ae466e764 WatchSource:0}: Error finding container 5186ab48de1f974198a203211d25ff96f7acbf2667f5455245705d6ae466e764: Status 404 returned error can't find the container with id 5186ab48de1f974198a203211d25ff96f7acbf2667f5455245705d6ae466e764
Jul 14 23:23:19.067445 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:19.067404    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37a8657d-7cm86" event={"ID":"c82f60e2-39e5-4c61-8fcc-331db1e25e24","Type":"ContainerStarted","Data":"5186ab48de1f974198a203211d25ff96f7acbf2667f5455245705d6ae466e764"}
Jul 14 23:23:20.072899 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:20.072863    2563 generic.go:356] "Generic (PLEG): container finished" podID="c82f60e2-39e5-4c61-8fcc-331db1e25e24" containerID="a60a2c0b0ab2712c51456dde28834728d34a8bbff86a6fa3fe70630b30ab5a73" exitCode=0
Jul 14 23:23:20.073300 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:20.072957    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37a8657d-7cm86" event={"ID":"c82f60e2-39e5-4c61-8fcc-331db1e25e24","Type":"ContainerDied","Data":"a60a2c0b0ab2712c51456dde28834728d34a8bbff86a6fa3fe70630b30ab5a73"}
Jul 14 23:23:21.208678 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:21.208650    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37a8657d-7cm86"
Jul 14 23:23:21.274242 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:21.274208    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c82f60e2-39e5-4c61-8fcc-331db1e25e24-maas-api-service-ca\") pod \"c82f60e2-39e5-4c61-8fcc-331db1e25e24\" (UID: \"c82f60e2-39e5-4c61-8fcc-331db1e25e24\") "
Jul 14 23:23:21.274635 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:21.274604    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c82f60e2-39e5-4c61-8fcc-331db1e25e24-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c82f60e2-39e5-4c61-8fcc-331db1e25e24" (UID: "c82f60e2-39e5-4c61-8fcc-331db1e25e24"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 23:23:21.374996 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:21.374913    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c82f60e2-39e5-4c61-8fcc-331db1e25e24-maas-api-service-ca\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:23:22.082671 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:22.082641    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37a8657d-7cm86"
Jul 14 23:23:22.082850 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:22.082644    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37a8657d-7cm86" event={"ID":"c82f60e2-39e5-4c61-8fcc-331db1e25e24","Type":"ContainerDied","Data":"5186ab48de1f974198a203211d25ff96f7acbf2667f5455245705d6ae466e764"}
Jul 14 23:23:22.082850 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:22.082767    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5186ab48de1f974198a203211d25ff96f7acbf2667f5455245705d6ae466e764"
Jul 14 23:23:29.187952 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:29.187913    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-37a8657d-679bcd9946-8qvpb"]
Jul 14 23:23:29.188451 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:29.188110    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-37a8657d-679bcd9946-8qvpb" podUID="654409bc-7e3b-438b-9aea-a4e4407550e9" containerName="maas-api" containerID="cri-o://b2131245446dac98e07ffab77d62020997d0a95d1591e2114401aae03910113c" gracePeriod=30
Jul 14 23:23:29.433800 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:29.433776    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-37a8657d-679bcd9946-8qvpb"
Jul 14 23:23:29.546387 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:29.546345    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/654409bc-7e3b-438b-9aea-a4e4407550e9-maas-api-tls\") pod \"654409bc-7e3b-438b-9aea-a4e4407550e9\" (UID: \"654409bc-7e3b-438b-9aea-a4e4407550e9\") "
Jul 14 23:23:29.546387 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:29.546395    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdphx\" (UniqueName: \"kubernetes.io/projected/654409bc-7e3b-438b-9aea-a4e4407550e9-kube-api-access-qdphx\") pod \"654409bc-7e3b-438b-9aea-a4e4407550e9\" (UID: \"654409bc-7e3b-438b-9aea-a4e4407550e9\") "
Jul 14 23:23:29.546641 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:29.546452    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/654409bc-7e3b-438b-9aea-a4e4407550e9-openshift-service-ca\") pod \"654409bc-7e3b-438b-9aea-a4e4407550e9\" (UID: \"654409bc-7e3b-438b-9aea-a4e4407550e9\") "
Jul 14 23:23:29.546872 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:29.546827    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/654409bc-7e3b-438b-9aea-a4e4407550e9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "654409bc-7e3b-438b-9aea-a4e4407550e9" (UID: "654409bc-7e3b-438b-9aea-a4e4407550e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 23:23:29.548725 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:29.548696    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/654409bc-7e3b-438b-9aea-a4e4407550e9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "654409bc-7e3b-438b-9aea-a4e4407550e9" (UID: "654409bc-7e3b-438b-9aea-a4e4407550e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 23:23:29.548725 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:29.548706    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/654409bc-7e3b-438b-9aea-a4e4407550e9-kube-api-access-qdphx" (OuterVolumeSpecName: "kube-api-access-qdphx") pod "654409bc-7e3b-438b-9aea-a4e4407550e9" (UID: "654409bc-7e3b-438b-9aea-a4e4407550e9"). InnerVolumeSpecName "kube-api-access-qdphx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:23:29.647518 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:29.647476    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/654409bc-7e3b-438b-9aea-a4e4407550e9-maas-api-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:23:29.647518 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:29.647513    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdphx\" (UniqueName: \"kubernetes.io/projected/654409bc-7e3b-438b-9aea-a4e4407550e9-kube-api-access-qdphx\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:23:29.647518 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:29.647526    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/654409bc-7e3b-438b-9aea-a4e4407550e9-openshift-service-ca\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:23:30.120016 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:30.119979    2563 generic.go:356] "Generic (PLEG): container finished" podID="654409bc-7e3b-438b-9aea-a4e4407550e9" containerID="b2131245446dac98e07ffab77d62020997d0a95d1591e2114401aae03910113c" exitCode=0
Jul 14 23:23:30.120184 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:30.120024    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-37a8657d-679bcd9946-8qvpb"
Jul 14 23:23:30.120184 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:30.120044    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-37a8657d-679bcd9946-8qvpb" event={"ID":"654409bc-7e3b-438b-9aea-a4e4407550e9","Type":"ContainerDied","Data":"b2131245446dac98e07ffab77d62020997d0a95d1591e2114401aae03910113c"}
Jul 14 23:23:30.120184 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:30.120088    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-37a8657d-679bcd9946-8qvpb" event={"ID":"654409bc-7e3b-438b-9aea-a4e4407550e9","Type":"ContainerDied","Data":"f071057d0035d6047fc8645932c125bcb0a6d7923dc24187a51bbf93d9631fca"}
Jul 14 23:23:30.120184 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:30.120109    2563 scope.go:117] "RemoveContainer" containerID="b2131245446dac98e07ffab77d62020997d0a95d1591e2114401aae03910113c"
Jul 14 23:23:30.129596 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:30.129578    2563 scope.go:117] "RemoveContainer" containerID="b2131245446dac98e07ffab77d62020997d0a95d1591e2114401aae03910113c"
Jul 14 23:23:30.129859 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:23:30.129841    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2131245446dac98e07ffab77d62020997d0a95d1591e2114401aae03910113c\": container with ID starting with b2131245446dac98e07ffab77d62020997d0a95d1591e2114401aae03910113c not found: ID does not exist" containerID="b2131245446dac98e07ffab77d62020997d0a95d1591e2114401aae03910113c"
Jul 14 23:23:30.129910 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:30.129869    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2131245446dac98e07ffab77d62020997d0a95d1591e2114401aae03910113c"} err="failed to get container status \"b2131245446dac98e07ffab77d62020997d0a95d1591e2114401aae03910113c\": rpc error: code = NotFound desc = could not find container \"b2131245446dac98e07ffab77d62020997d0a95d1591e2114401aae03910113c\": container with ID starting with b2131245446dac98e07ffab77d62020997d0a95d1591e2114401aae03910113c not found: ID does not exist"
Jul 14 23:23:30.141541 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:30.141517    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-37a8657d-679bcd9946-8qvpb"]
Jul 14 23:23:30.145918 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:30.145891    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-37a8657d-679bcd9946-8qvpb"]
Jul 14 23:23:31.228705 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:31.228669    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="654409bc-7e3b-438b-9aea-a4e4407550e9" path="/var/lib/kubelet/pods/654409bc-7e3b-438b-9aea-a4e4407550e9/volumes"
Jul 14 23:23:44.263607 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:44.263569    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"]
Jul 14 23:23:44.264092 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:44.264036    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1e0808e-008d-4b89-9728-6be8762bdb19" containerName="revoke-keys"
Jul 14 23:23:44.264092 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:44.264051    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1e0808e-008d-4b89-9728-6be8762bdb19" containerName="revoke-keys"
Jul 14 23:23:44.264092 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:44.264069    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="654409bc-7e3b-438b-9aea-a4e4407550e9" containerName="maas-api"
Jul 14 23:23:44.264092 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:44.264077    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="654409bc-7e3b-438b-9aea-a4e4407550e9" containerName="maas-api"
Jul 14 23:23:44.264344 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:44.264097    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c82f60e2-39e5-4c61-8fcc-331db1e25e24" containerName="revoke-keys"
Jul 14 23:23:44.264344 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:44.264106    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c82f60e2-39e5-4c61-8fcc-331db1e25e24" containerName="revoke-keys"
Jul 14 23:23:44.264344 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:44.264198    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="654409bc-7e3b-438b-9aea-a4e4407550e9" containerName="maas-api"
Jul 14 23:23:44.264344 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:44.264210    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c82f60e2-39e5-4c61-8fcc-331db1e25e24" containerName="revoke-keys"
Jul 14 23:23:44.268862 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:44.268839    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"
Jul 14 23:23:44.271793 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:44.271770    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 14 23:23:44.271911 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:44.271850    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-1731167b-gw-openshift-default-dockercfg-8cqdf\""
Jul 14 23:23:44.279477 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:44.279454    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"]
Jul 14 23:23:44.374826 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:44.374777    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wj8l\" (UniqueName: \"kubernetes.io/projected/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-kube-api-access-7wj8l\") pod \"e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc\" (UID: \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\") " pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"
Jul 14 23:23:44.375028 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:44.374844    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-istio-token\") pod \"e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc\" (UID: \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\") " pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"
Jul 14 23:23:44.375028 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:44.374931    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-istio-data\") pod \"e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc\" (UID: \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\") " pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"
Jul 14 23:23:44.375028 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:44.374996    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-workload-certs\") pod \"e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc\" (UID: \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\") " pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"
Jul 14 23:23:44.375028 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:44.375020    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-istio-envoy\") pod \"e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc\" (UID: \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\") " pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"
Jul 14 23:23:44.375239 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:44.375074    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-workload-socket\") pod \"e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc\" (UID: \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\") " pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"
Jul 14 23:23:44.375239 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:44.375181    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-istio-podinfo\") pod \"e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc\" (UID: \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\") " pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"
Jul 14 23:23:44.375239 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:44.375212    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-istiod-ca-cert\") pod \"e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc\" (UID: \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\") " pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"
Jul 14 23:23:44.375389 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:44.375325    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-credential-socket\") pod \"e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc\" (UID: \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\") " pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"
Jul 14 23:23:44.476722 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:44.476682    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wj8l\" (UniqueName: \"kubernetes.io/projected/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-kube-api-access-7wj8l\") pod \"e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc\" (UID: \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\") " pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"
Jul 14 23:23:44.476899 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:44.476733    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-istio-token\") pod \"e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc\" (UID: \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\") " pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"
Jul 14 23:23:44.476899 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:44.476771    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-istio-data\") pod \"e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc\" (UID: \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\") " pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"
Jul 14 23:23:44.476899 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:44.476804    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-workload-certs\") pod \"e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc\" (UID: \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\") " pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"
Jul 14 23:23:44.476899 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:44.476824    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-istio-envoy\") pod \"e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc\" (UID: \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\") " pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"
Jul 14 23:23:44.477091 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:44.476977    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-workload-socket\") pod \"e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc\" (UID: \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\") " pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"
Jul 14 23:23:44.477091 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:44.477040    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-istio-podinfo\") pod \"e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc\" (UID: \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\") " pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"
Jul 14 23:23:44.477091 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:44.477069    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-istiod-ca-cert\") pod \"e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc\" (UID: \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\") " pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"
Jul 14 23:23:44.477238 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:44.477115    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-credential-socket\") pod \"e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc\" (UID: \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\") " pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"
Jul 14 23:23:44.477238 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:44.477205    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-istio-data\") pod \"e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc\" (UID: \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\") " pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"
Jul 14 23:23:44.477385 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:44.477264    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-workload-certs\") pod \"e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc\" (UID: \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\") " pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"
Jul 14 23:23:44.477451 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:44.477434    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-credential-socket\") pod \"e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc\" (UID: \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\") " pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"
Jul 14 23:23:44.477507 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:44.477430    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-workload-socket\") pod \"e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc\" (UID: \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\") " pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"
Jul 14 23:23:44.477808 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:44.477786    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-istiod-ca-cert\") pod \"e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc\" (UID: \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\") " pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"
Jul 14 23:23:44.479238 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:44.479210    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-istio-envoy\") pod \"e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc\" (UID: \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\") " pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"
Jul 14 23:23:44.479526 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:44.479508    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-istio-podinfo\") pod \"e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc\" (UID: \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\") " pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"
Jul 14 23:23:44.486394 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:44.486367    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-istio-token\") pod \"e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc\" (UID: \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\") " pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"
Jul 14 23:23:44.486551 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:44.486529    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wj8l\" (UniqueName: \"kubernetes.io/projected/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-kube-api-access-7wj8l\") pod \"e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc\" (UID: \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\") " pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"
Jul 14 23:23:44.582688 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:44.582594    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"
Jul 14 23:23:44.720535 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:44.720463    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"]
Jul 14 23:23:44.723905 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:23:44.723876    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod862dbc52_1cb1_4b06_9a65_55bbb0a4e3d3.slice/crio-404f1c40957b54e6a77d0de662325edded452dd56e1e5a6790a28b6bdc0ebbe5 WatchSource:0}: Error finding container 404f1c40957b54e6a77d0de662325edded452dd56e1e5a6790a28b6bdc0ebbe5: Status 404 returned error can't find the container with id 404f1c40957b54e6a77d0de662325edded452dd56e1e5a6790a28b6bdc0ebbe5
Jul 14 23:23:45.184973 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:45.184932    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc" event={"ID":"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3","Type":"ContainerStarted","Data":"404f1c40957b54e6a77d0de662325edded452dd56e1e5a6790a28b6bdc0ebbe5"}
Jul 14 23:23:45.649884 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:45.649849    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1731167b-7bc969d7b-445nr"]
Jul 14 23:23:45.653519 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:45.653493    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1731167b-7bc969d7b-445nr"
Jul 14 23:23:45.656904 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:45.656882    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-1731167b-serving-cert\""
Jul 14 23:23:45.657511 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:45.657487    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 23:23:45.658487 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:45.658465    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7jvtc\""
Jul 14 23:23:45.658487 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:45.658479    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 23:23:45.680722 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:45.680658    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1731167b-7bc969d7b-445nr"]
Jul 14 23:23:45.789957 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:45.789921    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8f71f0d-d731-4f0f-9df9-4a2d7d774feb-openshift-service-ca\") pod \"maas-api-e2e-migrate-1731167b-7bc969d7b-445nr\" (UID: \"a8f71f0d-d731-4f0f-9df9-4a2d7d774feb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1731167b-7bc969d7b-445nr"
Jul 14 23:23:45.790150 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:45.789999    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp7zb\" (UniqueName: \"kubernetes.io/projected/a8f71f0d-d731-4f0f-9df9-4a2d7d774feb-kube-api-access-dp7zb\") pod \"maas-api-e2e-migrate-1731167b-7bc969d7b-445nr\" (UID: \"a8f71f0d-d731-4f0f-9df9-4a2d7d774feb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1731167b-7bc969d7b-445nr"
Jul 14 23:23:45.790150 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:45.790055    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8f71f0d-d731-4f0f-9df9-4a2d7d774feb-maas-api-tls\") pod \"maas-api-e2e-migrate-1731167b-7bc969d7b-445nr\" (UID: \"a8f71f0d-d731-4f0f-9df9-4a2d7d774feb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1731167b-7bc969d7b-445nr"
Jul 14 23:23:45.890598 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:45.890562    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8f71f0d-d731-4f0f-9df9-4a2d7d774feb-maas-api-tls\") pod \"maas-api-e2e-migrate-1731167b-7bc969d7b-445nr\" (UID: \"a8f71f0d-d731-4f0f-9df9-4a2d7d774feb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1731167b-7bc969d7b-445nr"
Jul 14 23:23:45.890790 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:45.890655    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8f71f0d-d731-4f0f-9df9-4a2d7d774feb-openshift-service-ca\") pod \"maas-api-e2e-migrate-1731167b-7bc969d7b-445nr\" (UID: \"a8f71f0d-d731-4f0f-9df9-4a2d7d774feb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1731167b-7bc969d7b-445nr"
Jul 14 23:23:45.890790 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:45.890728    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp7zb\" (UniqueName: \"kubernetes.io/projected/a8f71f0d-d731-4f0f-9df9-4a2d7d774feb-kube-api-access-dp7zb\") pod \"maas-api-e2e-migrate-1731167b-7bc969d7b-445nr\" (UID: \"a8f71f0d-d731-4f0f-9df9-4a2d7d774feb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1731167b-7bc969d7b-445nr"
Jul 14 23:23:45.891518 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:45.891488    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8f71f0d-d731-4f0f-9df9-4a2d7d774feb-openshift-service-ca\") pod \"maas-api-e2e-migrate-1731167b-7bc969d7b-445nr\" (UID: \"a8f71f0d-d731-4f0f-9df9-4a2d7d774feb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1731167b-7bc969d7b-445nr"
Jul 14 23:23:45.893405 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:45.893384    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8f71f0d-d731-4f0f-9df9-4a2d7d774feb-maas-api-tls\") pod \"maas-api-e2e-migrate-1731167b-7bc969d7b-445nr\" (UID: \"a8f71f0d-d731-4f0f-9df9-4a2d7d774feb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1731167b-7bc969d7b-445nr"
Jul 14 23:23:45.900037 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:45.899983    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp7zb\" (UniqueName: \"kubernetes.io/projected/a8f71f0d-d731-4f0f-9df9-4a2d7d774feb-kube-api-access-dp7zb\") pod \"maas-api-e2e-migrate-1731167b-7bc969d7b-445nr\" (UID: \"a8f71f0d-d731-4f0f-9df9-4a2d7d774feb\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1731167b-7bc969d7b-445nr"
Jul 14 23:23:45.966932 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:45.966896    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1731167b-7bc969d7b-445nr"
Jul 14 23:23:46.973137 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:46.973096    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:23:46.973508 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:46.973207    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:23:46.973508 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:46.973258    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:23:47.043230 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:47.043205    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1731167b-7bc969d7b-445nr"]
Jul 14 23:23:47.045205 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:23:47.045183    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8f71f0d_d731_4f0f_9df9_4a2d7d774feb.slice/crio-badeda5c0a25a2ce624891e987d5c64c9d957ca1e233cb6660439c79cac433da WatchSource:0}: Error finding container badeda5c0a25a2ce624891e987d5c64c9d957ca1e233cb6660439c79cac433da: Status 404 returned error can't find the container with id badeda5c0a25a2ce624891e987d5c64c9d957ca1e233cb6660439c79cac433da
Jul 14 23:23:47.195173 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:47.195085    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1731167b-7bc969d7b-445nr" event={"ID":"a8f71f0d-d731-4f0f-9df9-4a2d7d774feb","Type":"ContainerStarted","Data":"badeda5c0a25a2ce624891e987d5c64c9d957ca1e233cb6660439c79cac433da"}
Jul 14 23:23:47.196711 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:47.196682    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc" event={"ID":"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3","Type":"ContainerStarted","Data":"cf01ec04138c727c1abce6ccf3aacf75c0aca026fa9c1dd505bcdca2c8c8a4c2"}
Jul 14 23:23:47.219511 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:47.219465    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc" podStartSLOduration=0.972465458 podStartE2EDuration="3.21944792s" totalImagesPullingTime="2.246982462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:23:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:23:47.217207123 +0000 UTC m=+1614.520197916" watchObservedRunningTime="2026-07-14 23:23:47.21944792 +0000 UTC m=+1614.522438712"
Jul 14 23:23:47.583527 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:47.583491    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"
Jul 14 23:23:48.203542 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:48.203504    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1731167b-7bc969d7b-445nr" event={"ID":"a8f71f0d-d731-4f0f-9df9-4a2d7d774feb","Type":"ContainerStarted","Data":"443ea25a53e7ae1d009aea70577ffd827835ec7dad7d1db29e1aa4ef0daa6d00"}
Jul 14 23:23:48.203917 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:48.203719    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1731167b-7bc969d7b-445nr"
Jul 14 23:23:48.220269 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:48.220218    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1731167b-7bc969d7b-445nr" podStartSLOduration=2.88057231 podStartE2EDuration="3.220202062s" totalImagesPullingTime="339.629752ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:23:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:23:48.219556331 +0000 UTC m=+1615.522547122" watchObservedRunningTime="2026-07-14 23:23:48.220202062 +0000 UTC m=+1615.523192854"
Jul 14 23:23:48.588359 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:48.588325    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"
Jul 14 23:23:49.208428 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:49.208401    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"
Jul 14 23:23:49.209355 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:49.209338    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"
Jul 14 23:23:54.214827 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:23:54.214796    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1731167b-7bc969d7b-445nr"
Jul 14 23:24:00.515816 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:00.515776    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1731167b-7bc969d7b-445nr"]
Jul 14 23:24:00.516339 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:00.516056    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1731167b-7bc969d7b-445nr" podUID="a8f71f0d-d731-4f0f-9df9-4a2d7d774feb" containerName="maas-api" containerID="cri-o://443ea25a53e7ae1d009aea70577ffd827835ec7dad7d1db29e1aa4ef0daa6d00" gracePeriod=30
Jul 14 23:24:00.762055 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:00.762024    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1731167b-7bc969d7b-445nr"
Jul 14 23:24:00.822614 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:00.822533    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8f71f0d-d731-4f0f-9df9-4a2d7d774feb-maas-api-tls\") pod \"a8f71f0d-d731-4f0f-9df9-4a2d7d774feb\" (UID: \"a8f71f0d-d731-4f0f-9df9-4a2d7d774feb\") "
Jul 14 23:24:00.822748 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:00.822613    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dp7zb\" (UniqueName: \"kubernetes.io/projected/a8f71f0d-d731-4f0f-9df9-4a2d7d774feb-kube-api-access-dp7zb\") pod \"a8f71f0d-d731-4f0f-9df9-4a2d7d774feb\" (UID: \"a8f71f0d-d731-4f0f-9df9-4a2d7d774feb\") "
Jul 14 23:24:00.822748 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:00.822636    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8f71f0d-d731-4f0f-9df9-4a2d7d774feb-openshift-service-ca\") pod \"a8f71f0d-d731-4f0f-9df9-4a2d7d774feb\" (UID: \"a8f71f0d-d731-4f0f-9df9-4a2d7d774feb\") "
Jul 14 23:24:00.823045 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:00.823014    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8f71f0d-d731-4f0f-9df9-4a2d7d774feb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a8f71f0d-d731-4f0f-9df9-4a2d7d774feb" (UID: "a8f71f0d-d731-4f0f-9df9-4a2d7d774feb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 23:24:00.824761 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:00.824734    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8f71f0d-d731-4f0f-9df9-4a2d7d774feb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a8f71f0d-d731-4f0f-9df9-4a2d7d774feb" (UID: "a8f71f0d-d731-4f0f-9df9-4a2d7d774feb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 23:24:00.824880 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:00.824849    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8f71f0d-d731-4f0f-9df9-4a2d7d774feb-kube-api-access-dp7zb" (OuterVolumeSpecName: "kube-api-access-dp7zb") pod "a8f71f0d-d731-4f0f-9df9-4a2d7d774feb" (UID: "a8f71f0d-d731-4f0f-9df9-4a2d7d774feb"). InnerVolumeSpecName "kube-api-access-dp7zb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:24:00.924097 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:00.924064    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8f71f0d-d731-4f0f-9df9-4a2d7d774feb-maas-api-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:24:00.924097 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:00.924095    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp7zb\" (UniqueName: \"kubernetes.io/projected/a8f71f0d-d731-4f0f-9df9-4a2d7d774feb-kube-api-access-dp7zb\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:24:00.924097 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:00.924104    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8f71f0d-d731-4f0f-9df9-4a2d7d774feb-openshift-service-ca\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:24:01.266862 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:01.266828    2563 generic.go:356] "Generic (PLEG): container finished" podID="a8f71f0d-d731-4f0f-9df9-4a2d7d774feb" containerID="443ea25a53e7ae1d009aea70577ffd827835ec7dad7d1db29e1aa4ef0daa6d00" exitCode=0
Jul 14 23:24:01.266999 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:01.266869    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1731167b-7bc969d7b-445nr"
Jul 14 23:24:01.266999 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:01.266906    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1731167b-7bc969d7b-445nr" event={"ID":"a8f71f0d-d731-4f0f-9df9-4a2d7d774feb","Type":"ContainerDied","Data":"443ea25a53e7ae1d009aea70577ffd827835ec7dad7d1db29e1aa4ef0daa6d00"}
Jul 14 23:24:01.266999 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:01.266969    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1731167b-7bc969d7b-445nr" event={"ID":"a8f71f0d-d731-4f0f-9df9-4a2d7d774feb","Type":"ContainerDied","Data":"badeda5c0a25a2ce624891e987d5c64c9d957ca1e233cb6660439c79cac433da"}
Jul 14 23:24:01.267119 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:01.267006    2563 scope.go:117] "RemoveContainer" containerID="443ea25a53e7ae1d009aea70577ffd827835ec7dad7d1db29e1aa4ef0daa6d00"
Jul 14 23:24:01.277453 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:01.277437    2563 scope.go:117] "RemoveContainer" containerID="443ea25a53e7ae1d009aea70577ffd827835ec7dad7d1db29e1aa4ef0daa6d00"
Jul 14 23:24:01.277736 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:24:01.277719    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"443ea25a53e7ae1d009aea70577ffd827835ec7dad7d1db29e1aa4ef0daa6d00\": container with ID starting with 443ea25a53e7ae1d009aea70577ffd827835ec7dad7d1db29e1aa4ef0daa6d00 not found: ID does not exist" containerID="443ea25a53e7ae1d009aea70577ffd827835ec7dad7d1db29e1aa4ef0daa6d00"
Jul 14 23:24:01.277787 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:01.277746    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"443ea25a53e7ae1d009aea70577ffd827835ec7dad7d1db29e1aa4ef0daa6d00"} err="failed to get container status \"443ea25a53e7ae1d009aea70577ffd827835ec7dad7d1db29e1aa4ef0daa6d00\": rpc error: code = NotFound desc = could not find container \"443ea25a53e7ae1d009aea70577ffd827835ec7dad7d1db29e1aa4ef0daa6d00\": container with ID starting with 443ea25a53e7ae1d009aea70577ffd827835ec7dad7d1db29e1aa4ef0daa6d00 not found: ID does not exist"
Jul 14 23:24:01.286180 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:01.286158    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1731167b-7bc969d7b-445nr"]
Jul 14 23:24:01.291063 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:01.291042    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1731167b-7bc969d7b-445nr"]
Jul 14 23:24:02.273353 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:02.273320    2563 generic.go:356] "Generic (PLEG): container finished" podID="862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3" containerID="cf01ec04138c727c1abce6ccf3aacf75c0aca026fa9c1dd505bcdca2c8c8a4c2" exitCode=0
Jul 14 23:24:02.273815 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:02.273368    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc" event={"ID":"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3","Type":"ContainerDied","Data":"cf01ec04138c727c1abce6ccf3aacf75c0aca026fa9c1dd505bcdca2c8c8a4c2"}
Jul 14 23:24:02.273815 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:02.273713    2563 scope.go:117] "RemoveContainer" containerID="cf01ec04138c727c1abce6ccf3aacf75c0aca026fa9c1dd505bcdca2c8c8a4c2"
Jul 14 23:24:02.274697 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:02.274666    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:24:02.274757 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:02.274740    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:24:02.274796 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:02.274774    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:24:03.247066 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:03.247027    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8f71f0d-d731-4f0f-9df9-4a2d7d774feb" path="/var/lib/kubelet/pods/a8f71f0d-d731-4f0f-9df9-4a2d7d774feb/volumes"
Jul 14 23:24:03.279470 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:03.279435    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc" event={"ID":"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3","Type":"ContainerStarted","Data":"bc7f9530cc74d7a73887a02de83d110dfc80657256a170aa9575fd580b9c8549"}
Jul 14 23:24:03.582971 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:03.582885    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"
Jul 14 23:24:03.584048 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:03.584024    2563 patch_prober.go:28] interesting pod/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Jul 14 23:24:03.584164 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:03.584081    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc" podUID="862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Jul 14 23:24:04.208861 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:04.208823    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"
Jul 14 23:24:04.584012 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:04.583970    2563 patch_prober.go:28] interesting pod/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Jul 14 23:24:04.584439 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:04.584041    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc" podUID="862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Jul 14 23:24:05.583821 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:05.583781    2563 patch_prober.go:28] interesting pod/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Jul 14 23:24:05.583998 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:05.583855    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc" podUID="862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Jul 14 23:24:06.583566 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:06.583535    2563 patch_prober.go:28] interesting pod/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Jul 14 23:24:06.583954 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:06.583596    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc" podUID="862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Jul 14 23:24:07.587827 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:07.587788    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"
Jul 14 23:24:07.588912 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:07.588891    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"
Jul 14 23:24:15.316178 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:15.316147    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"]
Jul 14 23:24:15.316618 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:15.316410    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc" podUID="862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3" containerName="istio-proxy" containerID="cri-o://bc7f9530cc74d7a73887a02de83d110dfc80657256a170aa9575fd580b9c8549" gracePeriod=30
Jul 14 23:24:15.632121 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:15.632043    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"]
Jul 14 23:24:15.632589 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:15.632569    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8f71f0d-d731-4f0f-9df9-4a2d7d774feb" containerName="maas-api"
Jul 14 23:24:15.632635 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:15.632595    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8f71f0d-d731-4f0f-9df9-4a2d7d774feb" containerName="maas-api"
Jul 14 23:24:15.632685 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:15.632673    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8f71f0d-d731-4f0f-9df9-4a2d7d774feb" containerName="maas-api"
Jul 14 23:24:15.641037 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:15.641015    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"
Jul 14 23:24:15.643975 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:15.643953    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c1ba7d12-openshift-default-dockercfg-zpv5n\""
Jul 14 23:24:15.646727 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:15.646700    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"]
Jul 14 23:24:15.755406 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:15.755375    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a70b30a0-baff-466e-b04a-e47a712ff7ee-istio-envoy\") pod \"e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t\" (UID: \"a70b30a0-baff-466e-b04a-e47a712ff7ee\") " pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"
Jul 14 23:24:15.755558 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:15.755409    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a70b30a0-baff-466e-b04a-e47a712ff7ee-workload-certs\") pod \"e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t\" (UID: \"a70b30a0-baff-466e-b04a-e47a712ff7ee\") " pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"
Jul 14 23:24:15.755558 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:15.755430    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a70b30a0-baff-466e-b04a-e47a712ff7ee-istio-podinfo\") pod \"e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t\" (UID: \"a70b30a0-baff-466e-b04a-e47a712ff7ee\") " pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"
Jul 14 23:24:15.755558 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:15.755449    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a70b30a0-baff-466e-b04a-e47a712ff7ee-istiod-ca-cert\") pod \"e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t\" (UID: \"a70b30a0-baff-466e-b04a-e47a712ff7ee\") " pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"
Jul 14 23:24:15.755558 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:15.755483    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a70b30a0-baff-466e-b04a-e47a712ff7ee-workload-socket\") pod \"e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t\" (UID: \"a70b30a0-baff-466e-b04a-e47a712ff7ee\") " pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"
Jul 14 23:24:15.755558 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:15.755501    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a70b30a0-baff-466e-b04a-e47a712ff7ee-istio-data\") pod \"e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t\" (UID: \"a70b30a0-baff-466e-b04a-e47a712ff7ee\") " pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"
Jul 14 23:24:15.755558 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:15.755517    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2khp7\" (UniqueName: \"kubernetes.io/projected/a70b30a0-baff-466e-b04a-e47a712ff7ee-kube-api-access-2khp7\") pod \"e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t\" (UID: \"a70b30a0-baff-466e-b04a-e47a712ff7ee\") " pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"
Jul 14 23:24:15.755773 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:15.755623    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a70b30a0-baff-466e-b04a-e47a712ff7ee-credential-socket\") pod \"e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t\" (UID: \"a70b30a0-baff-466e-b04a-e47a712ff7ee\") " pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"
Jul 14 23:24:15.755773 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:15.755652    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a70b30a0-baff-466e-b04a-e47a712ff7ee-istio-token\") pod \"e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t\" (UID: \"a70b30a0-baff-466e-b04a-e47a712ff7ee\") " pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"
Jul 14 23:24:15.856662 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:15.856629    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a70b30a0-baff-466e-b04a-e47a712ff7ee-credential-socket\") pod \"e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t\" (UID: \"a70b30a0-baff-466e-b04a-e47a712ff7ee\") " pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"
Jul 14 23:24:15.856662 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:15.856663    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a70b30a0-baff-466e-b04a-e47a712ff7ee-istio-token\") pod \"e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t\" (UID: \"a70b30a0-baff-466e-b04a-e47a712ff7ee\") " pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"
Jul 14 23:24:15.856859 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:15.856693    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a70b30a0-baff-466e-b04a-e47a712ff7ee-istio-envoy\") pod \"e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t\" (UID: \"a70b30a0-baff-466e-b04a-e47a712ff7ee\") " pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"
Jul 14 23:24:15.856859 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:15.856716    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a70b30a0-baff-466e-b04a-e47a712ff7ee-workload-certs\") pod \"e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t\" (UID: \"a70b30a0-baff-466e-b04a-e47a712ff7ee\") " pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"
Jul 14 23:24:15.856859 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:15.856738    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a70b30a0-baff-466e-b04a-e47a712ff7ee-istio-podinfo\") pod \"e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t\" (UID: \"a70b30a0-baff-466e-b04a-e47a712ff7ee\") " pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"
Jul 14 23:24:15.856859 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:15.856767    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a70b30a0-baff-466e-b04a-e47a712ff7ee-istiod-ca-cert\") pod \"e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t\" (UID: \"a70b30a0-baff-466e-b04a-e47a712ff7ee\") " pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"
Jul 14 23:24:15.856859 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:15.856812    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a70b30a0-baff-466e-b04a-e47a712ff7ee-workload-socket\") pod \"e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t\" (UID: \"a70b30a0-baff-466e-b04a-e47a712ff7ee\") " pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"
Jul 14 23:24:15.856859 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:15.856837    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a70b30a0-baff-466e-b04a-e47a712ff7ee-istio-data\") pod \"e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t\" (UID: \"a70b30a0-baff-466e-b04a-e47a712ff7ee\") " pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"
Jul 14 23:24:15.856859 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:15.856857    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2khp7\" (UniqueName: \"kubernetes.io/projected/a70b30a0-baff-466e-b04a-e47a712ff7ee-kube-api-access-2khp7\") pod \"e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t\" (UID: \"a70b30a0-baff-466e-b04a-e47a712ff7ee\") " pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"
Jul 14 23:24:15.857251 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:15.857187    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a70b30a0-baff-466e-b04a-e47a712ff7ee-workload-certs\") pod \"e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t\" (UID: \"a70b30a0-baff-466e-b04a-e47a712ff7ee\") " pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"
Jul 14 23:24:15.857336 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:15.857257    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a70b30a0-baff-466e-b04a-e47a712ff7ee-workload-socket\") pod \"e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t\" (UID: \"a70b30a0-baff-466e-b04a-e47a712ff7ee\") " pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"
Jul 14 23:24:15.857382 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:15.857264    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a70b30a0-baff-466e-b04a-e47a712ff7ee-istio-data\") pod \"e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t\" (UID: \"a70b30a0-baff-466e-b04a-e47a712ff7ee\") " pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"
Jul 14 23:24:15.857427 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:15.857379    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a70b30a0-baff-466e-b04a-e47a712ff7ee-credential-socket\") pod \"e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t\" (UID: \"a70b30a0-baff-466e-b04a-e47a712ff7ee\") " pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"
Jul 14 23:24:15.857768 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:15.857743    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a70b30a0-baff-466e-b04a-e47a712ff7ee-istiod-ca-cert\") pod \"e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t\" (UID: \"a70b30a0-baff-466e-b04a-e47a712ff7ee\") " pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"
Jul 14 23:24:15.859187 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:15.859170    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a70b30a0-baff-466e-b04a-e47a712ff7ee-istio-envoy\") pod \"e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t\" (UID: \"a70b30a0-baff-466e-b04a-e47a712ff7ee\") " pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"
Jul 14 23:24:15.859473 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:15.859453    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a70b30a0-baff-466e-b04a-e47a712ff7ee-istio-podinfo\") pod \"e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t\" (UID: \"a70b30a0-baff-466e-b04a-e47a712ff7ee\") " pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"
Jul 14 23:24:15.865847 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:15.865820    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a70b30a0-baff-466e-b04a-e47a712ff7ee-istio-token\") pod \"e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t\" (UID: \"a70b30a0-baff-466e-b04a-e47a712ff7ee\") " pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"
Jul 14 23:24:15.866084 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:15.866062    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2khp7\" (UniqueName: \"kubernetes.io/projected/a70b30a0-baff-466e-b04a-e47a712ff7ee-kube-api-access-2khp7\") pod \"e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t\" (UID: \"a70b30a0-baff-466e-b04a-e47a712ff7ee\") " pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"
Jul 14 23:24:15.952556 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:15.952485    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"
Jul 14 23:24:16.091223 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:16.091199    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"]
Jul 14 23:24:16.093567 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:24:16.093541    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda70b30a0_baff_466e_b04a_e47a712ff7ee.slice/crio-b6803fe8ffb3169c404e4b5fa071b5df80eb3a34ad0134fd90bd4cfe4d1378b3 WatchSource:0}: Error finding container b6803fe8ffb3169c404e4b5fa071b5df80eb3a34ad0134fd90bd4cfe4d1378b3: Status 404 returned error can't find the container with id b6803fe8ffb3169c404e4b5fa071b5df80eb3a34ad0134fd90bd4cfe4d1378b3
Jul 14 23:24:16.096163 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:16.096131    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:24:16.096215 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:16.096199    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:24:16.096250 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:16.096226    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:24:16.338749 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:16.338712    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t" event={"ID":"a70b30a0-baff-466e-b04a-e47a712ff7ee","Type":"ContainerStarted","Data":"b4ebf5a5f8bd353e8b90e6d3ac50b00e02400588e9d03e0ffd7748f3388fcb43"}
Jul 14 23:24:16.339101 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:16.338763    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t" event={"ID":"a70b30a0-baff-466e-b04a-e47a712ff7ee","Type":"ContainerStarted","Data":"b6803fe8ffb3169c404e4b5fa071b5df80eb3a34ad0134fd90bd4cfe4d1378b3"}
Jul 14 23:24:16.360506 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:16.360448    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t" podStartSLOduration=1.360429259 podStartE2EDuration="1.360429259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:24:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:24:16.358668504 +0000 UTC m=+1643.661659296" watchObservedRunningTime="2026-07-14 23:24:16.360429259 +0000 UTC m=+1643.663420053"
Jul 14 23:24:16.953195 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:16.953153    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"
Jul 14 23:24:16.989110 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:16.989076    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c1ba7d12-79c896b84b-bbtrh"]
Jul 14 23:24:16.992796 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:16.992774    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1ba7d12-79c896b84b-bbtrh"
Jul 14 23:24:16.995763 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:16.995739    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7jvtc\""
Jul 14 23:24:16.995763 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:16.995746    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 23:24:16.997071 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:16.997052    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 23:24:16.997157 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:16.997087    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c1ba7d12-serving-cert\""
Jul 14 23:24:17.005234 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:17.005210    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c1ba7d12-79c896b84b-bbtrh"]
Jul 14 23:24:17.065973 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:17.065942    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0621c190-fe95-442e-a82c-d3cdbc14d1c7-openshift-service-ca\") pod \"maas-api-e2e-ait-c1ba7d12-79c896b84b-bbtrh\" (UID: \"0621c190-fe95-442e-a82c-d3cdbc14d1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1ba7d12-79c896b84b-bbtrh"
Jul 14 23:24:17.066134 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:17.065992    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwlfw\" (UniqueName: \"kubernetes.io/projected/0621c190-fe95-442e-a82c-d3cdbc14d1c7-kube-api-access-hwlfw\") pod \"maas-api-e2e-ait-c1ba7d12-79c896b84b-bbtrh\" (UID: \"0621c190-fe95-442e-a82c-d3cdbc14d1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1ba7d12-79c896b84b-bbtrh"
Jul 14 23:24:17.066134 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:17.066097    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0621c190-fe95-442e-a82c-d3cdbc14d1c7-maas-api-tls\") pod \"maas-api-e2e-ait-c1ba7d12-79c896b84b-bbtrh\" (UID: \"0621c190-fe95-442e-a82c-d3cdbc14d1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1ba7d12-79c896b84b-bbtrh"
Jul 14 23:24:17.166959 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:17.166913    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0621c190-fe95-442e-a82c-d3cdbc14d1c7-maas-api-tls\") pod \"maas-api-e2e-ait-c1ba7d12-79c896b84b-bbtrh\" (UID: \"0621c190-fe95-442e-a82c-d3cdbc14d1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1ba7d12-79c896b84b-bbtrh"
Jul 14 23:24:17.167148 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:17.167009    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0621c190-fe95-442e-a82c-d3cdbc14d1c7-openshift-service-ca\") pod \"maas-api-e2e-ait-c1ba7d12-79c896b84b-bbtrh\" (UID: \"0621c190-fe95-442e-a82c-d3cdbc14d1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1ba7d12-79c896b84b-bbtrh"
Jul 14 23:24:17.167148 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:17.167037    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwlfw\" (UniqueName: \"kubernetes.io/projected/0621c190-fe95-442e-a82c-d3cdbc14d1c7-kube-api-access-hwlfw\") pod \"maas-api-e2e-ait-c1ba7d12-79c896b84b-bbtrh\" (UID: \"0621c190-fe95-442e-a82c-d3cdbc14d1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1ba7d12-79c896b84b-bbtrh"
Jul 14 23:24:17.167148 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:24:17.167062    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-c1ba7d12-serving-cert: secret "maas-api-e2e-ait-c1ba7d12-serving-cert" not found
Jul 14 23:24:17.167148 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:24:17.167133    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0621c190-fe95-442e-a82c-d3cdbc14d1c7-maas-api-tls podName:0621c190-fe95-442e-a82c-d3cdbc14d1c7 nodeName:}" failed. No retries permitted until 2026-07-14 23:24:17.667113925 +0000 UTC m=+1644.970104698 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0621c190-fe95-442e-a82c-d3cdbc14d1c7-maas-api-tls") pod "maas-api-e2e-ait-c1ba7d12-79c896b84b-bbtrh" (UID: "0621c190-fe95-442e-a82c-d3cdbc14d1c7") : secret "maas-api-e2e-ait-c1ba7d12-serving-cert" not found
Jul 14 23:24:17.167715 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:17.167697    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0621c190-fe95-442e-a82c-d3cdbc14d1c7-openshift-service-ca\") pod \"maas-api-e2e-ait-c1ba7d12-79c896b84b-bbtrh\" (UID: \"0621c190-fe95-442e-a82c-d3cdbc14d1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1ba7d12-79c896b84b-bbtrh"
Jul 14 23:24:17.184886 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:17.184860    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwlfw\" (UniqueName: \"kubernetes.io/projected/0621c190-fe95-442e-a82c-d3cdbc14d1c7-kube-api-access-hwlfw\") pod \"maas-api-e2e-ait-c1ba7d12-79c896b84b-bbtrh\" (UID: \"0621c190-fe95-442e-a82c-d3cdbc14d1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1ba7d12-79c896b84b-bbtrh"
Jul 14 23:24:17.672678 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:17.672644    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0621c190-fe95-442e-a82c-d3cdbc14d1c7-maas-api-tls\") pod \"maas-api-e2e-ait-c1ba7d12-79c896b84b-bbtrh\" (UID: \"0621c190-fe95-442e-a82c-d3cdbc14d1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1ba7d12-79c896b84b-bbtrh"
Jul 14 23:24:17.675326 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:17.675302    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0621c190-fe95-442e-a82c-d3cdbc14d1c7-maas-api-tls\") pod \"maas-api-e2e-ait-c1ba7d12-79c896b84b-bbtrh\" (UID: \"0621c190-fe95-442e-a82c-d3cdbc14d1c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1ba7d12-79c896b84b-bbtrh"
Jul 14 23:24:17.904132 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:17.904092    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1ba7d12-79c896b84b-bbtrh"
Jul 14 23:24:17.960701 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:17.960664    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"
Jul 14 23:24:18.038545 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:18.038523    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c1ba7d12-79c896b84b-bbtrh"]
Jul 14 23:24:18.040610 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:24:18.040583    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0621c190_fe95_442e_a82c_d3cdbc14d1c7.slice/crio-c6b8acf673b4328e7ec542a608cf473b8df951a00519220b262a8c8a3e7c68fb WatchSource:0}: Error finding container c6b8acf673b4328e7ec542a608cf473b8df951a00519220b262a8c8a3e7c68fb: Status 404 returned error can't find the container with id c6b8acf673b4328e7ec542a608cf473b8df951a00519220b262a8c8a3e7c68fb
Jul 14 23:24:18.348536 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:18.348494    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1ba7d12-79c896b84b-bbtrh" event={"ID":"0621c190-fe95-442e-a82c-d3cdbc14d1c7","Type":"ContainerStarted","Data":"c6b8acf673b4328e7ec542a608cf473b8df951a00519220b262a8c8a3e7c68fb"}
Jul 14 23:24:18.348698 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:18.348678    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"
Jul 14 23:24:18.349575 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:18.349553    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"
Jul 14 23:24:19.355061 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:19.355027    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1ba7d12-79c896b84b-bbtrh" event={"ID":"0621c190-fe95-442e-a82c-d3cdbc14d1c7","Type":"ContainerStarted","Data":"e738cf9827987cc36b42cb6988c64888ab3e2290486191d330c5be759cc2890f"}
Jul 14 23:24:19.355508 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:19.355109    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1ba7d12-79c896b84b-bbtrh"
Jul 14 23:24:19.374751 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:19.374706    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1ba7d12-79c896b84b-bbtrh" podStartSLOduration=2.948867045 podStartE2EDuration="3.374692012s" totalImagesPullingTime="425.824967ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:24:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:24:19.374236533 +0000 UTC m=+1646.677227326" watchObservedRunningTime="2026-07-14 23:24:19.374692012 +0000 UTC m=+1646.677682803"
Jul 14 23:24:20.598983 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:20.598960    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"
Jul 14 23:24:20.700531 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:20.700497    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-istio-data\") pod \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\" (UID: \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\") "
Jul 14 23:24:20.700531 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:20.700532    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-istio-podinfo\") pod \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\" (UID: \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\") "
Jul 14 23:24:20.700776 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:20.700551    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-workload-socket\") pod \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\" (UID: \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\") "
Jul 14 23:24:20.700776 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:20.700595    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-istio-envoy\") pod \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\" (UID: \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\") "
Jul 14 23:24:20.700776 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:20.700619    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-istio-token\") pod \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\" (UID: \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\") "
Jul 14 23:24:20.700776 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:20.700648    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-istiod-ca-cert\") pod \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\" (UID: \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\") "
Jul 14 23:24:20.700776 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:20.700665    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-workload-certs\") pod \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\" (UID: \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\") "
Jul 14 23:24:20.700776 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:20.700681    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-credential-socket\") pod \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\" (UID: \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\") "
Jul 14 23:24:20.700776 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:20.700701    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wj8l\" (UniqueName: \"kubernetes.io/projected/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-kube-api-access-7wj8l\") pod \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\" (UID: \"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3\") "
Jul 14 23:24:20.701133 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:20.701103    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3" (UID: "862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:24:20.701249 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:20.701186    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3" (UID: "862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:24:20.701387 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:20.701350    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-istio-data" (OuterVolumeSpecName: "istio-data") pod "862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3" (UID: "862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:24:20.701453 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:20.701390    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3" (UID: "862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 23:24:20.701453 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:20.701416    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3" (UID: "862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:24:20.703140 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:20.703108    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-istio-token" (OuterVolumeSpecName: "istio-token") pod "862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3" (UID: "862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:24:20.703536 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:20.703516    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3" (UID: "862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:24:20.703610 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:20.703525    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3" (UID: "862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 23:24:20.703674 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:20.703653    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-kube-api-access-7wj8l" (OuterVolumeSpecName: "kube-api-access-7wj8l") pod "862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3" (UID: "862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3"). InnerVolumeSpecName "kube-api-access-7wj8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:24:20.801478 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:20.801427    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-istio-data\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:24:20.801478 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:20.801470    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-istio-podinfo\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:24:20.801478 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:20.801479    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-workload-socket\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:24:20.801478 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:20.801487    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-istio-envoy\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:24:20.801478 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:20.801495    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-istio-token\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:24:20.801785 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:20.801503    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-istiod-ca-cert\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:24:20.801785 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:20.801510    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-workload-certs\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:24:20.801785 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:20.801517    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-credential-socket\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:24:20.801785 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:20.801525    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wj8l\" (UniqueName: \"kubernetes.io/projected/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3-kube-api-access-7wj8l\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:24:21.365460 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:21.365423    2563 generic.go:356] "Generic (PLEG): container finished" podID="862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3" containerID="bc7f9530cc74d7a73887a02de83d110dfc80657256a170aa9575fd580b9c8549" exitCode=0
Jul 14 23:24:21.365620 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:21.365469    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"
Jul 14 23:24:21.365620 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:21.365511    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc" event={"ID":"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3","Type":"ContainerDied","Data":"bc7f9530cc74d7a73887a02de83d110dfc80657256a170aa9575fd580b9c8549"}
Jul 14 23:24:21.365620 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:21.365564    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc" event={"ID":"862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3","Type":"ContainerDied","Data":"404f1c40957b54e6a77d0de662325edded452dd56e1e5a6790a28b6bdc0ebbe5"}
Jul 14 23:24:21.365620 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:21.365588    2563 scope.go:117] "RemoveContainer" containerID="bc7f9530cc74d7a73887a02de83d110dfc80657256a170aa9575fd580b9c8549"
Jul 14 23:24:21.374083 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:21.374066    2563 scope.go:117] "RemoveContainer" containerID="cf01ec04138c727c1abce6ccf3aacf75c0aca026fa9c1dd505bcdca2c8c8a4c2"
Jul 14 23:24:21.381560 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:21.381544    2563 scope.go:117] "RemoveContainer" containerID="bc7f9530cc74d7a73887a02de83d110dfc80657256a170aa9575fd580b9c8549"
Jul 14 23:24:21.381806 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:24:21.381784    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc7f9530cc74d7a73887a02de83d110dfc80657256a170aa9575fd580b9c8549\": container with ID starting with bc7f9530cc74d7a73887a02de83d110dfc80657256a170aa9575fd580b9c8549 not found: ID does not exist" containerID="bc7f9530cc74d7a73887a02de83d110dfc80657256a170aa9575fd580b9c8549"
Jul 14 23:24:21.381884 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:21.381817    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc7f9530cc74d7a73887a02de83d110dfc80657256a170aa9575fd580b9c8549"} err="failed to get container status \"bc7f9530cc74d7a73887a02de83d110dfc80657256a170aa9575fd580b9c8549\": rpc error: code = NotFound desc = could not find container \"bc7f9530cc74d7a73887a02de83d110dfc80657256a170aa9575fd580b9c8549\": container with ID starting with bc7f9530cc74d7a73887a02de83d110dfc80657256a170aa9575fd580b9c8549 not found: ID does not exist"
Jul 14 23:24:21.381884 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:21.381841    2563 scope.go:117] "RemoveContainer" containerID="cf01ec04138c727c1abce6ccf3aacf75c0aca026fa9c1dd505bcdca2c8c8a4c2"
Jul 14 23:24:21.382114 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:24:21.382097    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf01ec04138c727c1abce6ccf3aacf75c0aca026fa9c1dd505bcdca2c8c8a4c2\": container with ID starting with cf01ec04138c727c1abce6ccf3aacf75c0aca026fa9c1dd505bcdca2c8c8a4c2 not found: ID does not exist" containerID="cf01ec04138c727c1abce6ccf3aacf75c0aca026fa9c1dd505bcdca2c8c8a4c2"
Jul 14 23:24:21.382156 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:21.382120    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf01ec04138c727c1abce6ccf3aacf75c0aca026fa9c1dd505bcdca2c8c8a4c2"} err="failed to get container status \"cf01ec04138c727c1abce6ccf3aacf75c0aca026fa9c1dd505bcdca2c8c8a4c2\": rpc error: code = NotFound desc = could not find container \"cf01ec04138c727c1abce6ccf3aacf75c0aca026fa9c1dd505bcdca2c8c8a4c2\": container with ID starting with cf01ec04138c727c1abce6ccf3aacf75c0aca026fa9c1dd505bcdca2c8c8a4c2 not found: ID does not exist"
Jul 14 23:24:21.385112 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:21.385090    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"]
Jul 14 23:24:21.389083 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:21.389059    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-1731167b-gw-openshift-default-569b98c499-lhvpc"]
Jul 14 23:24:23.234057 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:23.234014    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3" path="/var/lib/kubelet/pods/862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3/volumes"
Jul 14 23:24:25.365486 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:25.365455    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1ba7d12-79c896b84b-bbtrh"
Jul 14 23:24:27.394407 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:27.394302    2563 generic.go:356] "Generic (PLEG): container finished" podID="a70b30a0-baff-466e-b04a-e47a712ff7ee" containerID="b4ebf5a5f8bd353e8b90e6d3ac50b00e02400588e9d03e0ffd7748f3388fcb43" exitCode=0
Jul 14 23:24:27.394407 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:27.394309    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t" event={"ID":"a70b30a0-baff-466e-b04a-e47a712ff7ee","Type":"ContainerDied","Data":"b4ebf5a5f8bd353e8b90e6d3ac50b00e02400588e9d03e0ffd7748f3388fcb43"}
Jul 14 23:24:27.394967 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:27.394870    2563 scope.go:117] "RemoveContainer" containerID="b4ebf5a5f8bd353e8b90e6d3ac50b00e02400588e9d03e0ffd7748f3388fcb43"
Jul 14 23:24:27.395887 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:27.395859    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:24:27.395982 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:27.395925    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:24:27.395982 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:27.395956    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:24:28.401087 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:28.401046    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t" event={"ID":"a70b30a0-baff-466e-b04a-e47a712ff7ee","Type":"ContainerStarted","Data":"884996ff7c4fab7014b10a817714f15446db61096295cf3c09ba6bc573389f95"}
Jul 14 23:24:28.953567 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:28.953521    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"
Jul 14 23:24:28.954717 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:28.954692    2563 patch_prober.go:28] interesting pod/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 14 23:24:28.954836 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:28.954750    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t" podUID="a70b30a0-baff-466e-b04a-e47a712ff7ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 14 23:24:29.953770 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:29.953734    2563 patch_prober.go:28] interesting pod/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 14 23:24:29.954190 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:29.953790    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t" podUID="a70b30a0-baff-466e-b04a-e47a712ff7ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 14 23:24:30.953515 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:30.953477    2563 patch_prober.go:28] interesting pod/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 14 23:24:30.953728 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:30.953539    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t" podUID="a70b30a0-baff-466e-b04a-e47a712ff7ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 14 23:24:31.953390 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:31.953349    2563 patch_prober.go:28] interesting pod/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 14 23:24:31.954077 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:31.953407    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t" podUID="a70b30a0-baff-466e-b04a-e47a712ff7ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 14 23:24:32.957317 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:32.957265    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"
Jul 14 23:24:32.957753 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:32.957576    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"
Jul 14 23:24:32.958263 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:24:32.958243    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"
Jul 14 23:25:11.957592 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:11.957558    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1ba7d12-hkc4q"]
Jul 14 23:25:11.958042 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:11.957889    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3" containerName="istio-proxy"
Jul 14 23:25:11.958042 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:11.957899    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3" containerName="istio-proxy"
Jul 14 23:25:11.958042 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:11.957919    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3" containerName="istio-proxy"
Jul 14 23:25:11.958042 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:11.957925    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3" containerName="istio-proxy"
Jul 14 23:25:11.958042 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:11.957974    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3" containerName="istio-proxy"
Jul 14 23:25:11.958042 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:11.957984    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="862dbc52-1cb1-4b06-9a65-55bbb0a4e3d3" containerName="istio-proxy"
Jul 14 23:25:11.960961 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:11.960942    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1ba7d12-hkc4q"
Jul 14 23:25:11.963727 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:11.963707    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jw626\""
Jul 14 23:25:11.969049 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:11.969010    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1ba7d12-hkc4q"]
Jul 14 23:25:12.054348 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:12.054316    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a76d6a7-015a-4458-a14a-a03a015231e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c1ba7d12-hkc4q\" (UID: \"2a76d6a7-015a-4458-a14a-a03a015231e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1ba7d12-hkc4q"
Jul 14 23:25:12.154991 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:12.154950    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a76d6a7-015a-4458-a14a-a03a015231e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c1ba7d12-hkc4q\" (UID: \"2a76d6a7-015a-4458-a14a-a03a015231e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1ba7d12-hkc4q"
Jul 14 23:25:12.155637 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:12.155617    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a76d6a7-015a-4458-a14a-a03a015231e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c1ba7d12-hkc4q\" (UID: \"2a76d6a7-015a-4458-a14a-a03a015231e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1ba7d12-hkc4q"
Jul 14 23:25:12.271811 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:12.271764    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1ba7d12-hkc4q"
Jul 14 23:25:12.403995 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:12.403947    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1ba7d12-hkc4q"]
Jul 14 23:25:12.406230 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:25:12.406198    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a76d6a7_015a_4458_a14a_a03a015231e3.slice/crio-a92b4faedb5b551a3374de7e31e6b39c9b3f4dafc96385b23f8da96c6bcc9245 WatchSource:0}: Error finding container a92b4faedb5b551a3374de7e31e6b39c9b3f4dafc96385b23f8da96c6bcc9245: Status 404 returned error can't find the container with id a92b4faedb5b551a3374de7e31e6b39c9b3f4dafc96385b23f8da96c6bcc9245
Jul 14 23:25:12.610342 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:12.610308    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1ba7d12-hkc4q" event={"ID":"2a76d6a7-015a-4458-a14a-a03a015231e3","Type":"ContainerStarted","Data":"a92b4faedb5b551a3374de7e31e6b39c9b3f4dafc96385b23f8da96c6bcc9245"}
Jul 14 23:25:13.616426 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:13.616385    2563 generic.go:356] "Generic (PLEG): container finished" podID="2a76d6a7-015a-4458-a14a-a03a015231e3" containerID="99c106814a2964273e715a0df96864023073051aa13463767cd4050c6b61cb8a" exitCode=0
Jul 14 23:25:13.616824 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:13.616474    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1ba7d12-hkc4q" event={"ID":"2a76d6a7-015a-4458-a14a-a03a015231e3","Type":"ContainerDied","Data":"99c106814a2964273e715a0df96864023073051aa13463767cd4050c6b61cb8a"}
Jul 14 23:25:14.751410 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:14.751386    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1ba7d12-hkc4q"
Jul 14 23:25:14.878036 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:14.877941    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a76d6a7-015a-4458-a14a-a03a015231e3-maas-api-service-ca\") pod \"2a76d6a7-015a-4458-a14a-a03a015231e3\" (UID: \"2a76d6a7-015a-4458-a14a-a03a015231e3\") "
Jul 14 23:25:14.878353 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:14.878330    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a76d6a7-015a-4458-a14a-a03a015231e3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2a76d6a7-015a-4458-a14a-a03a015231e3" (UID: "2a76d6a7-015a-4458-a14a-a03a015231e3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 23:25:14.979401 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:14.979360    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a76d6a7-015a-4458-a14a-a03a015231e3-maas-api-service-ca\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:25:15.626448 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:15.626401    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1ba7d12-hkc4q" event={"ID":"2a76d6a7-015a-4458-a14a-a03a015231e3","Type":"ContainerDied","Data":"a92b4faedb5b551a3374de7e31e6b39c9b3f4dafc96385b23f8da96c6bcc9245"}
Jul 14 23:25:15.626448 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:15.626445    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1ba7d12-hkc4q"
Jul 14 23:25:15.626655 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:15.626469    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a92b4faedb5b551a3374de7e31e6b39c9b3f4dafc96385b23f8da96c6bcc9245"
Jul 14 23:25:22.527436 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:22.527402    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c1ba7d12-79c896b84b-bbtrh"]
Jul 14 23:25:22.527901 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:22.527647    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1ba7d12-79c896b84b-bbtrh" podUID="0621c190-fe95-442e-a82c-d3cdbc14d1c7" containerName="maas-api" containerID="cri-o://e738cf9827987cc36b42cb6988c64888ab3e2290486191d330c5be759cc2890f" gracePeriod=30
Jul 14 23:25:22.662762 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:22.662719    2563 generic.go:356] "Generic (PLEG): container finished" podID="0621c190-fe95-442e-a82c-d3cdbc14d1c7" containerID="e738cf9827987cc36b42cb6988c64888ab3e2290486191d330c5be759cc2890f" exitCode=0
Jul 14 23:25:22.662941 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:22.662798    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1ba7d12-79c896b84b-bbtrh" event={"ID":"0621c190-fe95-442e-a82c-d3cdbc14d1c7","Type":"ContainerDied","Data":"e738cf9827987cc36b42cb6988c64888ab3e2290486191d330c5be759cc2890f"}
Jul 14 23:25:22.786671 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:22.786616    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1ba7d12-79c896b84b-bbtrh"
Jul 14 23:25:22.957364 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:22.957324    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0621c190-fe95-442e-a82c-d3cdbc14d1c7-openshift-service-ca\") pod \"0621c190-fe95-442e-a82c-d3cdbc14d1c7\" (UID: \"0621c190-fe95-442e-a82c-d3cdbc14d1c7\") "
Jul 14 23:25:22.957364 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:22.957364    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwlfw\" (UniqueName: \"kubernetes.io/projected/0621c190-fe95-442e-a82c-d3cdbc14d1c7-kube-api-access-hwlfw\") pod \"0621c190-fe95-442e-a82c-d3cdbc14d1c7\" (UID: \"0621c190-fe95-442e-a82c-d3cdbc14d1c7\") "
Jul 14 23:25:22.957567 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:22.957384    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0621c190-fe95-442e-a82c-d3cdbc14d1c7-maas-api-tls\") pod \"0621c190-fe95-442e-a82c-d3cdbc14d1c7\" (UID: \"0621c190-fe95-442e-a82c-d3cdbc14d1c7\") "
Jul 14 23:25:22.957788 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:22.957759    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0621c190-fe95-442e-a82c-d3cdbc14d1c7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0621c190-fe95-442e-a82c-d3cdbc14d1c7" (UID: "0621c190-fe95-442e-a82c-d3cdbc14d1c7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 23:25:22.959655 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:22.959630    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0621c190-fe95-442e-a82c-d3cdbc14d1c7-kube-api-access-hwlfw" (OuterVolumeSpecName: "kube-api-access-hwlfw") pod "0621c190-fe95-442e-a82c-d3cdbc14d1c7" (UID: "0621c190-fe95-442e-a82c-d3cdbc14d1c7"). InnerVolumeSpecName "kube-api-access-hwlfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:25:22.959765 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:22.959671    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0621c190-fe95-442e-a82c-d3cdbc14d1c7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0621c190-fe95-442e-a82c-d3cdbc14d1c7" (UID: "0621c190-fe95-442e-a82c-d3cdbc14d1c7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 23:25:23.058365 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:23.058244    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0621c190-fe95-442e-a82c-d3cdbc14d1c7-openshift-service-ca\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:25:23.058365 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:23.058309    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwlfw\" (UniqueName: \"kubernetes.io/projected/0621c190-fe95-442e-a82c-d3cdbc14d1c7-kube-api-access-hwlfw\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:25:23.058365 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:23.058320    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0621c190-fe95-442e-a82c-d3cdbc14d1c7-maas-api-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:25:23.668212 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:23.668172    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1ba7d12-79c896b84b-bbtrh" event={"ID":"0621c190-fe95-442e-a82c-d3cdbc14d1c7","Type":"ContainerDied","Data":"c6b8acf673b4328e7ec542a608cf473b8df951a00519220b262a8c8a3e7c68fb"}
Jul 14 23:25:23.668695 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:23.668238    2563 scope.go:117] "RemoveContainer" containerID="e738cf9827987cc36b42cb6988c64888ab3e2290486191d330c5be759cc2890f"
Jul 14 23:25:23.668695 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:23.668190    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1ba7d12-79c896b84b-bbtrh"
Jul 14 23:25:23.686388 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:23.686363    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c1ba7d12-79c896b84b-bbtrh"]
Jul 14 23:25:23.690596 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:23.690571    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c1ba7d12-79c896b84b-bbtrh"]
Jul 14 23:25:25.229816 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:25.229781    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0621c190-fe95-442e-a82c-d3cdbc14d1c7" path="/var/lib/kubelet/pods/0621c190-fe95-442e-a82c-d3cdbc14d1c7/volumes"
Jul 14 23:25:38.062555 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:38.062512    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"]
Jul 14 23:25:38.062986 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:38.062732    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t" podUID="a70b30a0-baff-466e-b04a-e47a712ff7ee" containerName="istio-proxy" containerID="cri-o://884996ff7c4fab7014b10a817714f15446db61096295cf3c09ba6bc573389f95" gracePeriod=30
Jul 14 23:25:38.378223 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:38.378139    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"]
Jul 14 23:25:38.378604 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:38.378584    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0621c190-fe95-442e-a82c-d3cdbc14d1c7" containerName="maas-api"
Jul 14 23:25:38.378695 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:38.378606    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0621c190-fe95-442e-a82c-d3cdbc14d1c7" containerName="maas-api"
Jul 14 23:25:38.378695 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:38.378623    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a76d6a7-015a-4458-a14a-a03a015231e3" containerName="revoke-keys"
Jul 14 23:25:38.378695 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:38.378633    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a76d6a7-015a-4458-a14a-a03a015231e3" containerName="revoke-keys"
Jul 14 23:25:38.378845 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:38.378730    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a76d6a7-015a-4458-a14a-a03a015231e3" containerName="revoke-keys"
Jul 14 23:25:38.378845 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:38.378746    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0621c190-fe95-442e-a82c-d3cdbc14d1c7" containerName="maas-api"
Jul 14 23:25:38.382034 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:38.382012    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"
Jul 14 23:25:38.385119 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:38.385097    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-758a442d-openshift-default-dockercfg-lkcgt\""
Jul 14 23:25:38.395333 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:38.395311    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"]
Jul 14 23:25:38.482128 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:38.482076    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/287b6426-3b9e-4e79-9631-e54faaf548fb-istio-podinfo\") pod \"e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr\" (UID: \"287b6426-3b9e-4e79-9631-e54faaf548fb\") " pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"
Jul 14 23:25:38.482128 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:38.482120    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/287b6426-3b9e-4e79-9631-e54faaf548fb-istiod-ca-cert\") pod \"e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr\" (UID: \"287b6426-3b9e-4e79-9631-e54faaf548fb\") " pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"
Jul 14 23:25:38.482774 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:38.482142    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsg9j\" (UniqueName: \"kubernetes.io/projected/287b6426-3b9e-4e79-9631-e54faaf548fb-kube-api-access-zsg9j\") pod \"e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr\" (UID: \"287b6426-3b9e-4e79-9631-e54faaf548fb\") " pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"
Jul 14 23:25:38.482774 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:38.482272    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/287b6426-3b9e-4e79-9631-e54faaf548fb-workload-certs\") pod \"e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr\" (UID: \"287b6426-3b9e-4e79-9631-e54faaf548fb\") " pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"
Jul 14 23:25:38.482774 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:38.482324    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/287b6426-3b9e-4e79-9631-e54faaf548fb-istio-envoy\") pod \"e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr\" (UID: \"287b6426-3b9e-4e79-9631-e54faaf548fb\") " pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"
Jul 14 23:25:38.482774 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:38.482410    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/287b6426-3b9e-4e79-9631-e54faaf548fb-istio-token\") pod \"e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr\" (UID: \"287b6426-3b9e-4e79-9631-e54faaf548fb\") " pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"
Jul 14 23:25:38.482774 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:38.482451    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/287b6426-3b9e-4e79-9631-e54faaf548fb-workload-socket\") pod \"e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr\" (UID: \"287b6426-3b9e-4e79-9631-e54faaf548fb\") " pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"
Jul 14 23:25:38.482774 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:38.482474    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/287b6426-3b9e-4e79-9631-e54faaf548fb-credential-socket\") pod \"e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr\" (UID: \"287b6426-3b9e-4e79-9631-e54faaf548fb\") " pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"
Jul 14 23:25:38.482774 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:38.482505    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/287b6426-3b9e-4e79-9631-e54faaf548fb-istio-data\") pod \"e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr\" (UID: \"287b6426-3b9e-4e79-9631-e54faaf548fb\") " pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"
Jul 14 23:25:38.583220 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:38.583175    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/287b6426-3b9e-4e79-9631-e54faaf548fb-workload-socket\") pod \"e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr\" (UID: \"287b6426-3b9e-4e79-9631-e54faaf548fb\") " pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"
Jul 14 23:25:38.583220 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:38.583217    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/287b6426-3b9e-4e79-9631-e54faaf548fb-credential-socket\") pod \"e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr\" (UID: \"287b6426-3b9e-4e79-9631-e54faaf548fb\") " pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"
Jul 14 23:25:38.583509 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:38.583236    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/287b6426-3b9e-4e79-9631-e54faaf548fb-istio-data\") pod \"e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr\" (UID: \"287b6426-3b9e-4e79-9631-e54faaf548fb\") " pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"
Jul 14 23:25:38.583509 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:38.583313    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/287b6426-3b9e-4e79-9631-e54faaf548fb-istio-podinfo\") pod \"e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr\" (UID: \"287b6426-3b9e-4e79-9631-e54faaf548fb\") " pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"
Jul 14 23:25:38.583509 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:38.583341    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/287b6426-3b9e-4e79-9631-e54faaf548fb-istiod-ca-cert\") pod \"e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr\" (UID: \"287b6426-3b9e-4e79-9631-e54faaf548fb\") " pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"
Jul 14 23:25:38.583509 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:38.583361    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsg9j\" (UniqueName: \"kubernetes.io/projected/287b6426-3b9e-4e79-9631-e54faaf548fb-kube-api-access-zsg9j\") pod \"e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr\" (UID: \"287b6426-3b9e-4e79-9631-e54faaf548fb\") " pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"
Jul 14 23:25:38.583509 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:38.583418    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/287b6426-3b9e-4e79-9631-e54faaf548fb-workload-certs\") pod \"e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr\" (UID: \"287b6426-3b9e-4e79-9631-e54faaf548fb\") " pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"
Jul 14 23:25:38.583509 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:38.583441    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/287b6426-3b9e-4e79-9631-e54faaf548fb-istio-envoy\") pod \"e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr\" (UID: \"287b6426-3b9e-4e79-9631-e54faaf548fb\") " pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"
Jul 14 23:25:38.583509 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:38.583488    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/287b6426-3b9e-4e79-9631-e54faaf548fb-istio-token\") pod \"e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr\" (UID: \"287b6426-3b9e-4e79-9631-e54faaf548fb\") " pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"
Jul 14 23:25:38.583872 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:38.583679    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/287b6426-3b9e-4e79-9631-e54faaf548fb-credential-socket\") pod \"e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr\" (UID: \"287b6426-3b9e-4e79-9631-e54faaf548fb\") " pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"
Jul 14 23:25:38.583872 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:38.583722    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/287b6426-3b9e-4e79-9631-e54faaf548fb-workload-socket\") pod \"e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr\" (UID: \"287b6426-3b9e-4e79-9631-e54faaf548fb\") " pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"
Jul 14 23:25:38.583872 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:38.583754    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/287b6426-3b9e-4e79-9631-e54faaf548fb-istio-data\") pod \"e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr\" (UID: \"287b6426-3b9e-4e79-9631-e54faaf548fb\") " pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"
Jul 14 23:25:38.583988 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:38.583897    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/287b6426-3b9e-4e79-9631-e54faaf548fb-workload-certs\") pod \"e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr\" (UID: \"287b6426-3b9e-4e79-9631-e54faaf548fb\") " pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"
Jul 14 23:25:38.584132 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:38.584112    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/287b6426-3b9e-4e79-9631-e54faaf548fb-istiod-ca-cert\") pod \"e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr\" (UID: \"287b6426-3b9e-4e79-9631-e54faaf548fb\") " pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"
Jul 14 23:25:38.585883 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:38.585861    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/287b6426-3b9e-4e79-9631-e54faaf548fb-istio-podinfo\") pod \"e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr\" (UID: \"287b6426-3b9e-4e79-9631-e54faaf548fb\") " pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"
Jul 14 23:25:38.585998 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:38.585887    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/287b6426-3b9e-4e79-9631-e54faaf548fb-istio-envoy\") pod \"e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr\" (UID: \"287b6426-3b9e-4e79-9631-e54faaf548fb\") " pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"
Jul 14 23:25:38.593800 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:38.593768    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/287b6426-3b9e-4e79-9631-e54faaf548fb-istio-token\") pod \"e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr\" (UID: \"287b6426-3b9e-4e79-9631-e54faaf548fb\") " pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"
Jul 14 23:25:38.593951 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:38.593895    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsg9j\" (UniqueName: \"kubernetes.io/projected/287b6426-3b9e-4e79-9631-e54faaf548fb-kube-api-access-zsg9j\") pod \"e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr\" (UID: \"287b6426-3b9e-4e79-9631-e54faaf548fb\") " pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"
Jul 14 23:25:38.692807 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:38.692718    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"
Jul 14 23:25:38.828868 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:38.828843    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"]
Jul 14 23:25:38.830633 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:25:38.830605    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod287b6426_3b9e_4e79_9631_e54faaf548fb.slice/crio-997503162c7d5045b0418567eb75217de886280bf87a7ef23981378732141a65 WatchSource:0}: Error finding container 997503162c7d5045b0418567eb75217de886280bf87a7ef23981378732141a65: Status 404 returned error can't find the container with id 997503162c7d5045b0418567eb75217de886280bf87a7ef23981378732141a65
Jul 14 23:25:38.832757 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:38.832728    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:25:38.832815 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:38.832796    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:25:38.832857 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:38.832824    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:25:39.608047 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:39.608014    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-758a442d-68549d5d57-tmq8p"]
Jul 14 23:25:39.611673 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:39.611650    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-758a442d-68549d5d57-tmq8p"
Jul 14 23:25:39.614610 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:39.614589    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7jvtc\""
Jul 14 23:25:39.614743 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:39.614724    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 23:25:39.616133 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:39.616113    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 23:25:39.616257 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:39.616140    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-758a442d-serving-cert\""
Jul 14 23:25:39.620539 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:39.620517    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-758a442d-68549d5d57-tmq8p"]
Jul 14 23:25:39.692353 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:39.692320    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a69e5ee-1a1d-4801-bc5b-e3e5d2410163-openshift-service-ca\") pod \"maas-api-e2e-derive-758a442d-68549d5d57-tmq8p\" (UID: \"6a69e5ee-1a1d-4801-bc5b-e3e5d2410163\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-758a442d-68549d5d57-tmq8p"
Jul 14 23:25:39.692557 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:39.692360    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a69e5ee-1a1d-4801-bc5b-e3e5d2410163-maas-api-tls\") pod \"maas-api-e2e-derive-758a442d-68549d5d57-tmq8p\" (UID: \"6a69e5ee-1a1d-4801-bc5b-e3e5d2410163\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-758a442d-68549d5d57-tmq8p"
Jul 14 23:25:39.692557 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:39.692469    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8zb6\" (UniqueName: \"kubernetes.io/projected/6a69e5ee-1a1d-4801-bc5b-e3e5d2410163-kube-api-access-q8zb6\") pod \"maas-api-e2e-derive-758a442d-68549d5d57-tmq8p\" (UID: \"6a69e5ee-1a1d-4801-bc5b-e3e5d2410163\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-758a442d-68549d5d57-tmq8p"
Jul 14 23:25:39.746536 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:39.746499    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr" event={"ID":"287b6426-3b9e-4e79-9631-e54faaf548fb","Type":"ContainerStarted","Data":"48e6fcc7c40bed6d0b7756f02397d97b3b80caefa5e617465f7d4dfb23d2ab58"}
Jul 14 23:25:39.746737 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:39.746560    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr" event={"ID":"287b6426-3b9e-4e79-9631-e54faaf548fb","Type":"ContainerStarted","Data":"997503162c7d5045b0418567eb75217de886280bf87a7ef23981378732141a65"}
Jul 14 23:25:39.772658 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:39.772601    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr" podStartSLOduration=1.772584323 podStartE2EDuration="1.772584323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:25:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:25:39.771963573 +0000 UTC m=+1727.074954367" watchObservedRunningTime="2026-07-14 23:25:39.772584323 +0000 UTC m=+1727.075575116"
Jul 14 23:25:39.793086 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:39.793062    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8zb6\" (UniqueName: \"kubernetes.io/projected/6a69e5ee-1a1d-4801-bc5b-e3e5d2410163-kube-api-access-q8zb6\") pod \"maas-api-e2e-derive-758a442d-68549d5d57-tmq8p\" (UID: \"6a69e5ee-1a1d-4801-bc5b-e3e5d2410163\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-758a442d-68549d5d57-tmq8p"
Jul 14 23:25:39.793237 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:39.793121    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a69e5ee-1a1d-4801-bc5b-e3e5d2410163-openshift-service-ca\") pod \"maas-api-e2e-derive-758a442d-68549d5d57-tmq8p\" (UID: \"6a69e5ee-1a1d-4801-bc5b-e3e5d2410163\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-758a442d-68549d5d57-tmq8p"
Jul 14 23:25:39.793237 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:39.793145    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a69e5ee-1a1d-4801-bc5b-e3e5d2410163-maas-api-tls\") pod \"maas-api-e2e-derive-758a442d-68549d5d57-tmq8p\" (UID: \"6a69e5ee-1a1d-4801-bc5b-e3e5d2410163\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-758a442d-68549d5d57-tmq8p"
Jul 14 23:25:39.793379 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:25:39.793337    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-758a442d-serving-cert: secret "maas-api-e2e-derive-758a442d-serving-cert" not found
Jul 14 23:25:39.793433 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:25:39.793409    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a69e5ee-1a1d-4801-bc5b-e3e5d2410163-maas-api-tls podName:6a69e5ee-1a1d-4801-bc5b-e3e5d2410163 nodeName:}" failed. No retries permitted until 2026-07-14 23:25:40.293388667 +0000 UTC m=+1727.596379446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6a69e5ee-1a1d-4801-bc5b-e3e5d2410163-maas-api-tls") pod "maas-api-e2e-derive-758a442d-68549d5d57-tmq8p" (UID: "6a69e5ee-1a1d-4801-bc5b-e3e5d2410163") : secret "maas-api-e2e-derive-758a442d-serving-cert" not found
Jul 14 23:25:39.793822 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:39.793805    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a69e5ee-1a1d-4801-bc5b-e3e5d2410163-openshift-service-ca\") pod \"maas-api-e2e-derive-758a442d-68549d5d57-tmq8p\" (UID: \"6a69e5ee-1a1d-4801-bc5b-e3e5d2410163\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-758a442d-68549d5d57-tmq8p"
Jul 14 23:25:39.808555 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:39.808533    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8zb6\" (UniqueName: \"kubernetes.io/projected/6a69e5ee-1a1d-4801-bc5b-e3e5d2410163-kube-api-access-q8zb6\") pod \"maas-api-e2e-derive-758a442d-68549d5d57-tmq8p\" (UID: \"6a69e5ee-1a1d-4801-bc5b-e3e5d2410163\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-758a442d-68549d5d57-tmq8p"
Jul 14 23:25:40.296698 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:40.296664    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a69e5ee-1a1d-4801-bc5b-e3e5d2410163-maas-api-tls\") pod \"maas-api-e2e-derive-758a442d-68549d5d57-tmq8p\" (UID: \"6a69e5ee-1a1d-4801-bc5b-e3e5d2410163\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-758a442d-68549d5d57-tmq8p"
Jul 14 23:25:40.299299 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:40.299261    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a69e5ee-1a1d-4801-bc5b-e3e5d2410163-maas-api-tls\") pod \"maas-api-e2e-derive-758a442d-68549d5d57-tmq8p\" (UID: \"6a69e5ee-1a1d-4801-bc5b-e3e5d2410163\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-758a442d-68549d5d57-tmq8p"
Jul 14 23:25:40.525587 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:40.525558    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-758a442d-68549d5d57-tmq8p"
Jul 14 23:25:40.657145 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:40.657113    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-758a442d-68549d5d57-tmq8p"]
Jul 14 23:25:40.660856 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:25:40.660829    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a69e5ee_1a1d_4801_bc5b_e3e5d2410163.slice/crio-b341c14e5b6f5252618cdd5264d1d0ba701872e7317a6a0f9e4090c6f21e4667 WatchSource:0}: Error finding container b341c14e5b6f5252618cdd5264d1d0ba701872e7317a6a0f9e4090c6f21e4667: Status 404 returned error can't find the container with id b341c14e5b6f5252618cdd5264d1d0ba701872e7317a6a0f9e4090c6f21e4667
Jul 14 23:25:40.693774 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:40.693754    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"
Jul 14 23:25:40.698392 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:40.698372    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"
Jul 14 23:25:40.752063 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:40.752028    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-758a442d-68549d5d57-tmq8p" event={"ID":"6a69e5ee-1a1d-4801-bc5b-e3e5d2410163","Type":"ContainerStarted","Data":"b341c14e5b6f5252618cdd5264d1d0ba701872e7317a6a0f9e4090c6f21e4667"}
Jul 14 23:25:40.752215 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:40.752195    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"
Jul 14 23:25:40.753037 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:40.753022    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"
Jul 14 23:25:41.758345 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:41.758302    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-758a442d-68549d5d57-tmq8p" event={"ID":"6a69e5ee-1a1d-4801-bc5b-e3e5d2410163","Type":"ContainerStarted","Data":"9b8ebf290f4dad717645805d217362b1932b08946c605e5cb54d4c94713addf9"}
Jul 14 23:25:41.758802 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:41.758526    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-758a442d-68549d5d57-tmq8p"
Jul 14 23:25:41.780383 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:41.780340    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-758a442d-68549d5d57-tmq8p" podStartSLOduration=2.270143511 podStartE2EDuration="2.780326062s" totalImagesPullingTime="510.182551ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:25:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:25:41.779528054 +0000 UTC m=+1729.082518846" watchObservedRunningTime="2026-07-14 23:25:41.780326062 +0000 UTC m=+1729.083316851"
Jul 14 23:25:43.360448 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:43.360422    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"
Jul 14 23:25:43.533306 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:43.532607    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a70b30a0-baff-466e-b04a-e47a712ff7ee-istio-data\") pod \"a70b30a0-baff-466e-b04a-e47a712ff7ee\" (UID: \"a70b30a0-baff-466e-b04a-e47a712ff7ee\") "
Jul 14 23:25:43.533306 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:43.532666    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a70b30a0-baff-466e-b04a-e47a712ff7ee-istiod-ca-cert\") pod \"a70b30a0-baff-466e-b04a-e47a712ff7ee\" (UID: \"a70b30a0-baff-466e-b04a-e47a712ff7ee\") "
Jul 14 23:25:43.533306 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:43.532696    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a70b30a0-baff-466e-b04a-e47a712ff7ee-istio-envoy\") pod \"a70b30a0-baff-466e-b04a-e47a712ff7ee\" (UID: \"a70b30a0-baff-466e-b04a-e47a712ff7ee\") "
Jul 14 23:25:43.533306 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:43.532756    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a70b30a0-baff-466e-b04a-e47a712ff7ee-workload-socket\") pod \"a70b30a0-baff-466e-b04a-e47a712ff7ee\" (UID: \"a70b30a0-baff-466e-b04a-e47a712ff7ee\") "
Jul 14 23:25:43.533306 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:43.532790    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2khp7\" (UniqueName: \"kubernetes.io/projected/a70b30a0-baff-466e-b04a-e47a712ff7ee-kube-api-access-2khp7\") pod \"a70b30a0-baff-466e-b04a-e47a712ff7ee\" (UID: \"a70b30a0-baff-466e-b04a-e47a712ff7ee\") "
Jul 14 23:25:43.533306 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:43.532845    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a70b30a0-baff-466e-b04a-e47a712ff7ee-istio-podinfo\") pod \"a70b30a0-baff-466e-b04a-e47a712ff7ee\" (UID: \"a70b30a0-baff-466e-b04a-e47a712ff7ee\") "
Jul 14 23:25:43.533306 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:43.532874    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a70b30a0-baff-466e-b04a-e47a712ff7ee-workload-certs\") pod \"a70b30a0-baff-466e-b04a-e47a712ff7ee\" (UID: \"a70b30a0-baff-466e-b04a-e47a712ff7ee\") "
Jul 14 23:25:43.533306 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:43.532900    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a70b30a0-baff-466e-b04a-e47a712ff7ee-credential-socket\") pod \"a70b30a0-baff-466e-b04a-e47a712ff7ee\" (UID: \"a70b30a0-baff-466e-b04a-e47a712ff7ee\") "
Jul 14 23:25:43.533306 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:43.532930    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a70b30a0-baff-466e-b04a-e47a712ff7ee-istio-token\") pod \"a70b30a0-baff-466e-b04a-e47a712ff7ee\" (UID: \"a70b30a0-baff-466e-b04a-e47a712ff7ee\") "
Jul 14 23:25:43.533941 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:43.533452    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a70b30a0-baff-466e-b04a-e47a712ff7ee-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a70b30a0-baff-466e-b04a-e47a712ff7ee" (UID: "a70b30a0-baff-466e-b04a-e47a712ff7ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:25:43.538410 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:43.534329    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a70b30a0-baff-466e-b04a-e47a712ff7ee-istio-data" (OuterVolumeSpecName: "istio-data") pod "a70b30a0-baff-466e-b04a-e47a712ff7ee" (UID: "a70b30a0-baff-466e-b04a-e47a712ff7ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:25:43.538410 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:43.534613    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a70b30a0-baff-466e-b04a-e47a712ff7ee-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a70b30a0-baff-466e-b04a-e47a712ff7ee" (UID: "a70b30a0-baff-466e-b04a-e47a712ff7ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 23:25:43.538410 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:43.536202    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a70b30a0-baff-466e-b04a-e47a712ff7ee-istio-token" (OuterVolumeSpecName: "istio-token") pod "a70b30a0-baff-466e-b04a-e47a712ff7ee" (UID: "a70b30a0-baff-466e-b04a-e47a712ff7ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:25:43.538936 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:43.538909    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a70b30a0-baff-466e-b04a-e47a712ff7ee-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a70b30a0-baff-466e-b04a-e47a712ff7ee" (UID: "a70b30a0-baff-466e-b04a-e47a712ff7ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:25:43.540047 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:43.538972    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a70b30a0-baff-466e-b04a-e47a712ff7ee-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a70b30a0-baff-466e-b04a-e47a712ff7ee" (UID: "a70b30a0-baff-466e-b04a-e47a712ff7ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:25:43.540858 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:43.540831    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a70b30a0-baff-466e-b04a-e47a712ff7ee-kube-api-access-2khp7" (OuterVolumeSpecName: "kube-api-access-2khp7") pod "a70b30a0-baff-466e-b04a-e47a712ff7ee" (UID: "a70b30a0-baff-466e-b04a-e47a712ff7ee"). InnerVolumeSpecName "kube-api-access-2khp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:25:43.541147 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:43.541120    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a70b30a0-baff-466e-b04a-e47a712ff7ee-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a70b30a0-baff-466e-b04a-e47a712ff7ee" (UID: "a70b30a0-baff-466e-b04a-e47a712ff7ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:25:43.541595 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:43.541570    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a70b30a0-baff-466e-b04a-e47a712ff7ee-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a70b30a0-baff-466e-b04a-e47a712ff7ee" (UID: "a70b30a0-baff-466e-b04a-e47a712ff7ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 23:25:43.634365 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:43.634216    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a70b30a0-baff-466e-b04a-e47a712ff7ee-istio-podinfo\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:25:43.634365 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:43.634275    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a70b30a0-baff-466e-b04a-e47a712ff7ee-workload-certs\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:25:43.634365 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:43.634308    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a70b30a0-baff-466e-b04a-e47a712ff7ee-credential-socket\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:25:43.634365 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:43.634323    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a70b30a0-baff-466e-b04a-e47a712ff7ee-istio-token\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:25:43.634365 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:43.634336    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a70b30a0-baff-466e-b04a-e47a712ff7ee-istio-data\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:25:43.634365 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:43.634349    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a70b30a0-baff-466e-b04a-e47a712ff7ee-istiod-ca-cert\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:25:43.634365 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:43.634369    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a70b30a0-baff-466e-b04a-e47a712ff7ee-istio-envoy\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:25:43.634875 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:43.634382    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a70b30a0-baff-466e-b04a-e47a712ff7ee-workload-socket\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:25:43.634875 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:43.634395    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2khp7\" (UniqueName: \"kubernetes.io/projected/a70b30a0-baff-466e-b04a-e47a712ff7ee-kube-api-access-2khp7\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:25:43.769078 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:43.769041    2563 generic.go:356] "Generic (PLEG): container finished" podID="a70b30a0-baff-466e-b04a-e47a712ff7ee" containerID="884996ff7c4fab7014b10a817714f15446db61096295cf3c09ba6bc573389f95" exitCode=0
Jul 14 23:25:43.769256 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:43.769103    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"
Jul 14 23:25:43.769256 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:43.769118    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t" event={"ID":"a70b30a0-baff-466e-b04a-e47a712ff7ee","Type":"ContainerDied","Data":"884996ff7c4fab7014b10a817714f15446db61096295cf3c09ba6bc573389f95"}
Jul 14 23:25:43.769256 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:43.769198    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t" event={"ID":"a70b30a0-baff-466e-b04a-e47a712ff7ee","Type":"ContainerDied","Data":"b6803fe8ffb3169c404e4b5fa071b5df80eb3a34ad0134fd90bd4cfe4d1378b3"}
Jul 14 23:25:43.769256 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:43.769220    2563 scope.go:117] "RemoveContainer" containerID="884996ff7c4fab7014b10a817714f15446db61096295cf3c09ba6bc573389f95"
Jul 14 23:25:43.778724 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:43.778704    2563 scope.go:117] "RemoveContainer" containerID="b4ebf5a5f8bd353e8b90e6d3ac50b00e02400588e9d03e0ffd7748f3388fcb43"
Jul 14 23:25:43.786250 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:43.786234    2563 scope.go:117] "RemoveContainer" containerID="884996ff7c4fab7014b10a817714f15446db61096295cf3c09ba6bc573389f95"
Jul 14 23:25:43.786521 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:25:43.786501    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"884996ff7c4fab7014b10a817714f15446db61096295cf3c09ba6bc573389f95\": container with ID starting with 884996ff7c4fab7014b10a817714f15446db61096295cf3c09ba6bc573389f95 not found: ID does not exist" containerID="884996ff7c4fab7014b10a817714f15446db61096295cf3c09ba6bc573389f95"
Jul 14 23:25:43.786602 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:43.786533    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"884996ff7c4fab7014b10a817714f15446db61096295cf3c09ba6bc573389f95"} err="failed to get container status \"884996ff7c4fab7014b10a817714f15446db61096295cf3c09ba6bc573389f95\": rpc error: code = NotFound desc = could not find container \"884996ff7c4fab7014b10a817714f15446db61096295cf3c09ba6bc573389f95\": container with ID starting with 884996ff7c4fab7014b10a817714f15446db61096295cf3c09ba6bc573389f95 not found: ID does not exist"
Jul 14 23:25:43.786602 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:43.786554    2563 scope.go:117] "RemoveContainer" containerID="b4ebf5a5f8bd353e8b90e6d3ac50b00e02400588e9d03e0ffd7748f3388fcb43"
Jul 14 23:25:43.786762 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:25:43.786743    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4ebf5a5f8bd353e8b90e6d3ac50b00e02400588e9d03e0ffd7748f3388fcb43\": container with ID starting with b4ebf5a5f8bd353e8b90e6d3ac50b00e02400588e9d03e0ffd7748f3388fcb43 not found: ID does not exist" containerID="b4ebf5a5f8bd353e8b90e6d3ac50b00e02400588e9d03e0ffd7748f3388fcb43"
Jul 14 23:25:43.786809 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:43.786769    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4ebf5a5f8bd353e8b90e6d3ac50b00e02400588e9d03e0ffd7748f3388fcb43"} err="failed to get container status \"b4ebf5a5f8bd353e8b90e6d3ac50b00e02400588e9d03e0ffd7748f3388fcb43\": rpc error: code = NotFound desc = could not find container \"b4ebf5a5f8bd353e8b90e6d3ac50b00e02400588e9d03e0ffd7748f3388fcb43\": container with ID starting with b4ebf5a5f8bd353e8b90e6d3ac50b00e02400588e9d03e0ffd7748f3388fcb43 not found: ID does not exist"
Jul 14 23:25:43.792394 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:43.792374    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"]
Jul 14 23:25:43.796122 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:43.796101    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c1ba7d12-openshift-default-fd9c4d59c-bc92t"]
Jul 14 23:25:45.232995 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:45.232956    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a70b30a0-baff-466e-b04a-e47a712ff7ee" path="/var/lib/kubelet/pods/a70b30a0-baff-466e-b04a-e47a712ff7ee/volumes"
Jul 14 23:25:47.769542 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:47.769504    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-758a442d-68549d5d57-tmq8p"
Jul 14 23:25:50.806956 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:50.806919    2563 generic.go:356] "Generic (PLEG): container finished" podID="287b6426-3b9e-4e79-9631-e54faaf548fb" containerID="48e6fcc7c40bed6d0b7756f02397d97b3b80caefa5e617465f7d4dfb23d2ab58" exitCode=0
Jul 14 23:25:50.807451 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:50.806999    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr" event={"ID":"287b6426-3b9e-4e79-9631-e54faaf548fb","Type":"ContainerDied","Data":"48e6fcc7c40bed6d0b7756f02397d97b3b80caefa5e617465f7d4dfb23d2ab58"}
Jul 14 23:25:50.807451 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:50.807425    2563 scope.go:117] "RemoveContainer" containerID="48e6fcc7c40bed6d0b7756f02397d97b3b80caefa5e617465f7d4dfb23d2ab58"
Jul 14 23:25:50.808553 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:50.808518    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:25:50.808639 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:50.808611    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:25:50.808678 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:50.808653    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:25:51.814723 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:51.814674    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr" event={"ID":"287b6426-3b9e-4e79-9631-e54faaf548fb","Type":"ContainerStarted","Data":"b9d3671ee5395180e85060d17cc7481178dac4e55b9d77185bcf396c8f217c81"}
Jul 14 23:25:52.693433 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:52.693377    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"
Jul 14 23:25:52.694797 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:52.694766    2563 patch_prober.go:28] interesting pod/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Jul 14 23:25:52.694949 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:52.694829    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr" podUID="287b6426-3b9e-4e79-9631-e54faaf548fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Jul 14 23:25:53.693997 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:53.693886    2563 patch_prober.go:28] interesting pod/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Jul 14 23:25:53.693997 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:53.693953    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr" podUID="287b6426-3b9e-4e79-9631-e54faaf548fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Jul 14 23:25:54.693669 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:54.693629    2563 patch_prober.go:28] interesting pod/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Jul 14 23:25:54.693892 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:54.693714    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr" podUID="287b6426-3b9e-4e79-9631-e54faaf548fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Jul 14 23:25:55.725883 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:55.725849    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"
Jul 14 23:25:55.726395 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:55.726238    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"
Jul 14 23:25:55.726877 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:25:55.726856    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"
Jul 14 23:26:44.528702 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:26:44.528664    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-758a442d-68549d5d57-tmq8p"]
Jul 14 23:26:44.529211 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:26:44.528915    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-758a442d-68549d5d57-tmq8p" podUID="6a69e5ee-1a1d-4801-bc5b-e3e5d2410163" containerName="maas-api" containerID="cri-o://9b8ebf290f4dad717645805d217362b1932b08946c605e5cb54d4c94713addf9" gracePeriod=30
Jul 14 23:26:44.779638 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:26:44.779571    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-758a442d-68549d5d57-tmq8p"
Jul 14 23:26:44.933122 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:26:44.933088    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a69e5ee-1a1d-4801-bc5b-e3e5d2410163-maas-api-tls\") pod \"6a69e5ee-1a1d-4801-bc5b-e3e5d2410163\" (UID: \"6a69e5ee-1a1d-4801-bc5b-e3e5d2410163\") "
Jul 14 23:26:44.933324 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:26:44.933197    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a69e5ee-1a1d-4801-bc5b-e3e5d2410163-openshift-service-ca\") pod \"6a69e5ee-1a1d-4801-bc5b-e3e5d2410163\" (UID: \"6a69e5ee-1a1d-4801-bc5b-e3e5d2410163\") "
Jul 14 23:26:44.933324 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:26:44.933227    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8zb6\" (UniqueName: \"kubernetes.io/projected/6a69e5ee-1a1d-4801-bc5b-e3e5d2410163-kube-api-access-q8zb6\") pod \"6a69e5ee-1a1d-4801-bc5b-e3e5d2410163\" (UID: \"6a69e5ee-1a1d-4801-bc5b-e3e5d2410163\") "
Jul 14 23:26:44.933567 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:26:44.933542    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a69e5ee-1a1d-4801-bc5b-e3e5d2410163-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6a69e5ee-1a1d-4801-bc5b-e3e5d2410163" (UID: "6a69e5ee-1a1d-4801-bc5b-e3e5d2410163"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 23:26:44.935465 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:26:44.935438    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a69e5ee-1a1d-4801-bc5b-e3e5d2410163-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6a69e5ee-1a1d-4801-bc5b-e3e5d2410163" (UID: "6a69e5ee-1a1d-4801-bc5b-e3e5d2410163"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 23:26:44.935549 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:26:44.935459    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a69e5ee-1a1d-4801-bc5b-e3e5d2410163-kube-api-access-q8zb6" (OuterVolumeSpecName: "kube-api-access-q8zb6") pod "6a69e5ee-1a1d-4801-bc5b-e3e5d2410163" (UID: "6a69e5ee-1a1d-4801-bc5b-e3e5d2410163"). InnerVolumeSpecName "kube-api-access-q8zb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:26:45.034780 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:26:45.034685    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a69e5ee-1a1d-4801-bc5b-e3e5d2410163-openshift-service-ca\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:26:45.034780 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:26:45.034720    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8zb6\" (UniqueName: \"kubernetes.io/projected/6a69e5ee-1a1d-4801-bc5b-e3e5d2410163-kube-api-access-q8zb6\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:26:45.034780 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:26:45.034733    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a69e5ee-1a1d-4801-bc5b-e3e5d2410163-maas-api-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:26:45.062876 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:26:45.062839    2563 generic.go:356] "Generic (PLEG): container finished" podID="6a69e5ee-1a1d-4801-bc5b-e3e5d2410163" containerID="9b8ebf290f4dad717645805d217362b1932b08946c605e5cb54d4c94713addf9" exitCode=0
Jul 14 23:26:45.063021 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:26:45.062890    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-758a442d-68549d5d57-tmq8p"
Jul 14 23:26:45.063021 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:26:45.062902    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-758a442d-68549d5d57-tmq8p" event={"ID":"6a69e5ee-1a1d-4801-bc5b-e3e5d2410163","Type":"ContainerDied","Data":"9b8ebf290f4dad717645805d217362b1932b08946c605e5cb54d4c94713addf9"}
Jul 14 23:26:45.063021 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:26:45.062962    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-758a442d-68549d5d57-tmq8p" event={"ID":"6a69e5ee-1a1d-4801-bc5b-e3e5d2410163","Type":"ContainerDied","Data":"b341c14e5b6f5252618cdd5264d1d0ba701872e7317a6a0f9e4090c6f21e4667"}
Jul 14 23:26:45.063021 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:26:45.062991    2563 scope.go:117] "RemoveContainer" containerID="9b8ebf290f4dad717645805d217362b1932b08946c605e5cb54d4c94713addf9"
Jul 14 23:26:45.072570 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:26:45.072549    2563 scope.go:117] "RemoveContainer" containerID="9b8ebf290f4dad717645805d217362b1932b08946c605e5cb54d4c94713addf9"
Jul 14 23:26:45.072829 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:26:45.072812    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b8ebf290f4dad717645805d217362b1932b08946c605e5cb54d4c94713addf9\": container with ID starting with 9b8ebf290f4dad717645805d217362b1932b08946c605e5cb54d4c94713addf9 not found: ID does not exist" containerID="9b8ebf290f4dad717645805d217362b1932b08946c605e5cb54d4c94713addf9"
Jul 14 23:26:45.072874 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:26:45.072836    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b8ebf290f4dad717645805d217362b1932b08946c605e5cb54d4c94713addf9"} err="failed to get container status \"9b8ebf290f4dad717645805d217362b1932b08946c605e5cb54d4c94713addf9\": rpc error: code = NotFound desc = could not find container \"9b8ebf290f4dad717645805d217362b1932b08946c605e5cb54d4c94713addf9\": container with ID starting with 9b8ebf290f4dad717645805d217362b1932b08946c605e5cb54d4c94713addf9 not found: ID does not exist"
Jul 14 23:26:45.084077 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:26:45.084050    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-758a442d-68549d5d57-tmq8p"]
Jul 14 23:26:45.089544 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:26:45.089523    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-758a442d-68549d5d57-tmq8p"]
Jul 14 23:26:45.229161 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:26:45.229118    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a69e5ee-1a1d-4801-bc5b-e3e5d2410163" path="/var/lib/kubelet/pods/6a69e5ee-1a1d-4801-bc5b-e3e5d2410163/volumes"
Jul 14 23:26:59.255559 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:26:59.255524    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"]
Jul 14 23:26:59.255977 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:26:59.255751    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr" podUID="287b6426-3b9e-4e79-9631-e54faaf548fb" containerName="istio-proxy" containerID="cri-o://b9d3671ee5395180e85060d17cc7481178dac4e55b9d77185bcf396c8f217c81" gracePeriod=30
Jul 14 23:27:04.529185 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:27:04.529111    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"
Jul 14 23:27:04.601497 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:27:04.601465    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/287b6426-3b9e-4e79-9631-e54faaf548fb-istio-data\") pod \"287b6426-3b9e-4e79-9631-e54faaf548fb\" (UID: \"287b6426-3b9e-4e79-9631-e54faaf548fb\") "
Jul 14 23:27:04.601700 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:27:04.601522    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/287b6426-3b9e-4e79-9631-e54faaf548fb-credential-socket\") pod \"287b6426-3b9e-4e79-9631-e54faaf548fb\" (UID: \"287b6426-3b9e-4e79-9631-e54faaf548fb\") "
Jul 14 23:27:04.601700 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:27:04.601568    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/287b6426-3b9e-4e79-9631-e54faaf548fb-istio-token\") pod \"287b6426-3b9e-4e79-9631-e54faaf548fb\" (UID: \"287b6426-3b9e-4e79-9631-e54faaf548fb\") "
Jul 14 23:27:04.601700 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:27:04.601598    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/287b6426-3b9e-4e79-9631-e54faaf548fb-workload-certs\") pod \"287b6426-3b9e-4e79-9631-e54faaf548fb\" (UID: \"287b6426-3b9e-4e79-9631-e54faaf548fb\") "
Jul 14 23:27:04.601700 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:27:04.601621    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/287b6426-3b9e-4e79-9631-e54faaf548fb-workload-socket\") pod \"287b6426-3b9e-4e79-9631-e54faaf548fb\" (UID: \"287b6426-3b9e-4e79-9631-e54faaf548fb\") "
Jul 14 23:27:04.601700 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:27:04.601676    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/287b6426-3b9e-4e79-9631-e54faaf548fb-istio-envoy\") pod \"287b6426-3b9e-4e79-9631-e54faaf548fb\" (UID: \"287b6426-3b9e-4e79-9631-e54faaf548fb\") "
Jul 14 23:27:04.601977 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:27:04.601715    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/287b6426-3b9e-4e79-9631-e54faaf548fb-istio-podinfo\") pod \"287b6426-3b9e-4e79-9631-e54faaf548fb\" (UID: \"287b6426-3b9e-4e79-9631-e54faaf548fb\") "
Jul 14 23:27:04.602415 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:27:04.602275    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/287b6426-3b9e-4e79-9631-e54faaf548fb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "287b6426-3b9e-4e79-9631-e54faaf548fb" (UID: "287b6426-3b9e-4e79-9631-e54faaf548fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:27:04.602415 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:27:04.602352    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/287b6426-3b9e-4e79-9631-e54faaf548fb-istio-data" (OuterVolumeSpecName: "istio-data") pod "287b6426-3b9e-4e79-9631-e54faaf548fb" (UID: "287b6426-3b9e-4e79-9631-e54faaf548fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:27:04.602585 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:27:04.602457    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/287b6426-3b9e-4e79-9631-e54faaf548fb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "287b6426-3b9e-4e79-9631-e54faaf548fb" (UID: "287b6426-3b9e-4e79-9631-e54faaf548fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:27:04.602637 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:27:04.602600    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/287b6426-3b9e-4e79-9631-e54faaf548fb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "287b6426-3b9e-4e79-9631-e54faaf548fb" (UID: "287b6426-3b9e-4e79-9631-e54faaf548fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:27:04.604907 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:27:04.604884    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/287b6426-3b9e-4e79-9631-e54faaf548fb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "287b6426-3b9e-4e79-9631-e54faaf548fb" (UID: "287b6426-3b9e-4e79-9631-e54faaf548fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 23:27:04.605036 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:27:04.604963    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/287b6426-3b9e-4e79-9631-e54faaf548fb-istio-token" (OuterVolumeSpecName: "istio-token") pod "287b6426-3b9e-4e79-9631-e54faaf548fb" (UID: "287b6426-3b9e-4e79-9631-e54faaf548fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:27:04.605555 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:27:04.605533    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/287b6426-3b9e-4e79-9631-e54faaf548fb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "287b6426-3b9e-4e79-9631-e54faaf548fb" (UID: "287b6426-3b9e-4e79-9631-e54faaf548fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:27:04.702376 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:27:04.702333    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsg9j\" (UniqueName: \"kubernetes.io/projected/287b6426-3b9e-4e79-9631-e54faaf548fb-kube-api-access-zsg9j\") pod \"287b6426-3b9e-4e79-9631-e54faaf548fb\" (UID: \"287b6426-3b9e-4e79-9631-e54faaf548fb\") "
Jul 14 23:27:04.702376 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:27:04.702377    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/287b6426-3b9e-4e79-9631-e54faaf548fb-istiod-ca-cert\") pod \"287b6426-3b9e-4e79-9631-e54faaf548fb\" (UID: \"287b6426-3b9e-4e79-9631-e54faaf548fb\") "
Jul 14 23:27:04.702641 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:27:04.702575    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/287b6426-3b9e-4e79-9631-e54faaf548fb-istio-envoy\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:27:04.702641 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:27:04.702585    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/287b6426-3b9e-4e79-9631-e54faaf548fb-istio-podinfo\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:27:04.702641 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:27:04.702593    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/287b6426-3b9e-4e79-9631-e54faaf548fb-istio-data\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:27:04.702641 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:27:04.702603    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/287b6426-3b9e-4e79-9631-e54faaf548fb-credential-socket\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:27:04.702641 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:27:04.702610    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/287b6426-3b9e-4e79-9631-e54faaf548fb-istio-token\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:27:04.702641 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:27:04.702619    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/287b6426-3b9e-4e79-9631-e54faaf548fb-workload-certs\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:27:04.702641 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:27:04.702627    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/287b6426-3b9e-4e79-9631-e54faaf548fb-workload-socket\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:27:04.703002 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:27:04.702741    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/287b6426-3b9e-4e79-9631-e54faaf548fb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "287b6426-3b9e-4e79-9631-e54faaf548fb" (UID: "287b6426-3b9e-4e79-9631-e54faaf548fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 23:27:04.704726 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:27:04.704696    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/287b6426-3b9e-4e79-9631-e54faaf548fb-kube-api-access-zsg9j" (OuterVolumeSpecName: "kube-api-access-zsg9j") pod "287b6426-3b9e-4e79-9631-e54faaf548fb" (UID: "287b6426-3b9e-4e79-9631-e54faaf548fb"). InnerVolumeSpecName "kube-api-access-zsg9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:27:04.803259 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:27:04.803150    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsg9j\" (UniqueName: \"kubernetes.io/projected/287b6426-3b9e-4e79-9631-e54faaf548fb-kube-api-access-zsg9j\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:27:04.803259 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:27:04.803181    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/287b6426-3b9e-4e79-9631-e54faaf548fb-istiod-ca-cert\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:27:05.158114 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:27:05.158020    2563 generic.go:356] "Generic (PLEG): container finished" podID="287b6426-3b9e-4e79-9631-e54faaf548fb" containerID="b9d3671ee5395180e85060d17cc7481178dac4e55b9d77185bcf396c8f217c81" exitCode=0
Jul 14 23:27:05.158114 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:27:05.158066    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"
Jul 14 23:27:05.158310 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:27:05.158106    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr" event={"ID":"287b6426-3b9e-4e79-9631-e54faaf548fb","Type":"ContainerDied","Data":"b9d3671ee5395180e85060d17cc7481178dac4e55b9d77185bcf396c8f217c81"}
Jul 14 23:27:05.158310 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:27:05.158162    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr" event={"ID":"287b6426-3b9e-4e79-9631-e54faaf548fb","Type":"ContainerDied","Data":"997503162c7d5045b0418567eb75217de886280bf87a7ef23981378732141a65"}
Jul 14 23:27:05.158310 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:27:05.158186    2563 scope.go:117] "RemoveContainer" containerID="b9d3671ee5395180e85060d17cc7481178dac4e55b9d77185bcf396c8f217c81"
Jul 14 23:27:05.167500 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:27:05.167481    2563 scope.go:117] "RemoveContainer" containerID="48e6fcc7c40bed6d0b7756f02397d97b3b80caefa5e617465f7d4dfb23d2ab58"
Jul 14 23:27:05.176021 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:27:05.176000    2563 scope.go:117] "RemoveContainer" containerID="b9d3671ee5395180e85060d17cc7481178dac4e55b9d77185bcf396c8f217c81"
Jul 14 23:27:05.176335 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:27:05.176305    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9d3671ee5395180e85060d17cc7481178dac4e55b9d77185bcf396c8f217c81\": container with ID starting with b9d3671ee5395180e85060d17cc7481178dac4e55b9d77185bcf396c8f217c81 not found: ID does not exist" containerID="b9d3671ee5395180e85060d17cc7481178dac4e55b9d77185bcf396c8f217c81"
Jul 14 23:27:05.176448 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:27:05.176344    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9d3671ee5395180e85060d17cc7481178dac4e55b9d77185bcf396c8f217c81"} err="failed to get container status \"b9d3671ee5395180e85060d17cc7481178dac4e55b9d77185bcf396c8f217c81\": rpc error: code = NotFound desc = could not find container \"b9d3671ee5395180e85060d17cc7481178dac4e55b9d77185bcf396c8f217c81\": container with ID starting with b9d3671ee5395180e85060d17cc7481178dac4e55b9d77185bcf396c8f217c81 not found: ID does not exist"
Jul 14 23:27:05.176448 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:27:05.176366    2563 scope.go:117] "RemoveContainer" containerID="48e6fcc7c40bed6d0b7756f02397d97b3b80caefa5e617465f7d4dfb23d2ab58"
Jul 14 23:27:05.176649 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:27:05.176629    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48e6fcc7c40bed6d0b7756f02397d97b3b80caefa5e617465f7d4dfb23d2ab58\": container with ID starting with 48e6fcc7c40bed6d0b7756f02397d97b3b80caefa5e617465f7d4dfb23d2ab58 not found: ID does not exist" containerID="48e6fcc7c40bed6d0b7756f02397d97b3b80caefa5e617465f7d4dfb23d2ab58"
Jul 14 23:27:05.176696 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:27:05.176654    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48e6fcc7c40bed6d0b7756f02397d97b3b80caefa5e617465f7d4dfb23d2ab58"} err="failed to get container status \"48e6fcc7c40bed6d0b7756f02397d97b3b80caefa5e617465f7d4dfb23d2ab58\": rpc error: code = NotFound desc = could not find container \"48e6fcc7c40bed6d0b7756f02397d97b3b80caefa5e617465f7d4dfb23d2ab58\": container with ID starting with 48e6fcc7c40bed6d0b7756f02397d97b3b80caefa5e617465f7d4dfb23d2ab58 not found: ID does not exist"
Jul 14 23:27:05.181853 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:27:05.181827    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"]
Jul 14 23:27:05.187229 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:27:05.187207    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-758a442d-openshift-default-c8dd7fc75-9cwkr"]
Jul 14 23:27:05.230379 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:27:05.230344    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="287b6426-3b9e-4e79-9631-e54faaf548fb" path="/var/lib/kubelet/pods/287b6426-3b9e-4e79-9631-e54faaf548fb/volumes"
Jul 14 23:28:34.213219 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.213138    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"]
Jul 14 23:28:34.213695 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.213541    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="287b6426-3b9e-4e79-9631-e54faaf548fb" containerName="istio-proxy"
Jul 14 23:28:34.213695 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.213561    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="287b6426-3b9e-4e79-9631-e54faaf548fb" containerName="istio-proxy"
Jul 14 23:28:34.213695 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.213588    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a69e5ee-1a1d-4801-bc5b-e3e5d2410163" containerName="maas-api"
Jul 14 23:28:34.213695 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.213597    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a69e5ee-1a1d-4801-bc5b-e3e5d2410163" containerName="maas-api"
Jul 14 23:28:34.213695 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.213609    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a70b30a0-baff-466e-b04a-e47a712ff7ee" containerName="istio-proxy"
Jul 14 23:28:34.213695 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.213614    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a70b30a0-baff-466e-b04a-e47a712ff7ee" containerName="istio-proxy"
Jul 14 23:28:34.213695 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.213672    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a69e5ee-1a1d-4801-bc5b-e3e5d2410163" containerName="maas-api"
Jul 14 23:28:34.213695 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.213682    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="287b6426-3b9e-4e79-9631-e54faaf548fb" containerName="istio-proxy"
Jul 14 23:28:34.213695 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.213688    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a70b30a0-baff-466e-b04a-e47a712ff7ee" containerName="istio-proxy"
Jul 14 23:28:34.213695 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.213694    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="287b6426-3b9e-4e79-9631-e54faaf548fb" containerName="istio-proxy"
Jul 14 23:28:34.213695 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.213700    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a70b30a0-baff-466e-b04a-e47a712ff7ee" containerName="istio-proxy"
Jul 14 23:28:34.216745 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.216726    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"
Jul 14 23:28:34.221324 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.221298    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 14 23:28:34.221324 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.221312    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-9084746a-openshift-default-dockercfg-5nrmb\""
Jul 14 23:28:34.229992 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.229962    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"]
Jul 14 23:28:34.351918 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.351879    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-istio-data\") pod \"e2e-mt-9084746a-openshift-default-5644ff647-7f7wl\" (UID: \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\") " pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"
Jul 14 23:28:34.351918 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.351919    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-workload-socket\") pod \"e2e-mt-9084746a-openshift-default-5644ff647-7f7wl\" (UID: \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\") " pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"
Jul 14 23:28:34.352149 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.351940    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-istio-token\") pod \"e2e-mt-9084746a-openshift-default-5644ff647-7f7wl\" (UID: \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\") " pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"
Jul 14 23:28:34.352149 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.351998    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-istio-podinfo\") pod \"e2e-mt-9084746a-openshift-default-5644ff647-7f7wl\" (UID: \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\") " pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"
Jul 14 23:28:34.352149 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.352043    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2bdv\" (UniqueName: \"kubernetes.io/projected/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-kube-api-access-k2bdv\") pod \"e2e-mt-9084746a-openshift-default-5644ff647-7f7wl\" (UID: \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\") " pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"
Jul 14 23:28:34.352149 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.352069    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-istio-envoy\") pod \"e2e-mt-9084746a-openshift-default-5644ff647-7f7wl\" (UID: \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\") " pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"
Jul 14 23:28:34.352149 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.352105    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-credential-socket\") pod \"e2e-mt-9084746a-openshift-default-5644ff647-7f7wl\" (UID: \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\") " pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"
Jul 14 23:28:34.352149 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.352130    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-istiod-ca-cert\") pod \"e2e-mt-9084746a-openshift-default-5644ff647-7f7wl\" (UID: \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\") " pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"
Jul 14 23:28:34.352392 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.352158    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-workload-certs\") pod \"e2e-mt-9084746a-openshift-default-5644ff647-7f7wl\" (UID: \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\") " pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"
Jul 14 23:28:34.452859 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.452815    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-credential-socket\") pod \"e2e-mt-9084746a-openshift-default-5644ff647-7f7wl\" (UID: \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\") " pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"
Jul 14 23:28:34.452859 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.452857    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-istiod-ca-cert\") pod \"e2e-mt-9084746a-openshift-default-5644ff647-7f7wl\" (UID: \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\") " pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"
Jul 14 23:28:34.453107 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.452885    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-workload-certs\") pod \"e2e-mt-9084746a-openshift-default-5644ff647-7f7wl\" (UID: \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\") " pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"
Jul 14 23:28:34.453107 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.453004    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-istio-data\") pod \"e2e-mt-9084746a-openshift-default-5644ff647-7f7wl\" (UID: \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\") " pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"
Jul 14 23:28:34.453107 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.453044    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-workload-socket\") pod \"e2e-mt-9084746a-openshift-default-5644ff647-7f7wl\" (UID: \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\") " pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"
Jul 14 23:28:34.453107 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.453066    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-istio-token\") pod \"e2e-mt-9084746a-openshift-default-5644ff647-7f7wl\" (UID: \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\") " pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"
Jul 14 23:28:34.453107 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.453099    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-istio-podinfo\") pod \"e2e-mt-9084746a-openshift-default-5644ff647-7f7wl\" (UID: \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\") " pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"
Jul 14 23:28:34.453391 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.453157    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2bdv\" (UniqueName: \"kubernetes.io/projected/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-kube-api-access-k2bdv\") pod \"e2e-mt-9084746a-openshift-default-5644ff647-7f7wl\" (UID: \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\") " pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"
Jul 14 23:28:34.453391 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.453195    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-istio-envoy\") pod \"e2e-mt-9084746a-openshift-default-5644ff647-7f7wl\" (UID: \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\") " pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"
Jul 14 23:28:34.453490 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.453424    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-credential-socket\") pod \"e2e-mt-9084746a-openshift-default-5644ff647-7f7wl\" (UID: \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\") " pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"
Jul 14 23:28:34.453546 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.453530    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-workload-certs\") pod \"e2e-mt-9084746a-openshift-default-5644ff647-7f7wl\" (UID: \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\") " pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"
Jul 14 23:28:34.453706 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.453664    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-istiod-ca-cert\") pod \"e2e-mt-9084746a-openshift-default-5644ff647-7f7wl\" (UID: \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\") " pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"
Jul 14 23:28:34.453838 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.453816    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-workload-socket\") pod \"e2e-mt-9084746a-openshift-default-5644ff647-7f7wl\" (UID: \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\") " pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"
Jul 14 23:28:34.453903 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.453893    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-istio-data\") pod \"e2e-mt-9084746a-openshift-default-5644ff647-7f7wl\" (UID: \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\") " pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"
Jul 14 23:28:34.455945 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.455915    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-istio-podinfo\") pod \"e2e-mt-9084746a-openshift-default-5644ff647-7f7wl\" (UID: \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\") " pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"
Jul 14 23:28:34.456060 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.456039    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-istio-envoy\") pod \"e2e-mt-9084746a-openshift-default-5644ff647-7f7wl\" (UID: \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\") " pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"
Jul 14 23:28:34.461967 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.461938    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-istio-token\") pod \"e2e-mt-9084746a-openshift-default-5644ff647-7f7wl\" (UID: \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\") " pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"
Jul 14 23:28:34.462750 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.462728    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2bdv\" (UniqueName: \"kubernetes.io/projected/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-kube-api-access-k2bdv\") pod \"e2e-mt-9084746a-openshift-default-5644ff647-7f7wl\" (UID: \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\") " pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"
Jul 14 23:28:34.527169 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.527128    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"
Jul 14 23:28:34.668073 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.667919    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"]
Jul 14 23:28:34.670415 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:28:34.670367    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dfb4b1f_3c5e_4d6e_95ae_5ffcf478cf9f.slice/crio-d8d703af02735ea95b920ddac2f9a82556d18caeb3088dae2d25492f16c44216 WatchSource:0}: Error finding container d8d703af02735ea95b920ddac2f9a82556d18caeb3088dae2d25492f16c44216: Status 404 returned error can't find the container with id d8d703af02735ea95b920ddac2f9a82556d18caeb3088dae2d25492f16c44216
Jul 14 23:28:34.672531 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.672491    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:28:34.672608 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.672576    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:28:34.672645 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:34.672616    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:28:35.567727 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:35.567681    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl" event={"ID":"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f","Type":"ContainerStarted","Data":"9775e3286136b43b7fecab58ba3979f80d357b30985642e0050da9845303ff61"}
Jul 14 23:28:35.568093 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:35.567749    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl" event={"ID":"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f","Type":"ContainerStarted","Data":"d8d703af02735ea95b920ddac2f9a82556d18caeb3088dae2d25492f16c44216"}
Jul 14 23:28:35.591522 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:35.591477    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl" podStartSLOduration=1.5914628450000001 podStartE2EDuration="1.591462845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:28:35.588850955 +0000 UTC m=+1902.891841748" watchObservedRunningTime="2026-07-14 23:28:35.591462845 +0000 UTC m=+1902.894453636"
Jul 14 23:28:36.528274 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:36.528239    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"
Jul 14 23:28:36.533035 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:36.533008    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"
Jul 14 23:28:36.571957 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:36.571915    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"
Jul 14 23:28:36.572760 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:36.572741    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"
Jul 14 23:28:37.641862 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:37.641826    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9084746a-78c9744d8f-tplwt"]
Jul 14 23:28:37.642224 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:37.642144    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="287b6426-3b9e-4e79-9631-e54faaf548fb" containerName="istio-proxy"
Jul 14 23:28:37.642224 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:37.642155    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="287b6426-3b9e-4e79-9631-e54faaf548fb" containerName="istio-proxy"
Jul 14 23:28:37.642224 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:37.642172    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a70b30a0-baff-466e-b04a-e47a712ff7ee" containerName="istio-proxy"
Jul 14 23:28:37.642224 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:37.642177    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a70b30a0-baff-466e-b04a-e47a712ff7ee" containerName="istio-proxy"
Jul 14 23:28:37.645188 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:37.645169    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9084746a-78c9744d8f-tplwt"
Jul 14 23:28:37.650526 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:37.650502    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-9084746a-serving-cert\""
Jul 14 23:28:37.652086 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:37.652066    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 23:28:37.652193 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:37.652169    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 23:28:37.652263 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:37.652195    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7jvtc\""
Jul 14 23:28:37.663420 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:37.663399    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9084746a-78c9744d8f-tplwt"]
Jul 14 23:28:37.783878 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:37.783829    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba32842e-e390-4a31-8642-1d8929f356b6-openshift-service-ca\") pod \"maas-api-e2e-mt-9084746a-78c9744d8f-tplwt\" (UID: \"ba32842e-e390-4a31-8642-1d8929f356b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9084746a-78c9744d8f-tplwt"
Jul 14 23:28:37.784080 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:37.783951    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsbm5\" (UniqueName: \"kubernetes.io/projected/ba32842e-e390-4a31-8642-1d8929f356b6-kube-api-access-nsbm5\") pod \"maas-api-e2e-mt-9084746a-78c9744d8f-tplwt\" (UID: \"ba32842e-e390-4a31-8642-1d8929f356b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9084746a-78c9744d8f-tplwt"
Jul 14 23:28:37.784080 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:37.784003    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba32842e-e390-4a31-8642-1d8929f356b6-maas-api-tls\") pod \"maas-api-e2e-mt-9084746a-78c9744d8f-tplwt\" (UID: \"ba32842e-e390-4a31-8642-1d8929f356b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9084746a-78c9744d8f-tplwt"
Jul 14 23:28:37.884941 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:37.884902    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsbm5\" (UniqueName: \"kubernetes.io/projected/ba32842e-e390-4a31-8642-1d8929f356b6-kube-api-access-nsbm5\") pod \"maas-api-e2e-mt-9084746a-78c9744d8f-tplwt\" (UID: \"ba32842e-e390-4a31-8642-1d8929f356b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9084746a-78c9744d8f-tplwt"
Jul 14 23:28:37.885118 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:37.884951    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba32842e-e390-4a31-8642-1d8929f356b6-maas-api-tls\") pod \"maas-api-e2e-mt-9084746a-78c9744d8f-tplwt\" (UID: \"ba32842e-e390-4a31-8642-1d8929f356b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9084746a-78c9744d8f-tplwt"
Jul 14 23:28:37.885118 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:37.885032    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba32842e-e390-4a31-8642-1d8929f356b6-openshift-service-ca\") pod \"maas-api-e2e-mt-9084746a-78c9744d8f-tplwt\" (UID: \"ba32842e-e390-4a31-8642-1d8929f356b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9084746a-78c9744d8f-tplwt"
Jul 14 23:28:37.885118 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:28:37.885098    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-9084746a-serving-cert: secret "maas-api-e2e-mt-9084746a-serving-cert" not found
Jul 14 23:28:37.885252 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:28:37.885158    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba32842e-e390-4a31-8642-1d8929f356b6-maas-api-tls podName:ba32842e-e390-4a31-8642-1d8929f356b6 nodeName:}" failed. No retries permitted until 2026-07-14 23:28:38.385141802 +0000 UTC m=+1905.688132574 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ba32842e-e390-4a31-8642-1d8929f356b6-maas-api-tls") pod "maas-api-e2e-mt-9084746a-78c9744d8f-tplwt" (UID: "ba32842e-e390-4a31-8642-1d8929f356b6") : secret "maas-api-e2e-mt-9084746a-serving-cert" not found
Jul 14 23:28:37.885772 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:37.885753    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba32842e-e390-4a31-8642-1d8929f356b6-openshift-service-ca\") pod \"maas-api-e2e-mt-9084746a-78c9744d8f-tplwt\" (UID: \"ba32842e-e390-4a31-8642-1d8929f356b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9084746a-78c9744d8f-tplwt"
Jul 14 23:28:37.907743 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:37.907677    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsbm5\" (UniqueName: \"kubernetes.io/projected/ba32842e-e390-4a31-8642-1d8929f356b6-kube-api-access-nsbm5\") pod \"maas-api-e2e-mt-9084746a-78c9744d8f-tplwt\" (UID: \"ba32842e-e390-4a31-8642-1d8929f356b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9084746a-78c9744d8f-tplwt"
Jul 14 23:28:38.388631 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:38.388590    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba32842e-e390-4a31-8642-1d8929f356b6-maas-api-tls\") pod \"maas-api-e2e-mt-9084746a-78c9744d8f-tplwt\" (UID: \"ba32842e-e390-4a31-8642-1d8929f356b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9084746a-78c9744d8f-tplwt"
Jul 14 23:28:38.391576 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:38.391547    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba32842e-e390-4a31-8642-1d8929f356b6-maas-api-tls\") pod \"maas-api-e2e-mt-9084746a-78c9744d8f-tplwt\" (UID: \"ba32842e-e390-4a31-8642-1d8929f356b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9084746a-78c9744d8f-tplwt"
Jul 14 23:28:38.559072 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:38.559032    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9084746a-78c9744d8f-tplwt"
Jul 14 23:28:38.695388 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:38.695358    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9084746a-78c9744d8f-tplwt"]
Jul 14 23:28:38.697773 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:28:38.697741    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba32842e_e390_4a31_8642_1d8929f356b6.slice/crio-44cc3d82e1607a2fcc5be7884e89acb8018830d8d5dcfcdaef72569250cc19b0 WatchSource:0}: Error finding container 44cc3d82e1607a2fcc5be7884e89acb8018830d8d5dcfcdaef72569250cc19b0: Status 404 returned error can't find the container with id 44cc3d82e1607a2fcc5be7884e89acb8018830d8d5dcfcdaef72569250cc19b0
Jul 14 23:28:38.699220 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:38.699202    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 23:28:39.585806 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:39.585689    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9084746a-78c9744d8f-tplwt" event={"ID":"ba32842e-e390-4a31-8642-1d8929f356b6","Type":"ContainerStarted","Data":"e3da84e0e9fde79d73607f90b4870d0d1aec854eab46e521c701bf7ebf5e4852"}
Jul 14 23:28:39.585806 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:39.585763    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9084746a-78c9744d8f-tplwt"
Jul 14 23:28:39.585806 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:39.585782    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9084746a-78c9744d8f-tplwt" event={"ID":"ba32842e-e390-4a31-8642-1d8929f356b6","Type":"ContainerStarted","Data":"44cc3d82e1607a2fcc5be7884e89acb8018830d8d5dcfcdaef72569250cc19b0"}
Jul 14 23:28:39.604171 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:39.604120    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9084746a-78c9744d8f-tplwt" podStartSLOduration=2.010123161 podStartE2EDuration="2.604105166s" totalImagesPullingTime="593.982005ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:28:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:28:39.602645719 +0000 UTC m=+1906.905636511" watchObservedRunningTime="2026-07-14 23:28:39.604105166 +0000 UTC m=+1906.907095995"
Jul 14 23:28:45.595941 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:45.595910    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9084746a-78c9744d8f-tplwt"
Jul 14 23:28:49.390068 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:49.390037    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9084746a-nxbxh"]
Jul 14 23:28:49.393680 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:49.393661    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9084746a-nxbxh"
Jul 14 23:28:49.396614 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:49.396588    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jw626\""
Jul 14 23:28:49.401210 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:49.401174    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9084746a-nxbxh"]
Jul 14 23:28:49.488363 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:49.488313    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2730bbc7-5e83-418c-90b1-5c95a0a8a022-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9084746a-nxbxh\" (UID: \"2730bbc7-5e83-418c-90b1-5c95a0a8a022\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9084746a-nxbxh"
Jul 14 23:28:49.589757 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:49.589711    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2730bbc7-5e83-418c-90b1-5c95a0a8a022-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9084746a-nxbxh\" (UID: \"2730bbc7-5e83-418c-90b1-5c95a0a8a022\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9084746a-nxbxh"
Jul 14 23:28:49.590433 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:49.590410    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2730bbc7-5e83-418c-90b1-5c95a0a8a022-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9084746a-nxbxh\" (UID: \"2730bbc7-5e83-418c-90b1-5c95a0a8a022\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9084746a-nxbxh"
Jul 14 23:28:49.707064 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:49.706976    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9084746a-nxbxh"
Jul 14 23:28:49.852501 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:49.852471    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9084746a-nxbxh"]
Jul 14 23:28:49.855258 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:28:49.855212    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2730bbc7_5e83_418c_90b1_5c95a0a8a022.slice/crio-06bbb595cd9b873b4cc73cd24399cc94b87d4249e01b7103357e3781d0a73a83 WatchSource:0}: Error finding container 06bbb595cd9b873b4cc73cd24399cc94b87d4249e01b7103357e3781d0a73a83: Status 404 returned error can't find the container with id 06bbb595cd9b873b4cc73cd24399cc94b87d4249e01b7103357e3781d0a73a83
Jul 14 23:28:50.642769 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:50.642737    2563 generic.go:356] "Generic (PLEG): container finished" podID="2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f" containerID="9775e3286136b43b7fecab58ba3979f80d357b30985642e0050da9845303ff61" exitCode=0
Jul 14 23:28:50.643222 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:50.642804    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl" event={"ID":"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f","Type":"ContainerDied","Data":"9775e3286136b43b7fecab58ba3979f80d357b30985642e0050da9845303ff61"}
Jul 14 23:28:50.643222 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:50.643212    2563 scope.go:117] "RemoveContainer" containerID="9775e3286136b43b7fecab58ba3979f80d357b30985642e0050da9845303ff61"
Jul 14 23:28:50.644316 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:50.644264    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:28:50.644408 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:50.644332    2563 generic.go:356] "Generic (PLEG): container finished" podID="2730bbc7-5e83-418c-90b1-5c95a0a8a022" containerID="84cc659b738ac14a744f10208cac4f26264e899162822bf7a83db1f9ce6f466c" exitCode=0
Jul 14 23:28:50.644408 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:50.644361    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9084746a-nxbxh" event={"ID":"2730bbc7-5e83-418c-90b1-5c95a0a8a022","Type":"ContainerDied","Data":"84cc659b738ac14a744f10208cac4f26264e899162822bf7a83db1f9ce6f466c"}
Jul 14 23:28:50.644479 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:50.644408    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9084746a-nxbxh" event={"ID":"2730bbc7-5e83-418c-90b1-5c95a0a8a022","Type":"ContainerStarted","Data":"06bbb595cd9b873b4cc73cd24399cc94b87d4249e01b7103357e3781d0a73a83"}
Jul 14 23:28:50.644479 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:50.644362    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:28:50.644479 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:50.644465    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:28:51.572430 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:51.572390    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"
Jul 14 23:28:51.651095 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:51.651056    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl" event={"ID":"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f","Type":"ContainerStarted","Data":"a037203f2e7ea1b9f295a4f31f34a6b817aa294aef9c18b5d03f44796f1418a0"}
Jul 14 23:28:51.792062 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:51.792036    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9084746a-nxbxh"
Jul 14 23:28:51.912649 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:51.912528    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2730bbc7-5e83-418c-90b1-5c95a0a8a022-maas-api-service-ca\") pod \"2730bbc7-5e83-418c-90b1-5c95a0a8a022\" (UID: \"2730bbc7-5e83-418c-90b1-5c95a0a8a022\") "
Jul 14 23:28:51.912932 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:51.912903    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2730bbc7-5e83-418c-90b1-5c95a0a8a022-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2730bbc7-5e83-418c-90b1-5c95a0a8a022" (UID: "2730bbc7-5e83-418c-90b1-5c95a0a8a022"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 23:28:52.013622 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:52.013574    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2730bbc7-5e83-418c-90b1-5c95a0a8a022-maas-api-service-ca\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:28:52.527468 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:52.527423    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"
Jul 14 23:28:52.529021 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:52.528988    2563 patch_prober.go:28] interesting pod/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Jul 14 23:28:52.529271 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:52.529232    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl" podUID="2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Jul 14 23:28:52.657004 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:52.656974    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9084746a-nxbxh"
Jul 14 23:28:52.657462 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:52.657012    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9084746a-nxbxh" event={"ID":"2730bbc7-5e83-418c-90b1-5c95a0a8a022","Type":"ContainerDied","Data":"06bbb595cd9b873b4cc73cd24399cc94b87d4249e01b7103357e3781d0a73a83"}
Jul 14 23:28:52.657462 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:52.657076    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06bbb595cd9b873b4cc73cd24399cc94b87d4249e01b7103357e3781d0a73a83"
Jul 14 23:28:53.367393 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:53.367353    2563 scope.go:117] "RemoveContainer" containerID="0f3a86f343aa9ca47eeaf3bf68435414bcee416f5e425ed2a7c2dd44d5f9cef7"
Jul 14 23:28:53.528708 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:53.528662    2563 patch_prober.go:28] interesting pod/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Jul 14 23:28:53.528928 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:53.528744    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl" podUID="2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Jul 14 23:28:54.527788 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:54.527740    2563 patch_prober.go:28] interesting pod/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Jul 14 23:28:54.528313 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:54.527812    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl" podUID="2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Jul 14 23:28:55.531988 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:55.531959    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"
Jul 14 23:28:55.532456 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:55.532376    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"
Jul 14 23:28:55.533103 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:55.533090    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"
Jul 14 23:28:59.912975 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:59.912935    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9084746a-78c9744d8f-tplwt"]
Jul 14 23:28:59.913419 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:28:59.913194    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9084746a-78c9744d8f-tplwt" podUID="ba32842e-e390-4a31-8642-1d8929f356b6" containerName="maas-api" containerID="cri-o://e3da84e0e9fde79d73607f90b4870d0d1aec854eab46e521c701bf7ebf5e4852" gracePeriod=30
Jul 14 23:29:00.156694 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:00.156669    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9084746a-78c9744d8f-tplwt"
Jul 14 23:29:00.182400 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:00.182324    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba32842e-e390-4a31-8642-1d8929f356b6-openshift-service-ca\") pod \"ba32842e-e390-4a31-8642-1d8929f356b6\" (UID: \"ba32842e-e390-4a31-8642-1d8929f356b6\") "
Jul 14 23:29:00.182400 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:00.182380    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba32842e-e390-4a31-8642-1d8929f356b6-maas-api-tls\") pod \"ba32842e-e390-4a31-8642-1d8929f356b6\" (UID: \"ba32842e-e390-4a31-8642-1d8929f356b6\") "
Jul 14 23:29:00.182576 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:00.182443    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nsbm5\" (UniqueName: \"kubernetes.io/projected/ba32842e-e390-4a31-8642-1d8929f356b6-kube-api-access-nsbm5\") pod \"ba32842e-e390-4a31-8642-1d8929f356b6\" (UID: \"ba32842e-e390-4a31-8642-1d8929f356b6\") "
Jul 14 23:29:00.182732 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:00.182713    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba32842e-e390-4a31-8642-1d8929f356b6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ba32842e-e390-4a31-8642-1d8929f356b6" (UID: "ba32842e-e390-4a31-8642-1d8929f356b6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 23:29:00.184550 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:00.184525    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba32842e-e390-4a31-8642-1d8929f356b6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ba32842e-e390-4a31-8642-1d8929f356b6" (UID: "ba32842e-e390-4a31-8642-1d8929f356b6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 23:29:00.184643 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:00.184619    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba32842e-e390-4a31-8642-1d8929f356b6-kube-api-access-nsbm5" (OuterVolumeSpecName: "kube-api-access-nsbm5") pod "ba32842e-e390-4a31-8642-1d8929f356b6" (UID: "ba32842e-e390-4a31-8642-1d8929f356b6"). InnerVolumeSpecName "kube-api-access-nsbm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:29:00.283165 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:00.283131    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba32842e-e390-4a31-8642-1d8929f356b6-maas-api-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:29:00.283165 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:00.283157    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsbm5\" (UniqueName: \"kubernetes.io/projected/ba32842e-e390-4a31-8642-1d8929f356b6-kube-api-access-nsbm5\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:29:00.283165 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:00.283166    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba32842e-e390-4a31-8642-1d8929f356b6-openshift-service-ca\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:29:00.694617 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:00.694573    2563 generic.go:356] "Generic (PLEG): container finished" podID="ba32842e-e390-4a31-8642-1d8929f356b6" containerID="e3da84e0e9fde79d73607f90b4870d0d1aec854eab46e521c701bf7ebf5e4852" exitCode=0
Jul 14 23:29:00.694617 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:00.694617    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9084746a-78c9744d8f-tplwt"
Jul 14 23:29:00.694825 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:00.694656    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9084746a-78c9744d8f-tplwt" event={"ID":"ba32842e-e390-4a31-8642-1d8929f356b6","Type":"ContainerDied","Data":"e3da84e0e9fde79d73607f90b4870d0d1aec854eab46e521c701bf7ebf5e4852"}
Jul 14 23:29:00.694825 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:00.694712    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9084746a-78c9744d8f-tplwt" event={"ID":"ba32842e-e390-4a31-8642-1d8929f356b6","Type":"ContainerDied","Data":"44cc3d82e1607a2fcc5be7884e89acb8018830d8d5dcfcdaef72569250cc19b0"}
Jul 14 23:29:00.694825 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:00.694733    2563 scope.go:117] "RemoveContainer" containerID="e3da84e0e9fde79d73607f90b4870d0d1aec854eab46e521c701bf7ebf5e4852"
Jul 14 23:29:00.704229 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:00.704212    2563 scope.go:117] "RemoveContainer" containerID="e3da84e0e9fde79d73607f90b4870d0d1aec854eab46e521c701bf7ebf5e4852"
Jul 14 23:29:00.704496 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:29:00.704476    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3da84e0e9fde79d73607f90b4870d0d1aec854eab46e521c701bf7ebf5e4852\": container with ID starting with e3da84e0e9fde79d73607f90b4870d0d1aec854eab46e521c701bf7ebf5e4852 not found: ID does not exist" containerID="e3da84e0e9fde79d73607f90b4870d0d1aec854eab46e521c701bf7ebf5e4852"
Jul 14 23:29:00.704550 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:00.704504    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3da84e0e9fde79d73607f90b4870d0d1aec854eab46e521c701bf7ebf5e4852"} err="failed to get container status \"e3da84e0e9fde79d73607f90b4870d0d1aec854eab46e521c701bf7ebf5e4852\": rpc error: code = NotFound desc = could not find container \"e3da84e0e9fde79d73607f90b4870d0d1aec854eab46e521c701bf7ebf5e4852\": container with ID starting with e3da84e0e9fde79d73607f90b4870d0d1aec854eab46e521c701bf7ebf5e4852 not found: ID does not exist"
Jul 14 23:29:00.718370 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:00.718345    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9084746a-78c9744d8f-tplwt"]
Jul 14 23:29:00.724103 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:00.724081    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9084746a-78c9744d8f-tplwt"]
Jul 14 23:29:01.229621 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:01.229584    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba32842e-e390-4a31-8642-1d8929f356b6" path="/var/lib/kubelet/pods/ba32842e-e390-4a31-8642-1d8929f356b6/volumes"
Jul 14 23:29:15.174077 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:15.174034    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"]
Jul 14 23:29:15.174553 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:15.174271    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl" podUID="2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f" containerName="istio-proxy" containerID="cri-o://a037203f2e7ea1b9f295a4f31f34a6b817aa294aef9c18b5d03f44796f1418a0" gracePeriod=30
Jul 14 23:29:20.425919 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:20.425892    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"
Jul 14 23:29:20.551404 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:20.551373    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-istio-data\") pod \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\" (UID: \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\") "
Jul 14 23:29:20.551404 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:20.551409    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-credential-socket\") pod \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\" (UID: \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\") "
Jul 14 23:29:20.551664 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:20.551426    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-istiod-ca-cert\") pod \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\" (UID: \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\") "
Jul 14 23:29:20.551664 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:20.551464    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-istio-envoy\") pod \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\" (UID: \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\") "
Jul 14 23:29:20.551664 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:20.551499    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-istio-podinfo\") pod \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\" (UID: \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\") "
Jul 14 23:29:20.551664 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:20.551539    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2bdv\" (UniqueName: \"kubernetes.io/projected/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-kube-api-access-k2bdv\") pod \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\" (UID: \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\") "
Jul 14 23:29:20.551664 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:20.551581    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-workload-socket\") pod \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\" (UID: \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\") "
Jul 14 23:29:20.551664 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:20.551613    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-workload-certs\") pod \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\" (UID: \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\") "
Jul 14 23:29:20.551664 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:20.551638    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-istio-token\") pod \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\" (UID: \"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f\") "
Jul 14 23:29:20.552030 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:20.551766    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f" (UID: "2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:29:20.552030 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:20.551873    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f" (UID: "2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:29:20.552030 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:20.551898    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f" (UID: "2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 23:29:20.552030 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:20.551923    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-credential-socket\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:29:20.552229 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:20.552073    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f" (UID: "2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:29:20.552449 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:20.552426    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-istio-data" (OuterVolumeSpecName: "istio-data") pod "2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f" (UID: "2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:29:20.553992 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:20.553972    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f" (UID: "2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:29:20.554064 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:20.554049    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-istio-token" (OuterVolumeSpecName: "istio-token") pod "2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f" (UID: "2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:29:20.554414 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:20.554395    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f" (UID: "2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 23:29:20.554472 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:20.554434    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-kube-api-access-k2bdv" (OuterVolumeSpecName: "kube-api-access-k2bdv") pod "2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f" (UID: "2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f"). InnerVolumeSpecName "kube-api-access-k2bdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:29:20.653333 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:20.653254    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-workload-socket\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:29:20.653333 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:20.653331    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-workload-certs\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:29:20.653333 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:20.653344    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-istio-token\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:29:20.653599 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:20.653356    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-istio-data\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:29:20.653599 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:20.653368    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-istiod-ca-cert\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:29:20.653599 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:20.653379    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-istio-envoy\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:29:20.653599 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:20.653390    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-istio-podinfo\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:29:20.653599 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:20.653400    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2bdv\" (UniqueName: \"kubernetes.io/projected/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f-kube-api-access-k2bdv\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:29:20.785767 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:20.785727    2563 generic.go:356] "Generic (PLEG): container finished" podID="2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f" containerID="a037203f2e7ea1b9f295a4f31f34a6b817aa294aef9c18b5d03f44796f1418a0" exitCode=0
Jul 14 23:29:20.785916 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:20.785781    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl" event={"ID":"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f","Type":"ContainerDied","Data":"a037203f2e7ea1b9f295a4f31f34a6b817aa294aef9c18b5d03f44796f1418a0"}
Jul 14 23:29:20.785916 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:20.785825    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl" event={"ID":"2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f","Type":"ContainerDied","Data":"d8d703af02735ea95b920ddac2f9a82556d18caeb3088dae2d25492f16c44216"}
Jul 14 23:29:20.785916 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:20.785847    2563 scope.go:117] "RemoveContainer" containerID="a037203f2e7ea1b9f295a4f31f34a6b817aa294aef9c18b5d03f44796f1418a0"
Jul 14 23:29:20.785916 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:20.785781    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"
Jul 14 23:29:20.795207 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:20.795190    2563 scope.go:117] "RemoveContainer" containerID="9775e3286136b43b7fecab58ba3979f80d357b30985642e0050da9845303ff61"
Jul 14 23:29:20.803356 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:20.803336    2563 scope.go:117] "RemoveContainer" containerID="a037203f2e7ea1b9f295a4f31f34a6b817aa294aef9c18b5d03f44796f1418a0"
Jul 14 23:29:20.803653 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:29:20.803630    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a037203f2e7ea1b9f295a4f31f34a6b817aa294aef9c18b5d03f44796f1418a0\": container with ID starting with a037203f2e7ea1b9f295a4f31f34a6b817aa294aef9c18b5d03f44796f1418a0 not found: ID does not exist" containerID="a037203f2e7ea1b9f295a4f31f34a6b817aa294aef9c18b5d03f44796f1418a0"
Jul 14 23:29:20.803702 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:20.803663    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a037203f2e7ea1b9f295a4f31f34a6b817aa294aef9c18b5d03f44796f1418a0"} err="failed to get container status \"a037203f2e7ea1b9f295a4f31f34a6b817aa294aef9c18b5d03f44796f1418a0\": rpc error: code = NotFound desc = could not find container \"a037203f2e7ea1b9f295a4f31f34a6b817aa294aef9c18b5d03f44796f1418a0\": container with ID starting with a037203f2e7ea1b9f295a4f31f34a6b817aa294aef9c18b5d03f44796f1418a0 not found: ID does not exist"
Jul 14 23:29:20.803702 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:20.803681    2563 scope.go:117] "RemoveContainer" containerID="9775e3286136b43b7fecab58ba3979f80d357b30985642e0050da9845303ff61"
Jul 14 23:29:20.803925 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:29:20.803897    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9775e3286136b43b7fecab58ba3979f80d357b30985642e0050da9845303ff61\": container with ID starting with 9775e3286136b43b7fecab58ba3979f80d357b30985642e0050da9845303ff61 not found: ID does not exist" containerID="9775e3286136b43b7fecab58ba3979f80d357b30985642e0050da9845303ff61"
Jul 14 23:29:20.803968 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:20.803931    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9775e3286136b43b7fecab58ba3979f80d357b30985642e0050da9845303ff61"} err="failed to get container status \"9775e3286136b43b7fecab58ba3979f80d357b30985642e0050da9845303ff61\": rpc error: code = NotFound desc = could not find container \"9775e3286136b43b7fecab58ba3979f80d357b30985642e0050da9845303ff61\": container with ID starting with 9775e3286136b43b7fecab58ba3979f80d357b30985642e0050da9845303ff61 not found: ID does not exist"
Jul 14 23:29:20.809825 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:20.809803    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"]
Jul 14 23:29:20.814399 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:20.814378    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-9084746a-openshift-default-5644ff647-7f7wl"]
Jul 14 23:29:21.234875 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:21.230297    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f" path="/var/lib/kubelet/pods/2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f/volumes"
Jul 14 23:29:34.614213 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:34.614176    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"]
Jul 14 23:29:34.614624 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:34.614537    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba32842e-e390-4a31-8642-1d8929f356b6" containerName="maas-api"
Jul 14 23:29:34.614624 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:34.614550    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba32842e-e390-4a31-8642-1d8929f356b6" containerName="maas-api"
Jul 14 23:29:34.614624 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:34.614557    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2730bbc7-5e83-418c-90b1-5c95a0a8a022" containerName="revoke-keys"
Jul 14 23:29:34.614624 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:34.614563    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2730bbc7-5e83-418c-90b1-5c95a0a8a022" containerName="revoke-keys"
Jul 14 23:29:34.614624 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:34.614582    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f" containerName="istio-proxy"
Jul 14 23:29:34.614624 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:34.614588    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f" containerName="istio-proxy"
Jul 14 23:29:34.614624 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:34.614595    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f" containerName="istio-proxy"
Jul 14 23:29:34.614624 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:34.614600    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f" containerName="istio-proxy"
Jul 14 23:29:34.614862 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:34.614680    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2730bbc7-5e83-418c-90b1-5c95a0a8a022" containerName="revoke-keys"
Jul 14 23:29:34.614862 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:34.614690    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f" containerName="istio-proxy"
Jul 14 23:29:34.614862 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:34.614696    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba32842e-e390-4a31-8642-1d8929f356b6" containerName="maas-api"
Jul 14 23:29:34.616979 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:34.616963    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"
Jul 14 23:29:34.620188 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:34.620167    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-61774e12-openshift-default-dockercfg-5jjvt\""
Jul 14 23:29:34.620300 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:34.620165    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 14 23:29:34.634565 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:34.634541    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"]
Jul 14 23:29:34.680270 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:34.680228    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c8790f5-3eba-49c9-add9-7c451c117e00-istio-data\") pod \"e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6\" (UID: \"1c8790f5-3eba-49c9-add9-7c451c117e00\") " pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"
Jul 14 23:29:34.680476 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:34.680307    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c8790f5-3eba-49c9-add9-7c451c117e00-istio-podinfo\") pod \"e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6\" (UID: \"1c8790f5-3eba-49c9-add9-7c451c117e00\") " pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"
Jul 14 23:29:34.680476 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:34.680335    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c8790f5-3eba-49c9-add9-7c451c117e00-istiod-ca-cert\") pod \"e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6\" (UID: \"1c8790f5-3eba-49c9-add9-7c451c117e00\") " pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"
Jul 14 23:29:34.680476 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:34.680365    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2frr\" (UniqueName: \"kubernetes.io/projected/1c8790f5-3eba-49c9-add9-7c451c117e00-kube-api-access-x2frr\") pod \"e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6\" (UID: \"1c8790f5-3eba-49c9-add9-7c451c117e00\") " pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"
Jul 14 23:29:34.680606 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:34.680483    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c8790f5-3eba-49c9-add9-7c451c117e00-workload-certs\") pod \"e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6\" (UID: \"1c8790f5-3eba-49c9-add9-7c451c117e00\") " pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"
Jul 14 23:29:34.680606 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:34.680534    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8790f5-3eba-49c9-add9-7c451c117e00-workload-socket\") pod \"e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6\" (UID: \"1c8790f5-3eba-49c9-add9-7c451c117e00\") " pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"
Jul 14 23:29:34.680606 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:34.680600    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8790f5-3eba-49c9-add9-7c451c117e00-credential-socket\") pod \"e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6\" (UID: \"1c8790f5-3eba-49c9-add9-7c451c117e00\") " pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"
Jul 14 23:29:34.680726 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:34.680628    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c8790f5-3eba-49c9-add9-7c451c117e00-istio-token\") pod \"e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6\" (UID: \"1c8790f5-3eba-49c9-add9-7c451c117e00\") " pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"
Jul 14 23:29:34.680726 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:34.680668    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c8790f5-3eba-49c9-add9-7c451c117e00-istio-envoy\") pod \"e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6\" (UID: \"1c8790f5-3eba-49c9-add9-7c451c117e00\") " pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"
Jul 14 23:29:34.782027 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:34.781988    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c8790f5-3eba-49c9-add9-7c451c117e00-workload-certs\") pod \"e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6\" (UID: \"1c8790f5-3eba-49c9-add9-7c451c117e00\") " pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"
Jul 14 23:29:34.782027 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:34.782030    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8790f5-3eba-49c9-add9-7c451c117e00-workload-socket\") pod \"e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6\" (UID: \"1c8790f5-3eba-49c9-add9-7c451c117e00\") " pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"
Jul 14 23:29:34.782308 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:34.782055    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8790f5-3eba-49c9-add9-7c451c117e00-credential-socket\") pod \"e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6\" (UID: \"1c8790f5-3eba-49c9-add9-7c451c117e00\") " pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"
Jul 14 23:29:34.782308 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:34.782077    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c8790f5-3eba-49c9-add9-7c451c117e00-istio-token\") pod \"e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6\" (UID: \"1c8790f5-3eba-49c9-add9-7c451c117e00\") " pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"
Jul 14 23:29:34.782308 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:34.782115    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c8790f5-3eba-49c9-add9-7c451c117e00-istio-envoy\") pod \"e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6\" (UID: \"1c8790f5-3eba-49c9-add9-7c451c117e00\") " pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"
Jul 14 23:29:34.782308 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:34.782164    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c8790f5-3eba-49c9-add9-7c451c117e00-istio-data\") pod \"e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6\" (UID: \"1c8790f5-3eba-49c9-add9-7c451c117e00\") " pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"
Jul 14 23:29:34.782537 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:34.782332    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c8790f5-3eba-49c9-add9-7c451c117e00-istio-podinfo\") pod \"e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6\" (UID: \"1c8790f5-3eba-49c9-add9-7c451c117e00\") " pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"
Jul 14 23:29:34.782537 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:34.782361    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c8790f5-3eba-49c9-add9-7c451c117e00-istiod-ca-cert\") pod \"e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6\" (UID: \"1c8790f5-3eba-49c9-add9-7c451c117e00\") " pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"
Jul 14 23:29:34.782537 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:34.782400    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2frr\" (UniqueName: \"kubernetes.io/projected/1c8790f5-3eba-49c9-add9-7c451c117e00-kube-api-access-x2frr\") pod \"e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6\" (UID: \"1c8790f5-3eba-49c9-add9-7c451c117e00\") " pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"
Jul 14 23:29:34.782537 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:34.782498    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c8790f5-3eba-49c9-add9-7c451c117e00-workload-certs\") pod \"e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6\" (UID: \"1c8790f5-3eba-49c9-add9-7c451c117e00\") " pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"
Jul 14 23:29:34.782729 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:34.782685    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c8790f5-3eba-49c9-add9-7c451c117e00-istio-data\") pod \"e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6\" (UID: \"1c8790f5-3eba-49c9-add9-7c451c117e00\") " pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"
Jul 14 23:29:34.782833 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:34.782812    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8790f5-3eba-49c9-add9-7c451c117e00-credential-socket\") pod \"e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6\" (UID: \"1c8790f5-3eba-49c9-add9-7c451c117e00\") " pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"
Jul 14 23:29:34.782957 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:34.782935    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8790f5-3eba-49c9-add9-7c451c117e00-workload-socket\") pod \"e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6\" (UID: \"1c8790f5-3eba-49c9-add9-7c451c117e00\") " pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"
Jul 14 23:29:34.783189 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:34.783167    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c8790f5-3eba-49c9-add9-7c451c117e00-istiod-ca-cert\") pod \"e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6\" (UID: \"1c8790f5-3eba-49c9-add9-7c451c117e00\") " pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"
Jul 14 23:29:34.784891 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:34.784866    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c8790f5-3eba-49c9-add9-7c451c117e00-istio-envoy\") pod \"e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6\" (UID: \"1c8790f5-3eba-49c9-add9-7c451c117e00\") " pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"
Jul 14 23:29:34.785072 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:34.785053    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c8790f5-3eba-49c9-add9-7c451c117e00-istio-podinfo\") pod \"e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6\" (UID: \"1c8790f5-3eba-49c9-add9-7c451c117e00\") " pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"
Jul 14 23:29:34.795525 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:34.795451    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c8790f5-3eba-49c9-add9-7c451c117e00-istio-token\") pod \"e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6\" (UID: \"1c8790f5-3eba-49c9-add9-7c451c117e00\") " pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"
Jul 14 23:29:34.795675 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:34.795571    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2frr\" (UniqueName: \"kubernetes.io/projected/1c8790f5-3eba-49c9-add9-7c451c117e00-kube-api-access-x2frr\") pod \"e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6\" (UID: \"1c8790f5-3eba-49c9-add9-7c451c117e00\") " pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"
Jul 14 23:29:34.926747 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:34.926642    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"
Jul 14 23:29:35.081162 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:35.081127    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"]
Jul 14 23:29:35.083543 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:29:35.083516    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c8790f5_3eba_49c9_add9_7c451c117e00.slice/crio-ba18e02ffb11317e590cb27cc15c29519704f7a2de5ea17c7ac3103315acbd14 WatchSource:0}: Error finding container ba18e02ffb11317e590cb27cc15c29519704f7a2de5ea17c7ac3103315acbd14: Status 404 returned error can't find the container with id ba18e02ffb11317e590cb27cc15c29519704f7a2de5ea17c7ac3103315acbd14
Jul 14 23:29:35.085639 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:35.085603    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:29:35.085744 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:35.085668    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:29:35.085744 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:35.085703    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:29:35.858389 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:35.858337    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6" event={"ID":"1c8790f5-3eba-49c9-add9-7c451c117e00","Type":"ContainerStarted","Data":"9130d572be1c77ef952973d09f33ec52d72fb3db59d8dc4be381dc6d8602da29"}
Jul 14 23:29:35.858876 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:35.858407    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6" event={"ID":"1c8790f5-3eba-49c9-add9-7c451c117e00","Type":"ContainerStarted","Data":"ba18e02ffb11317e590cb27cc15c29519704f7a2de5ea17c7ac3103315acbd14"}
Jul 14 23:29:35.880226 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:35.880173    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6" podStartSLOduration=1.880159744 podStartE2EDuration="1.880159744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:29:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:29:35.878518132 +0000 UTC m=+1963.181508924" watchObservedRunningTime="2026-07-14 23:29:35.880159744 +0000 UTC m=+1963.183150535"
Jul 14 23:29:35.927390 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:35.927356    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"
Jul 14 23:29:36.932135 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:36.932102    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"
Jul 14 23:29:37.868448 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:37.868416    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"
Jul 14 23:29:37.869735 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:37.869710    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"
Jul 14 23:29:37.880674 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:37.880634    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-61774e12-7df7cb89fc-c8pqg"]
Jul 14 23:29:37.881010 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:37.880997    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dfb4b1f-3c5e-4d6e-95ae-5ffcf478cf9f" containerName="istio-proxy"
Jul 14 23:29:37.883658 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:37.883636    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61774e12-7df7cb89fc-c8pqg"
Jul 14 23:29:37.886755 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:37.886719    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 23:29:37.887084 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:37.886881    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 23:29:37.887084 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:37.886925    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-61774e12-serving-cert\""
Jul 14 23:29:37.887402 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:37.887379    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7jvtc\""
Jul 14 23:29:37.895675 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:37.895651    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-61774e12-7df7cb89fc-c8pqg"]
Jul 14 23:29:38.014606 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:38.014558    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hsgm\" (UniqueName: \"kubernetes.io/projected/a2b29666-10d9-4fd2-9039-2cd2ea1354df-kube-api-access-5hsgm\") pod \"maas-api-e2e-mt-61774e12-7df7cb89fc-c8pqg\" (UID: \"a2b29666-10d9-4fd2-9039-2cd2ea1354df\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61774e12-7df7cb89fc-c8pqg"
Jul 14 23:29:38.015094 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:38.014670    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2b29666-10d9-4fd2-9039-2cd2ea1354df-openshift-service-ca\") pod \"maas-api-e2e-mt-61774e12-7df7cb89fc-c8pqg\" (UID: \"a2b29666-10d9-4fd2-9039-2cd2ea1354df\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61774e12-7df7cb89fc-c8pqg"
Jul 14 23:29:38.015094 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:38.014822    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2b29666-10d9-4fd2-9039-2cd2ea1354df-maas-api-tls\") pod \"maas-api-e2e-mt-61774e12-7df7cb89fc-c8pqg\" (UID: \"a2b29666-10d9-4fd2-9039-2cd2ea1354df\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61774e12-7df7cb89fc-c8pqg"
Jul 14 23:29:38.116006 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:38.115972    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2b29666-10d9-4fd2-9039-2cd2ea1354df-maas-api-tls\") pod \"maas-api-e2e-mt-61774e12-7df7cb89fc-c8pqg\" (UID: \"a2b29666-10d9-4fd2-9039-2cd2ea1354df\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61774e12-7df7cb89fc-c8pqg"
Jul 14 23:29:38.116190 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:38.116019    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hsgm\" (UniqueName: \"kubernetes.io/projected/a2b29666-10d9-4fd2-9039-2cd2ea1354df-kube-api-access-5hsgm\") pod \"maas-api-e2e-mt-61774e12-7df7cb89fc-c8pqg\" (UID: \"a2b29666-10d9-4fd2-9039-2cd2ea1354df\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61774e12-7df7cb89fc-c8pqg"
Jul 14 23:29:38.116190 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:38.116048    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2b29666-10d9-4fd2-9039-2cd2ea1354df-openshift-service-ca\") pod \"maas-api-e2e-mt-61774e12-7df7cb89fc-c8pqg\" (UID: \"a2b29666-10d9-4fd2-9039-2cd2ea1354df\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61774e12-7df7cb89fc-c8pqg"
Jul 14 23:29:38.116718 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:38.116694    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2b29666-10d9-4fd2-9039-2cd2ea1354df-openshift-service-ca\") pod \"maas-api-e2e-mt-61774e12-7df7cb89fc-c8pqg\" (UID: \"a2b29666-10d9-4fd2-9039-2cd2ea1354df\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61774e12-7df7cb89fc-c8pqg"
Jul 14 23:29:38.118794 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:38.118736    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2b29666-10d9-4fd2-9039-2cd2ea1354df-maas-api-tls\") pod \"maas-api-e2e-mt-61774e12-7df7cb89fc-c8pqg\" (UID: \"a2b29666-10d9-4fd2-9039-2cd2ea1354df\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61774e12-7df7cb89fc-c8pqg"
Jul 14 23:29:38.125112 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:38.125080    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hsgm\" (UniqueName: \"kubernetes.io/projected/a2b29666-10d9-4fd2-9039-2cd2ea1354df-kube-api-access-5hsgm\") pod \"maas-api-e2e-mt-61774e12-7df7cb89fc-c8pqg\" (UID: \"a2b29666-10d9-4fd2-9039-2cd2ea1354df\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61774e12-7df7cb89fc-c8pqg"
Jul 14 23:29:38.196147 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:38.196117    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61774e12-7df7cb89fc-c8pqg"
Jul 14 23:29:38.374123 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:38.374097    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-61774e12-7df7cb89fc-c8pqg"]
Jul 14 23:29:38.375957 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:29:38.375929    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2b29666_10d9_4fd2_9039_2cd2ea1354df.slice/crio-28b03b293dfe340f35c08440c4ca2dec43973e1076372f7d78382f75188a12ab WatchSource:0}: Error finding container 28b03b293dfe340f35c08440c4ca2dec43973e1076372f7d78382f75188a12ab: Status 404 returned error can't find the container with id 28b03b293dfe340f35c08440c4ca2dec43973e1076372f7d78382f75188a12ab
Jul 14 23:29:38.874898 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:38.874860    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61774e12-7df7cb89fc-c8pqg" event={"ID":"a2b29666-10d9-4fd2-9039-2cd2ea1354df","Type":"ContainerStarted","Data":"4777c589d42ca22efc5eb39b69ee150f2011a5ad863b2a1929542d680f0717e5"}
Jul 14 23:29:38.875076 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:38.874915    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61774e12-7df7cb89fc-c8pqg" event={"ID":"a2b29666-10d9-4fd2-9039-2cd2ea1354df","Type":"ContainerStarted","Data":"28b03b293dfe340f35c08440c4ca2dec43973e1076372f7d78382f75188a12ab"}
Jul 14 23:29:38.894108 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:38.894050    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61774e12-7df7cb89fc-c8pqg" podStartSLOduration=1.538797262 podStartE2EDuration="1.894032211s" totalImagesPullingTime="355.234949ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:29:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:29:38.892682567 +0000 UTC m=+1966.195673360" watchObservedRunningTime="2026-07-14 23:29:38.894032211 +0000 UTC m=+1966.197023047"
Jul 14 23:29:39.879723 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:39.879675    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61774e12-7df7cb89fc-c8pqg"
Jul 14 23:29:45.889939 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:45.889855    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61774e12-7df7cb89fc-c8pqg"
Jul 14 23:29:51.936129 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:51.936091    2563 generic.go:356] "Generic (PLEG): container finished" podID="1c8790f5-3eba-49c9-add9-7c451c117e00" containerID="9130d572be1c77ef952973d09f33ec52d72fb3db59d8dc4be381dc6d8602da29" exitCode=0
Jul 14 23:29:51.936587 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:51.936150    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6" event={"ID":"1c8790f5-3eba-49c9-add9-7c451c117e00","Type":"ContainerDied","Data":"9130d572be1c77ef952973d09f33ec52d72fb3db59d8dc4be381dc6d8602da29"}
Jul 14 23:29:51.936587 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:51.936504    2563 scope.go:117] "RemoveContainer" containerID="9130d572be1c77ef952973d09f33ec52d72fb3db59d8dc4be381dc6d8602da29"
Jul 14 23:29:51.937503 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:51.937471    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:29:51.937579 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:51.937561    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:29:51.937626 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:51.937612    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:29:52.869051 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:52.868994    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"
Jul 14 23:29:52.943606 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:52.943566    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6" event={"ID":"1c8790f5-3eba-49c9-add9-7c451c117e00","Type":"ContainerStarted","Data":"31add902371d6ab184cab6e7d7c575ad70bb0dd1af59e26136996074005d4217"}
Jul 14 23:29:53.585696 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:53.585652    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-61774e12/e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9"]
Jul 14 23:29:53.589813 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:53.589783    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-61774e12/e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9"
Jul 14 23:29:53.592737 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:53.592705    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-61774e12\"/\"openshift-service-ca.crt\""
Jul 14 23:29:53.592737 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:53.592734    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-61774e12\"/\"kube-root-ca.crt\""
Jul 14 23:29:53.594037 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:53.594017    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-61774e12\"/\"default-dockercfg-5s7lj\""
Jul 14 23:29:53.594123 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:53.594102    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-61774e12\"/\"e2e-lifecycle-model-61774e12-kserve-self-signed-certs\""
Jul 14 23:29:53.597592 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:53.597571    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-61774e12/e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9"]
Jul 14 23:29:53.648400 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:53.648304    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa6e6dc7-1049-4695-acd2-93c27b192c51-dshm\") pod \"e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9\" (UID: \"aa6e6dc7-1049-4695-acd2-93c27b192c51\") " pod="ai-tenant-e2e-mt-61774e12/e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9"
Jul 14 23:29:53.648400 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:53.648382    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa6e6dc7-1049-4695-acd2-93c27b192c51-tls-certs\") pod \"e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9\" (UID: \"aa6e6dc7-1049-4695-acd2-93c27b192c51\") " pod="ai-tenant-e2e-mt-61774e12/e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9"
Jul 14 23:29:53.648636 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:53.648435    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa6e6dc7-1049-4695-acd2-93c27b192c51-home\") pod \"e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9\" (UID: \"aa6e6dc7-1049-4695-acd2-93c27b192c51\") " pod="ai-tenant-e2e-mt-61774e12/e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9"
Jul 14 23:29:53.648636 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:53.648495    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa6e6dc7-1049-4695-acd2-93c27b192c51-model-cache\") pod \"e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9\" (UID: \"aa6e6dc7-1049-4695-acd2-93c27b192c51\") " pod="ai-tenant-e2e-mt-61774e12/e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9"
Jul 14 23:29:53.648636 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:53.648520    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa6e6dc7-1049-4695-acd2-93c27b192c51-tmp-dir\") pod \"e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9\" (UID: \"aa6e6dc7-1049-4695-acd2-93c27b192c51\") " pod="ai-tenant-e2e-mt-61774e12/e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9"
Jul 14 23:29:53.648636 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:53.648577    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkgdv\" (UniqueName: \"kubernetes.io/projected/aa6e6dc7-1049-4695-acd2-93c27b192c51-kube-api-access-bkgdv\") pod \"e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9\" (UID: \"aa6e6dc7-1049-4695-acd2-93c27b192c51\") " pod="ai-tenant-e2e-mt-61774e12/e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9"
Jul 14 23:29:53.749372 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:53.749316    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa6e6dc7-1049-4695-acd2-93c27b192c51-model-cache\") pod \"e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9\" (UID: \"aa6e6dc7-1049-4695-acd2-93c27b192c51\") " pod="ai-tenant-e2e-mt-61774e12/e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9"
Jul 14 23:29:53.749372 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:53.749364    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa6e6dc7-1049-4695-acd2-93c27b192c51-tmp-dir\") pod \"e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9\" (UID: \"aa6e6dc7-1049-4695-acd2-93c27b192c51\") " pod="ai-tenant-e2e-mt-61774e12/e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9"
Jul 14 23:29:53.749648 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:53.749489    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkgdv\" (UniqueName: \"kubernetes.io/projected/aa6e6dc7-1049-4695-acd2-93c27b192c51-kube-api-access-bkgdv\") pod \"e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9\" (UID: \"aa6e6dc7-1049-4695-acd2-93c27b192c51\") " pod="ai-tenant-e2e-mt-61774e12/e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9"
Jul 14 23:29:53.749648 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:53.749536    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa6e6dc7-1049-4695-acd2-93c27b192c51-dshm\") pod \"e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9\" (UID: \"aa6e6dc7-1049-4695-acd2-93c27b192c51\") " pod="ai-tenant-e2e-mt-61774e12/e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9"
Jul 14 23:29:53.749648 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:53.749582    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa6e6dc7-1049-4695-acd2-93c27b192c51-tls-certs\") pod \"e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9\" (UID: \"aa6e6dc7-1049-4695-acd2-93c27b192c51\") " pod="ai-tenant-e2e-mt-61774e12/e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9"
Jul 14 23:29:53.749648 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:53.749645    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa6e6dc7-1049-4695-acd2-93c27b192c51-home\") pod \"e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9\" (UID: \"aa6e6dc7-1049-4695-acd2-93c27b192c51\") " pod="ai-tenant-e2e-mt-61774e12/e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9"
Jul 14 23:29:53.749862 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:53.749713    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa6e6dc7-1049-4695-acd2-93c27b192c51-tmp-dir\") pod \"e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9\" (UID: \"aa6e6dc7-1049-4695-acd2-93c27b192c51\") " pod="ai-tenant-e2e-mt-61774e12/e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9"
Jul 14 23:29:53.749862 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:53.749780    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa6e6dc7-1049-4695-acd2-93c27b192c51-model-cache\") pod \"e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9\" (UID: \"aa6e6dc7-1049-4695-acd2-93c27b192c51\") " pod="ai-tenant-e2e-mt-61774e12/e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9"
Jul 14 23:29:53.750022 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:53.749991    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa6e6dc7-1049-4695-acd2-93c27b192c51-home\") pod \"e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9\" (UID: \"aa6e6dc7-1049-4695-acd2-93c27b192c51\") " pod="ai-tenant-e2e-mt-61774e12/e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9"
Jul 14 23:29:53.752002 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:53.751981    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa6e6dc7-1049-4695-acd2-93c27b192c51-dshm\") pod \"e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9\" (UID: \"aa6e6dc7-1049-4695-acd2-93c27b192c51\") " pod="ai-tenant-e2e-mt-61774e12/e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9"
Jul 14 23:29:53.752312 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:53.752272    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa6e6dc7-1049-4695-acd2-93c27b192c51-tls-certs\") pod \"e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9\" (UID: \"aa6e6dc7-1049-4695-acd2-93c27b192c51\") " pod="ai-tenant-e2e-mt-61774e12/e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9"
Jul 14 23:29:53.758745 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:53.758721    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkgdv\" (UniqueName: \"kubernetes.io/projected/aa6e6dc7-1049-4695-acd2-93c27b192c51-kube-api-access-bkgdv\") pod \"e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9\" (UID: \"aa6e6dc7-1049-4695-acd2-93c27b192c51\") " pod="ai-tenant-e2e-mt-61774e12/e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9"
Jul 14 23:29:53.900767 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:53.900661    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-61774e12/e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9"
Jul 14 23:29:53.934561 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:53.934480    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"
Jul 14 23:29:53.936574 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:53.936176    2563 patch_prober.go:28] interesting pod/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Jul 14 23:29:53.936574 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:53.936232    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6" podUID="1c8790f5-3eba-49c9-add9-7c451c117e00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Jul 14 23:29:54.082632 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:54.082604    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-61774e12/e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9"]
Jul 14 23:29:54.084200 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:29:54.084172    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa6e6dc7_1049_4695_acd2_93c27b192c51.slice/crio-5fd885d34d8c228bc299777d6df3778d4d53482d293d8324c54696b1dd54cebb WatchSource:0}: Error finding container 5fd885d34d8c228bc299777d6df3778d4d53482d293d8324c54696b1dd54cebb: Status 404 returned error can't find the container with id 5fd885d34d8c228bc299777d6df3778d4d53482d293d8324c54696b1dd54cebb
Jul 14 23:29:54.927232 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:54.927187    2563 patch_prober.go:28] interesting pod/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Jul 14 23:29:54.927458 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:54.927258    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6" podUID="1c8790f5-3eba-49c9-add9-7c451c117e00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Jul 14 23:29:54.955627 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:54.955590    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-61774e12/e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9" event={"ID":"aa6e6dc7-1049-4695-acd2-93c27b192c51","Type":"ContainerStarted","Data":"f85e096bc6860dfad2299c8d6a9af52ac59cecb4c8284d7b56863f742e34b2ed"}
Jul 14 23:29:54.955627 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:54.955638    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-61774e12/e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9" event={"ID":"aa6e6dc7-1049-4695-acd2-93c27b192c51","Type":"ContainerStarted","Data":"5fd885d34d8c228bc299777d6df3778d4d53482d293d8324c54696b1dd54cebb"}
Jul 14 23:29:55.927989 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:55.927940    2563 patch_prober.go:28] interesting pod/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Jul 14 23:29:55.928507 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:55.928010    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6" podUID="1c8790f5-3eba-49c9-add9-7c451c117e00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Jul 14 23:29:56.931213 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:56.931174    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"
Jul 14 23:29:56.931721 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:56.931589    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"
Jul 14 23:29:56.932208 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:56.932191    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"
Jul 14 23:29:56.969363 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:29:56.969313    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-61774e12/e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9" podStartSLOduration=3.969272175 podStartE2EDuration="3.969272175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:29:54.9734154 +0000 UTC m=+1982.276406192" watchObservedRunningTime="2026-07-14 23:29:56.969272175 +0000 UTC m=+1984.272262976"
Jul 14 23:30:03.901873 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:03.901816    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-61774e12/e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9"
Jul 14 23:30:03.901873 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:03.901882    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-61774e12/e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9"
Jul 14 23:30:03.914555 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:03.914519    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-61774e12/e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9"
Jul 14 23:30:04.006444 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:04.006408    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-61774e12/e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9"
Jul 14 23:30:25.982834 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:25.982798    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-61774e12-7df7cb89fc-c8pqg"]
Jul 14 23:30:25.983469 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:25.983022    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61774e12-7df7cb89fc-c8pqg" podUID="a2b29666-10d9-4fd2-9039-2cd2ea1354df" containerName="maas-api" containerID="cri-o://4777c589d42ca22efc5eb39b69ee150f2011a5ad863b2a1929542d680f0717e5" gracePeriod=30
Jul 14 23:30:26.246622 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:26.246595    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61774e12-7df7cb89fc-c8pqg"
Jul 14 23:30:26.339765 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:26.339725    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2b29666-10d9-4fd2-9039-2cd2ea1354df-maas-api-tls\") pod \"a2b29666-10d9-4fd2-9039-2cd2ea1354df\" (UID: \"a2b29666-10d9-4fd2-9039-2cd2ea1354df\") "
Jul 14 23:30:26.339941 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:26.339823    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hsgm\" (UniqueName: \"kubernetes.io/projected/a2b29666-10d9-4fd2-9039-2cd2ea1354df-kube-api-access-5hsgm\") pod \"a2b29666-10d9-4fd2-9039-2cd2ea1354df\" (UID: \"a2b29666-10d9-4fd2-9039-2cd2ea1354df\") "
Jul 14 23:30:26.339941 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:26.339851    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2b29666-10d9-4fd2-9039-2cd2ea1354df-openshift-service-ca\") pod \"a2b29666-10d9-4fd2-9039-2cd2ea1354df\" (UID: \"a2b29666-10d9-4fd2-9039-2cd2ea1354df\") "
Jul 14 23:30:26.340261 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:26.340229    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2b29666-10d9-4fd2-9039-2cd2ea1354df-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a2b29666-10d9-4fd2-9039-2cd2ea1354df" (UID: "a2b29666-10d9-4fd2-9039-2cd2ea1354df"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 23:30:26.341960 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:26.341939    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2b29666-10d9-4fd2-9039-2cd2ea1354df-kube-api-access-5hsgm" (OuterVolumeSpecName: "kube-api-access-5hsgm") pod "a2b29666-10d9-4fd2-9039-2cd2ea1354df" (UID: "a2b29666-10d9-4fd2-9039-2cd2ea1354df"). InnerVolumeSpecName "kube-api-access-5hsgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:30:26.342061 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:26.341968    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2b29666-10d9-4fd2-9039-2cd2ea1354df-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a2b29666-10d9-4fd2-9039-2cd2ea1354df" (UID: "a2b29666-10d9-4fd2-9039-2cd2ea1354df"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 23:30:26.440650 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:26.440609    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hsgm\" (UniqueName: \"kubernetes.io/projected/a2b29666-10d9-4fd2-9039-2cd2ea1354df-kube-api-access-5hsgm\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:30:26.440650 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:26.440642    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2b29666-10d9-4fd2-9039-2cd2ea1354df-openshift-service-ca\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:30:26.440650 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:26.440652    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2b29666-10d9-4fd2-9039-2cd2ea1354df-maas-api-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:30:27.102869 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:27.102833    2563 generic.go:356] "Generic (PLEG): container finished" podID="a2b29666-10d9-4fd2-9039-2cd2ea1354df" containerID="4777c589d42ca22efc5eb39b69ee150f2011a5ad863b2a1929542d680f0717e5" exitCode=0
Jul 14 23:30:27.103331 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:27.102881    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61774e12-7df7cb89fc-c8pqg"
Jul 14 23:30:27.103331 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:27.102904    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61774e12-7df7cb89fc-c8pqg" event={"ID":"a2b29666-10d9-4fd2-9039-2cd2ea1354df","Type":"ContainerDied","Data":"4777c589d42ca22efc5eb39b69ee150f2011a5ad863b2a1929542d680f0717e5"}
Jul 14 23:30:27.103331 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:27.102945    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61774e12-7df7cb89fc-c8pqg" event={"ID":"a2b29666-10d9-4fd2-9039-2cd2ea1354df","Type":"ContainerDied","Data":"28b03b293dfe340f35c08440c4ca2dec43973e1076372f7d78382f75188a12ab"}
Jul 14 23:30:27.103331 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:27.102967    2563 scope.go:117] "RemoveContainer" containerID="4777c589d42ca22efc5eb39b69ee150f2011a5ad863b2a1929542d680f0717e5"
Jul 14 23:30:27.112324 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:27.112299    2563 scope.go:117] "RemoveContainer" containerID="4777c589d42ca22efc5eb39b69ee150f2011a5ad863b2a1929542d680f0717e5"
Jul 14 23:30:27.112571 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:30:27.112552    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4777c589d42ca22efc5eb39b69ee150f2011a5ad863b2a1929542d680f0717e5\": container with ID starting with 4777c589d42ca22efc5eb39b69ee150f2011a5ad863b2a1929542d680f0717e5 not found: ID does not exist" containerID="4777c589d42ca22efc5eb39b69ee150f2011a5ad863b2a1929542d680f0717e5"
Jul 14 23:30:27.112622 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:27.112580    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4777c589d42ca22efc5eb39b69ee150f2011a5ad863b2a1929542d680f0717e5"} err="failed to get container status \"4777c589d42ca22efc5eb39b69ee150f2011a5ad863b2a1929542d680f0717e5\": rpc error: code = NotFound desc = could not find container \"4777c589d42ca22efc5eb39b69ee150f2011a5ad863b2a1929542d680f0717e5\": container with ID starting with 4777c589d42ca22efc5eb39b69ee150f2011a5ad863b2a1929542d680f0717e5 not found: ID does not exist"
Jul 14 23:30:27.125493 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:27.125464    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-61774e12-7df7cb89fc-c8pqg"]
Jul 14 23:30:27.127310 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:27.127275    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-61774e12-7df7cb89fc-c8pqg"]
Jul 14 23:30:27.229069 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:27.229036    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2b29666-10d9-4fd2-9039-2cd2ea1354df" path="/var/lib/kubelet/pods/a2b29666-10d9-4fd2-9039-2cd2ea1354df/volumes"
Jul 14 23:30:35.582530 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:35.582496    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-61774e12/e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9"]
Jul 14 23:30:50.603944 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:50.603879    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-61774e12/e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9" podUID="aa6e6dc7-1049-4695-acd2-93c27b192c51" containerName="main" containerID="cri-o://f85e096bc6860dfad2299c8d6a9af52ac59cecb4c8284d7b56863f742e34b2ed" gracePeriod=45
Jul 14 23:30:50.862857 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:50.862791    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-61774e12/e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9"
Jul 14 23:30:50.945522 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:50.945487    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkgdv\" (UniqueName: \"kubernetes.io/projected/aa6e6dc7-1049-4695-acd2-93c27b192c51-kube-api-access-bkgdv\") pod \"aa6e6dc7-1049-4695-acd2-93c27b192c51\" (UID: \"aa6e6dc7-1049-4695-acd2-93c27b192c51\") "
Jul 14 23:30:50.945721 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:50.945533    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa6e6dc7-1049-4695-acd2-93c27b192c51-dshm\") pod \"aa6e6dc7-1049-4695-acd2-93c27b192c51\" (UID: \"aa6e6dc7-1049-4695-acd2-93c27b192c51\") "
Jul 14 23:30:50.945721 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:50.945573    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa6e6dc7-1049-4695-acd2-93c27b192c51-tls-certs\") pod \"aa6e6dc7-1049-4695-acd2-93c27b192c51\" (UID: \"aa6e6dc7-1049-4695-acd2-93c27b192c51\") "
Jul 14 23:30:50.945721 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:50.945594    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa6e6dc7-1049-4695-acd2-93c27b192c51-home\") pod \"aa6e6dc7-1049-4695-acd2-93c27b192c51\" (UID: \"aa6e6dc7-1049-4695-acd2-93c27b192c51\") "
Jul 14 23:30:50.945721 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:50.945620    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa6e6dc7-1049-4695-acd2-93c27b192c51-tmp-dir\") pod \"aa6e6dc7-1049-4695-acd2-93c27b192c51\" (UID: \"aa6e6dc7-1049-4695-acd2-93c27b192c51\") "
Jul 14 23:30:50.945721 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:50.945692    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa6e6dc7-1049-4695-acd2-93c27b192c51-model-cache\") pod \"aa6e6dc7-1049-4695-acd2-93c27b192c51\" (UID: \"aa6e6dc7-1049-4695-acd2-93c27b192c51\") "
Jul 14 23:30:50.945985 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:50.945901    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa6e6dc7-1049-4695-acd2-93c27b192c51-home" (OuterVolumeSpecName: "home") pod "aa6e6dc7-1049-4695-acd2-93c27b192c51" (UID: "aa6e6dc7-1049-4695-acd2-93c27b192c51"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:30:50.945985 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:50.945953    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa6e6dc7-1049-4695-acd2-93c27b192c51-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aa6e6dc7-1049-4695-acd2-93c27b192c51" (UID: "aa6e6dc7-1049-4695-acd2-93c27b192c51"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:30:50.946101 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:50.945998    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa6e6dc7-1049-4695-acd2-93c27b192c51-model-cache" (OuterVolumeSpecName: "model-cache") pod "aa6e6dc7-1049-4695-acd2-93c27b192c51" (UID: "aa6e6dc7-1049-4695-acd2-93c27b192c51"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:30:50.948082 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:50.948050    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa6e6dc7-1049-4695-acd2-93c27b192c51-kube-api-access-bkgdv" (OuterVolumeSpecName: "kube-api-access-bkgdv") pod "aa6e6dc7-1049-4695-acd2-93c27b192c51" (UID: "aa6e6dc7-1049-4695-acd2-93c27b192c51"). InnerVolumeSpecName "kube-api-access-bkgdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:30:50.948082 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:50.948062    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa6e6dc7-1049-4695-acd2-93c27b192c51-dshm" (OuterVolumeSpecName: "dshm") pod "aa6e6dc7-1049-4695-acd2-93c27b192c51" (UID: "aa6e6dc7-1049-4695-acd2-93c27b192c51"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:30:50.948258 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:50.948117    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa6e6dc7-1049-4695-acd2-93c27b192c51-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aa6e6dc7-1049-4695-acd2-93c27b192c51" (UID: "aa6e6dc7-1049-4695-acd2-93c27b192c51"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 23:30:51.047244 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:51.047210    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa6e6dc7-1049-4695-acd2-93c27b192c51-dshm\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:30:51.047244 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:51.047243    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa6e6dc7-1049-4695-acd2-93c27b192c51-tls-certs\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:30:51.047468 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:51.047255    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa6e6dc7-1049-4695-acd2-93c27b192c51-home\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:30:51.047468 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:51.047266    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa6e6dc7-1049-4695-acd2-93c27b192c51-tmp-dir\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:30:51.047468 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:51.047277    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa6e6dc7-1049-4695-acd2-93c27b192c51-model-cache\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:30:51.047468 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:51.047314    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkgdv\" (UniqueName: \"kubernetes.io/projected/aa6e6dc7-1049-4695-acd2-93c27b192c51-kube-api-access-bkgdv\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:30:51.223302 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:51.223196    2563 generic.go:356] "Generic (PLEG): container finished" podID="aa6e6dc7-1049-4695-acd2-93c27b192c51" containerID="f85e096bc6860dfad2299c8d6a9af52ac59cecb4c8284d7b56863f742e34b2ed" exitCode=0
Jul 14 23:30:51.223302 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:51.223256    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-61774e12/e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9"
Jul 14 23:30:51.223302 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:51.223292    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-61774e12/e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9" event={"ID":"aa6e6dc7-1049-4695-acd2-93c27b192c51","Type":"ContainerDied","Data":"f85e096bc6860dfad2299c8d6a9af52ac59cecb4c8284d7b56863f742e34b2ed"}
Jul 14 23:30:51.223551 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:51.223381    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-61774e12/e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9" event={"ID":"aa6e6dc7-1049-4695-acd2-93c27b192c51","Type":"ContainerDied","Data":"5fd885d34d8c228bc299777d6df3778d4d53482d293d8324c54696b1dd54cebb"}
Jul 14 23:30:51.223551 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:51.223402    2563 scope.go:117] "RemoveContainer" containerID="f85e096bc6860dfad2299c8d6a9af52ac59cecb4c8284d7b56863f742e34b2ed"
Jul 14 23:30:51.232792 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:51.232777    2563 scope.go:117] "RemoveContainer" containerID="f85e096bc6860dfad2299c8d6a9af52ac59cecb4c8284d7b56863f742e34b2ed"
Jul 14 23:30:51.233070 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:30:51.233051    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f85e096bc6860dfad2299c8d6a9af52ac59cecb4c8284d7b56863f742e34b2ed\": container with ID starting with f85e096bc6860dfad2299c8d6a9af52ac59cecb4c8284d7b56863f742e34b2ed not found: ID does not exist" containerID="f85e096bc6860dfad2299c8d6a9af52ac59cecb4c8284d7b56863f742e34b2ed"
Jul 14 23:30:51.233119 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:51.233078    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f85e096bc6860dfad2299c8d6a9af52ac59cecb4c8284d7b56863f742e34b2ed"} err="failed to get container status \"f85e096bc6860dfad2299c8d6a9af52ac59cecb4c8284d7b56863f742e34b2ed\": rpc error: code = NotFound desc = could not find container \"f85e096bc6860dfad2299c8d6a9af52ac59cecb4c8284d7b56863f742e34b2ed\": container with ID starting with f85e096bc6860dfad2299c8d6a9af52ac59cecb4c8284d7b56863f742e34b2ed not found: ID does not exist"
Jul 14 23:30:51.247975 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:51.247947    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-61774e12/e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9"]
Jul 14 23:30:51.253697 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:51.253673    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-61774e12/e2e-lifecycle-model-61774e12-kserve-5ccdfd759-cjlj9"]
Jul 14 23:30:53.230324 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:30:53.230262    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa6e6dc7-1049-4695-acd2-93c27b192c51" path="/var/lib/kubelet/pods/aa6e6dc7-1049-4695-acd2-93c27b192c51/volumes"
Jul 14 23:31:22.945594 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:22.945382    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"]
Jul 14 23:31:22.946540 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:22.946503    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6" podUID="1c8790f5-3eba-49c9-add9-7c451c117e00" containerName="istio-proxy" containerID="cri-o://31add902371d6ab184cab6e7d7c575ad70bb0dd1af59e26136996074005d4217" gracePeriod=30
Jul 14 23:31:26.933879 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:26.933840    2563 patch_prober.go:28] interesting pod/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 14 23:31:26.934268 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:26.933915    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6" podUID="1c8790f5-3eba-49c9-add9-7c451c117e00" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 14 23:31:28.207994 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:28.207965    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"
Jul 14 23:31:28.263492 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:28.263452    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c8790f5-3eba-49c9-add9-7c451c117e00-istiod-ca-cert\") pod \"1c8790f5-3eba-49c9-add9-7c451c117e00\" (UID: \"1c8790f5-3eba-49c9-add9-7c451c117e00\") "
Jul 14 23:31:28.263492 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:28.263495    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c8790f5-3eba-49c9-add9-7c451c117e00-istio-podinfo\") pod \"1c8790f5-3eba-49c9-add9-7c451c117e00\" (UID: \"1c8790f5-3eba-49c9-add9-7c451c117e00\") "
Jul 14 23:31:28.263749 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:28.263519    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c8790f5-3eba-49c9-add9-7c451c117e00-istio-token\") pod \"1c8790f5-3eba-49c9-add9-7c451c117e00\" (UID: \"1c8790f5-3eba-49c9-add9-7c451c117e00\") "
Jul 14 23:31:28.263749 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:28.263545    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c8790f5-3eba-49c9-add9-7c451c117e00-istio-envoy\") pod \"1c8790f5-3eba-49c9-add9-7c451c117e00\" (UID: \"1c8790f5-3eba-49c9-add9-7c451c117e00\") "
Jul 14 23:31:28.263749 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:28.263578    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8790f5-3eba-49c9-add9-7c451c117e00-workload-socket\") pod \"1c8790f5-3eba-49c9-add9-7c451c117e00\" (UID: \"1c8790f5-3eba-49c9-add9-7c451c117e00\") "
Jul 14 23:31:28.263997 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:28.263885    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c8790f5-3eba-49c9-add9-7c451c117e00-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1c8790f5-3eba-49c9-add9-7c451c117e00" (UID: "1c8790f5-3eba-49c9-add9-7c451c117e00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 23:31:28.264116 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:28.263999    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c8790f5-3eba-49c9-add9-7c451c117e00-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1c8790f5-3eba-49c9-add9-7c451c117e00" (UID: "1c8790f5-3eba-49c9-add9-7c451c117e00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:31:28.280470 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:28.280403    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c8790f5-3eba-49c9-add9-7c451c117e00-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1c8790f5-3eba-49c9-add9-7c451c117e00" (UID: "1c8790f5-3eba-49c9-add9-7c451c117e00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:31:28.285493 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:28.285444    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c8790f5-3eba-49c9-add9-7c451c117e00-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1c8790f5-3eba-49c9-add9-7c451c117e00" (UID: "1c8790f5-3eba-49c9-add9-7c451c117e00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 23:31:28.285632 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:28.285540    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c8790f5-3eba-49c9-add9-7c451c117e00-istio-token" (OuterVolumeSpecName: "istio-token") pod "1c8790f5-3eba-49c9-add9-7c451c117e00" (UID: "1c8790f5-3eba-49c9-add9-7c451c117e00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:31:28.364209 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:28.364173    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c8790f5-3eba-49c9-add9-7c451c117e00-istio-data\") pod \"1c8790f5-3eba-49c9-add9-7c451c117e00\" (UID: \"1c8790f5-3eba-49c9-add9-7c451c117e00\") "
Jul 14 23:31:28.364209 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:28.364209    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2frr\" (UniqueName: \"kubernetes.io/projected/1c8790f5-3eba-49c9-add9-7c451c117e00-kube-api-access-x2frr\") pod \"1c8790f5-3eba-49c9-add9-7c451c117e00\" (UID: \"1c8790f5-3eba-49c9-add9-7c451c117e00\") "
Jul 14 23:31:28.364468 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:28.364235    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c8790f5-3eba-49c9-add9-7c451c117e00-workload-certs\") pod \"1c8790f5-3eba-49c9-add9-7c451c117e00\" (UID: \"1c8790f5-3eba-49c9-add9-7c451c117e00\") "
Jul 14 23:31:28.364468 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:28.364260    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8790f5-3eba-49c9-add9-7c451c117e00-credential-socket\") pod \"1c8790f5-3eba-49c9-add9-7c451c117e00\" (UID: \"1c8790f5-3eba-49c9-add9-7c451c117e00\") "
Jul 14 23:31:28.364587 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:28.364471    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c8790f5-3eba-49c9-add9-7c451c117e00-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1c8790f5-3eba-49c9-add9-7c451c117e00" (UID: "1c8790f5-3eba-49c9-add9-7c451c117e00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:31:28.364587 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:28.364486    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c8790f5-3eba-49c9-add9-7c451c117e00-istiod-ca-cert\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:31:28.364587 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:28.364502    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c8790f5-3eba-49c9-add9-7c451c117e00-istio-podinfo\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:31:28.364587 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:28.364515    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c8790f5-3eba-49c9-add9-7c451c117e00-istio-token\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:31:28.364587 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:28.364526    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c8790f5-3eba-49c9-add9-7c451c117e00-istio-envoy\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:31:28.364587 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:28.364538    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8790f5-3eba-49c9-add9-7c451c117e00-workload-socket\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:31:28.364587 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:28.364572    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c8790f5-3eba-49c9-add9-7c451c117e00-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1c8790f5-3eba-49c9-add9-7c451c117e00" (UID: "1c8790f5-3eba-49c9-add9-7c451c117e00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:31:28.365106 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:28.365076    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c8790f5-3eba-49c9-add9-7c451c117e00-istio-data" (OuterVolumeSpecName: "istio-data") pod "1c8790f5-3eba-49c9-add9-7c451c117e00" (UID: "1c8790f5-3eba-49c9-add9-7c451c117e00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:31:28.367091 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:28.367069    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c8790f5-3eba-49c9-add9-7c451c117e00-kube-api-access-x2frr" (OuterVolumeSpecName: "kube-api-access-x2frr") pod "1c8790f5-3eba-49c9-add9-7c451c117e00" (UID: "1c8790f5-3eba-49c9-add9-7c451c117e00"). InnerVolumeSpecName "kube-api-access-x2frr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:31:28.390967 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:28.390922    2563 generic.go:356] "Generic (PLEG): container finished" podID="1c8790f5-3eba-49c9-add9-7c451c117e00" containerID="31add902371d6ab184cab6e7d7c575ad70bb0dd1af59e26136996074005d4217" exitCode=0
Jul 14 23:31:28.391166 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:28.391021    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"
Jul 14 23:31:28.391166 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:28.391060    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6" event={"ID":"1c8790f5-3eba-49c9-add9-7c451c117e00","Type":"ContainerDied","Data":"31add902371d6ab184cab6e7d7c575ad70bb0dd1af59e26136996074005d4217"}
Jul 14 23:31:28.391166 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:28.391127    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6" event={"ID":"1c8790f5-3eba-49c9-add9-7c451c117e00","Type":"ContainerDied","Data":"ba18e02ffb11317e590cb27cc15c29519704f7a2de5ea17c7ac3103315acbd14"}
Jul 14 23:31:28.391166 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:28.391158    2563 scope.go:117] "RemoveContainer" containerID="31add902371d6ab184cab6e7d7c575ad70bb0dd1af59e26136996074005d4217"
Jul 14 23:31:28.400904 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:28.400887    2563 scope.go:117] "RemoveContainer" containerID="9130d572be1c77ef952973d09f33ec52d72fb3db59d8dc4be381dc6d8602da29"
Jul 14 23:31:28.408964 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:28.408946    2563 scope.go:117] "RemoveContainer" containerID="31add902371d6ab184cab6e7d7c575ad70bb0dd1af59e26136996074005d4217"
Jul 14 23:31:28.409219 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:31:28.409201    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31add902371d6ab184cab6e7d7c575ad70bb0dd1af59e26136996074005d4217\": container with ID starting with 31add902371d6ab184cab6e7d7c575ad70bb0dd1af59e26136996074005d4217 not found: ID does not exist" containerID="31add902371d6ab184cab6e7d7c575ad70bb0dd1af59e26136996074005d4217"
Jul 14 23:31:28.409329 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:28.409226    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31add902371d6ab184cab6e7d7c575ad70bb0dd1af59e26136996074005d4217"} err="failed to get container status \"31add902371d6ab184cab6e7d7c575ad70bb0dd1af59e26136996074005d4217\": rpc error: code = NotFound desc = could not find container \"31add902371d6ab184cab6e7d7c575ad70bb0dd1af59e26136996074005d4217\": container with ID starting with 31add902371d6ab184cab6e7d7c575ad70bb0dd1af59e26136996074005d4217 not found: ID does not exist"
Jul 14 23:31:28.409329 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:28.409241    2563 scope.go:117] "RemoveContainer" containerID="9130d572be1c77ef952973d09f33ec52d72fb3db59d8dc4be381dc6d8602da29"
Jul 14 23:31:28.409463 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:31:28.409445    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9130d572be1c77ef952973d09f33ec52d72fb3db59d8dc4be381dc6d8602da29\": container with ID starting with 9130d572be1c77ef952973d09f33ec52d72fb3db59d8dc4be381dc6d8602da29 not found: ID does not exist" containerID="9130d572be1c77ef952973d09f33ec52d72fb3db59d8dc4be381dc6d8602da29"
Jul 14 23:31:28.409518 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:28.409467    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9130d572be1c77ef952973d09f33ec52d72fb3db59d8dc4be381dc6d8602da29"} err="failed to get container status \"9130d572be1c77ef952973d09f33ec52d72fb3db59d8dc4be381dc6d8602da29\": rpc error: code = NotFound desc = could not find container \"9130d572be1c77ef952973d09f33ec52d72fb3db59d8dc4be381dc6d8602da29\": container with ID starting with 9130d572be1c77ef952973d09f33ec52d72fb3db59d8dc4be381dc6d8602da29 not found: ID does not exist"
Jul 14 23:31:28.415490 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:28.415458    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"]
Jul 14 23:31:28.421791 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:28.421759    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-61774e12-openshift-default-6f655f76bc-lx8g6"]
Jul 14 23:31:28.465303 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:28.465245    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c8790f5-3eba-49c9-add9-7c451c117e00-istio-data\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:31:28.465303 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:28.465302    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2frr\" (UniqueName: \"kubernetes.io/projected/1c8790f5-3eba-49c9-add9-7c451c117e00-kube-api-access-x2frr\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:31:28.465564 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:28.465319    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c8790f5-3eba-49c9-add9-7c451c117e00-workload-certs\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:31:28.465564 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:28.465334    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c8790f5-3eba-49c9-add9-7c451c117e00-credential-socket\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:31:29.234520 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:31:29.234421    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c8790f5-3eba-49c9-add9-7c451c117e00" path="/var/lib/kubelet/pods/1c8790f5-3eba-49c9-add9-7c451c117e00/volumes"
Jul 14 23:32:18.113770 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.113730    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"]
Jul 14 23:32:18.114377 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.114253    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c8790f5-3eba-49c9-add9-7c451c117e00" containerName="istio-proxy"
Jul 14 23:32:18.114377 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.114273    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c8790f5-3eba-49c9-add9-7c451c117e00" containerName="istio-proxy"
Jul 14 23:32:18.114377 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.114317    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2b29666-10d9-4fd2-9039-2cd2ea1354df" containerName="maas-api"
Jul 14 23:32:18.114377 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.114326    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2b29666-10d9-4fd2-9039-2cd2ea1354df" containerName="maas-api"
Jul 14 23:32:18.114377 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.114348    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c8790f5-3eba-49c9-add9-7c451c117e00" containerName="istio-proxy"
Jul 14 23:32:18.114377 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.114357    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c8790f5-3eba-49c9-add9-7c451c117e00" containerName="istio-proxy"
Jul 14 23:32:18.114377 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.114368    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa6e6dc7-1049-4695-acd2-93c27b192c51" containerName="main"
Jul 14 23:32:18.114377 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.114375    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa6e6dc7-1049-4695-acd2-93c27b192c51" containerName="main"
Jul 14 23:32:18.114788 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.114459    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c8790f5-3eba-49c9-add9-7c451c117e00" containerName="istio-proxy"
Jul 14 23:32:18.114788 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.114473    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c8790f5-3eba-49c9-add9-7c451c117e00" containerName="istio-proxy"
Jul 14 23:32:18.114788 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.114485    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2b29666-10d9-4fd2-9039-2cd2ea1354df" containerName="maas-api"
Jul 14 23:32:18.114788 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.114495    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa6e6dc7-1049-4695-acd2-93c27b192c51" containerName="main"
Jul 14 23:32:18.117534 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.117514    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"
Jul 14 23:32:18.120792 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.120772    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 14 23:32:18.120885 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.120795    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d130d1-openshift-default-dockercfg-42b88\""
Jul 14 23:32:18.129172 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.129147    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"]
Jul 14 23:32:18.186179 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.186150    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-istio-data\") pod \"e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm\" (UID: \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\") " pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"
Jul 14 23:32:18.186368 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.186186    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-workload-socket\") pod \"e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm\" (UID: \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\") " pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"
Jul 14 23:32:18.186368 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.186207    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-workload-certs\") pod \"e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm\" (UID: \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\") " pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"
Jul 14 23:32:18.186368 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.186223    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-istio-podinfo\") pod \"e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm\" (UID: \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\") " pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"
Jul 14 23:32:18.186368 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.186276    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-istio-token\") pod \"e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm\" (UID: \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\") " pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"
Jul 14 23:32:18.186368 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.186341    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-istio-envoy\") pod \"e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm\" (UID: \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\") " pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"
Jul 14 23:32:18.186562 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.186387    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-credential-socket\") pod \"e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm\" (UID: \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\") " pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"
Jul 14 23:32:18.186562 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.186410    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mcjj\" (UniqueName: \"kubernetes.io/projected/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-kube-api-access-5mcjj\") pod \"e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm\" (UID: \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\") " pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"
Jul 14 23:32:18.186562 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.186464    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-istiod-ca-cert\") pod \"e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm\" (UID: \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\") " pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"
Jul 14 23:32:18.286835 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.286799    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-istiod-ca-cert\") pod \"e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm\" (UID: \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\") " pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"
Jul 14 23:32:18.287010 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.286840    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-istio-data\") pod \"e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm\" (UID: \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\") " pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"
Jul 14 23:32:18.287010 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.286972    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-workload-socket\") pod \"e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm\" (UID: \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\") " pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"
Jul 14 23:32:18.287138 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.287018    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-workload-certs\") pod \"e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm\" (UID: \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\") " pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"
Jul 14 23:32:18.287138 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.287047    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-istio-podinfo\") pod \"e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm\" (UID: \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\") " pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"
Jul 14 23:32:18.287138 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.287078    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-istio-token\") pod \"e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm\" (UID: \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\") " pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"
Jul 14 23:32:18.287138 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.287112    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-istio-envoy\") pod \"e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm\" (UID: \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\") " pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"
Jul 14 23:32:18.287362 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.287160    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-istio-data\") pod \"e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm\" (UID: \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\") " pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"
Jul 14 23:32:18.287362 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.287205    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-credential-socket\") pod \"e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm\" (UID: \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\") " pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"
Jul 14 23:32:18.287362 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.287229    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mcjj\" (UniqueName: \"kubernetes.io/projected/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-kube-api-access-5mcjj\") pod \"e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm\" (UID: \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\") " pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"
Jul 14 23:32:18.287513 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.287383    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-workload-socket\") pod \"e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm\" (UID: \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\") " pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"
Jul 14 23:32:18.287704 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.287684    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-workload-certs\") pod \"e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm\" (UID: \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\") " pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"
Jul 14 23:32:18.287782 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.287709    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-credential-socket\") pod \"e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm\" (UID: \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\") " pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"
Jul 14 23:32:18.288091 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.288069    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-istiod-ca-cert\") pod \"e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm\" (UID: \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\") " pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"
Jul 14 23:32:18.289648 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.289625    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-istio-envoy\") pod \"e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm\" (UID: \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\") " pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"
Jul 14 23:32:18.289862 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.289841    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-istio-podinfo\") pod \"e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm\" (UID: \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\") " pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"
Jul 14 23:32:18.295793 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.295766    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-istio-token\") pod \"e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm\" (UID: \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\") " pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"
Jul 14 23:32:18.295935 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.295917    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mcjj\" (UniqueName: \"kubernetes.io/projected/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-kube-api-access-5mcjj\") pod \"e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm\" (UID: \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\") " pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"
Jul 14 23:32:18.427811 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.427714    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"
Jul 14 23:32:18.568526 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.568498    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"]
Jul 14 23:32:18.570940 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:32:18.570912    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8fd0204_0adb_46d5_b4f5_bf15ca9bbc5d.slice/crio-bbb17930b6b26e8250b33db771197f358ab4651692aa14c65d4d1d496efa8813 WatchSource:0}: Error finding container bbb17930b6b26e8250b33db771197f358ab4651692aa14c65d4d1d496efa8813: Status 404 returned error can't find the container with id bbb17930b6b26e8250b33db771197f358ab4651692aa14c65d4d1d496efa8813
Jul 14 23:32:18.572876 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.572839    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:32:18.572970 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.572907    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:32:18.572970 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.572934    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:32:18.616491 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:18.616460    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm" event={"ID":"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d","Type":"ContainerStarted","Data":"bbb17930b6b26e8250b33db771197f358ab4651692aa14c65d4d1d496efa8813"}
Jul 14 23:32:19.622615 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:19.622578    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm" event={"ID":"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d","Type":"ContainerStarted","Data":"160896e35e9513decb257d360cf37df266acb0140095012857b36379c962ed93"}
Jul 14 23:32:19.648033 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:19.647983    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm" podStartSLOduration=1.647967634 podStartE2EDuration="1.647967634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:32:19.645469549 +0000 UTC m=+2126.948460342" watchObservedRunningTime="2026-07-14 23:32:19.647967634 +0000 UTC m=+2126.950958426"
Jul 14 23:32:20.428471 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:20.428433    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"
Jul 14 23:32:20.432909 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:20.432885    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"
Jul 14 23:32:20.627643 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:20.627610    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"
Jul 14 23:32:20.628654 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:20.628635    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"
Jul 14 23:32:21.687944 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:21.687912    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d130d1-77749fddc7-7mzd4"]
Jul 14 23:32:21.691194 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:21.691177    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d130d1-77749fddc7-7mzd4"
Jul 14 23:32:21.695414 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:21.695389    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-d130d1-serving-cert\""
Jul 14 23:32:21.695545 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:21.695416    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 23:32:21.696729 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:21.696706    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7jvtc\""
Jul 14 23:32:21.696820 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:21.696748    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 23:32:21.705518 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:21.705491    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d130d1-77749fddc7-7mzd4"]
Jul 14 23:32:21.713374 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:21.713352    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea4261ea-5839-4c00-bc5b-6e33d605ffff-maas-api-tls\") pod \"maas-api-e2e-inf-a-d130d1-77749fddc7-7mzd4\" (UID: \"ea4261ea-5839-4c00-bc5b-6e33d605ffff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d130d1-77749fddc7-7mzd4"
Jul 14 23:32:21.713479 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:21.713440    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9xm6\" (UniqueName: \"kubernetes.io/projected/ea4261ea-5839-4c00-bc5b-6e33d605ffff-kube-api-access-x9xm6\") pod \"maas-api-e2e-inf-a-d130d1-77749fddc7-7mzd4\" (UID: \"ea4261ea-5839-4c00-bc5b-6e33d605ffff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d130d1-77749fddc7-7mzd4"
Jul 14 23:32:21.713544 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:21.713487    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea4261ea-5839-4c00-bc5b-6e33d605ffff-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d130d1-77749fddc7-7mzd4\" (UID: \"ea4261ea-5839-4c00-bc5b-6e33d605ffff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d130d1-77749fddc7-7mzd4"
Jul 14 23:32:21.813975 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:21.813937    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9xm6\" (UniqueName: \"kubernetes.io/projected/ea4261ea-5839-4c00-bc5b-6e33d605ffff-kube-api-access-x9xm6\") pod \"maas-api-e2e-inf-a-d130d1-77749fddc7-7mzd4\" (UID: \"ea4261ea-5839-4c00-bc5b-6e33d605ffff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d130d1-77749fddc7-7mzd4"
Jul 14 23:32:21.814185 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:21.813985    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea4261ea-5839-4c00-bc5b-6e33d605ffff-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d130d1-77749fddc7-7mzd4\" (UID: \"ea4261ea-5839-4c00-bc5b-6e33d605ffff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d130d1-77749fddc7-7mzd4"
Jul 14 23:32:21.814185 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:21.814039    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea4261ea-5839-4c00-bc5b-6e33d605ffff-maas-api-tls\") pod \"maas-api-e2e-inf-a-d130d1-77749fddc7-7mzd4\" (UID: \"ea4261ea-5839-4c00-bc5b-6e33d605ffff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d130d1-77749fddc7-7mzd4"
Jul 14 23:32:21.814185 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:32:21.814133    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-d130d1-serving-cert: secret "maas-api-e2e-inf-a-d130d1-serving-cert" not found
Jul 14 23:32:21.814185 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:32:21.814186    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea4261ea-5839-4c00-bc5b-6e33d605ffff-maas-api-tls podName:ea4261ea-5839-4c00-bc5b-6e33d605ffff nodeName:}" failed. No retries permitted until 2026-07-14 23:32:22.314168626 +0000 UTC m=+2129.617159395 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ea4261ea-5839-4c00-bc5b-6e33d605ffff-maas-api-tls") pod "maas-api-e2e-inf-a-d130d1-77749fddc7-7mzd4" (UID: "ea4261ea-5839-4c00-bc5b-6e33d605ffff") : secret "maas-api-e2e-inf-a-d130d1-serving-cert" not found
Jul 14 23:32:21.814853 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:21.814829    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea4261ea-5839-4c00-bc5b-6e33d605ffff-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d130d1-77749fddc7-7mzd4\" (UID: \"ea4261ea-5839-4c00-bc5b-6e33d605ffff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d130d1-77749fddc7-7mzd4"
Jul 14 23:32:21.823746 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:21.823712    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9xm6\" (UniqueName: \"kubernetes.io/projected/ea4261ea-5839-4c00-bc5b-6e33d605ffff-kube-api-access-x9xm6\") pod \"maas-api-e2e-inf-a-d130d1-77749fddc7-7mzd4\" (UID: \"ea4261ea-5839-4c00-bc5b-6e33d605ffff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d130d1-77749fddc7-7mzd4"
Jul 14 23:32:22.319602 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:22.319561    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea4261ea-5839-4c00-bc5b-6e33d605ffff-maas-api-tls\") pod \"maas-api-e2e-inf-a-d130d1-77749fddc7-7mzd4\" (UID: \"ea4261ea-5839-4c00-bc5b-6e33d605ffff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d130d1-77749fddc7-7mzd4"
Jul 14 23:32:22.322062 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:22.322035    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea4261ea-5839-4c00-bc5b-6e33d605ffff-maas-api-tls\") pod \"maas-api-e2e-inf-a-d130d1-77749fddc7-7mzd4\" (UID: \"ea4261ea-5839-4c00-bc5b-6e33d605ffff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d130d1-77749fddc7-7mzd4"
Jul 14 23:32:22.601822 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:22.601727    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d130d1-77749fddc7-7mzd4"
Jul 14 23:32:22.743572 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:22.743544    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d130d1-77749fddc7-7mzd4"]
Jul 14 23:32:22.745687 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:32:22.745656    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea4261ea_5839_4c00_bc5b_6e33d605ffff.slice/crio-12df5b1455a4b3d33747e1a10721aa2a8706d3262e341c8bf3c0cd8a1e7e6c90 WatchSource:0}: Error finding container 12df5b1455a4b3d33747e1a10721aa2a8706d3262e341c8bf3c0cd8a1e7e6c90: Status 404 returned error can't find the container with id 12df5b1455a4b3d33747e1a10721aa2a8706d3262e341c8bf3c0cd8a1e7e6c90
Jul 14 23:32:23.642329 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:23.642227    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d130d1-77749fddc7-7mzd4" event={"ID":"ea4261ea-5839-4c00-bc5b-6e33d605ffff","Type":"ContainerStarted","Data":"6428fd699673eb5f82dbc7420a0013ccd22d3f58654103ddfbeb8bb842515eca"}
Jul 14 23:32:23.642329 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:23.642294    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d130d1-77749fddc7-7mzd4" event={"ID":"ea4261ea-5839-4c00-bc5b-6e33d605ffff","Type":"ContainerStarted","Data":"12df5b1455a4b3d33747e1a10721aa2a8706d3262e341c8bf3c0cd8a1e7e6c90"}
Jul 14 23:32:23.642598 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:23.642391    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d130d1-77749fddc7-7mzd4"
Jul 14 23:32:23.662371 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:23.662319    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d130d1-77749fddc7-7mzd4" podStartSLOduration=2.157226233 podStartE2EDuration="2.662305295s" totalImagesPullingTime="505.079062ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:32:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:32:23.660334329 +0000 UTC m=+2130.963325122" watchObservedRunningTime="2026-07-14 23:32:23.662305295 +0000 UTC m=+2130.965296081"
Jul 14 23:32:26.733166 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:26.733132    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"]
Jul 14 23:32:26.736953 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:26.736935    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"
Jul 14 23:32:26.741479 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:26.741457    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-4b47a7-openshift-default-dockercfg-s7pr4\""
Jul 14 23:32:26.746368 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:26.746336    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-istio-data\") pod \"e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg\" (UID: \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\") " pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"
Jul 14 23:32:26.746458 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:26.746426    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-istio-envoy\") pod \"e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg\" (UID: \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\") " pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"
Jul 14 23:32:26.746515 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:26.746458    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmghk\" (UniqueName: \"kubernetes.io/projected/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-kube-api-access-zmghk\") pod \"e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg\" (UID: \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\") " pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"
Jul 14 23:32:26.746567 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:26.746497    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-workload-socket\") pod \"e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg\" (UID: \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\") " pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"
Jul 14 23:32:26.746567 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:26.746536    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-credential-socket\") pod \"e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg\" (UID: \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\") " pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"
Jul 14 23:32:26.746687 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:26.746570    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-workload-certs\") pod \"e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg\" (UID: \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\") " pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"
Jul 14 23:32:26.746687 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:26.746609    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-istiod-ca-cert\") pod \"e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg\" (UID: \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\") " pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"
Jul 14 23:32:26.746687 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:26.746675    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-istio-podinfo\") pod \"e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg\" (UID: \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\") " pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"
Jul 14 23:32:26.746839 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:26.746691    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-istio-token\") pod \"e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg\" (UID: \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\") " pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"
Jul 14 23:32:26.755490 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:26.755465    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"]
Jul 14 23:32:26.847733 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:26.847690    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-istio-envoy\") pod \"e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg\" (UID: \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\") " pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"
Jul 14 23:32:26.847924 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:26.847751    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmghk\" (UniqueName: \"kubernetes.io/projected/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-kube-api-access-zmghk\") pod \"e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg\" (UID: \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\") " pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"
Jul 14 23:32:26.847924 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:26.847860    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-workload-socket\") pod \"e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg\" (UID: \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\") " pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"
Jul 14 23:32:26.847924 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:26.847898    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-credential-socket\") pod \"e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg\" (UID: \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\") " pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"
Jul 14 23:32:26.848106 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:26.847921    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-workload-certs\") pod \"e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg\" (UID: \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\") " pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"
Jul 14 23:32:26.848106 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:26.848010    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-istiod-ca-cert\") pod \"e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg\" (UID: \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\") " pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"
Jul 14 23:32:26.848106 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:26.848054    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-istio-podinfo\") pod \"e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg\" (UID: \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\") " pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"
Jul 14 23:32:26.848106 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:26.848079    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-istio-token\") pod \"e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg\" (UID: \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\") " pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"
Jul 14 23:32:26.848334 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:26.848109    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-istio-data\") pod \"e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg\" (UID: \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\") " pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"
Jul 14 23:32:26.848334 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:26.848265    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-workload-socket\") pod \"e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg\" (UID: \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\") " pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"
Jul 14 23:32:26.848442 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:26.848425    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-credential-socket\") pod \"e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg\" (UID: \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\") " pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"
Jul 14 23:32:26.848659 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:26.848628    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-workload-certs\") pod \"e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg\" (UID: \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\") " pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"
Jul 14 23:32:26.848773 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:26.848684    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-istio-data\") pod \"e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg\" (UID: \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\") " pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"
Jul 14 23:32:26.849247 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:26.849220    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-istiod-ca-cert\") pod \"e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg\" (UID: \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\") " pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"
Jul 14 23:32:26.850458 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:26.850435    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-istio-envoy\") pod \"e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg\" (UID: \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\") " pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"
Jul 14 23:32:26.851345 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:26.851320    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-istio-podinfo\") pod \"e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg\" (UID: \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\") " pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"
Jul 14 23:32:26.856629 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:26.856598    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmghk\" (UniqueName: \"kubernetes.io/projected/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-kube-api-access-zmghk\") pod \"e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg\" (UID: \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\") " pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"
Jul 14 23:32:26.856913 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:26.856896    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-istio-token\") pod \"e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg\" (UID: \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\") " pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"
Jul 14 23:32:27.047813 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:27.047775    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"
Jul 14 23:32:27.184711 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:27.184684    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"]
Jul 14 23:32:27.186514 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:32:27.186483    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aa9c0e7_abd5_4b5e_bf10_49725d83d9c1.slice/crio-dc54646b3b1e6a8190320965cfc74c6f3607ea741912bff425306a77677ddacd WatchSource:0}: Error finding container dc54646b3b1e6a8190320965cfc74c6f3607ea741912bff425306a77677ddacd: Status 404 returned error can't find the container with id dc54646b3b1e6a8190320965cfc74c6f3607ea741912bff425306a77677ddacd
Jul 14 23:32:27.188646 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:27.188616    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:32:27.188747 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:27.188695    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:32:27.188747 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:27.188738    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:32:27.661687 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:27.661644    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg" event={"ID":"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1","Type":"ContainerStarted","Data":"37a7d1a4eadff759564908cef547ad07b4fa154de8e6c4da30e051bbce2a687d"}
Jul 14 23:32:27.661891 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:27.661703    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg" event={"ID":"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1","Type":"ContainerStarted","Data":"dc54646b3b1e6a8190320965cfc74c6f3607ea741912bff425306a77677ddacd"}
Jul 14 23:32:27.683612 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:27.683562    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg" podStartSLOduration=1.683546964 podStartE2EDuration="1.683546964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:32:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:32:27.681599174 +0000 UTC m=+2134.984589966" watchObservedRunningTime="2026-07-14 23:32:27.683546964 +0000 UTC m=+2134.986537756"
Jul 14 23:32:28.048836 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:28.048795    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"
Jul 14 23:32:28.054167 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:28.054142    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"
Jul 14 23:32:28.666154 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:28.666119    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"
Jul 14 23:32:28.666981 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:28.666961    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"
Jul 14 23:32:29.652312 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:29.652253    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d130d1-77749fddc7-7mzd4"
Jul 14 23:32:29.704166 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:29.704133    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4b47a7-84b6b8c5f8-qmpbh"]
Jul 14 23:32:29.707496 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:29.707479    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b47a7-84b6b8c5f8-qmpbh"
Jul 14 23:32:29.710488 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:29.710467    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-4b47a7-serving-cert\""
Jul 14 23:32:29.719617 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:29.719592    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4b47a7-84b6b8c5f8-qmpbh"]
Jul 14 23:32:29.772768 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:29.772717    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c57xw\" (UniqueName: \"kubernetes.io/projected/0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d-kube-api-access-c57xw\") pod \"maas-api-e2e-inf-b-4b47a7-84b6b8c5f8-qmpbh\" (UID: \"0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b47a7-84b6b8c5f8-qmpbh"
Jul 14 23:32:29.772768 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:29.772777    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4b47a7-84b6b8c5f8-qmpbh\" (UID: \"0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b47a7-84b6b8c5f8-qmpbh"
Jul 14 23:32:29.773042 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:29.772850    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d-maas-api-tls\") pod \"maas-api-e2e-inf-b-4b47a7-84b6b8c5f8-qmpbh\" (UID: \"0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b47a7-84b6b8c5f8-qmpbh"
Jul 14 23:32:29.873564 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:29.873527    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c57xw\" (UniqueName: \"kubernetes.io/projected/0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d-kube-api-access-c57xw\") pod \"maas-api-e2e-inf-b-4b47a7-84b6b8c5f8-qmpbh\" (UID: \"0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b47a7-84b6b8c5f8-qmpbh"
Jul 14 23:32:29.873747 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:29.873573    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4b47a7-84b6b8c5f8-qmpbh\" (UID: \"0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b47a7-84b6b8c5f8-qmpbh"
Jul 14 23:32:29.873747 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:29.873611    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d-maas-api-tls\") pod \"maas-api-e2e-inf-b-4b47a7-84b6b8c5f8-qmpbh\" (UID: \"0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b47a7-84b6b8c5f8-qmpbh"
Jul 14 23:32:29.873747 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:32:29.873743    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-4b47a7-serving-cert: secret "maas-api-e2e-inf-b-4b47a7-serving-cert" not found
Jul 14 23:32:29.873934 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:32:29.873805    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d-maas-api-tls podName:0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d nodeName:}" failed. No retries permitted until 2026-07-14 23:32:30.373783458 +0000 UTC m=+2137.676774228 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d-maas-api-tls") pod "maas-api-e2e-inf-b-4b47a7-84b6b8c5f8-qmpbh" (UID: "0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d") : secret "maas-api-e2e-inf-b-4b47a7-serving-cert" not found
Jul 14 23:32:29.874303 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:29.874262    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4b47a7-84b6b8c5f8-qmpbh\" (UID: \"0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b47a7-84b6b8c5f8-qmpbh"
Jul 14 23:32:29.882874 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:29.882847    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c57xw\" (UniqueName: \"kubernetes.io/projected/0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d-kube-api-access-c57xw\") pod \"maas-api-e2e-inf-b-4b47a7-84b6b8c5f8-qmpbh\" (UID: \"0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b47a7-84b6b8c5f8-qmpbh"
Jul 14 23:32:30.377913 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:30.377873    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d-maas-api-tls\") pod \"maas-api-e2e-inf-b-4b47a7-84b6b8c5f8-qmpbh\" (UID: \"0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b47a7-84b6b8c5f8-qmpbh"
Jul 14 23:32:30.380654 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:30.380621    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d-maas-api-tls\") pod \"maas-api-e2e-inf-b-4b47a7-84b6b8c5f8-qmpbh\" (UID: \"0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b47a7-84b6b8c5f8-qmpbh"
Jul 14 23:32:30.620353 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:30.620307    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b47a7-84b6b8c5f8-qmpbh"
Jul 14 23:32:30.794429 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:30.794405    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4b47a7-84b6b8c5f8-qmpbh"]
Jul 14 23:32:31.688146 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:31.688110    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b47a7-84b6b8c5f8-qmpbh" event={"ID":"0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d","Type":"ContainerStarted","Data":"87277bf90395fce6198faa447a6b65e809287355f5a034216d5ec88729ba3728"}
Jul 14 23:32:31.688351 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:31.688191    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b47a7-84b6b8c5f8-qmpbh" event={"ID":"0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d","Type":"ContainerStarted","Data":"a62d5b82c181d9bc03a08b251ac45ed3ba906c320b2364d81bfa95af2f17a7d6"}
Jul 14 23:32:31.688351 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:31.688258    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b47a7-84b6b8c5f8-qmpbh"
Jul 14 23:32:31.706997 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:31.706950    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b47a7-84b6b8c5f8-qmpbh" podStartSLOduration=2.331360534 podStartE2EDuration="2.706935454s" totalImagesPullingTime="375.57492ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:32:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:32:31.705411336 +0000 UTC m=+2139.008402128" watchObservedRunningTime="2026-07-14 23:32:31.706935454 +0000 UTC m=+2139.009926262"
Jul 14 23:32:36.669483 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:36.669440    2563 patch_prober.go:28] interesting pod/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 14 23:32:36.669868 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:36.669507    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm" podUID="d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 14 23:32:37.699539 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:37.699456    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b47a7-84b6b8c5f8-qmpbh"
Jul 14 23:32:37.721812 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:37.720693    2563 generic.go:356] "Generic (PLEG): container finished" podID="d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d" containerID="160896e35e9513decb257d360cf37df266acb0140095012857b36379c962ed93" exitCode=0
Jul 14 23:32:37.721812 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:37.720756    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm" event={"ID":"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d","Type":"ContainerDied","Data":"160896e35e9513decb257d360cf37df266acb0140095012857b36379c962ed93"}
Jul 14 23:32:37.721812 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:37.721208    2563 scope.go:117] "RemoveContainer" containerID="160896e35e9513decb257d360cf37df266acb0140095012857b36379c962ed93"
Jul 14 23:32:37.723903 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:37.723555    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:32:37.723903 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:37.723642    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:32:37.723903 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:37.723689    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:32:38.737927 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:38.737882    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm" event={"ID":"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d","Type":"ContainerStarted","Data":"c8cc61cc71a82eb98aaaeede887d16c8af577beb735568e78b4b5d031aaddc4c"}
Jul 14 23:32:39.428796 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:39.428753    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"
Jul 14 23:32:39.429198 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:39.429106    2563 patch_prober.go:28] interesting pod/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Jul 14 23:32:39.429410 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:39.429386    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm" podUID="d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Jul 14 23:32:40.428721 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:40.428664    2563 patch_prober.go:28] interesting pod/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Jul 14 23:32:40.429410 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:40.429359    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm" podUID="d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Jul 14 23:32:41.428792 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:41.428746    2563 patch_prober.go:28] interesting pod/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Jul 14 23:32:41.429359 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:41.428830    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm" podUID="d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Jul 14 23:32:42.429164 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:42.429113    2563 patch_prober.go:28] interesting pod/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Jul 14 23:32:42.429701 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:42.429201    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm" podUID="d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Jul 14 23:32:43.432583 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:43.432542    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"
Jul 14 23:32:43.433051 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:43.432938    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"
Jul 14 23:32:43.433751 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:43.433730    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"
Jul 14 23:32:43.842670 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:43.841240    2563 patch_prober.go:28] interesting pod/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": read tcp 10.133.0.2:34974->10.133.0.57:15021: read: connection reset by peer" start-of-body=
Jul 14 23:32:43.842670 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:43.841329    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg" podUID="8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": read tcp 10.133.0.2:34974->10.133.0.57:15021: read: connection reset by peer"
Jul 14 23:32:44.771477 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:44.771388    2563 generic.go:356] "Generic (PLEG): container finished" podID="8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1" containerID="37a7d1a4eadff759564908cef547ad07b4fa154de8e6c4da30e051bbce2a687d" exitCode=0
Jul 14 23:32:44.771477 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:44.771461    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg" event={"ID":"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1","Type":"ContainerDied","Data":"37a7d1a4eadff759564908cef547ad07b4fa154de8e6c4da30e051bbce2a687d"}
Jul 14 23:32:44.772068 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:44.772050    2563 scope.go:117] "RemoveContainer" containerID="37a7d1a4eadff759564908cef547ad07b4fa154de8e6c4da30e051bbce2a687d"
Jul 14 23:32:44.773010 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:44.772983    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:32:44.773064 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:44.773050    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:32:44.773099 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:44.773081    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:32:45.778187 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:45.778155    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg" event={"ID":"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1","Type":"ContainerStarted","Data":"d8b47475289218b9f40ed12a5ede008c79f00cf85d860d1dece278b1cc0bfbeb"}
Jul 14 23:32:46.048941 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:46.048833    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"
Jul 14 23:32:46.049144 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:46.049123    2563 patch_prober.go:28] interesting pod/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Jul 14 23:32:46.049205 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:46.049185    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg" podUID="8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Jul 14 23:32:47.048155 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:47.048114    2563 patch_prober.go:28] interesting pod/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Jul 14 23:32:47.048596 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:47.048171    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg" podUID="8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Jul 14 23:32:47.088380 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:47.088342    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-d130d1/test-model-d130d1-kserve-56bbcd4f55-pccjf"]
Jul 14 23:32:47.091590 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:47.091565    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d130d1/test-model-d130d1-kserve-56bbcd4f55-pccjf"
Jul 14 23:32:47.095106 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:47.095078    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d130d1\"/\"kube-root-ca.crt\""
Jul 14 23:32:47.096673 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:47.096654    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d130d1\"/\"openshift-service-ca.crt\""
Jul 14 23:32:47.096787 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:47.096659    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d130d1\"/\"test-model-d130d1-kserve-self-signed-certs\""
Jul 14 23:32:47.096787 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:47.096691    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d130d1\"/\"default-dockercfg-nkf24\""
Jul 14 23:32:47.103023 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:47.103000    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d130d1/test-model-d130d1-kserve-56bbcd4f55-pccjf"]
Jul 14 23:32:47.143314 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:47.142475    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flxp2\" (UniqueName: \"kubernetes.io/projected/e1fa8be6-3275-4833-b100-bddb9d7a14a2-kube-api-access-flxp2\") pod \"test-model-d130d1-kserve-56bbcd4f55-pccjf\" (UID: \"e1fa8be6-3275-4833-b100-bddb9d7a14a2\") " pod="ai-tenant-e2e-inf-a-d130d1/test-model-d130d1-kserve-56bbcd4f55-pccjf"
Jul 14 23:32:47.143314 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:47.142540    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1fa8be6-3275-4833-b100-bddb9d7a14a2-dshm\") pod \"test-model-d130d1-kserve-56bbcd4f55-pccjf\" (UID: \"e1fa8be6-3275-4833-b100-bddb9d7a14a2\") " pod="ai-tenant-e2e-inf-a-d130d1/test-model-d130d1-kserve-56bbcd4f55-pccjf"
Jul 14 23:32:47.143314 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:47.142584    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1fa8be6-3275-4833-b100-bddb9d7a14a2-home\") pod \"test-model-d130d1-kserve-56bbcd4f55-pccjf\" (UID: \"e1fa8be6-3275-4833-b100-bddb9d7a14a2\") " pod="ai-tenant-e2e-inf-a-d130d1/test-model-d130d1-kserve-56bbcd4f55-pccjf"
Jul 14 23:32:47.143314 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:47.142614    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1fa8be6-3275-4833-b100-bddb9d7a14a2-tmp-dir\") pod \"test-model-d130d1-kserve-56bbcd4f55-pccjf\" (UID: \"e1fa8be6-3275-4833-b100-bddb9d7a14a2\") " pod="ai-tenant-e2e-inf-a-d130d1/test-model-d130d1-kserve-56bbcd4f55-pccjf"
Jul 14 23:32:47.143314 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:47.142665    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1fa8be6-3275-4833-b100-bddb9d7a14a2-model-cache\") pod \"test-model-d130d1-kserve-56bbcd4f55-pccjf\" (UID: \"e1fa8be6-3275-4833-b100-bddb9d7a14a2\") " pod="ai-tenant-e2e-inf-a-d130d1/test-model-d130d1-kserve-56bbcd4f55-pccjf"
Jul 14 23:32:47.143314 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:47.142738    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1fa8be6-3275-4833-b100-bddb9d7a14a2-tls-certs\") pod \"test-model-d130d1-kserve-56bbcd4f55-pccjf\" (UID: \"e1fa8be6-3275-4833-b100-bddb9d7a14a2\") " pod="ai-tenant-e2e-inf-a-d130d1/test-model-d130d1-kserve-56bbcd4f55-pccjf"
Jul 14 23:32:47.243352 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:47.243316    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1fa8be6-3275-4833-b100-bddb9d7a14a2-home\") pod \"test-model-d130d1-kserve-56bbcd4f55-pccjf\" (UID: \"e1fa8be6-3275-4833-b100-bddb9d7a14a2\") " pod="ai-tenant-e2e-inf-a-d130d1/test-model-d130d1-kserve-56bbcd4f55-pccjf"
Jul 14 23:32:47.243352 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:47.243354    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1fa8be6-3275-4833-b100-bddb9d7a14a2-tmp-dir\") pod \"test-model-d130d1-kserve-56bbcd4f55-pccjf\" (UID: \"e1fa8be6-3275-4833-b100-bddb9d7a14a2\") " pod="ai-tenant-e2e-inf-a-d130d1/test-model-d130d1-kserve-56bbcd4f55-pccjf"
Jul 14 23:32:47.243612 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:47.243382    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1fa8be6-3275-4833-b100-bddb9d7a14a2-model-cache\") pod \"test-model-d130d1-kserve-56bbcd4f55-pccjf\" (UID: \"e1fa8be6-3275-4833-b100-bddb9d7a14a2\") " pod="ai-tenant-e2e-inf-a-d130d1/test-model-d130d1-kserve-56bbcd4f55-pccjf"
Jul 14 23:32:47.243612 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:47.243427    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1fa8be6-3275-4833-b100-bddb9d7a14a2-tls-certs\") pod \"test-model-d130d1-kserve-56bbcd4f55-pccjf\" (UID: \"e1fa8be6-3275-4833-b100-bddb9d7a14a2\") " pod="ai-tenant-e2e-inf-a-d130d1/test-model-d130d1-kserve-56bbcd4f55-pccjf"
Jul 14 23:32:47.243612 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:47.243499    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flxp2\" (UniqueName: \"kubernetes.io/projected/e1fa8be6-3275-4833-b100-bddb9d7a14a2-kube-api-access-flxp2\") pod \"test-model-d130d1-kserve-56bbcd4f55-pccjf\" (UID: \"e1fa8be6-3275-4833-b100-bddb9d7a14a2\") " pod="ai-tenant-e2e-inf-a-d130d1/test-model-d130d1-kserve-56bbcd4f55-pccjf"
Jul 14 23:32:47.243612 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:47.243528    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1fa8be6-3275-4833-b100-bddb9d7a14a2-dshm\") pod \"test-model-d130d1-kserve-56bbcd4f55-pccjf\" (UID: \"e1fa8be6-3275-4833-b100-bddb9d7a14a2\") " pod="ai-tenant-e2e-inf-a-d130d1/test-model-d130d1-kserve-56bbcd4f55-pccjf"
Jul 14 23:32:47.243842 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:47.243776    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1fa8be6-3275-4833-b100-bddb9d7a14a2-home\") pod \"test-model-d130d1-kserve-56bbcd4f55-pccjf\" (UID: \"e1fa8be6-3275-4833-b100-bddb9d7a14a2\") " pod="ai-tenant-e2e-inf-a-d130d1/test-model-d130d1-kserve-56bbcd4f55-pccjf"
Jul 14 23:32:47.243842 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:47.243800    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1fa8be6-3275-4833-b100-bddb9d7a14a2-tmp-dir\") pod \"test-model-d130d1-kserve-56bbcd4f55-pccjf\" (UID: \"e1fa8be6-3275-4833-b100-bddb9d7a14a2\") " pod="ai-tenant-e2e-inf-a-d130d1/test-model-d130d1-kserve-56bbcd4f55-pccjf"
Jul 14 23:32:47.243956 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:47.243905    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1fa8be6-3275-4833-b100-bddb9d7a14a2-model-cache\") pod \"test-model-d130d1-kserve-56bbcd4f55-pccjf\" (UID: \"e1fa8be6-3275-4833-b100-bddb9d7a14a2\") " pod="ai-tenant-e2e-inf-a-d130d1/test-model-d130d1-kserve-56bbcd4f55-pccjf"
Jul 14 23:32:47.246128 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:47.246104    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1fa8be6-3275-4833-b100-bddb9d7a14a2-tls-certs\") pod \"test-model-d130d1-kserve-56bbcd4f55-pccjf\" (UID: \"e1fa8be6-3275-4833-b100-bddb9d7a14a2\") " pod="ai-tenant-e2e-inf-a-d130d1/test-model-d130d1-kserve-56bbcd4f55-pccjf"
Jul 14 23:32:47.246458 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:47.246438    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1fa8be6-3275-4833-b100-bddb9d7a14a2-dshm\") pod \"test-model-d130d1-kserve-56bbcd4f55-pccjf\" (UID: \"e1fa8be6-3275-4833-b100-bddb9d7a14a2\") " pod="ai-tenant-e2e-inf-a-d130d1/test-model-d130d1-kserve-56bbcd4f55-pccjf"
Jul 14 23:32:47.253880 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:47.253821    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flxp2\" (UniqueName: \"kubernetes.io/projected/e1fa8be6-3275-4833-b100-bddb9d7a14a2-kube-api-access-flxp2\") pod \"test-model-d130d1-kserve-56bbcd4f55-pccjf\" (UID: \"e1fa8be6-3275-4833-b100-bddb9d7a14a2\") " pod="ai-tenant-e2e-inf-a-d130d1/test-model-d130d1-kserve-56bbcd4f55-pccjf"
Jul 14 23:32:47.401524 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:47.401431    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d130d1/test-model-d130d1-kserve-56bbcd4f55-pccjf"
Jul 14 23:32:47.580111 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:47.580085    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d130d1/test-model-d130d1-kserve-56bbcd4f55-pccjf"]
Jul 14 23:32:47.582414 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:32:47.582375    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1fa8be6_3275_4833_b100_bddb9d7a14a2.slice/crio-7ced590e1d4d64855a2e374e4ed1d58600926a0828dce6e281f7a45051cb4824 WatchSource:0}: Error finding container 7ced590e1d4d64855a2e374e4ed1d58600926a0828dce6e281f7a45051cb4824: Status 404 returned error can't find the container with id 7ced590e1d4d64855a2e374e4ed1d58600926a0828dce6e281f7a45051cb4824
Jul 14 23:32:47.790193 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:47.790150    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d130d1/test-model-d130d1-kserve-56bbcd4f55-pccjf" event={"ID":"e1fa8be6-3275-4833-b100-bddb9d7a14a2","Type":"ContainerStarted","Data":"3bbe05ec0dce646fa8e722923f937e2f96b34795f3cfeb2208b173476fe19156"}
Jul 14 23:32:47.790193 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:47.790199    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d130d1/test-model-d130d1-kserve-56bbcd4f55-pccjf" event={"ID":"e1fa8be6-3275-4833-b100-bddb9d7a14a2","Type":"ContainerStarted","Data":"7ced590e1d4d64855a2e374e4ed1d58600926a0828dce6e281f7a45051cb4824"}
Jul 14 23:32:47.807560 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:47.807369    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-d130d1/test-model-d130d1-kserve-56bbcd4f55-pccjf" podStartSLOduration=0.80735093 podStartE2EDuration="807.35093ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:32:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:32:47.807238106 +0000 UTC m=+2155.110228926" watchObservedRunningTime="2026-07-14 23:32:47.80735093 +0000 UTC m=+2155.110341722"
Jul 14 23:32:48.049034 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:48.048986    2563 patch_prober.go:28] interesting pod/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Jul 14 23:32:48.049512 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:48.049053    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg" podUID="8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Jul 14 23:32:49.048465 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:49.048426    2563 patch_prober.go:28] interesting pod/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Jul 14 23:32:49.048734 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:49.048488    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg" podUID="8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Jul 14 23:32:50.051848 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:50.051816    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"
Jul 14 23:32:50.052326 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:50.052309    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"
Jul 14 23:32:50.053117 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:50.053098    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"
Jul 14 23:32:57.402386 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:57.402340    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d130d1/test-model-d130d1-kserve-56bbcd4f55-pccjf"
Jul 14 23:32:57.402919 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:57.402643    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-d130d1/test-model-d130d1-kserve-56bbcd4f55-pccjf"
Jul 14 23:32:57.415370 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:57.415335    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-d130d1/test-model-d130d1-kserve-56bbcd4f55-pccjf"
Jul 14 23:32:57.848559 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:32:57.848528    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d130d1/test-model-d130d1-kserve-56bbcd4f55-pccjf"
Jul 14 23:33:10.602194 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:10.602157    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-4b47a7/test-model-4b47a7-kserve-5dd656448c-qmhhk"]
Jul 14 23:33:10.608402 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:10.608374    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4b47a7/test-model-4b47a7-kserve-5dd656448c-qmhhk"
Jul 14 23:33:10.611696 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:10.611665    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4b47a7\"/\"kube-root-ca.crt\""
Jul 14 23:33:10.611990 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:10.611970    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4b47a7\"/\"test-model-4b47a7-kserve-self-signed-certs\""
Jul 14 23:33:10.612065 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:10.611973    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4b47a7\"/\"default-dockercfg-nc7rj\""
Jul 14 23:33:10.612979 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:10.612953    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4b47a7/test-model-4b47a7-kserve-5dd656448c-qmhhk"]
Jul 14 23:33:10.613223 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:10.613207    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4b47a7\"/\"openshift-service-ca.crt\""
Jul 14 23:33:10.755036 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:10.754999    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8ff88dc-4677-41a6-bc63-82924267fdd8-home\") pod \"test-model-4b47a7-kserve-5dd656448c-qmhhk\" (UID: \"c8ff88dc-4677-41a6-bc63-82924267fdd8\") " pod="ai-tenant-e2e-inf-b-4b47a7/test-model-4b47a7-kserve-5dd656448c-qmhhk"
Jul 14 23:33:10.755036 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:10.755032    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8ff88dc-4677-41a6-bc63-82924267fdd8-dshm\") pod \"test-model-4b47a7-kserve-5dd656448c-qmhhk\" (UID: \"c8ff88dc-4677-41a6-bc63-82924267fdd8\") " pod="ai-tenant-e2e-inf-b-4b47a7/test-model-4b47a7-kserve-5dd656448c-qmhhk"
Jul 14 23:33:10.755258 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:10.755055    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bkxs\" (UniqueName: \"kubernetes.io/projected/c8ff88dc-4677-41a6-bc63-82924267fdd8-kube-api-access-6bkxs\") pod \"test-model-4b47a7-kserve-5dd656448c-qmhhk\" (UID: \"c8ff88dc-4677-41a6-bc63-82924267fdd8\") " pod="ai-tenant-e2e-inf-b-4b47a7/test-model-4b47a7-kserve-5dd656448c-qmhhk"
Jul 14 23:33:10.755258 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:10.755138    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8ff88dc-4677-41a6-bc63-82924267fdd8-model-cache\") pod \"test-model-4b47a7-kserve-5dd656448c-qmhhk\" (UID: \"c8ff88dc-4677-41a6-bc63-82924267fdd8\") " pod="ai-tenant-e2e-inf-b-4b47a7/test-model-4b47a7-kserve-5dd656448c-qmhhk"
Jul 14 23:33:10.755258 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:10.755161    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8ff88dc-4677-41a6-bc63-82924267fdd8-tmp-dir\") pod \"test-model-4b47a7-kserve-5dd656448c-qmhhk\" (UID: \"c8ff88dc-4677-41a6-bc63-82924267fdd8\") " pod="ai-tenant-e2e-inf-b-4b47a7/test-model-4b47a7-kserve-5dd656448c-qmhhk"
Jul 14 23:33:10.755258 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:10.755228    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8ff88dc-4677-41a6-bc63-82924267fdd8-tls-certs\") pod \"test-model-4b47a7-kserve-5dd656448c-qmhhk\" (UID: \"c8ff88dc-4677-41a6-bc63-82924267fdd8\") " pod="ai-tenant-e2e-inf-b-4b47a7/test-model-4b47a7-kserve-5dd656448c-qmhhk"
Jul 14 23:33:10.855965 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:10.855876    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8ff88dc-4677-41a6-bc63-82924267fdd8-model-cache\") pod \"test-model-4b47a7-kserve-5dd656448c-qmhhk\" (UID: \"c8ff88dc-4677-41a6-bc63-82924267fdd8\") " pod="ai-tenant-e2e-inf-b-4b47a7/test-model-4b47a7-kserve-5dd656448c-qmhhk"
Jul 14 23:33:10.855965 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:10.855911    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8ff88dc-4677-41a6-bc63-82924267fdd8-tmp-dir\") pod \"test-model-4b47a7-kserve-5dd656448c-qmhhk\" (UID: \"c8ff88dc-4677-41a6-bc63-82924267fdd8\") " pod="ai-tenant-e2e-inf-b-4b47a7/test-model-4b47a7-kserve-5dd656448c-qmhhk"
Jul 14 23:33:10.855965 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:10.855936    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8ff88dc-4677-41a6-bc63-82924267fdd8-tls-certs\") pod \"test-model-4b47a7-kserve-5dd656448c-qmhhk\" (UID: \"c8ff88dc-4677-41a6-bc63-82924267fdd8\") " pod="ai-tenant-e2e-inf-b-4b47a7/test-model-4b47a7-kserve-5dd656448c-qmhhk"
Jul 14 23:33:10.856227 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:10.856059    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8ff88dc-4677-41a6-bc63-82924267fdd8-home\") pod \"test-model-4b47a7-kserve-5dd656448c-qmhhk\" (UID: \"c8ff88dc-4677-41a6-bc63-82924267fdd8\") " pod="ai-tenant-e2e-inf-b-4b47a7/test-model-4b47a7-kserve-5dd656448c-qmhhk"
Jul 14 23:33:10.856227 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:10.856085    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8ff88dc-4677-41a6-bc63-82924267fdd8-dshm\") pod \"test-model-4b47a7-kserve-5dd656448c-qmhhk\" (UID: \"c8ff88dc-4677-41a6-bc63-82924267fdd8\") " pod="ai-tenant-e2e-inf-b-4b47a7/test-model-4b47a7-kserve-5dd656448c-qmhhk"
Jul 14 23:33:10.856227 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:10.856120    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bkxs\" (UniqueName: \"kubernetes.io/projected/c8ff88dc-4677-41a6-bc63-82924267fdd8-kube-api-access-6bkxs\") pod \"test-model-4b47a7-kserve-5dd656448c-qmhhk\" (UID: \"c8ff88dc-4677-41a6-bc63-82924267fdd8\") " pod="ai-tenant-e2e-inf-b-4b47a7/test-model-4b47a7-kserve-5dd656448c-qmhhk"
Jul 14 23:33:10.856428 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:10.856339    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8ff88dc-4677-41a6-bc63-82924267fdd8-tmp-dir\") pod \"test-model-4b47a7-kserve-5dd656448c-qmhhk\" (UID: \"c8ff88dc-4677-41a6-bc63-82924267fdd8\") " pod="ai-tenant-e2e-inf-b-4b47a7/test-model-4b47a7-kserve-5dd656448c-qmhhk"
Jul 14 23:33:10.856428 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:10.856348    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8ff88dc-4677-41a6-bc63-82924267fdd8-model-cache\") pod \"test-model-4b47a7-kserve-5dd656448c-qmhhk\" (UID: \"c8ff88dc-4677-41a6-bc63-82924267fdd8\") " pod="ai-tenant-e2e-inf-b-4b47a7/test-model-4b47a7-kserve-5dd656448c-qmhhk"
Jul 14 23:33:10.856522 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:10.856452    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8ff88dc-4677-41a6-bc63-82924267fdd8-home\") pod \"test-model-4b47a7-kserve-5dd656448c-qmhhk\" (UID: \"c8ff88dc-4677-41a6-bc63-82924267fdd8\") " pod="ai-tenant-e2e-inf-b-4b47a7/test-model-4b47a7-kserve-5dd656448c-qmhhk"
Jul 14 23:33:10.858376 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:10.858344    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8ff88dc-4677-41a6-bc63-82924267fdd8-dshm\") pod \"test-model-4b47a7-kserve-5dd656448c-qmhhk\" (UID: \"c8ff88dc-4677-41a6-bc63-82924267fdd8\") " pod="ai-tenant-e2e-inf-b-4b47a7/test-model-4b47a7-kserve-5dd656448c-qmhhk"
Jul 14 23:33:10.858684 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:10.858664    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8ff88dc-4677-41a6-bc63-82924267fdd8-tls-certs\") pod \"test-model-4b47a7-kserve-5dd656448c-qmhhk\" (UID: \"c8ff88dc-4677-41a6-bc63-82924267fdd8\") " pod="ai-tenant-e2e-inf-b-4b47a7/test-model-4b47a7-kserve-5dd656448c-qmhhk"
Jul 14 23:33:10.864885 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:10.864863    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bkxs\" (UniqueName: \"kubernetes.io/projected/c8ff88dc-4677-41a6-bc63-82924267fdd8-kube-api-access-6bkxs\") pod \"test-model-4b47a7-kserve-5dd656448c-qmhhk\" (UID: \"c8ff88dc-4677-41a6-bc63-82924267fdd8\") " pod="ai-tenant-e2e-inf-b-4b47a7/test-model-4b47a7-kserve-5dd656448c-qmhhk"
Jul 14 23:33:10.919723 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:10.919690    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4b47a7/test-model-4b47a7-kserve-5dd656448c-qmhhk"
Jul 14 23:33:11.053302 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:11.053262    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4b47a7/test-model-4b47a7-kserve-5dd656448c-qmhhk"]
Jul 14 23:33:11.055430 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:33:11.055400    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8ff88dc_4677_41a6_bc63_82924267fdd8.slice/crio-f1027f761f5473a7f764a50dba580c0e617370ccf51637ba8fdc52d7b75748b3 WatchSource:0}: Error finding container f1027f761f5473a7f764a50dba580c0e617370ccf51637ba8fdc52d7b75748b3: Status 404 returned error can't find the container with id f1027f761f5473a7f764a50dba580c0e617370ccf51637ba8fdc52d7b75748b3
Jul 14 23:33:11.909035 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:11.908994    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4b47a7/test-model-4b47a7-kserve-5dd656448c-qmhhk" event={"ID":"c8ff88dc-4677-41a6-bc63-82924267fdd8","Type":"ContainerStarted","Data":"4907e95570f7c6d053d29fee31723a6d454dba974fd7fd5e797f24e20878a468"}
Jul 14 23:33:11.909448 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:11.909046    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4b47a7/test-model-4b47a7-kserve-5dd656448c-qmhhk" event={"ID":"c8ff88dc-4677-41a6-bc63-82924267fdd8","Type":"ContainerStarted","Data":"f1027f761f5473a7f764a50dba580c0e617370ccf51637ba8fdc52d7b75748b3"}
Jul 14 23:33:11.928184 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:11.928136    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-4b47a7/test-model-4b47a7-kserve-5dd656448c-qmhhk" podStartSLOduration=1.928119402 podStartE2EDuration="1.928119402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:33:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:33:11.926724134 +0000 UTC m=+2179.229714925" watchObservedRunningTime="2026-07-14 23:33:11.928119402 +0000 UTC m=+2179.231110194"
Jul 14 23:33:20.920477 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:20.920426    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-4b47a7/test-model-4b47a7-kserve-5dd656448c-qmhhk"
Jul 14 23:33:20.920477 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:20.920486    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-4b47a7/test-model-4b47a7-kserve-5dd656448c-qmhhk"
Jul 14 23:33:20.932950 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:20.932923    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-4b47a7/test-model-4b47a7-kserve-5dd656448c-qmhhk"
Jul 14 23:33:20.963184 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:20.963152    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-4b47a7/test-model-4b47a7-kserve-5dd656448c-qmhhk"
Jul 14 23:33:27.441775 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:27.441739    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d130d1/test-model-d130d1-kserve-56bbcd4f55-pccjf"]
Jul 14 23:33:28.383124 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:28.383092    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4b47a7/test-model-4b47a7-kserve-5dd656448c-qmhhk"]
Jul 14 23:33:38.975340 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:38.975303    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d130d1-77749fddc7-7mzd4"]
Jul 14 23:33:38.975743 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:38.975512    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d130d1-77749fddc7-7mzd4" podUID="ea4261ea-5839-4c00-bc5b-6e33d605ffff" containerName="maas-api" containerID="cri-o://6428fd699673eb5f82dbc7420a0013ccd22d3f58654103ddfbeb8bb842515eca" gracePeriod=30
Jul 14 23:33:39.224327 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:39.224273    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d130d1-77749fddc7-7mzd4"
Jul 14 23:33:39.318386 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:39.318357    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea4261ea-5839-4c00-bc5b-6e33d605ffff-openshift-service-ca\") pod \"ea4261ea-5839-4c00-bc5b-6e33d605ffff\" (UID: \"ea4261ea-5839-4c00-bc5b-6e33d605ffff\") "
Jul 14 23:33:39.318548 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:39.318402    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea4261ea-5839-4c00-bc5b-6e33d605ffff-maas-api-tls\") pod \"ea4261ea-5839-4c00-bc5b-6e33d605ffff\" (UID: \"ea4261ea-5839-4c00-bc5b-6e33d605ffff\") "
Jul 14 23:33:39.318548 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:39.318423    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9xm6\" (UniqueName: \"kubernetes.io/projected/ea4261ea-5839-4c00-bc5b-6e33d605ffff-kube-api-access-x9xm6\") pod \"ea4261ea-5839-4c00-bc5b-6e33d605ffff\" (UID: \"ea4261ea-5839-4c00-bc5b-6e33d605ffff\") "
Jul 14 23:33:39.318758 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:39.318732    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea4261ea-5839-4c00-bc5b-6e33d605ffff-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ea4261ea-5839-4c00-bc5b-6e33d605ffff" (UID: "ea4261ea-5839-4c00-bc5b-6e33d605ffff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 23:33:39.320593 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:39.320565    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea4261ea-5839-4c00-bc5b-6e33d605ffff-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ea4261ea-5839-4c00-bc5b-6e33d605ffff" (UID: "ea4261ea-5839-4c00-bc5b-6e33d605ffff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 23:33:39.320716 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:39.320691    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea4261ea-5839-4c00-bc5b-6e33d605ffff-kube-api-access-x9xm6" (OuterVolumeSpecName: "kube-api-access-x9xm6") pod "ea4261ea-5839-4c00-bc5b-6e33d605ffff" (UID: "ea4261ea-5839-4c00-bc5b-6e33d605ffff"). InnerVolumeSpecName "kube-api-access-x9xm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:33:39.419794 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:39.419767    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea4261ea-5839-4c00-bc5b-6e33d605ffff-openshift-service-ca\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:33:39.419794 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:39.419796    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea4261ea-5839-4c00-bc5b-6e33d605ffff-maas-api-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:33:39.419953 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:39.419809    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9xm6\" (UniqueName: \"kubernetes.io/projected/ea4261ea-5839-4c00-bc5b-6e33d605ffff-kube-api-access-x9xm6\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:33:40.046298 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:40.046249    2563 generic.go:356] "Generic (PLEG): container finished" podID="ea4261ea-5839-4c00-bc5b-6e33d605ffff" containerID="6428fd699673eb5f82dbc7420a0013ccd22d3f58654103ddfbeb8bb842515eca" exitCode=0
Jul 14 23:33:40.046724 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:40.046307    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d130d1-77749fddc7-7mzd4"
Jul 14 23:33:40.046724 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:40.046339    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d130d1-77749fddc7-7mzd4" event={"ID":"ea4261ea-5839-4c00-bc5b-6e33d605ffff","Type":"ContainerDied","Data":"6428fd699673eb5f82dbc7420a0013ccd22d3f58654103ddfbeb8bb842515eca"}
Jul 14 23:33:40.046724 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:40.046400    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d130d1-77749fddc7-7mzd4" event={"ID":"ea4261ea-5839-4c00-bc5b-6e33d605ffff","Type":"ContainerDied","Data":"12df5b1455a4b3d33747e1a10721aa2a8706d3262e341c8bf3c0cd8a1e7e6c90"}
Jul 14 23:33:40.046724 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:40.046424    2563 scope.go:117] "RemoveContainer" containerID="6428fd699673eb5f82dbc7420a0013ccd22d3f58654103ddfbeb8bb842515eca"
Jul 14 23:33:40.055609 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:40.055587    2563 scope.go:117] "RemoveContainer" containerID="6428fd699673eb5f82dbc7420a0013ccd22d3f58654103ddfbeb8bb842515eca"
Jul 14 23:33:40.055894 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:33:40.055876    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6428fd699673eb5f82dbc7420a0013ccd22d3f58654103ddfbeb8bb842515eca\": container with ID starting with 6428fd699673eb5f82dbc7420a0013ccd22d3f58654103ddfbeb8bb842515eca not found: ID does not exist" containerID="6428fd699673eb5f82dbc7420a0013ccd22d3f58654103ddfbeb8bb842515eca"
Jul 14 23:33:40.055962 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:40.055904    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6428fd699673eb5f82dbc7420a0013ccd22d3f58654103ddfbeb8bb842515eca"} err="failed to get container status \"6428fd699673eb5f82dbc7420a0013ccd22d3f58654103ddfbeb8bb842515eca\": rpc error: code = NotFound desc = could not find container \"6428fd699673eb5f82dbc7420a0013ccd22d3f58654103ddfbeb8bb842515eca\": container with ID starting with 6428fd699673eb5f82dbc7420a0013ccd22d3f58654103ddfbeb8bb842515eca not found: ID does not exist"
Jul 14 23:33:40.068337 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:40.068315    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d130d1-77749fddc7-7mzd4"]
Jul 14 23:33:40.072826 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:40.072806    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d130d1-77749fddc7-7mzd4"]
Jul 14 23:33:41.229412 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:41.229379    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea4261ea-5839-4c00-bc5b-6e33d605ffff" path="/var/lib/kubelet/pods/ea4261ea-5839-4c00-bc5b-6e33d605ffff/volumes"
Jul 14 23:33:42.462905 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:42.462865    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-d130d1/test-model-d130d1-kserve-56bbcd4f55-pccjf" podUID="e1fa8be6-3275-4833-b100-bddb9d7a14a2" containerName="main" containerID="cri-o://3bbe05ec0dce646fa8e722923f937e2f96b34795f3cfeb2208b173476fe19156" gracePeriod=45
Jul 14 23:33:42.720261 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:42.720200    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d130d1/test-model-d130d1-kserve-56bbcd4f55-pccjf"
Jul 14 23:33:42.848036 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:42.848004    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1fa8be6-3275-4833-b100-bddb9d7a14a2-model-cache\") pod \"e1fa8be6-3275-4833-b100-bddb9d7a14a2\" (UID: \"e1fa8be6-3275-4833-b100-bddb9d7a14a2\") "
Jul 14 23:33:42.848220 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:42.848044    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-flxp2\" (UniqueName: \"kubernetes.io/projected/e1fa8be6-3275-4833-b100-bddb9d7a14a2-kube-api-access-flxp2\") pod \"e1fa8be6-3275-4833-b100-bddb9d7a14a2\" (UID: \"e1fa8be6-3275-4833-b100-bddb9d7a14a2\") "
Jul 14 23:33:42.848220 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:42.848061    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1fa8be6-3275-4833-b100-bddb9d7a14a2-home\") pod \"e1fa8be6-3275-4833-b100-bddb9d7a14a2\" (UID: \"e1fa8be6-3275-4833-b100-bddb9d7a14a2\") "
Jul 14 23:33:42.848220 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:42.848148    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1fa8be6-3275-4833-b100-bddb9d7a14a2-tls-certs\") pod \"e1fa8be6-3275-4833-b100-bddb9d7a14a2\" (UID: \"e1fa8be6-3275-4833-b100-bddb9d7a14a2\") "
Jul 14 23:33:42.848220 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:42.848176    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1fa8be6-3275-4833-b100-bddb9d7a14a2-dshm\") pod \"e1fa8be6-3275-4833-b100-bddb9d7a14a2\" (UID: \"e1fa8be6-3275-4833-b100-bddb9d7a14a2\") "
Jul 14 23:33:42.848474 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:42.848234    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1fa8be6-3275-4833-b100-bddb9d7a14a2-tmp-dir\") pod \"e1fa8be6-3275-4833-b100-bddb9d7a14a2\" (UID: \"e1fa8be6-3275-4833-b100-bddb9d7a14a2\") "
Jul 14 23:33:42.848474 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:42.848366    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1fa8be6-3275-4833-b100-bddb9d7a14a2-model-cache" (OuterVolumeSpecName: "model-cache") pod "e1fa8be6-3275-4833-b100-bddb9d7a14a2" (UID: "e1fa8be6-3275-4833-b100-bddb9d7a14a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:33:42.848474 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:42.848396    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1fa8be6-3275-4833-b100-bddb9d7a14a2-home" (OuterVolumeSpecName: "home") pod "e1fa8be6-3275-4833-b100-bddb9d7a14a2" (UID: "e1fa8be6-3275-4833-b100-bddb9d7a14a2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:33:42.848594 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:42.848517    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1fa8be6-3275-4833-b100-bddb9d7a14a2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e1fa8be6-3275-4833-b100-bddb9d7a14a2" (UID: "e1fa8be6-3275-4833-b100-bddb9d7a14a2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:33:42.848594 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:42.848580    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1fa8be6-3275-4833-b100-bddb9d7a14a2-model-cache\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:33:42.848690 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:42.848601    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1fa8be6-3275-4833-b100-bddb9d7a14a2-home\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:33:42.850459 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:42.850433    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1fa8be6-3275-4833-b100-bddb9d7a14a2-kube-api-access-flxp2" (OuterVolumeSpecName: "kube-api-access-flxp2") pod "e1fa8be6-3275-4833-b100-bddb9d7a14a2" (UID: "e1fa8be6-3275-4833-b100-bddb9d7a14a2"). InnerVolumeSpecName "kube-api-access-flxp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:33:42.850560 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:42.850471    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1fa8be6-3275-4833-b100-bddb9d7a14a2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e1fa8be6-3275-4833-b100-bddb9d7a14a2" (UID: "e1fa8be6-3275-4833-b100-bddb9d7a14a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 23:33:42.850560 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:42.850485    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1fa8be6-3275-4833-b100-bddb9d7a14a2-dshm" (OuterVolumeSpecName: "dshm") pod "e1fa8be6-3275-4833-b100-bddb9d7a14a2" (UID: "e1fa8be6-3275-4833-b100-bddb9d7a14a2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:33:42.949000 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:42.948972    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flxp2\" (UniqueName: \"kubernetes.io/projected/e1fa8be6-3275-4833-b100-bddb9d7a14a2-kube-api-access-flxp2\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:33:42.949000 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:42.948995    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1fa8be6-3275-4833-b100-bddb9d7a14a2-tls-certs\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:33:42.949000 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:42.949005    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1fa8be6-3275-4833-b100-bddb9d7a14a2-dshm\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:33:42.949185 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:42.949013    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1fa8be6-3275-4833-b100-bddb9d7a14a2-tmp-dir\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:33:43.062185 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:43.062149    2563 generic.go:356] "Generic (PLEG): container finished" podID="e1fa8be6-3275-4833-b100-bddb9d7a14a2" containerID="3bbe05ec0dce646fa8e722923f937e2f96b34795f3cfeb2208b173476fe19156" exitCode=0
Jul 14 23:33:43.062383 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:43.062186    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d130d1/test-model-d130d1-kserve-56bbcd4f55-pccjf" event={"ID":"e1fa8be6-3275-4833-b100-bddb9d7a14a2","Type":"ContainerDied","Data":"3bbe05ec0dce646fa8e722923f937e2f96b34795f3cfeb2208b173476fe19156"}
Jul 14 23:33:43.062383 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:43.062199    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d130d1/test-model-d130d1-kserve-56bbcd4f55-pccjf"
Jul 14 23:33:43.062383 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:43.062228    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d130d1/test-model-d130d1-kserve-56bbcd4f55-pccjf" event={"ID":"e1fa8be6-3275-4833-b100-bddb9d7a14a2","Type":"ContainerDied","Data":"7ced590e1d4d64855a2e374e4ed1d58600926a0828dce6e281f7a45051cb4824"}
Jul 14 23:33:43.062383 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:43.062249    2563 scope.go:117] "RemoveContainer" containerID="3bbe05ec0dce646fa8e722923f937e2f96b34795f3cfeb2208b173476fe19156"
Jul 14 23:33:43.071779 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:43.071764    2563 scope.go:117] "RemoveContainer" containerID="3bbe05ec0dce646fa8e722923f937e2f96b34795f3cfeb2208b173476fe19156"
Jul 14 23:33:43.072002 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:33:43.071987    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bbe05ec0dce646fa8e722923f937e2f96b34795f3cfeb2208b173476fe19156\": container with ID starting with 3bbe05ec0dce646fa8e722923f937e2f96b34795f3cfeb2208b173476fe19156 not found: ID does not exist" containerID="3bbe05ec0dce646fa8e722923f937e2f96b34795f3cfeb2208b173476fe19156"
Jul 14 23:33:43.072059 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:43.072010    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bbe05ec0dce646fa8e722923f937e2f96b34795f3cfeb2208b173476fe19156"} err="failed to get container status \"3bbe05ec0dce646fa8e722923f937e2f96b34795f3cfeb2208b173476fe19156\": rpc error: code = NotFound desc = could not find container \"3bbe05ec0dce646fa8e722923f937e2f96b34795f3cfeb2208b173476fe19156\": container with ID starting with 3bbe05ec0dce646fa8e722923f937e2f96b34795f3cfeb2208b173476fe19156 not found: ID does not exist"
Jul 14 23:33:43.086121 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:43.086094    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d130d1/test-model-d130d1-kserve-56bbcd4f55-pccjf"]
Jul 14 23:33:43.089418 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:43.089399    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-d130d1/test-model-d130d1-kserve-56bbcd4f55-pccjf"]
Jul 14 23:33:43.229430 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:43.229405    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1fa8be6-3275-4833-b100-bddb9d7a14a2" path="/var/lib/kubelet/pods/e1fa8be6-3275-4833-b100-bddb9d7a14a2/volumes"
Jul 14 23:33:43.403580 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:43.403466    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-4b47a7/test-model-4b47a7-kserve-5dd656448c-qmhhk" podUID="c8ff88dc-4677-41a6-bc63-82924267fdd8" containerName="main" containerID="cri-o://4907e95570f7c6d053d29fee31723a6d454dba974fd7fd5e797f24e20878a468" gracePeriod=45
Jul 14 23:33:43.652091 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:43.652066    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4b47a7/test-model-4b47a7-kserve-5dd656448c-qmhhk"
Jul 14 23:33:43.756180 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:43.756153    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8ff88dc-4677-41a6-bc63-82924267fdd8-model-cache\") pod \"c8ff88dc-4677-41a6-bc63-82924267fdd8\" (UID: \"c8ff88dc-4677-41a6-bc63-82924267fdd8\") "
Jul 14 23:33:43.756365 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:43.756192    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8ff88dc-4677-41a6-bc63-82924267fdd8-tls-certs\") pod \"c8ff88dc-4677-41a6-bc63-82924267fdd8\" (UID: \"c8ff88dc-4677-41a6-bc63-82924267fdd8\") "
Jul 14 23:33:43.756365 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:43.756215    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8ff88dc-4677-41a6-bc63-82924267fdd8-home\") pod \"c8ff88dc-4677-41a6-bc63-82924267fdd8\" (UID: \"c8ff88dc-4677-41a6-bc63-82924267fdd8\") "
Jul 14 23:33:43.756365 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:43.756307    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8ff88dc-4677-41a6-bc63-82924267fdd8-dshm\") pod \"c8ff88dc-4677-41a6-bc63-82924267fdd8\" (UID: \"c8ff88dc-4677-41a6-bc63-82924267fdd8\") "
Jul 14 23:33:43.756365 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:43.756361    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8ff88dc-4677-41a6-bc63-82924267fdd8-tmp-dir\") pod \"c8ff88dc-4677-41a6-bc63-82924267fdd8\" (UID: \"c8ff88dc-4677-41a6-bc63-82924267fdd8\") "
Jul 14 23:33:43.756652 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:43.756384    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bkxs\" (UniqueName: \"kubernetes.io/projected/c8ff88dc-4677-41a6-bc63-82924267fdd8-kube-api-access-6bkxs\") pod \"c8ff88dc-4677-41a6-bc63-82924267fdd8\" (UID: \"c8ff88dc-4677-41a6-bc63-82924267fdd8\") "
Jul 14 23:33:43.756652 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:43.756450    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ff88dc-4677-41a6-bc63-82924267fdd8-model-cache" (OuterVolumeSpecName: "model-cache") pod "c8ff88dc-4677-41a6-bc63-82924267fdd8" (UID: "c8ff88dc-4677-41a6-bc63-82924267fdd8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:33:43.756652 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:43.756545    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ff88dc-4677-41a6-bc63-82924267fdd8-home" (OuterVolumeSpecName: "home") pod "c8ff88dc-4677-41a6-bc63-82924267fdd8" (UID: "c8ff88dc-4677-41a6-bc63-82924267fdd8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:33:43.756652 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:43.756616    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ff88dc-4677-41a6-bc63-82924267fdd8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c8ff88dc-4677-41a6-bc63-82924267fdd8" (UID: "c8ff88dc-4677-41a6-bc63-82924267fdd8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:33:43.756854 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:43.756723    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8ff88dc-4677-41a6-bc63-82924267fdd8-tmp-dir\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:33:43.756854 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:43.756744    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8ff88dc-4677-41a6-bc63-82924267fdd8-model-cache\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:33:43.756854 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:43.756756    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8ff88dc-4677-41a6-bc63-82924267fdd8-home\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:33:43.758778 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:43.758751    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ff88dc-4677-41a6-bc63-82924267fdd8-dshm" (OuterVolumeSpecName: "dshm") pod "c8ff88dc-4677-41a6-bc63-82924267fdd8" (UID: "c8ff88dc-4677-41a6-bc63-82924267fdd8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:33:43.758778 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:43.758767    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8ff88dc-4677-41a6-bc63-82924267fdd8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c8ff88dc-4677-41a6-bc63-82924267fdd8" (UID: "c8ff88dc-4677-41a6-bc63-82924267fdd8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 23:33:43.758910 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:43.758872    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8ff88dc-4677-41a6-bc63-82924267fdd8-kube-api-access-6bkxs" (OuterVolumeSpecName: "kube-api-access-6bkxs") pod "c8ff88dc-4677-41a6-bc63-82924267fdd8" (UID: "c8ff88dc-4677-41a6-bc63-82924267fdd8"). InnerVolumeSpecName "kube-api-access-6bkxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:33:43.858107 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:43.858079    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8ff88dc-4677-41a6-bc63-82924267fdd8-tls-certs\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:33:43.858107 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:43.858104    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8ff88dc-4677-41a6-bc63-82924267fdd8-dshm\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:33:43.858294 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:43.858115    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bkxs\" (UniqueName: \"kubernetes.io/projected/c8ff88dc-4677-41a6-bc63-82924267fdd8-kube-api-access-6bkxs\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:33:44.070325 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:44.070227    2563 generic.go:356] "Generic (PLEG): container finished" podID="c8ff88dc-4677-41a6-bc63-82924267fdd8" containerID="4907e95570f7c6d053d29fee31723a6d454dba974fd7fd5e797f24e20878a468" exitCode=0
Jul 14 23:33:44.070325 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:44.070271    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4b47a7/test-model-4b47a7-kserve-5dd656448c-qmhhk"
Jul 14 23:33:44.070325 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:44.070309    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4b47a7/test-model-4b47a7-kserve-5dd656448c-qmhhk" event={"ID":"c8ff88dc-4677-41a6-bc63-82924267fdd8","Type":"ContainerDied","Data":"4907e95570f7c6d053d29fee31723a6d454dba974fd7fd5e797f24e20878a468"}
Jul 14 23:33:44.070573 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:44.070363    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4b47a7/test-model-4b47a7-kserve-5dd656448c-qmhhk" event={"ID":"c8ff88dc-4677-41a6-bc63-82924267fdd8","Type":"ContainerDied","Data":"f1027f761f5473a7f764a50dba580c0e617370ccf51637ba8fdc52d7b75748b3"}
Jul 14 23:33:44.070573 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:44.070382    2563 scope.go:117] "RemoveContainer" containerID="4907e95570f7c6d053d29fee31723a6d454dba974fd7fd5e797f24e20878a468"
Jul 14 23:33:44.079603 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:44.079588    2563 scope.go:117] "RemoveContainer" containerID="4907e95570f7c6d053d29fee31723a6d454dba974fd7fd5e797f24e20878a468"
Jul 14 23:33:44.079867 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:33:44.079844    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4907e95570f7c6d053d29fee31723a6d454dba974fd7fd5e797f24e20878a468\": container with ID starting with 4907e95570f7c6d053d29fee31723a6d454dba974fd7fd5e797f24e20878a468 not found: ID does not exist" containerID="4907e95570f7c6d053d29fee31723a6d454dba974fd7fd5e797f24e20878a468"
Jul 14 23:33:44.079916 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:44.079874    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4907e95570f7c6d053d29fee31723a6d454dba974fd7fd5e797f24e20878a468"} err="failed to get container status \"4907e95570f7c6d053d29fee31723a6d454dba974fd7fd5e797f24e20878a468\": rpc error: code = NotFound desc = could not find container \"4907e95570f7c6d053d29fee31723a6d454dba974fd7fd5e797f24e20878a468\": container with ID starting with 4907e95570f7c6d053d29fee31723a6d454dba974fd7fd5e797f24e20878a468 not found: ID does not exist"
Jul 14 23:33:44.095338 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:44.095305    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4b47a7/test-model-4b47a7-kserve-5dd656448c-qmhhk"]
Jul 14 23:33:44.099324 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:44.099275    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-4b47a7/test-model-4b47a7-kserve-5dd656448c-qmhhk"]
Jul 14 23:33:45.229928 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:45.229891    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8ff88dc-4677-41a6-bc63-82924267fdd8" path="/var/lib/kubelet/pods/c8ff88dc-4677-41a6-bc63-82924267fdd8/volumes"
Jul 14 23:33:54.300841 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:54.300804    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"]
Jul 14 23:33:54.301267 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:54.301079    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm" podUID="d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d" containerName="istio-proxy" containerID="cri-o://c8cc61cc71a82eb98aaaeede887d16c8af577beb735568e78b4b5d031aaddc4c" gracePeriod=30
Jul 14 23:33:58.435843 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:58.435806    2563 patch_prober.go:28] interesting pod/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 14 23:33:58.436310 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:58.435866    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm" podUID="d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 14 23:33:59.557213 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:59.557184    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"
Jul 14 23:33:59.702328 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:59.702228    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-workload-certs\") pod \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\" (UID: \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\") "
Jul 14 23:33:59.702328 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:59.702299    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-istio-data\") pod \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\" (UID: \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\") "
Jul 14 23:33:59.702328 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:59.702318    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-credential-socket\") pod \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\" (UID: \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\") "
Jul 14 23:33:59.702619 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:59.702365    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mcjj\" (UniqueName: \"kubernetes.io/projected/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-kube-api-access-5mcjj\") pod \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\" (UID: \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\") "
Jul 14 23:33:59.702619 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:59.702384    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-istio-podinfo\") pod \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\" (UID: \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\") "
Jul 14 23:33:59.702619 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:59.702418    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-istio-envoy\") pod \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\" (UID: \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\") "
Jul 14 23:33:59.702619 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:59.702503    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-workload-socket\") pod \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\" (UID: \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\") "
Jul 14 23:33:59.702619 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:59.702525    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d" (UID: "d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:33:59.702619 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:59.702547    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-istiod-ca-cert\") pod \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\" (UID: \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\") "
Jul 14 23:33:59.702619 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:59.702618    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-istio-token\") pod \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\" (UID: \"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d\") "
Jul 14 23:33:59.702995 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:59.702674    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d" (UID: "d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:33:59.702995 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:59.702866    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d" (UID: "d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 23:33:59.702995 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:59.702876    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d" (UID: "d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:33:59.702995 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:59.702928    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-workload-certs\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:33:59.702995 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:59.702948    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-credential-socket\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:33:59.703297 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:59.703228    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-istio-data" (OuterVolumeSpecName: "istio-data") pod "d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d" (UID: "d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:33:59.704820 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:59.704787    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d" (UID: "d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 23:33:59.704940 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:59.704863    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-istio-token" (OuterVolumeSpecName: "istio-token") pod "d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d" (UID: "d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:33:59.705086 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:59.705066    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-kube-api-access-5mcjj" (OuterVolumeSpecName: "kube-api-access-5mcjj") pod "d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d" (UID: "d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d"). InnerVolumeSpecName "kube-api-access-5mcjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:33:59.705086 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:59.705069    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d" (UID: "d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:33:59.803646 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:59.803606    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-istio-data\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:33:59.803646 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:59.803639    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mcjj\" (UniqueName: \"kubernetes.io/projected/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-kube-api-access-5mcjj\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:33:59.803646 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:59.803653    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-istio-podinfo\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:33:59.803888 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:59.803666    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-istio-envoy\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:33:59.803888 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:59.803678    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-workload-socket\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:33:59.803888 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:59.803689    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-istiod-ca-cert\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:33:59.803888 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:33:59.803697    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d-istio-token\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:34:00.147262 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:00.147229    2563 generic.go:356] "Generic (PLEG): container finished" podID="d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d" containerID="c8cc61cc71a82eb98aaaeede887d16c8af577beb735568e78b4b5d031aaddc4c" exitCode=0
Jul 14 23:34:00.147459 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:00.147272    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm" event={"ID":"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d","Type":"ContainerDied","Data":"c8cc61cc71a82eb98aaaeede887d16c8af577beb735568e78b4b5d031aaddc4c"}
Jul 14 23:34:00.147459 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:00.147276    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"
Jul 14 23:34:00.147459 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:00.147338    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm" event={"ID":"d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d","Type":"ContainerDied","Data":"bbb17930b6b26e8250b33db771197f358ab4651692aa14c65d4d1d496efa8813"}
Jul 14 23:34:00.147459 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:00.147360    2563 scope.go:117] "RemoveContainer" containerID="c8cc61cc71a82eb98aaaeede887d16c8af577beb735568e78b4b5d031aaddc4c"
Jul 14 23:34:00.156263 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:00.156235    2563 scope.go:117] "RemoveContainer" containerID="160896e35e9513decb257d360cf37df266acb0140095012857b36379c962ed93"
Jul 14 23:34:00.164079 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:00.164057    2563 scope.go:117] "RemoveContainer" containerID="c8cc61cc71a82eb98aaaeede887d16c8af577beb735568e78b4b5d031aaddc4c"
Jul 14 23:34:00.164378 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:34:00.164350    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8cc61cc71a82eb98aaaeede887d16c8af577beb735568e78b4b5d031aaddc4c\": container with ID starting with c8cc61cc71a82eb98aaaeede887d16c8af577beb735568e78b4b5d031aaddc4c not found: ID does not exist" containerID="c8cc61cc71a82eb98aaaeede887d16c8af577beb735568e78b4b5d031aaddc4c"
Jul 14 23:34:00.164439 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:00.164389    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8cc61cc71a82eb98aaaeede887d16c8af577beb735568e78b4b5d031aaddc4c"} err="failed to get container status \"c8cc61cc71a82eb98aaaeede887d16c8af577beb735568e78b4b5d031aaddc4c\": rpc error: code = NotFound desc = could not find container \"c8cc61cc71a82eb98aaaeede887d16c8af577beb735568e78b4b5d031aaddc4c\": container with ID starting with c8cc61cc71a82eb98aaaeede887d16c8af577beb735568e78b4b5d031aaddc4c not found: ID does not exist"
Jul 14 23:34:00.164439 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:00.164406    2563 scope.go:117] "RemoveContainer" containerID="160896e35e9513decb257d360cf37df266acb0140095012857b36379c962ed93"
Jul 14 23:34:00.164660 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:34:00.164643    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"160896e35e9513decb257d360cf37df266acb0140095012857b36379c962ed93\": container with ID starting with 160896e35e9513decb257d360cf37df266acb0140095012857b36379c962ed93 not found: ID does not exist" containerID="160896e35e9513decb257d360cf37df266acb0140095012857b36379c962ed93"
Jul 14 23:34:00.164711 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:00.164672    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"160896e35e9513decb257d360cf37df266acb0140095012857b36379c962ed93"} err="failed to get container status \"160896e35e9513decb257d360cf37df266acb0140095012857b36379c962ed93\": rpc error: code = NotFound desc = could not find container \"160896e35e9513decb257d360cf37df266acb0140095012857b36379c962ed93\": container with ID starting with 160896e35e9513decb257d360cf37df266acb0140095012857b36379c962ed93 not found: ID does not exist"
Jul 14 23:34:00.169752 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:00.169729    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"]
Jul 14 23:34:00.173443 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:00.173420    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d130d1-openshift-default-7d75ddd5f5-q2nzm"]
Jul 14 23:34:01.229960 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:01.229925    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d" path="/var/lib/kubelet/pods/d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d/volumes"
Jul 14 23:34:05.119273 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:05.119242    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4b47a7-84b6b8c5f8-qmpbh"]
Jul 14 23:34:05.119724 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:05.119480    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b47a7-84b6b8c5f8-qmpbh" podUID="0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d" containerName="maas-api" containerID="cri-o://87277bf90395fce6198faa447a6b65e809287355f5a034216d5ec88729ba3728" gracePeriod=30
Jul 14 23:34:05.371556 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:05.371489    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b47a7-84b6b8c5f8-qmpbh"
Jul 14 23:34:05.452177 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:05.452147    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c57xw\" (UniqueName: \"kubernetes.io/projected/0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d-kube-api-access-c57xw\") pod \"0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d\" (UID: \"0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d\") "
Jul 14 23:34:05.452361 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:05.452190    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d-openshift-service-ca\") pod \"0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d\" (UID: \"0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d\") "
Jul 14 23:34:05.452361 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:05.452244    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d-maas-api-tls\") pod \"0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d\" (UID: \"0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d\") "
Jul 14 23:34:05.452616 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:05.452581    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d" (UID: "0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 23:34:05.454426 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:05.454397    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d-kube-api-access-c57xw" (OuterVolumeSpecName: "kube-api-access-c57xw") pod "0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d" (UID: "0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d"). InnerVolumeSpecName "kube-api-access-c57xw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:34:05.454426 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:05.454418    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d" (UID: "0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 23:34:05.553223 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:05.553183    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c57xw\" (UniqueName: \"kubernetes.io/projected/0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d-kube-api-access-c57xw\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:34:05.553223 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:05.553214    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d-openshift-service-ca\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:34:05.553223 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:05.553223    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d-maas-api-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:34:06.176963 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:06.176931    2563 generic.go:356] "Generic (PLEG): container finished" podID="0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d" containerID="87277bf90395fce6198faa447a6b65e809287355f5a034216d5ec88729ba3728" exitCode=0
Jul 14 23:34:06.177375 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:06.176976    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b47a7-84b6b8c5f8-qmpbh"
Jul 14 23:34:06.177375 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:06.177013    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b47a7-84b6b8c5f8-qmpbh" event={"ID":"0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d","Type":"ContainerDied","Data":"87277bf90395fce6198faa447a6b65e809287355f5a034216d5ec88729ba3728"}
Jul 14 23:34:06.177375 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:06.177065    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b47a7-84b6b8c5f8-qmpbh" event={"ID":"0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d","Type":"ContainerDied","Data":"a62d5b82c181d9bc03a08b251ac45ed3ba906c320b2364d81bfa95af2f17a7d6"}
Jul 14 23:34:06.177375 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:06.177088    2563 scope.go:117] "RemoveContainer" containerID="87277bf90395fce6198faa447a6b65e809287355f5a034216d5ec88729ba3728"
Jul 14 23:34:06.186196 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:06.186179    2563 scope.go:117] "RemoveContainer" containerID="87277bf90395fce6198faa447a6b65e809287355f5a034216d5ec88729ba3728"
Jul 14 23:34:06.186489 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:34:06.186470    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87277bf90395fce6198faa447a6b65e809287355f5a034216d5ec88729ba3728\": container with ID starting with 87277bf90395fce6198faa447a6b65e809287355f5a034216d5ec88729ba3728 not found: ID does not exist" containerID="87277bf90395fce6198faa447a6b65e809287355f5a034216d5ec88729ba3728"
Jul 14 23:34:06.186552 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:06.186496    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87277bf90395fce6198faa447a6b65e809287355f5a034216d5ec88729ba3728"} err="failed to get container status \"87277bf90395fce6198faa447a6b65e809287355f5a034216d5ec88729ba3728\": rpc error: code = NotFound desc = could not find container \"87277bf90395fce6198faa447a6b65e809287355f5a034216d5ec88729ba3728\": container with ID starting with 87277bf90395fce6198faa447a6b65e809287355f5a034216d5ec88729ba3728 not found: ID does not exist"
Jul 14 23:34:06.197587 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:06.197557    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4b47a7-84b6b8c5f8-qmpbh"]
Jul 14 23:34:06.202815 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:06.202791    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4b47a7-84b6b8c5f8-qmpbh"]
Jul 14 23:34:07.230036 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:07.229994    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d" path="/var/lib/kubelet/pods/0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d/volumes"
Jul 14 23:34:20.445549 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:20.445446    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"]
Jul 14 23:34:20.446499 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:20.446431    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg" podUID="8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1" containerName="istio-proxy" containerID="cri-o://d8b47475289218b9f40ed12a5ede008c79f00cf85d860d1dece278b1cc0bfbeb" gracePeriod=30
Jul 14 23:34:21.329592 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.329554    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"]
Jul 14 23:34:21.329912 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.329899    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1fa8be6-3275-4833-b100-bddb9d7a14a2" containerName="main"
Jul 14 23:34:21.329967 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.329917    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1fa8be6-3275-4833-b100-bddb9d7a14a2" containerName="main"
Jul 14 23:34:21.329967 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.329930    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8ff88dc-4677-41a6-bc63-82924267fdd8" containerName="main"
Jul 14 23:34:21.329967 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.329936    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8ff88dc-4677-41a6-bc63-82924267fdd8" containerName="main"
Jul 14 23:34:21.329967 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.329946    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea4261ea-5839-4c00-bc5b-6e33d605ffff" containerName="maas-api"
Jul 14 23:34:21.329967 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.329953    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea4261ea-5839-4c00-bc5b-6e33d605ffff" containerName="maas-api"
Jul 14 23:34:21.329967 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.329961    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d" containerName="istio-proxy"
Jul 14 23:34:21.329967 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.329966    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d" containerName="istio-proxy"
Jul 14 23:34:21.330172 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.329976    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d" containerName="maas-api"
Jul 14 23:34:21.330172 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.329981    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d" containerName="maas-api"
Jul 14 23:34:21.330172 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.329987    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d" containerName="istio-proxy"
Jul 14 23:34:21.330172 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.329994    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d" containerName="istio-proxy"
Jul 14 23:34:21.330172 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.330050    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c8c9b86-67b7-4fcf-97a5-0fa0ea6cc60d" containerName="maas-api"
Jul 14 23:34:21.330172 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.330058    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8ff88dc-4677-41a6-bc63-82924267fdd8" containerName="main"
Jul 14 23:34:21.330172 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.330066    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea4261ea-5839-4c00-bc5b-6e33d605ffff" containerName="maas-api"
Jul 14 23:34:21.330172 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.330074    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d" containerName="istio-proxy"
Jul 14 23:34:21.330172 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.330083    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1fa8be6-3275-4833-b100-bddb9d7a14a2" containerName="main"
Jul 14 23:34:21.335086 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.335066    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"
Jul 14 23:34:21.338317 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.338273    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-5e557a-openshift-default-dockercfg-754c6\""
Jul 14 23:34:21.344147 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.344110    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"]
Jul 14 23:34:21.383171 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.383137    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c951f6ad-edb3-4d82-a58b-137c5e6f9784-istio-token\") pod \"e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5\" (UID: \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\") " pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"
Jul 14 23:34:21.383365 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.383186    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c951f6ad-edb3-4d82-a58b-137c5e6f9784-workload-certs\") pod \"e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5\" (UID: \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\") " pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"
Jul 14 23:34:21.383365 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.383244    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c951f6ad-edb3-4d82-a58b-137c5e6f9784-istio-envoy\") pod \"e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5\" (UID: \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\") " pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"
Jul 14 23:34:21.383365 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.383302    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkxp4\" (UniqueName: \"kubernetes.io/projected/c951f6ad-edb3-4d82-a58b-137c5e6f9784-kube-api-access-mkxp4\") pod \"e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5\" (UID: \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\") " pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"
Jul 14 23:34:21.383365 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.383370    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c951f6ad-edb3-4d82-a58b-137c5e6f9784-workload-socket\") pod \"e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5\" (UID: \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\") " pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"
Jul 14 23:34:21.383549 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.383386    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c951f6ad-edb3-4d82-a58b-137c5e6f9784-istiod-ca-cert\") pod \"e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5\" (UID: \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\") " pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"
Jul 14 23:34:21.383549 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.383416    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c951f6ad-edb3-4d82-a58b-137c5e6f9784-istio-data\") pod \"e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5\" (UID: \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\") " pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"
Jul 14 23:34:21.383549 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.383434    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c951f6ad-edb3-4d82-a58b-137c5e6f9784-credential-socket\") pod \"e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5\" (UID: \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\") " pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"
Jul 14 23:34:21.383549 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.383473    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c951f6ad-edb3-4d82-a58b-137c5e6f9784-istio-podinfo\") pod \"e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5\" (UID: \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\") " pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"
Jul 14 23:34:21.484088 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.484049    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c951f6ad-edb3-4d82-a58b-137c5e6f9784-workload-socket\") pod \"e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5\" (UID: \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\") " pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"
Jul 14 23:34:21.484088 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.484087    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c951f6ad-edb3-4d82-a58b-137c5e6f9784-istiod-ca-cert\") pod \"e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5\" (UID: \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\") " pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"
Jul 14 23:34:21.484653 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.484150    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c951f6ad-edb3-4d82-a58b-137c5e6f9784-istio-data\") pod \"e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5\" (UID: \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\") " pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"
Jul 14 23:34:21.484653 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.484183    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c951f6ad-edb3-4d82-a58b-137c5e6f9784-credential-socket\") pod \"e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5\" (UID: \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\") " pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"
Jul 14 23:34:21.484653 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.484265    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c951f6ad-edb3-4d82-a58b-137c5e6f9784-istio-podinfo\") pod \"e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5\" (UID: \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\") " pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"
Jul 14 23:34:21.484653 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.484343    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c951f6ad-edb3-4d82-a58b-137c5e6f9784-istio-token\") pod \"e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5\" (UID: \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\") " pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"
Jul 14 23:34:21.484653 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.484405    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c951f6ad-edb3-4d82-a58b-137c5e6f9784-workload-certs\") pod \"e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5\" (UID: \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\") " pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"
Jul 14 23:34:21.484653 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.484450    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c951f6ad-edb3-4d82-a58b-137c5e6f9784-istio-envoy\") pod \"e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5\" (UID: \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\") " pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"
Jul 14 23:34:21.484653 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.484475    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkxp4\" (UniqueName: \"kubernetes.io/projected/c951f6ad-edb3-4d82-a58b-137c5e6f9784-kube-api-access-mkxp4\") pod \"e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5\" (UID: \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\") " pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"
Jul 14 23:34:21.484653 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.484625    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c951f6ad-edb3-4d82-a58b-137c5e6f9784-credential-socket\") pod \"e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5\" (UID: \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\") " pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"
Jul 14 23:34:21.485022 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.484659    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c951f6ad-edb3-4d82-a58b-137c5e6f9784-istio-data\") pod \"e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5\" (UID: \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\") " pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"
Jul 14 23:34:21.485022 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.484682    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c951f6ad-edb3-4d82-a58b-137c5e6f9784-workload-socket\") pod \"e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5\" (UID: \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\") " pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"
Jul 14 23:34:21.485022 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.484886    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c951f6ad-edb3-4d82-a58b-137c5e6f9784-workload-certs\") pod \"e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5\" (UID: \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\") " pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"
Jul 14 23:34:21.485022 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.484926    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c951f6ad-edb3-4d82-a58b-137c5e6f9784-istiod-ca-cert\") pod \"e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5\" (UID: \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\") " pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"
Jul 14 23:34:21.486887 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.486860    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c951f6ad-edb3-4d82-a58b-137c5e6f9784-istio-envoy\") pod \"e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5\" (UID: \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\") " pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"
Jul 14 23:34:21.487411 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.487393    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c951f6ad-edb3-4d82-a58b-137c5e6f9784-istio-podinfo\") pod \"e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5\" (UID: \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\") " pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"
Jul 14 23:34:21.493816 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.493789    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c951f6ad-edb3-4d82-a58b-137c5e6f9784-istio-token\") pod \"e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5\" (UID: \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\") " pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"
Jul 14 23:34:21.494059 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.494036    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkxp4\" (UniqueName: \"kubernetes.io/projected/c951f6ad-edb3-4d82-a58b-137c5e6f9784-kube-api-access-mkxp4\") pod \"e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5\" (UID: \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\") " pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"
Jul 14 23:34:21.647872 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.647781    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"
Jul 14 23:34:21.793618 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.793582    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"]
Jul 14 23:34:21.799661 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:34:21.799626    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc951f6ad_edb3_4d82_a58b_137c5e6f9784.slice/crio-23c7316827c98cdbbb897c4ce3ca8119fdc7b21c6fc1db055901ae4fa67cd726 WatchSource:0}: Error finding container 23c7316827c98cdbbb897c4ce3ca8119fdc7b21c6fc1db055901ae4fa67cd726: Status 404 returned error can't find the container with id 23c7316827c98cdbbb897c4ce3ca8119fdc7b21c6fc1db055901ae4fa67cd726
Jul 14 23:34:21.801628 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.801596    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:34:21.801709 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.801658    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:34:21.801709 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:21.801700    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:34:22.252200 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:22.252163    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5" event={"ID":"c951f6ad-edb3-4d82-a58b-137c5e6f9784","Type":"ContainerStarted","Data":"57b6ee23d3f96071369a2bd8814c6cfbf29ca80af87767df24b1702dd721b694"}
Jul 14 23:34:22.252444 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:22.252218    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5" event={"ID":"c951f6ad-edb3-4d82-a58b-137c5e6f9784","Type":"ContainerStarted","Data":"23c7316827c98cdbbb897c4ce3ca8119fdc7b21c6fc1db055901ae4fa67cd726"}
Jul 14 23:34:22.277904 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:22.277851    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5" podStartSLOduration=1.2778354 podStartE2EDuration="1.2778354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:34:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:34:22.275811932 +0000 UTC m=+2249.578802725" watchObservedRunningTime="2026-07-14 23:34:22.2778354 +0000 UTC m=+2249.580826188"
Jul 14 23:34:22.648740 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:22.648650    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"
Jul 14 23:34:22.653615 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:22.653591    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"
Jul 14 23:34:23.256688 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:23.256651    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"
Jul 14 23:34:23.257798 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:23.257771    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"
Jul 14 23:34:24.292841 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:24.292803    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5e557a-6f4567d887-4pk5b"]
Jul 14 23:34:24.293437 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:24.293415    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8fd0204-0adb-46d5-b4f5-bf15ca9bbc5d" containerName="istio-proxy"
Jul 14 23:34:24.296807 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:24.296784    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e557a-6f4567d887-4pk5b"
Jul 14 23:34:24.299786 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:24.299758    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 23:34:24.299905 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:24.299760    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 23:34:24.301098 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:24.301078    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7jvtc\""
Jul 14 23:34:24.301219 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:24.301080    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-5e557a-serving-cert\""
Jul 14 23:34:24.306669 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:24.306645    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5e557a-6f4567d887-4pk5b"]
Jul 14 23:34:24.411091 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:24.411053    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02457bcd-3ecf-45ea-99b2-154c20e918f3-maas-api-tls\") pod \"maas-api-e2e-api-a-5e557a-6f4567d887-4pk5b\" (UID: \"02457bcd-3ecf-45ea-99b2-154c20e918f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e557a-6f4567d887-4pk5b"
Jul 14 23:34:24.411275 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:24.411147    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02457bcd-3ecf-45ea-99b2-154c20e918f3-openshift-service-ca\") pod \"maas-api-e2e-api-a-5e557a-6f4567d887-4pk5b\" (UID: \"02457bcd-3ecf-45ea-99b2-154c20e918f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e557a-6f4567d887-4pk5b"
Jul 14 23:34:24.411275 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:24.411207    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6m9r\" (UniqueName: \"kubernetes.io/projected/02457bcd-3ecf-45ea-99b2-154c20e918f3-kube-api-access-f6m9r\") pod \"maas-api-e2e-api-a-5e557a-6f4567d887-4pk5b\" (UID: \"02457bcd-3ecf-45ea-99b2-154c20e918f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e557a-6f4567d887-4pk5b"
Jul 14 23:34:24.512170 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:24.512139    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02457bcd-3ecf-45ea-99b2-154c20e918f3-openshift-service-ca\") pod \"maas-api-e2e-api-a-5e557a-6f4567d887-4pk5b\" (UID: \"02457bcd-3ecf-45ea-99b2-154c20e918f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e557a-6f4567d887-4pk5b"
Jul 14 23:34:24.512396 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:24.512194    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6m9r\" (UniqueName: \"kubernetes.io/projected/02457bcd-3ecf-45ea-99b2-154c20e918f3-kube-api-access-f6m9r\") pod \"maas-api-e2e-api-a-5e557a-6f4567d887-4pk5b\" (UID: \"02457bcd-3ecf-45ea-99b2-154c20e918f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e557a-6f4567d887-4pk5b"
Jul 14 23:34:24.512396 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:24.512226    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02457bcd-3ecf-45ea-99b2-154c20e918f3-maas-api-tls\") pod \"maas-api-e2e-api-a-5e557a-6f4567d887-4pk5b\" (UID: \"02457bcd-3ecf-45ea-99b2-154c20e918f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e557a-6f4567d887-4pk5b"
Jul 14 23:34:24.512859 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:24.512834    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02457bcd-3ecf-45ea-99b2-154c20e918f3-openshift-service-ca\") pod \"maas-api-e2e-api-a-5e557a-6f4567d887-4pk5b\" (UID: \"02457bcd-3ecf-45ea-99b2-154c20e918f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e557a-6f4567d887-4pk5b"
Jul 14 23:34:24.514720 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:24.514701    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02457bcd-3ecf-45ea-99b2-154c20e918f3-maas-api-tls\") pod \"maas-api-e2e-api-a-5e557a-6f4567d887-4pk5b\" (UID: \"02457bcd-3ecf-45ea-99b2-154c20e918f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e557a-6f4567d887-4pk5b"
Jul 14 23:34:24.525027 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:24.525001    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6m9r\" (UniqueName: \"kubernetes.io/projected/02457bcd-3ecf-45ea-99b2-154c20e918f3-kube-api-access-f6m9r\") pod \"maas-api-e2e-api-a-5e557a-6f4567d887-4pk5b\" (UID: \"02457bcd-3ecf-45ea-99b2-154c20e918f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e557a-6f4567d887-4pk5b"
Jul 14 23:34:24.609683 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:24.609597    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e557a-6f4567d887-4pk5b"
Jul 14 23:34:24.754180 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:24.754154    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5e557a-6f4567d887-4pk5b"]
Jul 14 23:34:24.756683 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:34:24.756652    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02457bcd_3ecf_45ea_99b2_154c20e918f3.slice/crio-aeb143a0567fb7299cdae7783ed3162708d419d91a6734fe38e89597909a1b5e WatchSource:0}: Error finding container aeb143a0567fb7299cdae7783ed3162708d419d91a6734fe38e89597909a1b5e: Status 404 returned error can't find the container with id aeb143a0567fb7299cdae7783ed3162708d419d91a6734fe38e89597909a1b5e
Jul 14 23:34:24.758029 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:24.758011    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 23:34:25.268123 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:25.268083    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e557a-6f4567d887-4pk5b" event={"ID":"02457bcd-3ecf-45ea-99b2-154c20e918f3","Type":"ContainerStarted","Data":"aeb143a0567fb7299cdae7783ed3162708d419d91a6734fe38e89597909a1b5e"}
Jul 14 23:34:25.706989 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:25.706965    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"
Jul 14 23:34:25.823311 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:25.823193    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-workload-certs\") pod \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\" (UID: \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\") "
Jul 14 23:34:25.823311 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:25.823232    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-istio-podinfo\") pod \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\" (UID: \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\") "
Jul 14 23:34:25.823311 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:25.823304    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-istiod-ca-cert\") pod \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\" (UID: \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\") "
Jul 14 23:34:25.823603 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:25.823330    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-workload-socket\") pod \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\" (UID: \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\") "
Jul 14 23:34:25.823603 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:25.823358    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-istio-envoy\") pod \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\" (UID: \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\") "
Jul 14 23:34:25.823603 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:25.823459    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-istio-data\") pod \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\" (UID: \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\") "
Jul 14 23:34:25.823603 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:25.823495    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-istio-token\") pod \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\" (UID: \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\") "
Jul 14 23:34:25.823603 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:25.823529    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-credential-socket\") pod \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\" (UID: \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\") "
Jul 14 23:34:25.823603 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:25.823544    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1" (UID: "8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:34:25.823603 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:25.823578    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmghk\" (UniqueName: \"kubernetes.io/projected/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-kube-api-access-zmghk\") pod \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\" (UID: \"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1\") "
Jul 14 23:34:25.823971 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:25.823727    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1" (UID: "8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:34:25.823971 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:25.823765    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1" (UID: "8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 23:34:25.823971 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:25.823884    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-workload-certs\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:34:25.823971 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:25.823901    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-istiod-ca-cert\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:34:25.823971 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:25.823915    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-credential-socket\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:34:25.823971 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:25.823918    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1" (UID: "8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:34:25.824566 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:25.824464    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-istio-data" (OuterVolumeSpecName: "istio-data") pod "8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1" (UID: "8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:34:25.825838 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:25.825807    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1" (UID: "8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:34:25.825962 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:25.825871    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-istio-token" (OuterVolumeSpecName: "istio-token") pod "8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1" (UID: "8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:34:25.825962 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:25.825918    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1" (UID: "8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 23:34:25.825962 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:25.825927    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-kube-api-access-zmghk" (OuterVolumeSpecName: "kube-api-access-zmghk") pod "8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1" (UID: "8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1"). InnerVolumeSpecName "kube-api-access-zmghk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:34:25.924843 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:25.924805    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-workload-socket\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:34:25.924843 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:25.924837    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-istio-envoy\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:34:25.924843 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:25.924846    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-istio-data\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:34:25.924843 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:25.924853    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-istio-token\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:34:25.925104 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:25.924861    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmghk\" (UniqueName: \"kubernetes.io/projected/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-kube-api-access-zmghk\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:34:25.925104 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:25.924869    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1-istio-podinfo\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:34:26.277260 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:26.277218    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e557a-6f4567d887-4pk5b" event={"ID":"02457bcd-3ecf-45ea-99b2-154c20e918f3","Type":"ContainerStarted","Data":"e7600f3c2ebf08849265a226b5fe71adba55fc1dc371cb167c49c899c990a210"}
Jul 14 23:34:26.277491 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:26.277351    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e557a-6f4567d887-4pk5b"
Jul 14 23:34:26.279083 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:26.279054    2563 generic.go:356] "Generic (PLEG): container finished" podID="8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1" containerID="d8b47475289218b9f40ed12a5ede008c79f00cf85d860d1dece278b1cc0bfbeb" exitCode=0
Jul 14 23:34:26.279205 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:26.279094    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg" event={"ID":"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1","Type":"ContainerDied","Data":"d8b47475289218b9f40ed12a5ede008c79f00cf85d860d1dece278b1cc0bfbeb"}
Jul 14 23:34:26.279205 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:26.279129    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg" event={"ID":"8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1","Type":"ContainerDied","Data":"dc54646b3b1e6a8190320965cfc74c6f3607ea741912bff425306a77677ddacd"}
Jul 14 23:34:26.279205 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:26.279150    2563 scope.go:117] "RemoveContainer" containerID="d8b47475289218b9f40ed12a5ede008c79f00cf85d860d1dece278b1cc0bfbeb"
Jul 14 23:34:26.279205 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:26.279154    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"
Jul 14 23:34:26.288401 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:26.288383    2563 scope.go:117] "RemoveContainer" containerID="37a7d1a4eadff759564908cef547ad07b4fa154de8e6c4da30e051bbce2a687d"
Jul 14 23:34:26.295395 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:26.295346    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e557a-6f4567d887-4pk5b" podStartSLOduration=1.517696633 podStartE2EDuration="2.295328812s" totalImagesPullingTime="777.632179ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:34:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:34:26.293371445 +0000 UTC m=+2253.596362238" watchObservedRunningTime="2026-07-14 23:34:26.295328812 +0000 UTC m=+2253.598319605"
Jul 14 23:34:26.297021 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:26.297004    2563 scope.go:117] "RemoveContainer" containerID="d8b47475289218b9f40ed12a5ede008c79f00cf85d860d1dece278b1cc0bfbeb"
Jul 14 23:34:26.297316 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:34:26.297264    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8b47475289218b9f40ed12a5ede008c79f00cf85d860d1dece278b1cc0bfbeb\": container with ID starting with d8b47475289218b9f40ed12a5ede008c79f00cf85d860d1dece278b1cc0bfbeb not found: ID does not exist" containerID="d8b47475289218b9f40ed12a5ede008c79f00cf85d860d1dece278b1cc0bfbeb"
Jul 14 23:34:26.297398 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:26.297328    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8b47475289218b9f40ed12a5ede008c79f00cf85d860d1dece278b1cc0bfbeb"} err="failed to get container status \"d8b47475289218b9f40ed12a5ede008c79f00cf85d860d1dece278b1cc0bfbeb\": rpc error: code = NotFound desc = could not find container \"d8b47475289218b9f40ed12a5ede008c79f00cf85d860d1dece278b1cc0bfbeb\": container with ID starting with d8b47475289218b9f40ed12a5ede008c79f00cf85d860d1dece278b1cc0bfbeb not found: ID does not exist"
Jul 14 23:34:26.297398 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:26.297353    2563 scope.go:117] "RemoveContainer" containerID="37a7d1a4eadff759564908cef547ad07b4fa154de8e6c4da30e051bbce2a687d"
Jul 14 23:34:26.297596 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:34:26.297578    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37a7d1a4eadff759564908cef547ad07b4fa154de8e6c4da30e051bbce2a687d\": container with ID starting with 37a7d1a4eadff759564908cef547ad07b4fa154de8e6c4da30e051bbce2a687d not found: ID does not exist" containerID="37a7d1a4eadff759564908cef547ad07b4fa154de8e6c4da30e051bbce2a687d"
Jul 14 23:34:26.297636 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:26.297602    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37a7d1a4eadff759564908cef547ad07b4fa154de8e6c4da30e051bbce2a687d"} err="failed to get container status \"37a7d1a4eadff759564908cef547ad07b4fa154de8e6c4da30e051bbce2a687d\": rpc error: code = NotFound desc = could not find container \"37a7d1a4eadff759564908cef547ad07b4fa154de8e6c4da30e051bbce2a687d\": container with ID starting with 37a7d1a4eadff759564908cef547ad07b4fa154de8e6c4da30e051bbce2a687d not found: ID does not exist"
Jul 14 23:34:26.309205 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:26.309178    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"]
Jul 14 23:34:26.315955 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:26.315933    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-4b47a7-openshift-default-66fb74c965-7x6cg"]
Jul 14 23:34:27.229887 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:27.229857    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1" path="/var/lib/kubelet/pods/8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1/volumes"
Jul 14 23:34:29.470064 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:29.470021    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"]
Jul 14 23:34:29.470627 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:29.470425    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1" containerName="istio-proxy"
Jul 14 23:34:29.470627 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:29.470440    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1" containerName="istio-proxy"
Jul 14 23:34:29.470627 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:29.470449    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1" containerName="istio-proxy"
Jul 14 23:34:29.470627 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:29.470454    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1" containerName="istio-proxy"
Jul 14 23:34:29.470627 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:29.470508    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1" containerName="istio-proxy"
Jul 14 23:34:29.470627 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:29.470515    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aa9c0e7-abd5-4b5e-bf10-49725d83d9c1" containerName="istio-proxy"
Jul 14 23:34:29.473221 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:29.473202    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"
Jul 14 23:34:29.476187 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:29.476160    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-a8299a-openshift-default-dockercfg-dvr9l\""
Jul 14 23:34:29.484715 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:29.484681    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"]
Jul 14 23:34:29.555857 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:29.555814    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69l84\" (UniqueName: \"kubernetes.io/projected/304ab33d-29dd-4f39-95cb-03bc227b6634-kube-api-access-69l84\") pod \"e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd\" (UID: \"304ab33d-29dd-4f39-95cb-03bc227b6634\") " pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"
Jul 14 23:34:29.556061 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:29.555880    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/304ab33d-29dd-4f39-95cb-03bc227b6634-istio-data\") pod \"e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd\" (UID: \"304ab33d-29dd-4f39-95cb-03bc227b6634\") " pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"
Jul 14 23:34:29.556061 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:29.555981    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/304ab33d-29dd-4f39-95cb-03bc227b6634-credential-socket\") pod \"e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd\" (UID: \"304ab33d-29dd-4f39-95cb-03bc227b6634\") " pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"
Jul 14 23:34:29.556061 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:29.556018    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/304ab33d-29dd-4f39-95cb-03bc227b6634-istiod-ca-cert\") pod \"e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd\" (UID: \"304ab33d-29dd-4f39-95cb-03bc227b6634\") " pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"
Jul 14 23:34:29.556061 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:29.556049    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/304ab33d-29dd-4f39-95cb-03bc227b6634-workload-certs\") pod \"e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd\" (UID: \"304ab33d-29dd-4f39-95cb-03bc227b6634\") " pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"
Jul 14 23:34:29.556264 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:29.556076    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/304ab33d-29dd-4f39-95cb-03bc227b6634-istio-envoy\") pod \"e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd\" (UID: \"304ab33d-29dd-4f39-95cb-03bc227b6634\") " pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"
Jul 14 23:34:29.556264 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:29.556103    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/304ab33d-29dd-4f39-95cb-03bc227b6634-istio-podinfo\") pod \"e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd\" (UID: \"304ab33d-29dd-4f39-95cb-03bc227b6634\") " pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"
Jul 14 23:34:29.556264 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:29.556139    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/304ab33d-29dd-4f39-95cb-03bc227b6634-workload-socket\") pod \"e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd\" (UID: \"304ab33d-29dd-4f39-95cb-03bc227b6634\") " pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"
Jul 14 23:34:29.556264 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:29.556198    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/304ab33d-29dd-4f39-95cb-03bc227b6634-istio-token\") pod \"e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd\" (UID: \"304ab33d-29dd-4f39-95cb-03bc227b6634\") " pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"
Jul 14 23:34:29.657766 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:29.657713    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/304ab33d-29dd-4f39-95cb-03bc227b6634-istio-token\") pod \"e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd\" (UID: \"304ab33d-29dd-4f39-95cb-03bc227b6634\") " pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"
Jul 14 23:34:29.657988 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:29.657784    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69l84\" (UniqueName: \"kubernetes.io/projected/304ab33d-29dd-4f39-95cb-03bc227b6634-kube-api-access-69l84\") pod \"e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd\" (UID: \"304ab33d-29dd-4f39-95cb-03bc227b6634\") " pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"
Jul 14 23:34:29.657988 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:29.657850    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/304ab33d-29dd-4f39-95cb-03bc227b6634-istio-data\") pod \"e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd\" (UID: \"304ab33d-29dd-4f39-95cb-03bc227b6634\") " pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"
Jul 14 23:34:29.657988 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:29.657903    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/304ab33d-29dd-4f39-95cb-03bc227b6634-credential-socket\") pod \"e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd\" (UID: \"304ab33d-29dd-4f39-95cb-03bc227b6634\") " pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"
Jul 14 23:34:29.657988 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:29.657928    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/304ab33d-29dd-4f39-95cb-03bc227b6634-istiod-ca-cert\") pod \"e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd\" (UID: \"304ab33d-29dd-4f39-95cb-03bc227b6634\") " pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"
Jul 14 23:34:29.657988 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:29.657959    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/304ab33d-29dd-4f39-95cb-03bc227b6634-workload-certs\") pod \"e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd\" (UID: \"304ab33d-29dd-4f39-95cb-03bc227b6634\") " pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"
Jul 14 23:34:29.657988 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:29.657985    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/304ab33d-29dd-4f39-95cb-03bc227b6634-istio-envoy\") pod \"e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd\" (UID: \"304ab33d-29dd-4f39-95cb-03bc227b6634\") " pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"
Jul 14 23:34:29.658336 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:29.658013    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/304ab33d-29dd-4f39-95cb-03bc227b6634-istio-podinfo\") pod \"e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd\" (UID: \"304ab33d-29dd-4f39-95cb-03bc227b6634\") " pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"
Jul 14 23:34:29.658336 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:29.658049    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/304ab33d-29dd-4f39-95cb-03bc227b6634-workload-socket\") pod \"e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd\" (UID: \"304ab33d-29dd-4f39-95cb-03bc227b6634\") " pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"
Jul 14 23:34:29.658563 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:29.658533    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/304ab33d-29dd-4f39-95cb-03bc227b6634-istio-data\") pod \"e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd\" (UID: \"304ab33d-29dd-4f39-95cb-03bc227b6634\") " pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"
Jul 14 23:34:29.658664 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:29.658570    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/304ab33d-29dd-4f39-95cb-03bc227b6634-workload-socket\") pod \"e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd\" (UID: \"304ab33d-29dd-4f39-95cb-03bc227b6634\") " pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"
Jul 14 23:34:29.658818 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:29.658786    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/304ab33d-29dd-4f39-95cb-03bc227b6634-istiod-ca-cert\") pod \"e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd\" (UID: \"304ab33d-29dd-4f39-95cb-03bc227b6634\") " pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"
Jul 14 23:34:29.658818 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:29.658807    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/304ab33d-29dd-4f39-95cb-03bc227b6634-workload-certs\") pod \"e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd\" (UID: \"304ab33d-29dd-4f39-95cb-03bc227b6634\") " pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"
Jul 14 23:34:29.659007 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:29.658873    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/304ab33d-29dd-4f39-95cb-03bc227b6634-credential-socket\") pod \"e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd\" (UID: \"304ab33d-29dd-4f39-95cb-03bc227b6634\") " pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"
Jul 14 23:34:29.661102 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:29.661066    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/304ab33d-29dd-4f39-95cb-03bc227b6634-istio-envoy\") pod \"e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd\" (UID: \"304ab33d-29dd-4f39-95cb-03bc227b6634\") " pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"
Jul 14 23:34:29.661262 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:29.661245    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/304ab33d-29dd-4f39-95cb-03bc227b6634-istio-podinfo\") pod \"e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd\" (UID: \"304ab33d-29dd-4f39-95cb-03bc227b6634\") " pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"
Jul 14 23:34:29.667145 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:29.667112    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69l84\" (UniqueName: \"kubernetes.io/projected/304ab33d-29dd-4f39-95cb-03bc227b6634-kube-api-access-69l84\") pod \"e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd\" (UID: \"304ab33d-29dd-4f39-95cb-03bc227b6634\") " pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"
Jul 14 23:34:29.667320 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:29.667118    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/304ab33d-29dd-4f39-95cb-03bc227b6634-istio-token\") pod \"e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd\" (UID: \"304ab33d-29dd-4f39-95cb-03bc227b6634\") " pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"
Jul 14 23:34:29.786459 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:29.786414    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"
Jul 14 23:34:29.943937 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:29.943909    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"]
Jul 14 23:34:29.946717 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:34:29.946681    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod304ab33d_29dd_4f39_95cb_03bc227b6634.slice/crio-99c2c7cadf3f0bb00150a5798e4a418059ae79de63830ed84a9e7bf371741214 WatchSource:0}: Error finding container 99c2c7cadf3f0bb00150a5798e4a418059ae79de63830ed84a9e7bf371741214: Status 404 returned error can't find the container with id 99c2c7cadf3f0bb00150a5798e4a418059ae79de63830ed84a9e7bf371741214
Jul 14 23:34:29.949655 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:29.949613    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:34:29.949778 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:29.949706    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:34:29.949778 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:29.949744    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:34:30.299794 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:30.299750    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd" event={"ID":"304ab33d-29dd-4f39-95cb-03bc227b6634","Type":"ContainerStarted","Data":"637107c0f2bf1a110fd3e35dadb629d24beffeaad510e811a4bec00b970dd509"}
Jul 14 23:34:30.299974 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:30.299820    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd" event={"ID":"304ab33d-29dd-4f39-95cb-03bc227b6634","Type":"ContainerStarted","Data":"99c2c7cadf3f0bb00150a5798e4a418059ae79de63830ed84a9e7bf371741214"}
Jul 14 23:34:30.324712 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:30.324640    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd" podStartSLOduration=1.3246157809999999 podStartE2EDuration="1.324615781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:34:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:34:30.323060937 +0000 UTC m=+2257.626051730" watchObservedRunningTime="2026-07-14 23:34:30.324615781 +0000 UTC m=+2257.627606583"
Jul 14 23:34:30.786757 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:30.786720    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"
Jul 14 23:34:31.792371 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:31.792334    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"
Jul 14 23:34:32.290638 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:32.290599    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e557a-6f4567d887-4pk5b"
Jul 14 23:34:32.309334 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:32.309276    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"
Jul 14 23:34:32.310633 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:32.310606    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"
Jul 14 23:34:37.338518 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:37.338485    2563 generic.go:356] "Generic (PLEG): container finished" podID="c951f6ad-edb3-4d82-a58b-137c5e6f9784" containerID="57b6ee23d3f96071369a2bd8814c6cfbf29ca80af87767df24b1702dd721b694" exitCode=0
Jul 14 23:34:37.338940 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:37.338567    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5" event={"ID":"c951f6ad-edb3-4d82-a58b-137c5e6f9784","Type":"ContainerDied","Data":"57b6ee23d3f96071369a2bd8814c6cfbf29ca80af87767df24b1702dd721b694"}
Jul 14 23:34:37.338940 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:37.338929    2563 scope.go:117] "RemoveContainer" containerID="57b6ee23d3f96071369a2bd8814c6cfbf29ca80af87767df24b1702dd721b694"
Jul 14 23:34:37.339923 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:37.339895    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:34:37.340010 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:37.339981    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:34:37.340047 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:37.340037    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:34:38.257205 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:38.257164    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"
Jul 14 23:34:38.349087 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:38.349047    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5" event={"ID":"c951f6ad-edb3-4d82-a58b-137c5e6f9784","Type":"ContainerStarted","Data":"93bee8415cc05d92b73de914ac822c77d1e01ec380c7a0e0c42ab4ada03da9c8"}
Jul 14 23:34:38.648190 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:38.648096    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"
Jul 14 23:34:38.649093 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:38.649069    2563 patch_prober.go:28] interesting pod/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Jul 14 23:34:38.649149 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:38.649116    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5" podUID="c951f6ad-edb3-4d82-a58b-137c5e6f9784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Jul 14 23:34:39.648614 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:39.648575    2563 patch_prober.go:28] interesting pod/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Jul 14 23:34:39.649045 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:39.648644    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5" podUID="c951f6ad-edb3-4d82-a58b-137c5e6f9784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Jul 14 23:34:40.648663 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:40.648615    2563 patch_prober.go:28] interesting pod/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Jul 14 23:34:40.649186 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:40.648705    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5" podUID="c951f6ad-edb3-4d82-a58b-137c5e6f9784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Jul 14 23:34:41.649723 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:41.649190    2563 patch_prober.go:28] interesting pod/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Jul 14 23:34:41.649723 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:41.649255    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5" podUID="c951f6ad-edb3-4d82-a58b-137c5e6f9784" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Jul 14 23:34:42.654669 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:42.654639    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"
Jul 14 23:34:42.655080 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:42.654970    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"
Jul 14 23:34:42.655793 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:42.655766    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"
Jul 14 23:34:47.311313 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:47.311210    2563 patch_prober.go:28] interesting pod/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Jul 14 23:34:47.311716 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:47.311275    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd" podUID="304ab33d-29dd-4f39-95cb-03bc227b6634" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Jul 14 23:34:47.396822 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:47.396726    2563 generic.go:356] "Generic (PLEG): container finished" podID="304ab33d-29dd-4f39-95cb-03bc227b6634" containerID="637107c0f2bf1a110fd3e35dadb629d24beffeaad510e811a4bec00b970dd509" exitCode=0
Jul 14 23:34:47.396822 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:47.396805    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd" event={"ID":"304ab33d-29dd-4f39-95cb-03bc227b6634","Type":"ContainerDied","Data":"637107c0f2bf1a110fd3e35dadb629d24beffeaad510e811a4bec00b970dd509"}
Jul 14 23:34:47.397228 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:47.397214    2563 scope.go:117] "RemoveContainer" containerID="637107c0f2bf1a110fd3e35dadb629d24beffeaad510e811a4bec00b970dd509"
Jul 14 23:34:47.398231 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:47.398203    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:34:47.398312 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:47.398277    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:34:47.398365 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:47.398338    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:34:48.403858 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:48.403820    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd" event={"ID":"304ab33d-29dd-4f39-95cb-03bc227b6634","Type":"ContainerStarted","Data":"4ccd288011da8084ee136a9a8d77f31736c8699e7a350e2a1bd1cea78c23ff41"}
Jul 14 23:34:48.787031 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:48.786986    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"
Jul 14 23:34:48.787264 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:48.787240    2563 patch_prober.go:28] interesting pod/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Jul 14 23:34:48.787359 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:48.787326    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd" podUID="304ab33d-29dd-4f39-95cb-03bc227b6634" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Jul 14 23:34:49.787606 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:49.787557    2563 patch_prober.go:28] interesting pod/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Jul 14 23:34:49.788104 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:49.787625    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd" podUID="304ab33d-29dd-4f39-95cb-03bc227b6634" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Jul 14 23:34:50.787402 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:50.787360    2563 patch_prober.go:28] interesting pod/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Jul 14 23:34:50.787608 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:50.787435    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd" podUID="304ab33d-29dd-4f39-95cb-03bc227b6634" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Jul 14 23:34:51.787021 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:51.786977    2563 patch_prober.go:28] interesting pod/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Jul 14 23:34:51.787501 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:51.787042    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd" podUID="304ab33d-29dd-4f39-95cb-03bc227b6634" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Jul 14 23:34:52.792135 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:52.792096    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"
Jul 14 23:34:52.792676 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:52.792543    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"
Jul 14 23:34:52.793522 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:52.793497    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"
Jul 14 23:34:55.296386 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:55.296341    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5e557a-6f4567d887-4pk5b"]
Jul 14 23:34:55.296821 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:55.296615    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e557a-6f4567d887-4pk5b" podUID="02457bcd-3ecf-45ea-99b2-154c20e918f3" containerName="maas-api" containerID="cri-o://e7600f3c2ebf08849265a226b5fe71adba55fc1dc371cb167c49c899c990a210" gracePeriod=30
Jul 14 23:34:55.445259 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:55.445209    2563 generic.go:356] "Generic (PLEG): container finished" podID="02457bcd-3ecf-45ea-99b2-154c20e918f3" containerID="e7600f3c2ebf08849265a226b5fe71adba55fc1dc371cb167c49c899c990a210" exitCode=0
Jul 14 23:34:55.445460 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:55.445315    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e557a-6f4567d887-4pk5b" event={"ID":"02457bcd-3ecf-45ea-99b2-154c20e918f3","Type":"ContainerDied","Data":"e7600f3c2ebf08849265a226b5fe71adba55fc1dc371cb167c49c899c990a210"}
Jul 14 23:34:55.561773 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:55.561706    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e557a-6f4567d887-4pk5b"
Jul 14 23:34:55.701171 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:55.701123    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6m9r\" (UniqueName: \"kubernetes.io/projected/02457bcd-3ecf-45ea-99b2-154c20e918f3-kube-api-access-f6m9r\") pod \"02457bcd-3ecf-45ea-99b2-154c20e918f3\" (UID: \"02457bcd-3ecf-45ea-99b2-154c20e918f3\") "
Jul 14 23:34:55.701406 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:55.701212    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02457bcd-3ecf-45ea-99b2-154c20e918f3-maas-api-tls\") pod \"02457bcd-3ecf-45ea-99b2-154c20e918f3\" (UID: \"02457bcd-3ecf-45ea-99b2-154c20e918f3\") "
Jul 14 23:34:55.701406 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:55.701239    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02457bcd-3ecf-45ea-99b2-154c20e918f3-openshift-service-ca\") pod \"02457bcd-3ecf-45ea-99b2-154c20e918f3\" (UID: \"02457bcd-3ecf-45ea-99b2-154c20e918f3\") "
Jul 14 23:34:55.701721 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:55.701689    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02457bcd-3ecf-45ea-99b2-154c20e918f3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "02457bcd-3ecf-45ea-99b2-154c20e918f3" (UID: "02457bcd-3ecf-45ea-99b2-154c20e918f3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 23:34:55.703816 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:55.703786    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02457bcd-3ecf-45ea-99b2-154c20e918f3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "02457bcd-3ecf-45ea-99b2-154c20e918f3" (UID: "02457bcd-3ecf-45ea-99b2-154c20e918f3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 23:34:55.703902 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:55.703788    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02457bcd-3ecf-45ea-99b2-154c20e918f3-kube-api-access-f6m9r" (OuterVolumeSpecName: "kube-api-access-f6m9r") pod "02457bcd-3ecf-45ea-99b2-154c20e918f3" (UID: "02457bcd-3ecf-45ea-99b2-154c20e918f3"). InnerVolumeSpecName "kube-api-access-f6m9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:34:55.802700 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:55.802654    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6m9r\" (UniqueName: \"kubernetes.io/projected/02457bcd-3ecf-45ea-99b2-154c20e918f3-kube-api-access-f6m9r\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:34:55.802700 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:55.802696    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02457bcd-3ecf-45ea-99b2-154c20e918f3-maas-api-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:34:55.802700 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:55.802707    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02457bcd-3ecf-45ea-99b2-154c20e918f3-openshift-service-ca\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:34:56.452226 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:56.452191    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e557a-6f4567d887-4pk5b"
Jul 14 23:34:56.452857 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:56.452236    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e557a-6f4567d887-4pk5b" event={"ID":"02457bcd-3ecf-45ea-99b2-154c20e918f3","Type":"ContainerDied","Data":"aeb143a0567fb7299cdae7783ed3162708d419d91a6734fe38e89597909a1b5e"}
Jul 14 23:34:56.452857 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:56.452340    2563 scope.go:117] "RemoveContainer" containerID="e7600f3c2ebf08849265a226b5fe71adba55fc1dc371cb167c49c899c990a210"
Jul 14 23:34:56.477243 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:56.477198    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5e557a-6f4567d887-4pk5b"]
Jul 14 23:34:56.482632 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:56.482592    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5e557a-6f4567d887-4pk5b"]
Jul 14 23:34:57.230352 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:34:57.230263    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02457bcd-3ecf-45ea-99b2-154c20e918f3" path="/var/lib/kubelet/pods/02457bcd-3ecf-45ea-99b2-154c20e918f3/volumes"
Jul 14 23:35:10.618137 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:10.618098    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"]
Jul 14 23:35:10.618686 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:10.618382    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5" podUID="c951f6ad-edb3-4d82-a58b-137c5e6f9784" containerName="istio-proxy" containerID="cri-o://93bee8415cc05d92b73de914ac822c77d1e01ec380c7a0e0c42ab4ada03da9c8" gracePeriod=30
Jul 14 23:35:12.657123 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:12.657075    2563 patch_prober.go:28] interesting pod/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 14 23:35:12.657610 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:12.657164    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5" podUID="c951f6ad-edb3-4d82-a58b-137c5e6f9784" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 14 23:35:15.880059 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:15.880031    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"
Jul 14 23:35:15.984972 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:15.984851    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c951f6ad-edb3-4d82-a58b-137c5e6f9784-workload-socket\") pod \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\" (UID: \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\") "
Jul 14 23:35:15.984972 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:15.984911    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c951f6ad-edb3-4d82-a58b-137c5e6f9784-istio-data\") pod \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\" (UID: \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\") "
Jul 14 23:35:15.984972 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:15.984941    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c951f6ad-edb3-4d82-a58b-137c5e6f9784-credential-socket\") pod \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\" (UID: \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\") "
Jul 14 23:35:15.984972 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:15.984978    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mkxp4\" (UniqueName: \"kubernetes.io/projected/c951f6ad-edb3-4d82-a58b-137c5e6f9784-kube-api-access-mkxp4\") pod \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\" (UID: \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\") "
Jul 14 23:35:15.985331 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:15.985026    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c951f6ad-edb3-4d82-a58b-137c5e6f9784-istiod-ca-cert\") pod \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\" (UID: \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\") "
Jul 14 23:35:15.985331 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:15.985054    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c951f6ad-edb3-4d82-a58b-137c5e6f9784-istio-token\") pod \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\" (UID: \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\") "
Jul 14 23:35:15.985331 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:15.985167    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c951f6ad-edb3-4d82-a58b-137c5e6f9784-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c951f6ad-edb3-4d82-a58b-137c5e6f9784" (UID: "c951f6ad-edb3-4d82-a58b-137c5e6f9784"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:35:15.985331 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:15.985214    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c951f6ad-edb3-4d82-a58b-137c5e6f9784-istio-podinfo\") pod \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\" (UID: \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\") "
Jul 14 23:35:15.985331 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:15.985312    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c951f6ad-edb3-4d82-a58b-137c5e6f9784-workload-certs\") pod \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\" (UID: \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\") "
Jul 14 23:35:15.985578 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:15.985338    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c951f6ad-edb3-4d82-a58b-137c5e6f9784-istio-envoy\") pod \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\" (UID: \"c951f6ad-edb3-4d82-a58b-137c5e6f9784\") "
Jul 14 23:35:15.985578 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:15.985385    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c951f6ad-edb3-4d82-a58b-137c5e6f9784-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c951f6ad-edb3-4d82-a58b-137c5e6f9784" (UID: "c951f6ad-edb3-4d82-a58b-137c5e6f9784"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:35:15.985578 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:15.985521    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c951f6ad-edb3-4d82-a58b-137c5e6f9784-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c951f6ad-edb3-4d82-a58b-137c5e6f9784" (UID: "c951f6ad-edb3-4d82-a58b-137c5e6f9784"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:35:15.985578 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:15.985557    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c951f6ad-edb3-4d82-a58b-137c5e6f9784-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c951f6ad-edb3-4d82-a58b-137c5e6f9784" (UID: "c951f6ad-edb3-4d82-a58b-137c5e6f9784"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 23:35:15.985782 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:15.985654    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c951f6ad-edb3-4d82-a58b-137c5e6f9784-workload-certs\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:35:15.985782 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:15.985675    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c951f6ad-edb3-4d82-a58b-137c5e6f9784-workload-socket\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:35:15.985782 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:15.985688    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c951f6ad-edb3-4d82-a58b-137c5e6f9784-credential-socket\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:35:15.985782 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:15.985702    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c951f6ad-edb3-4d82-a58b-137c5e6f9784-istiod-ca-cert\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:35:15.986377 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:15.986347    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c951f6ad-edb3-4d82-a58b-137c5e6f9784-istio-data" (OuterVolumeSpecName: "istio-data") pod "c951f6ad-edb3-4d82-a58b-137c5e6f9784" (UID: "c951f6ad-edb3-4d82-a58b-137c5e6f9784"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:35:15.987586 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:15.987547    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c951f6ad-edb3-4d82-a58b-137c5e6f9784-kube-api-access-mkxp4" (OuterVolumeSpecName: "kube-api-access-mkxp4") pod "c951f6ad-edb3-4d82-a58b-137c5e6f9784" (UID: "c951f6ad-edb3-4d82-a58b-137c5e6f9784"). InnerVolumeSpecName "kube-api-access-mkxp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:35:15.987586 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:15.987579    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c951f6ad-edb3-4d82-a58b-137c5e6f9784-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c951f6ad-edb3-4d82-a58b-137c5e6f9784" (UID: "c951f6ad-edb3-4d82-a58b-137c5e6f9784"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:35:15.987883 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:15.987863    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c951f6ad-edb3-4d82-a58b-137c5e6f9784-istio-token" (OuterVolumeSpecName: "istio-token") pod "c951f6ad-edb3-4d82-a58b-137c5e6f9784" (UID: "c951f6ad-edb3-4d82-a58b-137c5e6f9784"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:35:15.987988 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:15.987899    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c951f6ad-edb3-4d82-a58b-137c5e6f9784-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c951f6ad-edb3-4d82-a58b-137c5e6f9784" (UID: "c951f6ad-edb3-4d82-a58b-137c5e6f9784"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 23:35:16.086562 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:16.086514    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c951f6ad-edb3-4d82-a58b-137c5e6f9784-istio-data\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:35:16.086562 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:16.086554    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkxp4\" (UniqueName: \"kubernetes.io/projected/c951f6ad-edb3-4d82-a58b-137c5e6f9784-kube-api-access-mkxp4\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:35:16.086562 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:16.086565    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c951f6ad-edb3-4d82-a58b-137c5e6f9784-istio-token\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:35:16.086562 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:16.086574    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c951f6ad-edb3-4d82-a58b-137c5e6f9784-istio-podinfo\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:35:16.086850 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:16.086583    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c951f6ad-edb3-4d82-a58b-137c5e6f9784-istio-envoy\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:35:16.555932 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:16.555889    2563 generic.go:356] "Generic (PLEG): container finished" podID="c951f6ad-edb3-4d82-a58b-137c5e6f9784" containerID="93bee8415cc05d92b73de914ac822c77d1e01ec380c7a0e0c42ab4ada03da9c8" exitCode=0
Jul 14 23:35:16.556143 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:16.555940    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"
Jul 14 23:35:16.556143 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:16.555964    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5" event={"ID":"c951f6ad-edb3-4d82-a58b-137c5e6f9784","Type":"ContainerDied","Data":"93bee8415cc05d92b73de914ac822c77d1e01ec380c7a0e0c42ab4ada03da9c8"}
Jul 14 23:35:16.556143 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:16.556019    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5" event={"ID":"c951f6ad-edb3-4d82-a58b-137c5e6f9784","Type":"ContainerDied","Data":"23c7316827c98cdbbb897c4ce3ca8119fdc7b21c6fc1db055901ae4fa67cd726"}
Jul 14 23:35:16.556143 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:16.556049    2563 scope.go:117] "RemoveContainer" containerID="93bee8415cc05d92b73de914ac822c77d1e01ec380c7a0e0c42ab4ada03da9c8"
Jul 14 23:35:16.566389 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:16.566355    2563 scope.go:117] "RemoveContainer" containerID="57b6ee23d3f96071369a2bd8814c6cfbf29ca80af87767df24b1702dd721b694"
Jul 14 23:35:16.575421 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:16.575397    2563 scope.go:117] "RemoveContainer" containerID="93bee8415cc05d92b73de914ac822c77d1e01ec380c7a0e0c42ab4ada03da9c8"
Jul 14 23:35:16.575748 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:35:16.575727    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93bee8415cc05d92b73de914ac822c77d1e01ec380c7a0e0c42ab4ada03da9c8\": container with ID starting with 93bee8415cc05d92b73de914ac822c77d1e01ec380c7a0e0c42ab4ada03da9c8 not found: ID does not exist" containerID="93bee8415cc05d92b73de914ac822c77d1e01ec380c7a0e0c42ab4ada03da9c8"
Jul 14 23:35:16.575822 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:16.575759    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93bee8415cc05d92b73de914ac822c77d1e01ec380c7a0e0c42ab4ada03da9c8"} err="failed to get container status \"93bee8415cc05d92b73de914ac822c77d1e01ec380c7a0e0c42ab4ada03da9c8\": rpc error: code = NotFound desc = could not find container \"93bee8415cc05d92b73de914ac822c77d1e01ec380c7a0e0c42ab4ada03da9c8\": container with ID starting with 93bee8415cc05d92b73de914ac822c77d1e01ec380c7a0e0c42ab4ada03da9c8 not found: ID does not exist"
Jul 14 23:35:16.575822 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:16.575780    2563 scope.go:117] "RemoveContainer" containerID="57b6ee23d3f96071369a2bd8814c6cfbf29ca80af87767df24b1702dd721b694"
Jul 14 23:35:16.576059 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:35:16.576043    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57b6ee23d3f96071369a2bd8814c6cfbf29ca80af87767df24b1702dd721b694\": container with ID starting with 57b6ee23d3f96071369a2bd8814c6cfbf29ca80af87767df24b1702dd721b694 not found: ID does not exist" containerID="57b6ee23d3f96071369a2bd8814c6cfbf29ca80af87767df24b1702dd721b694"
Jul 14 23:35:16.576101 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:16.576063    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57b6ee23d3f96071369a2bd8814c6cfbf29ca80af87767df24b1702dd721b694"} err="failed to get container status \"57b6ee23d3f96071369a2bd8814c6cfbf29ca80af87767df24b1702dd721b694\": rpc error: code = NotFound desc = could not find container \"57b6ee23d3f96071369a2bd8814c6cfbf29ca80af87767df24b1702dd721b694\": container with ID starting with 57b6ee23d3f96071369a2bd8814c6cfbf29ca80af87767df24b1702dd721b694 not found: ID does not exist"
Jul 14 23:35:16.581021 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:16.580984    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"]
Jul 14 23:35:16.585160 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:16.585128    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-5e557a-openshift-default-6c74fc84dc-2drc5"]
Jul 14 23:35:17.229812 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:17.229773    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c951f6ad-edb3-4d82-a58b-137c5e6f9784" path="/var/lib/kubelet/pods/c951f6ad-edb3-4d82-a58b-137c5e6f9784/volumes"
Jul 14 23:35:36.742462 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:36.742414    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"]
Jul 14 23:35:36.742994 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:36.742718    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd" podUID="304ab33d-29dd-4f39-95cb-03bc227b6634" containerName="istio-proxy" containerID="cri-o://4ccd288011da8084ee136a9a8d77f31736c8699e7a350e2a1bd1cea78c23ff41" gracePeriod=30
Jul 14 23:35:37.665903 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:37.665871    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"]
Jul 14 23:35:37.666202 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:37.666189    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c951f6ad-edb3-4d82-a58b-137c5e6f9784" containerName="istio-proxy"
Jul 14 23:35:37.666253 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:37.666203    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c951f6ad-edb3-4d82-a58b-137c5e6f9784" containerName="istio-proxy"
Jul 14 23:35:37.666253 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:37.666219    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c951f6ad-edb3-4d82-a58b-137c5e6f9784" containerName="istio-proxy"
Jul 14 23:35:37.666253 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:37.666225    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c951f6ad-edb3-4d82-a58b-137c5e6f9784" containerName="istio-proxy"
Jul 14 23:35:37.666253 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:37.666233    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02457bcd-3ecf-45ea-99b2-154c20e918f3" containerName="maas-api"
Jul 14 23:35:37.666253 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:37.666239    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="02457bcd-3ecf-45ea-99b2-154c20e918f3" containerName="maas-api"
Jul 14 23:35:37.666457 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:37.666314    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="02457bcd-3ecf-45ea-99b2-154c20e918f3" containerName="maas-api"
Jul 14 23:35:37.666457 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:37.666323    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c951f6ad-edb3-4d82-a58b-137c5e6f9784" containerName="istio-proxy"
Jul 14 23:35:37.666457 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:37.666331    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c951f6ad-edb3-4d82-a58b-137c5e6f9784" containerName="istio-proxy"
Jul 14 23:35:37.669214 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:37.669196    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"
Jul 14 23:35:37.671985 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:37.671961    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-d62786-openshift-default-dockercfg-w2l4c\""
Jul 14 23:35:37.682441 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:37.682404    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"]
Jul 14 23:35:37.777694 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:37.777641    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0718434c-4640-407f-8215-872ae7c6a09d-istio-podinfo\") pod \"e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq\" (UID: \"0718434c-4640-407f-8215-872ae7c6a09d\") " pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"
Jul 14 23:35:37.777694 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:37.777697    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0718434c-4640-407f-8215-872ae7c6a09d-istio-envoy\") pod \"e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq\" (UID: \"0718434c-4640-407f-8215-872ae7c6a09d\") " pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"
Jul 14 23:35:37.778305 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:37.777721    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0718434c-4640-407f-8215-872ae7c6a09d-istio-data\") pod \"e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq\" (UID: \"0718434c-4640-407f-8215-872ae7c6a09d\") " pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"
Jul 14 23:35:37.778305 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:37.777744    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0718434c-4640-407f-8215-872ae7c6a09d-istio-token\") pod \"e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq\" (UID: \"0718434c-4640-407f-8215-872ae7c6a09d\") " pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"
Jul 14 23:35:37.778305 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:37.777913    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pst6\" (UniqueName: \"kubernetes.io/projected/0718434c-4640-407f-8215-872ae7c6a09d-kube-api-access-7pst6\") pod \"e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq\" (UID: \"0718434c-4640-407f-8215-872ae7c6a09d\") " pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"
Jul 14 23:35:37.778305 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:37.777957    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0718434c-4640-407f-8215-872ae7c6a09d-istiod-ca-cert\") pod \"e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq\" (UID: \"0718434c-4640-407f-8215-872ae7c6a09d\") " pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"
Jul 14 23:35:37.778305 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:37.778037    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0718434c-4640-407f-8215-872ae7c6a09d-workload-socket\") pod \"e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq\" (UID: \"0718434c-4640-407f-8215-872ae7c6a09d\") " pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"
Jul 14 23:35:37.778305 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:37.778086    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0718434c-4640-407f-8215-872ae7c6a09d-credential-socket\") pod \"e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq\" (UID: \"0718434c-4640-407f-8215-872ae7c6a09d\") " pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"
Jul 14 23:35:37.778305 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:37.778111    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0718434c-4640-407f-8215-872ae7c6a09d-workload-certs\") pod \"e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq\" (UID: \"0718434c-4640-407f-8215-872ae7c6a09d\") " pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"
Jul 14 23:35:37.795786 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:37.795749    2563 patch_prober.go:28] interesting pod/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 14 23:35:37.795962 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:37.795809    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd" podUID="304ab33d-29dd-4f39-95cb-03bc227b6634" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 14 23:35:37.879448 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:37.879403    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pst6\" (UniqueName: \"kubernetes.io/projected/0718434c-4640-407f-8215-872ae7c6a09d-kube-api-access-7pst6\") pod \"e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq\" (UID: \"0718434c-4640-407f-8215-872ae7c6a09d\") " pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"
Jul 14 23:35:37.879448 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:37.879449    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0718434c-4640-407f-8215-872ae7c6a09d-istiod-ca-cert\") pod \"e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq\" (UID: \"0718434c-4640-407f-8215-872ae7c6a09d\") " pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"
Jul 14 23:35:37.879697 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:37.879590    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0718434c-4640-407f-8215-872ae7c6a09d-workload-socket\") pod \"e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq\" (UID: \"0718434c-4640-407f-8215-872ae7c6a09d\") " pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"
Jul 14 23:35:37.879697 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:37.879646    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0718434c-4640-407f-8215-872ae7c6a09d-credential-socket\") pod \"e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq\" (UID: \"0718434c-4640-407f-8215-872ae7c6a09d\") " pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"
Jul 14 23:35:37.879697 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:37.879674    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0718434c-4640-407f-8215-872ae7c6a09d-workload-certs\") pod \"e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq\" (UID: \"0718434c-4640-407f-8215-872ae7c6a09d\") " pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"
Jul 14 23:35:37.879846 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:37.879747    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0718434c-4640-407f-8215-872ae7c6a09d-istio-podinfo\") pod \"e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq\" (UID: \"0718434c-4640-407f-8215-872ae7c6a09d\") " pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"
Jul 14 23:35:37.879846 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:37.879778    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0718434c-4640-407f-8215-872ae7c6a09d-istio-envoy\") pod \"e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq\" (UID: \"0718434c-4640-407f-8215-872ae7c6a09d\") " pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"
Jul 14 23:35:37.880204 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:37.879875    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0718434c-4640-407f-8215-872ae7c6a09d-istio-data\") pod \"e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq\" (UID: \"0718434c-4640-407f-8215-872ae7c6a09d\") " pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"
Jul 14 23:35:37.880355 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:37.880226    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0718434c-4640-407f-8215-872ae7c6a09d-istio-token\") pod \"e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq\" (UID: \"0718434c-4640-407f-8215-872ae7c6a09d\") " pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"
Jul 14 23:35:37.880355 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:37.880234    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0718434c-4640-407f-8215-872ae7c6a09d-istio-data\") pod \"e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq\" (UID: \"0718434c-4640-407f-8215-872ae7c6a09d\") " pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"
Jul 14 23:35:37.880355 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:37.880072    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0718434c-4640-407f-8215-872ae7c6a09d-credential-socket\") pod \"e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq\" (UID: \"0718434c-4640-407f-8215-872ae7c6a09d\") " pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"
Jul 14 23:35:37.880355 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:37.880146    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0718434c-4640-407f-8215-872ae7c6a09d-workload-certs\") pod \"e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq\" (UID: \"0718434c-4640-407f-8215-872ae7c6a09d\") " pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"
Jul 14 23:35:37.880355 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:37.880044    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0718434c-4640-407f-8215-872ae7c6a09d-workload-socket\") pod \"e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq\" (UID: \"0718434c-4640-407f-8215-872ae7c6a09d\") " pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"
Jul 14 23:35:37.880355 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:37.880172    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0718434c-4640-407f-8215-872ae7c6a09d-istiod-ca-cert\") pod \"e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq\" (UID: \"0718434c-4640-407f-8215-872ae7c6a09d\") " pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"
Jul 14 23:35:37.882212 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:37.882180    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0718434c-4640-407f-8215-872ae7c6a09d-istio-envoy\") pod \"e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq\" (UID: \"0718434c-4640-407f-8215-872ae7c6a09d\") " pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"
Jul 14 23:35:37.882525 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:37.882506    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0718434c-4640-407f-8215-872ae7c6a09d-istio-podinfo\") pod \"e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq\" (UID: \"0718434c-4640-407f-8215-872ae7c6a09d\") " pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"
Jul 14 23:35:37.887979 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:37.887945    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pst6\" (UniqueName: \"kubernetes.io/projected/0718434c-4640-407f-8215-872ae7c6a09d-kube-api-access-7pst6\") pod \"e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq\" (UID: \"0718434c-4640-407f-8215-872ae7c6a09d\") " pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"
Jul 14 23:35:37.887979 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:37.887961    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0718434c-4640-407f-8215-872ae7c6a09d-istio-token\") pod \"e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq\" (UID: \"0718434c-4640-407f-8215-872ae7c6a09d\") " pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"
Jul 14 23:35:37.980177 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:37.980075    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"
Jul 14 23:35:38.125051 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:38.125024    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"]
Jul 14 23:35:38.127726 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:35:38.127692    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0718434c_4640_407f_8215_872ae7c6a09d.slice/crio-a9deb7c538db9b02538170bdbea124d0edcca2a8c28a0d0e9152a50d174fd142 WatchSource:0}: Error finding container a9deb7c538db9b02538170bdbea124d0edcca2a8c28a0d0e9152a50d174fd142: Status 404 returned error can't find the container with id a9deb7c538db9b02538170bdbea124d0edcca2a8c28a0d0e9152a50d174fd142
Jul 14 23:35:38.129892 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:38.129855    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:35:38.129987 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:38.129925    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:35:38.129987 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:38.129958    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:35:38.667603 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:38.667563    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq" event={"ID":"0718434c-4640-407f-8215-872ae7c6a09d","Type":"ContainerStarted","Data":"eab5b6d53d10b6a1d502e072fe892c84a94db2a70aa1bdef1139620af96ba024"}
Jul 14 23:35:38.667811 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:38.667625    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq" event={"ID":"0718434c-4640-407f-8215-872ae7c6a09d","Type":"ContainerStarted","Data":"a9deb7c538db9b02538170bdbea124d0edcca2a8c28a0d0e9152a50d174fd142"}
Jul 14 23:35:38.690297 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:38.690239    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq" podStartSLOduration=1.690223254 podStartE2EDuration="1.690223254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:35:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:35:38.688702896 +0000 UTC m=+2325.991693688" watchObservedRunningTime="2026-07-14 23:35:38.690223254 +0000 UTC m=+2325.993214099"
Jul 14 23:35:38.980570 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:38.980485    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"
Jul 14 23:35:39.985976 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:39.985945    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"
Jul 14 23:35:40.677017 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:40.676980    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"
Jul 14 23:35:40.677958 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:40.677941    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"
Jul 14 23:35:40.718427 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:40.718378    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d62786-74788c9dc9-lfs6q"]
Jul 14 23:35:40.722179 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:40.722149    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d62786-74788c9dc9-lfs6q"
Jul 14 23:35:40.726501 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:40.725827    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 23:35:40.726501 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:40.725864    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-d62786-serving-cert\""
Jul 14 23:35:40.726501 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:40.726029    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7jvtc\""
Jul 14 23:35:40.726501 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:40.726094    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 23:35:40.735529 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:40.735498    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d62786-74788c9dc9-lfs6q"]
Jul 14 23:35:40.805684 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:40.805641    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c933a07f-bc78-48ac-ba27-9c59fe97a35f-maas-api-tls\") pod \"maas-api-e2e-shared-a-d62786-74788c9dc9-lfs6q\" (UID: \"c933a07f-bc78-48ac-ba27-9c59fe97a35f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d62786-74788c9dc9-lfs6q"
Jul 14 23:35:40.805897 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:40.805707    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbc7x\" (UniqueName: \"kubernetes.io/projected/c933a07f-bc78-48ac-ba27-9c59fe97a35f-kube-api-access-nbc7x\") pod \"maas-api-e2e-shared-a-d62786-74788c9dc9-lfs6q\" (UID: \"c933a07f-bc78-48ac-ba27-9c59fe97a35f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d62786-74788c9dc9-lfs6q"
Jul 14 23:35:40.805897 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:40.805853    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c933a07f-bc78-48ac-ba27-9c59fe97a35f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-d62786-74788c9dc9-lfs6q\" (UID: \"c933a07f-bc78-48ac-ba27-9c59fe97a35f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d62786-74788c9dc9-lfs6q"
Jul 14 23:35:40.906803 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:40.906765    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c933a07f-bc78-48ac-ba27-9c59fe97a35f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-d62786-74788c9dc9-lfs6q\" (UID: \"c933a07f-bc78-48ac-ba27-9c59fe97a35f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d62786-74788c9dc9-lfs6q"
Jul 14 23:35:40.906964 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:40.906812    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c933a07f-bc78-48ac-ba27-9c59fe97a35f-maas-api-tls\") pod \"maas-api-e2e-shared-a-d62786-74788c9dc9-lfs6q\" (UID: \"c933a07f-bc78-48ac-ba27-9c59fe97a35f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d62786-74788c9dc9-lfs6q"
Jul 14 23:35:40.906964 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:35:40.906910    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-d62786-serving-cert: secret "maas-api-e2e-shared-a-d62786-serving-cert" not found
Jul 14 23:35:40.906964 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:40.906930    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbc7x\" (UniqueName: \"kubernetes.io/projected/c933a07f-bc78-48ac-ba27-9c59fe97a35f-kube-api-access-nbc7x\") pod \"maas-api-e2e-shared-a-d62786-74788c9dc9-lfs6q\" (UID: \"c933a07f-bc78-48ac-ba27-9c59fe97a35f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d62786-74788c9dc9-lfs6q"
Jul 14 23:35:40.907116 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:35:40.906977    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c933a07f-bc78-48ac-ba27-9c59fe97a35f-maas-api-tls podName:c933a07f-bc78-48ac-ba27-9c59fe97a35f nodeName:}" failed. No retries permitted until 2026-07-14 23:35:41.406959351 +0000 UTC m=+2328.709950121 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c933a07f-bc78-48ac-ba27-9c59fe97a35f-maas-api-tls") pod "maas-api-e2e-shared-a-d62786-74788c9dc9-lfs6q" (UID: "c933a07f-bc78-48ac-ba27-9c59fe97a35f") : secret "maas-api-e2e-shared-a-d62786-serving-cert" not found
Jul 14 23:35:40.907547 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:40.907527    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c933a07f-bc78-48ac-ba27-9c59fe97a35f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-d62786-74788c9dc9-lfs6q\" (UID: \"c933a07f-bc78-48ac-ba27-9c59fe97a35f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d62786-74788c9dc9-lfs6q"
Jul 14 23:35:40.918425 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:40.918383    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbc7x\" (UniqueName: \"kubernetes.io/projected/c933a07f-bc78-48ac-ba27-9c59fe97a35f-kube-api-access-nbc7x\") pod \"maas-api-e2e-shared-a-d62786-74788c9dc9-lfs6q\" (UID: \"c933a07f-bc78-48ac-ba27-9c59fe97a35f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d62786-74788c9dc9-lfs6q"
Jul 14 23:35:41.412197 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:41.412156    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c933a07f-bc78-48ac-ba27-9c59fe97a35f-maas-api-tls\") pod \"maas-api-e2e-shared-a-d62786-74788c9dc9-lfs6q\" (UID: \"c933a07f-bc78-48ac-ba27-9c59fe97a35f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d62786-74788c9dc9-lfs6q"
Jul 14 23:35:41.414843 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:41.414815    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c933a07f-bc78-48ac-ba27-9c59fe97a35f-maas-api-tls\") pod \"maas-api-e2e-shared-a-d62786-74788c9dc9-lfs6q\" (UID: \"c933a07f-bc78-48ac-ba27-9c59fe97a35f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d62786-74788c9dc9-lfs6q"
Jul 14 23:35:41.639616 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:41.639585    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d62786-74788c9dc9-lfs6q"
Jul 14 23:35:41.781006 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:41.780970    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d62786-74788c9dc9-lfs6q"]
Jul 14 23:35:41.810259 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:35:41.810224    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc933a07f_bc78_48ac_ba27_9c59fe97a35f.slice/crio-3b7b3dd91d106645b41a73a68d478a251bcf89adc89743aa4962d1434b04e849 WatchSource:0}: Error finding container 3b7b3dd91d106645b41a73a68d478a251bcf89adc89743aa4962d1434b04e849: Status 404 returned error can't find the container with id 3b7b3dd91d106645b41a73a68d478a251bcf89adc89743aa4962d1434b04e849
Jul 14 23:35:41.998352 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:41.998323    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"
Jul 14 23:35:42.118545 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:42.118504    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/304ab33d-29dd-4f39-95cb-03bc227b6634-workload-socket\") pod \"304ab33d-29dd-4f39-95cb-03bc227b6634\" (UID: \"304ab33d-29dd-4f39-95cb-03bc227b6634\") "
Jul 14 23:35:42.118740 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:42.118572    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/304ab33d-29dd-4f39-95cb-03bc227b6634-workload-certs\") pod \"304ab33d-29dd-4f39-95cb-03bc227b6634\" (UID: \"304ab33d-29dd-4f39-95cb-03bc227b6634\") "
Jul 14 23:35:42.118740 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:42.118607    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/304ab33d-29dd-4f39-95cb-03bc227b6634-istio-token\") pod \"304ab33d-29dd-4f39-95cb-03bc227b6634\" (UID: \"304ab33d-29dd-4f39-95cb-03bc227b6634\") "
Jul 14 23:35:42.118740 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:42.118650    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/304ab33d-29dd-4f39-95cb-03bc227b6634-istio-data\") pod \"304ab33d-29dd-4f39-95cb-03bc227b6634\" (UID: \"304ab33d-29dd-4f39-95cb-03bc227b6634\") "
Jul 14 23:35:42.118740 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:42.118685    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/304ab33d-29dd-4f39-95cb-03bc227b6634-istiod-ca-cert\") pod \"304ab33d-29dd-4f39-95cb-03bc227b6634\" (UID: \"304ab33d-29dd-4f39-95cb-03bc227b6634\") "
Jul 14 23:35:42.118740 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:42.118738    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/304ab33d-29dd-4f39-95cb-03bc227b6634-credential-socket\") pod \"304ab33d-29dd-4f39-95cb-03bc227b6634\" (UID: \"304ab33d-29dd-4f39-95cb-03bc227b6634\") "
Jul 14 23:35:42.119029 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:42.118761    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/304ab33d-29dd-4f39-95cb-03bc227b6634-istio-podinfo\") pod \"304ab33d-29dd-4f39-95cb-03bc227b6634\" (UID: \"304ab33d-29dd-4f39-95cb-03bc227b6634\") "
Jul 14 23:35:42.119029 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:42.118811    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69l84\" (UniqueName: \"kubernetes.io/projected/304ab33d-29dd-4f39-95cb-03bc227b6634-kube-api-access-69l84\") pod \"304ab33d-29dd-4f39-95cb-03bc227b6634\" (UID: \"304ab33d-29dd-4f39-95cb-03bc227b6634\") "
Jul 14 23:35:42.119029 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:42.118833    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/304ab33d-29dd-4f39-95cb-03bc227b6634-istio-envoy\") pod \"304ab33d-29dd-4f39-95cb-03bc227b6634\" (UID: \"304ab33d-29dd-4f39-95cb-03bc227b6634\") "
Jul 14 23:35:42.119029 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:42.118848    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/304ab33d-29dd-4f39-95cb-03bc227b6634-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "304ab33d-29dd-4f39-95cb-03bc227b6634" (UID: "304ab33d-29dd-4f39-95cb-03bc227b6634"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:35:42.119029 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:42.118960    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/304ab33d-29dd-4f39-95cb-03bc227b6634-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "304ab33d-29dd-4f39-95cb-03bc227b6634" (UID: "304ab33d-29dd-4f39-95cb-03bc227b6634"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:35:42.119316 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:42.119030    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/304ab33d-29dd-4f39-95cb-03bc227b6634-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "304ab33d-29dd-4f39-95cb-03bc227b6634" (UID: "304ab33d-29dd-4f39-95cb-03bc227b6634"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:35:42.119316 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:42.119100    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/304ab33d-29dd-4f39-95cb-03bc227b6634-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "304ab33d-29dd-4f39-95cb-03bc227b6634" (UID: "304ab33d-29dd-4f39-95cb-03bc227b6634"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 23:35:42.119316 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:42.119173    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/304ab33d-29dd-4f39-95cb-03bc227b6634-credential-socket\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:35:42.119316 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:42.119190    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/304ab33d-29dd-4f39-95cb-03bc227b6634-workload-socket\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:35:42.119316 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:42.119207    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/304ab33d-29dd-4f39-95cb-03bc227b6634-workload-certs\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:35:42.119316 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:42.119225    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/304ab33d-29dd-4f39-95cb-03bc227b6634-istiod-ca-cert\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:35:42.119574 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:42.119485    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/304ab33d-29dd-4f39-95cb-03bc227b6634-istio-data" (OuterVolumeSpecName: "istio-data") pod "304ab33d-29dd-4f39-95cb-03bc227b6634" (UID: "304ab33d-29dd-4f39-95cb-03bc227b6634"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:35:42.121166 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:42.121145    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/304ab33d-29dd-4f39-95cb-03bc227b6634-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "304ab33d-29dd-4f39-95cb-03bc227b6634" (UID: "304ab33d-29dd-4f39-95cb-03bc227b6634"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:35:42.121266 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:42.121231    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/304ab33d-29dd-4f39-95cb-03bc227b6634-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "304ab33d-29dd-4f39-95cb-03bc227b6634" (UID: "304ab33d-29dd-4f39-95cb-03bc227b6634"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 23:35:42.121626 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:42.121605    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/304ab33d-29dd-4f39-95cb-03bc227b6634-kube-api-access-69l84" (OuterVolumeSpecName: "kube-api-access-69l84") pod "304ab33d-29dd-4f39-95cb-03bc227b6634" (UID: "304ab33d-29dd-4f39-95cb-03bc227b6634"). InnerVolumeSpecName "kube-api-access-69l84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:35:42.121694 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:42.121636    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/304ab33d-29dd-4f39-95cb-03bc227b6634-istio-token" (OuterVolumeSpecName: "istio-token") pod "304ab33d-29dd-4f39-95cb-03bc227b6634" (UID: "304ab33d-29dd-4f39-95cb-03bc227b6634"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:35:42.220512 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:42.220467    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/304ab33d-29dd-4f39-95cb-03bc227b6634-istio-token\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:35:42.220512 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:42.220502    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/304ab33d-29dd-4f39-95cb-03bc227b6634-istio-data\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:35:42.220512 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:42.220511    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/304ab33d-29dd-4f39-95cb-03bc227b6634-istio-podinfo\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:35:42.220512 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:42.220519    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69l84\" (UniqueName: \"kubernetes.io/projected/304ab33d-29dd-4f39-95cb-03bc227b6634-kube-api-access-69l84\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:35:42.220797 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:42.220528    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/304ab33d-29dd-4f39-95cb-03bc227b6634-istio-envoy\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:35:42.688526 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:42.688433    2563 generic.go:356] "Generic (PLEG): container finished" podID="304ab33d-29dd-4f39-95cb-03bc227b6634" containerID="4ccd288011da8084ee136a9a8d77f31736c8699e7a350e2a1bd1cea78c23ff41" exitCode=0
Jul 14 23:35:42.688526 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:42.688490    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"
Jul 14 23:35:42.689078 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:42.688526    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd" event={"ID":"304ab33d-29dd-4f39-95cb-03bc227b6634","Type":"ContainerDied","Data":"4ccd288011da8084ee136a9a8d77f31736c8699e7a350e2a1bd1cea78c23ff41"}
Jul 14 23:35:42.689078 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:42.688579    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd" event={"ID":"304ab33d-29dd-4f39-95cb-03bc227b6634","Type":"ContainerDied","Data":"99c2c7cadf3f0bb00150a5798e4a418059ae79de63830ed84a9e7bf371741214"}
Jul 14 23:35:42.689078 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:42.688608    2563 scope.go:117] "RemoveContainer" containerID="4ccd288011da8084ee136a9a8d77f31736c8699e7a350e2a1bd1cea78c23ff41"
Jul 14 23:35:42.690952 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:42.690928    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d62786-74788c9dc9-lfs6q" event={"ID":"c933a07f-bc78-48ac-ba27-9c59fe97a35f","Type":"ContainerStarted","Data":"0ec911c5bd3dd3097339692fcf8a2b4e121dbd290467a8fdd61aa814dcaf6d1d"}
Jul 14 23:35:42.691091 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:42.690972    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d62786-74788c9dc9-lfs6q" event={"ID":"c933a07f-bc78-48ac-ba27-9c59fe97a35f","Type":"ContainerStarted","Data":"3b7b3dd91d106645b41a73a68d478a251bcf89adc89743aa4962d1434b04e849"}
Jul 14 23:35:42.698823 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:42.698805    2563 scope.go:117] "RemoveContainer" containerID="637107c0f2bf1a110fd3e35dadb629d24beffeaad510e811a4bec00b970dd509"
Jul 14 23:35:42.707637 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:42.707611    2563 scope.go:117] "RemoveContainer" containerID="4ccd288011da8084ee136a9a8d77f31736c8699e7a350e2a1bd1cea78c23ff41"
Jul 14 23:35:42.707922 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:35:42.707901    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ccd288011da8084ee136a9a8d77f31736c8699e7a350e2a1bd1cea78c23ff41\": container with ID starting with 4ccd288011da8084ee136a9a8d77f31736c8699e7a350e2a1bd1cea78c23ff41 not found: ID does not exist" containerID="4ccd288011da8084ee136a9a8d77f31736c8699e7a350e2a1bd1cea78c23ff41"
Jul 14 23:35:42.707979 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:42.707936    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ccd288011da8084ee136a9a8d77f31736c8699e7a350e2a1bd1cea78c23ff41"} err="failed to get container status \"4ccd288011da8084ee136a9a8d77f31736c8699e7a350e2a1bd1cea78c23ff41\": rpc error: code = NotFound desc = could not find container \"4ccd288011da8084ee136a9a8d77f31736c8699e7a350e2a1bd1cea78c23ff41\": container with ID starting with 4ccd288011da8084ee136a9a8d77f31736c8699e7a350e2a1bd1cea78c23ff41 not found: ID does not exist"
Jul 14 23:35:42.707979 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:42.707960    2563 scope.go:117] "RemoveContainer" containerID="637107c0f2bf1a110fd3e35dadb629d24beffeaad510e811a4bec00b970dd509"
Jul 14 23:35:42.708367 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:35:42.708330    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"637107c0f2bf1a110fd3e35dadb629d24beffeaad510e811a4bec00b970dd509\": container with ID starting with 637107c0f2bf1a110fd3e35dadb629d24beffeaad510e811a4bec00b970dd509 not found: ID does not exist" containerID="637107c0f2bf1a110fd3e35dadb629d24beffeaad510e811a4bec00b970dd509"
Jul 14 23:35:42.708481 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:42.708368    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"637107c0f2bf1a110fd3e35dadb629d24beffeaad510e811a4bec00b970dd509"} err="failed to get container status \"637107c0f2bf1a110fd3e35dadb629d24beffeaad510e811a4bec00b970dd509\": rpc error: code = NotFound desc = could not find container \"637107c0f2bf1a110fd3e35dadb629d24beffeaad510e811a4bec00b970dd509\": container with ID starting with 637107c0f2bf1a110fd3e35dadb629d24beffeaad510e811a4bec00b970dd509 not found: ID does not exist"
Jul 14 23:35:42.710516 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:42.710476    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d62786-74788c9dc9-lfs6q" podStartSLOduration=2.264429189 podStartE2EDuration="2.710462267s" totalImagesPullingTime="446.033078ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:35:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:35:42.708564562 +0000 UTC m=+2330.011555352" watchObservedRunningTime="2026-07-14 23:35:42.710462267 +0000 UTC m=+2330.013453070"
Jul 14 23:35:42.723022 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:42.722990    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"]
Jul 14 23:35:42.730311 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:42.730268    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-a8299a-openshift-default-86994ff997-5zpkd"]
Jul 14 23:35:43.229498 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:43.229461    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="304ab33d-29dd-4f39-95cb-03bc227b6634" path="/var/lib/kubelet/pods/304ab33d-29dd-4f39-95cb-03bc227b6634/volumes"
Jul 14 23:35:43.696520 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:43.696492    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d62786-74788c9dc9-lfs6q"
Jul 14 23:35:45.771469 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:45.771383    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"]
Jul 14 23:35:45.771866 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:45.771850    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="304ab33d-29dd-4f39-95cb-03bc227b6634" containerName="istio-proxy"
Jul 14 23:35:45.771917 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:45.771870    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="304ab33d-29dd-4f39-95cb-03bc227b6634" containerName="istio-proxy"
Jul 14 23:35:45.771917 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:45.771885    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="304ab33d-29dd-4f39-95cb-03bc227b6634" containerName="istio-proxy"
Jul 14 23:35:45.771917 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:45.771891    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="304ab33d-29dd-4f39-95cb-03bc227b6634" containerName="istio-proxy"
Jul 14 23:35:45.772031 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:45.771947    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="304ab33d-29dd-4f39-95cb-03bc227b6634" containerName="istio-proxy"
Jul 14 23:35:45.772031 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:45.771955    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="304ab33d-29dd-4f39-95cb-03bc227b6634" containerName="istio-proxy"
Jul 14 23:35:45.776650 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:45.776625    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"
Jul 14 23:35:45.780687 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:45.780658    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-638b58-openshift-default-dockercfg-dxtm7\""
Jul 14 23:35:45.787186 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:45.787155    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"]
Jul 14 23:35:45.855440 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:45.855400    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-istio-podinfo\") pod \"e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm\" (UID: \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\") " pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"
Jul 14 23:35:45.855649 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:45.855450    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-istiod-ca-cert\") pod \"e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm\" (UID: \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\") " pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"
Jul 14 23:35:45.855649 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:45.855538    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-workload-certs\") pod \"e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm\" (UID: \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\") " pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"
Jul 14 23:35:45.855649 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:45.855568    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-istio-envoy\") pod \"e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm\" (UID: \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\") " pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"
Jul 14 23:35:45.855649 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:45.855599    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-workload-socket\") pod \"e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm\" (UID: \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\") " pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"
Jul 14 23:35:45.855649 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:45.855620    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-istio-token\") pod \"e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm\" (UID: \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\") " pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"
Jul 14 23:35:45.855649 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:45.855649    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-istio-data\") pod \"e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm\" (UID: \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\") " pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"
Jul 14 23:35:45.855894 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:45.855670    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-credential-socket\") pod \"e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm\" (UID: \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\") " pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"
Jul 14 23:35:45.855894 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:45.855689    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frtxf\" (UniqueName: \"kubernetes.io/projected/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-kube-api-access-frtxf\") pod \"e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm\" (UID: \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\") " pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"
Jul 14 23:35:45.956334 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:45.956264    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-workload-certs\") pod \"e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm\" (UID: \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\") " pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"
Jul 14 23:35:45.956334 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:45.956332    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-istio-envoy\") pod \"e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm\" (UID: \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\") " pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"
Jul 14 23:35:45.956590 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:45.956369    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-workload-socket\") pod \"e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm\" (UID: \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\") " pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"
Jul 14 23:35:45.956590 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:45.956387    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-istio-token\") pod \"e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm\" (UID: \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\") " pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"
Jul 14 23:35:45.956590 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:45.956407    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-istio-data\") pod \"e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm\" (UID: \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\") " pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"
Jul 14 23:35:45.956590 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:45.956436    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-credential-socket\") pod \"e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm\" (UID: \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\") " pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"
Jul 14 23:35:45.956590 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:45.956460    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frtxf\" (UniqueName: \"kubernetes.io/projected/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-kube-api-access-frtxf\") pod \"e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm\" (UID: \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\") " pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"
Jul 14 23:35:45.956590 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:45.956539    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-istio-podinfo\") pod \"e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm\" (UID: \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\") " pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"
Jul 14 23:35:45.956590 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:45.956583    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-istiod-ca-cert\") pod \"e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm\" (UID: \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\") " pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"
Jul 14 23:35:45.956949 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:45.956852    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-workload-certs\") pod \"e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm\" (UID: \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\") " pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"
Jul 14 23:35:45.956949 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:45.956893    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-workload-socket\") pod \"e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm\" (UID: \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\") " pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"
Jul 14 23:35:45.957200 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:45.957176    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-istio-data\") pod \"e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm\" (UID: \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\") " pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"
Jul 14 23:35:45.957262 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:45.957206    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-credential-socket\") pod \"e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm\" (UID: \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\") " pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"
Jul 14 23:35:45.957360 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:45.957335    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-istiod-ca-cert\") pod \"e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm\" (UID: \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\") " pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"
Jul 14 23:35:45.958995 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:45.958963    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-istio-envoy\") pod \"e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm\" (UID: \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\") " pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"
Jul 14 23:35:45.959200 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:45.959182    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-istio-podinfo\") pod \"e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm\" (UID: \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\") " pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"
Jul 14 23:35:45.965655 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:45.965621    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-istio-token\") pod \"e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm\" (UID: \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\") " pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"
Jul 14 23:35:45.965775 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:45.965757    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frtxf\" (UniqueName: \"kubernetes.io/projected/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-kube-api-access-frtxf\") pod \"e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm\" (UID: \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\") " pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"
Jul 14 23:35:46.088781 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:46.088679    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"
Jul 14 23:35:46.250423 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:46.250384    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"]
Jul 14 23:35:46.252541 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:35:46.252494    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2a753c8_3cdc_418b_b490_ec0fe6fd4a30.slice/crio-9f2bf5f0daf7d320398633cb5c117ca4996e51a9b96749bb28c61aa79ecd0866 WatchSource:0}: Error finding container 9f2bf5f0daf7d320398633cb5c117ca4996e51a9b96749bb28c61aa79ecd0866: Status 404 returned error can't find the container with id 9f2bf5f0daf7d320398633cb5c117ca4996e51a9b96749bb28c61aa79ecd0866
Jul 14 23:35:46.254870 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:46.254830    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:35:46.254970 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:46.254907    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:35:46.254970 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:46.254936    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:35:46.712779 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:46.712745    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm" event={"ID":"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30","Type":"ContainerStarted","Data":"f52d1466e684ae5faea8fbe5f21260467af0c7c9387b2c5129c21d5a895b63a3"}
Jul 14 23:35:46.712981 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:46.712797    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm" event={"ID":"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30","Type":"ContainerStarted","Data":"9f2bf5f0daf7d320398633cb5c117ca4996e51a9b96749bb28c61aa79ecd0866"}
Jul 14 23:35:46.738295 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:46.738217    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm" podStartSLOduration=1.73819988 podStartE2EDuration="1.73819988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:35:46.736872114 +0000 UTC m=+2334.039862947" watchObservedRunningTime="2026-07-14 23:35:46.73819988 +0000 UTC m=+2334.041190671"
Jul 14 23:35:47.089818 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:47.089764    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"
Jul 14 23:35:47.095697 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:47.095661    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"
Jul 14 23:35:47.721305 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:47.720877    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"
Jul 14 23:35:47.729713 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:47.729673    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"
Jul 14 23:35:48.894547 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:48.894493    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-638b58-5c7d44b967-qc9dv"]
Jul 14 23:35:48.898122 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:48.898098    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-638b58-5c7d44b967-qc9dv"
Jul 14 23:35:48.900878 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:48.900853    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-638b58-serving-cert\""
Jul 14 23:35:48.909714 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:48.909681    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-638b58-5c7d44b967-qc9dv"]
Jul 14 23:35:48.987688 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:48.987645    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fcb78ef-281c-451f-9cd9-64e932f5cd29-maas-api-tls\") pod \"maas-api-e2e-shared-b-638b58-5c7d44b967-qc9dv\" (UID: \"4fcb78ef-281c-451f-9cd9-64e932f5cd29\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-638b58-5c7d44b967-qc9dv"
Jul 14 23:35:48.987910 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:48.987715    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fcb78ef-281c-451f-9cd9-64e932f5cd29-openshift-service-ca\") pod \"maas-api-e2e-shared-b-638b58-5c7d44b967-qc9dv\" (UID: \"4fcb78ef-281c-451f-9cd9-64e932f5cd29\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-638b58-5c7d44b967-qc9dv"
Jul 14 23:35:48.987910 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:48.987818    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f2k9\" (UniqueName: \"kubernetes.io/projected/4fcb78ef-281c-451f-9cd9-64e932f5cd29-kube-api-access-2f2k9\") pod \"maas-api-e2e-shared-b-638b58-5c7d44b967-qc9dv\" (UID: \"4fcb78ef-281c-451f-9cd9-64e932f5cd29\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-638b58-5c7d44b967-qc9dv"
Jul 14 23:35:49.088635 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:49.088586    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fcb78ef-281c-451f-9cd9-64e932f5cd29-openshift-service-ca\") pod \"maas-api-e2e-shared-b-638b58-5c7d44b967-qc9dv\" (UID: \"4fcb78ef-281c-451f-9cd9-64e932f5cd29\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-638b58-5c7d44b967-qc9dv"
Jul 14 23:35:49.088635 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:49.088640    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f2k9\" (UniqueName: \"kubernetes.io/projected/4fcb78ef-281c-451f-9cd9-64e932f5cd29-kube-api-access-2f2k9\") pod \"maas-api-e2e-shared-b-638b58-5c7d44b967-qc9dv\" (UID: \"4fcb78ef-281c-451f-9cd9-64e932f5cd29\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-638b58-5c7d44b967-qc9dv"
Jul 14 23:35:49.088884 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:49.088686    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fcb78ef-281c-451f-9cd9-64e932f5cd29-maas-api-tls\") pod \"maas-api-e2e-shared-b-638b58-5c7d44b967-qc9dv\" (UID: \"4fcb78ef-281c-451f-9cd9-64e932f5cd29\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-638b58-5c7d44b967-qc9dv"
Jul 14 23:35:49.088884 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:35:49.088813    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-638b58-serving-cert: secret "maas-api-e2e-shared-b-638b58-serving-cert" not found
Jul 14 23:35:49.088884 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:35:49.088872    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fcb78ef-281c-451f-9cd9-64e932f5cd29-maas-api-tls podName:4fcb78ef-281c-451f-9cd9-64e932f5cd29 nodeName:}" failed. No retries permitted until 2026-07-14 23:35:49.588855919 +0000 UTC m=+2336.891846690 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4fcb78ef-281c-451f-9cd9-64e932f5cd29-maas-api-tls") pod "maas-api-e2e-shared-b-638b58-5c7d44b967-qc9dv" (UID: "4fcb78ef-281c-451f-9cd9-64e932f5cd29") : secret "maas-api-e2e-shared-b-638b58-serving-cert" not found
Jul 14 23:35:49.089325 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:49.089273    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fcb78ef-281c-451f-9cd9-64e932f5cd29-openshift-service-ca\") pod \"maas-api-e2e-shared-b-638b58-5c7d44b967-qc9dv\" (UID: \"4fcb78ef-281c-451f-9cd9-64e932f5cd29\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-638b58-5c7d44b967-qc9dv"
Jul 14 23:35:49.101110 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:49.101074    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f2k9\" (UniqueName: \"kubernetes.io/projected/4fcb78ef-281c-451f-9cd9-64e932f5cd29-kube-api-access-2f2k9\") pod \"maas-api-e2e-shared-b-638b58-5c7d44b967-qc9dv\" (UID: \"4fcb78ef-281c-451f-9cd9-64e932f5cd29\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-638b58-5c7d44b967-qc9dv"
Jul 14 23:35:49.594067 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:49.594014    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fcb78ef-281c-451f-9cd9-64e932f5cd29-maas-api-tls\") pod \"maas-api-e2e-shared-b-638b58-5c7d44b967-qc9dv\" (UID: \"4fcb78ef-281c-451f-9cd9-64e932f5cd29\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-638b58-5c7d44b967-qc9dv"
Jul 14 23:35:49.596749 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:49.596709    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fcb78ef-281c-451f-9cd9-64e932f5cd29-maas-api-tls\") pod \"maas-api-e2e-shared-b-638b58-5c7d44b967-qc9dv\" (UID: \"4fcb78ef-281c-451f-9cd9-64e932f5cd29\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-638b58-5c7d44b967-qc9dv"
Jul 14 23:35:49.706427 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:49.706384    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d62786-74788c9dc9-lfs6q"
Jul 14 23:35:49.811685 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:49.811651    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-638b58-5c7d44b967-qc9dv"
Jul 14 23:35:50.000418 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:50.000390    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-638b58-5c7d44b967-qc9dv"]
Jul 14 23:35:50.002191 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:35:50.002164    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fcb78ef_281c_451f_9cd9_64e932f5cd29.slice/crio-9adf229b71d10185e7cf2b78e8aae6158f581d99a766b0f6c6d699e4cee57041 WatchSource:0}: Error finding container 9adf229b71d10185e7cf2b78e8aae6158f581d99a766b0f6c6d699e4cee57041: Status 404 returned error can't find the container with id 9adf229b71d10185e7cf2b78e8aae6158f581d99a766b0f6c6d699e4cee57041
Jul 14 23:35:50.758315 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:50.758262    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-638b58-5c7d44b967-qc9dv" event={"ID":"4fcb78ef-281c-451f-9cd9-64e932f5cd29","Type":"ContainerStarted","Data":"d206d25ab25a3d2009347851e59f0c2a62cf0eb5122335d00d916b326c80c81e"}
Jul 14 23:35:50.758531 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:50.758342    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-638b58-5c7d44b967-qc9dv" event={"ID":"4fcb78ef-281c-451f-9cd9-64e932f5cd29","Type":"ContainerStarted","Data":"9adf229b71d10185e7cf2b78e8aae6158f581d99a766b0f6c6d699e4cee57041"}
Jul 14 23:35:50.758531 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:50.758377    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-638b58-5c7d44b967-qc9dv"
Jul 14 23:35:50.776778 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:50.776721    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-638b58-5c7d44b967-qc9dv" podStartSLOduration=2.389718768 podStartE2EDuration="2.776701843s" totalImagesPullingTime="386.983075ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:35:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:35:50.776490363 +0000 UTC m=+2338.079481154" watchObservedRunningTime="2026-07-14 23:35:50.776701843 +0000 UTC m=+2338.079692635"
Jul 14 23:35:54.781691 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:54.781650    2563 generic.go:356] "Generic (PLEG): container finished" podID="0718434c-4640-407f-8215-872ae7c6a09d" containerID="eab5b6d53d10b6a1d502e072fe892c84a94db2a70aa1bdef1139620af96ba024" exitCode=0
Jul 14 23:35:54.782238 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:54.781728    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq" event={"ID":"0718434c-4640-407f-8215-872ae7c6a09d","Type":"ContainerDied","Data":"eab5b6d53d10b6a1d502e072fe892c84a94db2a70aa1bdef1139620af96ba024"}
Jul 14 23:35:54.782353 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:54.782330    2563 scope.go:117] "RemoveContainer" containerID="eab5b6d53d10b6a1d502e072fe892c84a94db2a70aa1bdef1139620af96ba024"
Jul 14 23:35:54.783417 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:54.783387    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:35:54.783494 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:54.783460    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:35:54.783494 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:54.783490    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:35:55.677898 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:55.677787    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"
Jul 14 23:35:55.788240 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:55.788199    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq" event={"ID":"0718434c-4640-407f-8215-872ae7c6a09d","Type":"ContainerStarted","Data":"51f058db4ec441ad4d7294b630dc33707af3d41b95b27db7ab0c300c676eaef5"}
Jul 14 23:35:55.980936 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:55.980828    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"
Jul 14 23:35:55.982372 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:55.982328    2563 patch_prober.go:28] interesting pod/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Jul 14 23:35:55.982549 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:55.982417    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq" podUID="0718434c-4640-407f-8215-872ae7c6a09d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Jul 14 23:35:56.770349 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:56.770316    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-638b58-5c7d44b967-qc9dv"
Jul 14 23:35:56.981059 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:56.981025    2563 patch_prober.go:28] interesting pod/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Jul 14 23:35:56.981522 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:56.981081    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq" podUID="0718434c-4640-407f-8215-872ae7c6a09d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Jul 14 23:35:57.981607 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:57.981561    2563 patch_prober.go:28] interesting pod/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Jul 14 23:35:57.982109 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:57.981636    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq" podUID="0718434c-4640-407f-8215-872ae7c6a09d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Jul 14 23:35:58.980647 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:58.980607    2563 patch_prober.go:28] interesting pod/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Jul 14 23:35:58.980920 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:58.980894    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq" podUID="0718434c-4640-407f-8215-872ae7c6a09d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Jul 14 23:35:59.980775 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:59.980731    2563 patch_prober.go:28] interesting pod/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Jul 14 23:35:59.981221 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:35:59.980798    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq" podUID="0718434c-4640-407f-8215-872ae7c6a09d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Jul 14 23:36:00.985461 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:00.985427    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"
Jul 14 23:36:00.985943 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:00.985814    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"
Jul 14 23:36:00.986755 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:00.986734    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"
Jul 14 23:36:02.992604 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:02.992556    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-d62786/auth-test-model-d62786-kserve-6f86689575-gxsrr"]
Jul 14 23:36:02.996025 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:02.995993    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d62786/auth-test-model-d62786-kserve-6f86689575-gxsrr"
Jul 14 23:36:02.999084 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:02.999044    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-d62786\"/\"kube-root-ca.crt\""
Jul 14 23:36:03.000923 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:03.000880    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-d62786\"/\"openshift-service-ca.crt\""
Jul 14 23:36:03.001103 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:03.000880    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d62786\"/\"auth-test-model-d62786-kserve-self-signed-certs\""
Jul 14 23:36:03.001184 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:03.000932    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d62786\"/\"default-dockercfg-mvxpj\""
Jul 14 23:36:03.004176 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:03.004138    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d62786/auth-test-model-d62786-kserve-6f86689575-gxsrr"]
Jul 14 23:36:03.024594 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:03.024555    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adaa4248-0b75-41b3-9d9e-48c8f1026162-home\") pod \"auth-test-model-d62786-kserve-6f86689575-gxsrr\" (UID: \"adaa4248-0b75-41b3-9d9e-48c8f1026162\") " pod="ai-tenant-e2e-shared-a-d62786/auth-test-model-d62786-kserve-6f86689575-gxsrr"
Jul 14 23:36:03.024594 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:03.024597    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adaa4248-0b75-41b3-9d9e-48c8f1026162-dshm\") pod \"auth-test-model-d62786-kserve-6f86689575-gxsrr\" (UID: \"adaa4248-0b75-41b3-9d9e-48c8f1026162\") " pod="ai-tenant-e2e-shared-a-d62786/auth-test-model-d62786-kserve-6f86689575-gxsrr"
Jul 14 23:36:03.024880 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:03.024711    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adaa4248-0b75-41b3-9d9e-48c8f1026162-tmp-dir\") pod \"auth-test-model-d62786-kserve-6f86689575-gxsrr\" (UID: \"adaa4248-0b75-41b3-9d9e-48c8f1026162\") " pod="ai-tenant-e2e-shared-a-d62786/auth-test-model-d62786-kserve-6f86689575-gxsrr"
Jul 14 23:36:03.024880 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:03.024745    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adaa4248-0b75-41b3-9d9e-48c8f1026162-tls-certs\") pod \"auth-test-model-d62786-kserve-6f86689575-gxsrr\" (UID: \"adaa4248-0b75-41b3-9d9e-48c8f1026162\") " pod="ai-tenant-e2e-shared-a-d62786/auth-test-model-d62786-kserve-6f86689575-gxsrr"
Jul 14 23:36:03.024880 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:03.024769    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adaa4248-0b75-41b3-9d9e-48c8f1026162-model-cache\") pod \"auth-test-model-d62786-kserve-6f86689575-gxsrr\" (UID: \"adaa4248-0b75-41b3-9d9e-48c8f1026162\") " pod="ai-tenant-e2e-shared-a-d62786/auth-test-model-d62786-kserve-6f86689575-gxsrr"
Jul 14 23:36:03.024880 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:03.024792    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhf8g\" (UniqueName: \"kubernetes.io/projected/adaa4248-0b75-41b3-9d9e-48c8f1026162-kube-api-access-bhf8g\") pod \"auth-test-model-d62786-kserve-6f86689575-gxsrr\" (UID: \"adaa4248-0b75-41b3-9d9e-48c8f1026162\") " pod="ai-tenant-e2e-shared-a-d62786/auth-test-model-d62786-kserve-6f86689575-gxsrr"
Jul 14 23:36:03.041935 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:03.041887    2563 patch_prober.go:28] interesting pod/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": read tcp 10.133.0.2:59888->10.133.0.66:15021: read: connection reset by peer" start-of-body=
Jul 14 23:36:03.042199 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:03.041970    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm" podUID="c2a753c8-3cdc-418b-b490-ec0fe6fd4a30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": read tcp 10.133.0.2:59888->10.133.0.66:15021: read: connection reset by peer"
Jul 14 23:36:03.125691 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:03.125649    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adaa4248-0b75-41b3-9d9e-48c8f1026162-home\") pod \"auth-test-model-d62786-kserve-6f86689575-gxsrr\" (UID: \"adaa4248-0b75-41b3-9d9e-48c8f1026162\") " pod="ai-tenant-e2e-shared-a-d62786/auth-test-model-d62786-kserve-6f86689575-gxsrr"
Jul 14 23:36:03.125884 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:03.125695    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adaa4248-0b75-41b3-9d9e-48c8f1026162-dshm\") pod \"auth-test-model-d62786-kserve-6f86689575-gxsrr\" (UID: \"adaa4248-0b75-41b3-9d9e-48c8f1026162\") " pod="ai-tenant-e2e-shared-a-d62786/auth-test-model-d62786-kserve-6f86689575-gxsrr"
Jul 14 23:36:03.125884 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:03.125762    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adaa4248-0b75-41b3-9d9e-48c8f1026162-tmp-dir\") pod \"auth-test-model-d62786-kserve-6f86689575-gxsrr\" (UID: \"adaa4248-0b75-41b3-9d9e-48c8f1026162\") " pod="ai-tenant-e2e-shared-a-d62786/auth-test-model-d62786-kserve-6f86689575-gxsrr"
Jul 14 23:36:03.125884 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:03.125779    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adaa4248-0b75-41b3-9d9e-48c8f1026162-tls-certs\") pod \"auth-test-model-d62786-kserve-6f86689575-gxsrr\" (UID: \"adaa4248-0b75-41b3-9d9e-48c8f1026162\") " pod="ai-tenant-e2e-shared-a-d62786/auth-test-model-d62786-kserve-6f86689575-gxsrr"
Jul 14 23:36:03.125884 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:03.125797    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adaa4248-0b75-41b3-9d9e-48c8f1026162-model-cache\") pod \"auth-test-model-d62786-kserve-6f86689575-gxsrr\" (UID: \"adaa4248-0b75-41b3-9d9e-48c8f1026162\") " pod="ai-tenant-e2e-shared-a-d62786/auth-test-model-d62786-kserve-6f86689575-gxsrr"
Jul 14 23:36:03.125884 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:03.125818    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhf8g\" (UniqueName: \"kubernetes.io/projected/adaa4248-0b75-41b3-9d9e-48c8f1026162-kube-api-access-bhf8g\") pod \"auth-test-model-d62786-kserve-6f86689575-gxsrr\" (UID: \"adaa4248-0b75-41b3-9d9e-48c8f1026162\") " pod="ai-tenant-e2e-shared-a-d62786/auth-test-model-d62786-kserve-6f86689575-gxsrr"
Jul 14 23:36:03.126192 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:03.126159    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adaa4248-0b75-41b3-9d9e-48c8f1026162-home\") pod \"auth-test-model-d62786-kserve-6f86689575-gxsrr\" (UID: \"adaa4248-0b75-41b3-9d9e-48c8f1026162\") " pod="ai-tenant-e2e-shared-a-d62786/auth-test-model-d62786-kserve-6f86689575-gxsrr"
Jul 14 23:36:03.126262 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:03.126187    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adaa4248-0b75-41b3-9d9e-48c8f1026162-tmp-dir\") pod \"auth-test-model-d62786-kserve-6f86689575-gxsrr\" (UID: \"adaa4248-0b75-41b3-9d9e-48c8f1026162\") " pod="ai-tenant-e2e-shared-a-d62786/auth-test-model-d62786-kserve-6f86689575-gxsrr"
Jul 14 23:36:03.126262 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:03.126229    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adaa4248-0b75-41b3-9d9e-48c8f1026162-model-cache\") pod \"auth-test-model-d62786-kserve-6f86689575-gxsrr\" (UID: \"adaa4248-0b75-41b3-9d9e-48c8f1026162\") " pod="ai-tenant-e2e-shared-a-d62786/auth-test-model-d62786-kserve-6f86689575-gxsrr"
Jul 14 23:36:03.128424 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:03.128389    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adaa4248-0b75-41b3-9d9e-48c8f1026162-dshm\") pod \"auth-test-model-d62786-kserve-6f86689575-gxsrr\" (UID: \"adaa4248-0b75-41b3-9d9e-48c8f1026162\") " pod="ai-tenant-e2e-shared-a-d62786/auth-test-model-d62786-kserve-6f86689575-gxsrr"
Jul 14 23:36:03.128679 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:03.128662    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adaa4248-0b75-41b3-9d9e-48c8f1026162-tls-certs\") pod \"auth-test-model-d62786-kserve-6f86689575-gxsrr\" (UID: \"adaa4248-0b75-41b3-9d9e-48c8f1026162\") " pod="ai-tenant-e2e-shared-a-d62786/auth-test-model-d62786-kserve-6f86689575-gxsrr"
Jul 14 23:36:03.136519 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:03.136485    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhf8g\" (UniqueName: \"kubernetes.io/projected/adaa4248-0b75-41b3-9d9e-48c8f1026162-kube-api-access-bhf8g\") pod \"auth-test-model-d62786-kserve-6f86689575-gxsrr\" (UID: \"adaa4248-0b75-41b3-9d9e-48c8f1026162\") " pod="ai-tenant-e2e-shared-a-d62786/auth-test-model-d62786-kserve-6f86689575-gxsrr"
Jul 14 23:36:03.309767 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:03.309661    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d62786/auth-test-model-d62786-kserve-6f86689575-gxsrr"
Jul 14 23:36:03.459806 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:03.459696    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d62786/auth-test-model-d62786-kserve-6f86689575-gxsrr"]
Jul 14 23:36:03.462689 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:36:03.462643    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podadaa4248_0b75_41b3_9d9e_48c8f1026162.slice/crio-c0c79580d518bbf15658773fba3e2e1757dcfd9fca496fa416fd1232586dd4e3 WatchSource:0}: Error finding container c0c79580d518bbf15658773fba3e2e1757dcfd9fca496fa416fd1232586dd4e3: Status 404 returned error can't find the container with id c0c79580d518bbf15658773fba3e2e1757dcfd9fca496fa416fd1232586dd4e3
Jul 14 23:36:03.831745 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:03.831708    2563 generic.go:356] "Generic (PLEG): container finished" podID="c2a753c8-3cdc-418b-b490-ec0fe6fd4a30" containerID="f52d1466e684ae5faea8fbe5f21260467af0c7c9387b2c5129c21d5a895b63a3" exitCode=0
Jul 14 23:36:03.831967 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:03.831789    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm" event={"ID":"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30","Type":"ContainerDied","Data":"f52d1466e684ae5faea8fbe5f21260467af0c7c9387b2c5129c21d5a895b63a3"}
Jul 14 23:36:03.832256 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:03.832235    2563 scope.go:117] "RemoveContainer" containerID="f52d1466e684ae5faea8fbe5f21260467af0c7c9387b2c5129c21d5a895b63a3"
Jul 14 23:36:03.833355 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:03.833328    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:36:03.833440 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:03.833391    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d62786/auth-test-model-d62786-kserve-6f86689575-gxsrr" event={"ID":"adaa4248-0b75-41b3-9d9e-48c8f1026162","Type":"ContainerStarted","Data":"a205a2fca30593783dacc3d7d8a6b2deffa4122508444070c0d55bd97fcdb067"}
Jul 14 23:36:03.833440 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:03.833411    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:36:03.833440 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:03.833433    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d62786/auth-test-model-d62786-kserve-6f86689575-gxsrr" event={"ID":"adaa4248-0b75-41b3-9d9e-48c8f1026162","Type":"ContainerStarted","Data":"c0c79580d518bbf15658773fba3e2e1757dcfd9fca496fa416fd1232586dd4e3"}
Jul 14 23:36:03.833557 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:03.833457    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 14 23:36:03.875466 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:03.875409    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-d62786/auth-test-model-d62786-kserve-6f86689575-gxsrr" podStartSLOduration=1.875389727 podStartE2EDuration="1.875389727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:36:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:36:03.874693334 +0000 UTC m=+2351.177684126" watchObservedRunningTime="2026-07-14 23:36:03.875389727 +0000 UTC m=+2351.178380519"
Jul 14 23:36:04.843232 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:04.843190    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm" event={"ID":"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30","Type":"ContainerStarted","Data":"1ab11a47375f21ce93184457994e965149e8694fe55fba2f658dcad1dc3189c0"}
Jul 14 23:36:05.089452 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:05.089407    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"
Jul 14 23:36:05.089774 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:05.089751    2563 patch_prober.go:28] interesting pod/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Jul 14 23:36:05.089840 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:05.089800    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm" podUID="c2a753c8-3cdc-418b-b490-ec0fe6fd4a30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Jul 14 23:36:06.090158 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:06.090118    2563 patch_prober.go:28] interesting pod/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Jul 14 23:36:06.090625 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:06.090182    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm" podUID="c2a753c8-3cdc-418b-b490-ec0fe6fd4a30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Jul 14 23:36:07.089604 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:07.089567    2563 patch_prober.go:28] interesting pod/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Jul 14 23:36:07.089803 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:07.089624    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm" podUID="c2a753c8-3cdc-418b-b490-ec0fe6fd4a30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Jul 14 23:36:08.089534 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:08.089485    2563 patch_prober.go:28] interesting pod/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Jul 14 23:36:08.090039 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:08.089564    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm" podUID="c2a753c8-3cdc-418b-b490-ec0fe6fd4a30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Jul 14 23:36:09.093869 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:09.093833    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"
Jul 14 23:36:09.094357 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:09.094243    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"
Jul 14 23:36:09.094926 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:09.094909    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"
Jul 14 23:36:13.310555 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:13.310517    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-d62786/auth-test-model-d62786-kserve-6f86689575-gxsrr"
Jul 14 23:36:13.310555 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:13.310567    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-d62786/auth-test-model-d62786-kserve-6f86689575-gxsrr"
Jul 14 23:36:13.323236 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:13.323194    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-d62786/auth-test-model-d62786-kserve-6f86689575-gxsrr"
Jul 14 23:36:13.898270 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:13.898230    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-d62786/auth-test-model-d62786-kserve-6f86689575-gxsrr"
Jul 14 23:36:19.697514 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:19.697474    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-638b58/auth-test-model-638b58-kserve-54b66cd79-p8knm"]
Jul 14 23:36:19.700630 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:19.700612    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-638b58/auth-test-model-638b58-kserve-54b66cd79-p8knm"
Jul 14 23:36:19.703599 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:19.703575    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-638b58\"/\"openshift-service-ca.crt\""
Jul 14 23:36:19.704981 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:19.704944    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-638b58\"/\"default-dockercfg-xvhb8\""
Jul 14 23:36:19.705101 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:19.705007    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-638b58\"/\"kube-root-ca.crt\""
Jul 14 23:36:19.705101 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:19.705012    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-638b58\"/\"auth-test-model-638b58-kserve-self-signed-certs\""
Jul 14 23:36:19.711081 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:19.711050    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-638b58/auth-test-model-638b58-kserve-54b66cd79-p8knm"]
Jul 14 23:36:19.784854 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:19.784818    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20f50c93-9a09-4fc9-9e48-954558575482-home\") pod \"auth-test-model-638b58-kserve-54b66cd79-p8knm\" (UID: \"20f50c93-9a09-4fc9-9e48-954558575482\") " pod="ai-tenant-e2e-shared-b-638b58/auth-test-model-638b58-kserve-54b66cd79-p8knm"
Jul 14 23:36:19.785049 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:19.784880    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20f50c93-9a09-4fc9-9e48-954558575482-dshm\") pod \"auth-test-model-638b58-kserve-54b66cd79-p8knm\" (UID: \"20f50c93-9a09-4fc9-9e48-954558575482\") " pod="ai-tenant-e2e-shared-b-638b58/auth-test-model-638b58-kserve-54b66cd79-p8knm"
Jul 14 23:36:19.785049 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:19.784933    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20f50c93-9a09-4fc9-9e48-954558575482-model-cache\") pod \"auth-test-model-638b58-kserve-54b66cd79-p8knm\" (UID: \"20f50c93-9a09-4fc9-9e48-954558575482\") " pod="ai-tenant-e2e-shared-b-638b58/auth-test-model-638b58-kserve-54b66cd79-p8knm"
Jul 14 23:36:19.785049 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:19.784979    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8zwf\" (UniqueName: \"kubernetes.io/projected/20f50c93-9a09-4fc9-9e48-954558575482-kube-api-access-v8zwf\") pod \"auth-test-model-638b58-kserve-54b66cd79-p8knm\" (UID: \"20f50c93-9a09-4fc9-9e48-954558575482\") " pod="ai-tenant-e2e-shared-b-638b58/auth-test-model-638b58-kserve-54b66cd79-p8knm"
Jul 14 23:36:19.785049 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:19.785019    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20f50c93-9a09-4fc9-9e48-954558575482-tls-certs\") pod \"auth-test-model-638b58-kserve-54b66cd79-p8knm\" (UID: \"20f50c93-9a09-4fc9-9e48-954558575482\") " pod="ai-tenant-e2e-shared-b-638b58/auth-test-model-638b58-kserve-54b66cd79-p8knm"
Jul 14 23:36:19.785213 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:19.785097    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20f50c93-9a09-4fc9-9e48-954558575482-tmp-dir\") pod \"auth-test-model-638b58-kserve-54b66cd79-p8knm\" (UID: \"20f50c93-9a09-4fc9-9e48-954558575482\") " pod="ai-tenant-e2e-shared-b-638b58/auth-test-model-638b58-kserve-54b66cd79-p8knm"
Jul 14 23:36:19.886257 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:19.886220    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20f50c93-9a09-4fc9-9e48-954558575482-tmp-dir\") pod \"auth-test-model-638b58-kserve-54b66cd79-p8knm\" (UID: \"20f50c93-9a09-4fc9-9e48-954558575482\") " pod="ai-tenant-e2e-shared-b-638b58/auth-test-model-638b58-kserve-54b66cd79-p8knm"
Jul 14 23:36:19.886257 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:19.886267    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20f50c93-9a09-4fc9-9e48-954558575482-home\") pod \"auth-test-model-638b58-kserve-54b66cd79-p8knm\" (UID: \"20f50c93-9a09-4fc9-9e48-954558575482\") " pod="ai-tenant-e2e-shared-b-638b58/auth-test-model-638b58-kserve-54b66cd79-p8knm"
Jul 14 23:36:19.886559 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:19.886333    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20f50c93-9a09-4fc9-9e48-954558575482-dshm\") pod \"auth-test-model-638b58-kserve-54b66cd79-p8knm\" (UID: \"20f50c93-9a09-4fc9-9e48-954558575482\") " pod="ai-tenant-e2e-shared-b-638b58/auth-test-model-638b58-kserve-54b66cd79-p8knm"
Jul 14 23:36:19.886559 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:19.886348    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20f50c93-9a09-4fc9-9e48-954558575482-model-cache\") pod \"auth-test-model-638b58-kserve-54b66cd79-p8knm\" (UID: \"20f50c93-9a09-4fc9-9e48-954558575482\") " pod="ai-tenant-e2e-shared-b-638b58/auth-test-model-638b58-kserve-54b66cd79-p8knm"
Jul 14 23:36:19.886559 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:19.886373    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8zwf\" (UniqueName: \"kubernetes.io/projected/20f50c93-9a09-4fc9-9e48-954558575482-kube-api-access-v8zwf\") pod \"auth-test-model-638b58-kserve-54b66cd79-p8knm\" (UID: \"20f50c93-9a09-4fc9-9e48-954558575482\") " pod="ai-tenant-e2e-shared-b-638b58/auth-test-model-638b58-kserve-54b66cd79-p8knm"
Jul 14 23:36:19.886559 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:19.886497    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20f50c93-9a09-4fc9-9e48-954558575482-tls-certs\") pod \"auth-test-model-638b58-kserve-54b66cd79-p8knm\" (UID: \"20f50c93-9a09-4fc9-9e48-954558575482\") " pod="ai-tenant-e2e-shared-b-638b58/auth-test-model-638b58-kserve-54b66cd79-p8knm"
Jul 14 23:36:19.886766 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:19.886746    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20f50c93-9a09-4fc9-9e48-954558575482-tmp-dir\") pod \"auth-test-model-638b58-kserve-54b66cd79-p8knm\" (UID: \"20f50c93-9a09-4fc9-9e48-954558575482\") " pod="ai-tenant-e2e-shared-b-638b58/auth-test-model-638b58-kserve-54b66cd79-p8knm"
Jul 14 23:36:19.886828 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:19.886787    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20f50c93-9a09-4fc9-9e48-954558575482-model-cache\") pod \"auth-test-model-638b58-kserve-54b66cd79-p8knm\" (UID: \"20f50c93-9a09-4fc9-9e48-954558575482\") " pod="ai-tenant-e2e-shared-b-638b58/auth-test-model-638b58-kserve-54b66cd79-p8knm"
Jul 14 23:36:19.886883 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:19.886856    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20f50c93-9a09-4fc9-9e48-954558575482-home\") pod \"auth-test-model-638b58-kserve-54b66cd79-p8knm\" (UID: \"20f50c93-9a09-4fc9-9e48-954558575482\") " pod="ai-tenant-e2e-shared-b-638b58/auth-test-model-638b58-kserve-54b66cd79-p8knm"
Jul 14 23:36:19.888902 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:19.888872    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20f50c93-9a09-4fc9-9e48-954558575482-dshm\") pod \"auth-test-model-638b58-kserve-54b66cd79-p8knm\" (UID: \"20f50c93-9a09-4fc9-9e48-954558575482\") " pod="ai-tenant-e2e-shared-b-638b58/auth-test-model-638b58-kserve-54b66cd79-p8knm"
Jul 14 23:36:19.889104 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:19.889089    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20f50c93-9a09-4fc9-9e48-954558575482-tls-certs\") pod \"auth-test-model-638b58-kserve-54b66cd79-p8knm\" (UID: \"20f50c93-9a09-4fc9-9e48-954558575482\") " pod="ai-tenant-e2e-shared-b-638b58/auth-test-model-638b58-kserve-54b66cd79-p8knm"
Jul 14 23:36:19.895688 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:19.895664    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8zwf\" (UniqueName: \"kubernetes.io/projected/20f50c93-9a09-4fc9-9e48-954558575482-kube-api-access-v8zwf\") pod \"auth-test-model-638b58-kserve-54b66cd79-p8knm\" (UID: \"20f50c93-9a09-4fc9-9e48-954558575482\") " pod="ai-tenant-e2e-shared-b-638b58/auth-test-model-638b58-kserve-54b66cd79-p8knm"
Jul 14 23:36:20.012557 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:20.012510    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-638b58/auth-test-model-638b58-kserve-54b66cd79-p8knm"
Jul 14 23:36:20.152246 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:20.152214    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-638b58/auth-test-model-638b58-kserve-54b66cd79-p8knm"]
Jul 14 23:36:20.155520 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:36:20.155490    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20f50c93_9a09_4fc9_9e48_954558575482.slice/crio-16b6293f39e0ce49bd79a33b5517e2c6288e9e0b0b4c5137486b1baccfc709e2 WatchSource:0}: Error finding container 16b6293f39e0ce49bd79a33b5517e2c6288e9e0b0b4c5137486b1baccfc709e2: Status 404 returned error can't find the container with id 16b6293f39e0ce49bd79a33b5517e2c6288e9e0b0b4c5137486b1baccfc709e2
Jul 14 23:36:20.920762 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:20.920722    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-638b58/auth-test-model-638b58-kserve-54b66cd79-p8knm" event={"ID":"20f50c93-9a09-4fc9-9e48-954558575482","Type":"ContainerStarted","Data":"78a8a95124018226c5af4652d1444e8d888f02eb06eeb184d98035509c43c528"}
Jul 14 23:36:20.921174 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:20.920774    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-638b58/auth-test-model-638b58-kserve-54b66cd79-p8knm" event={"ID":"20f50c93-9a09-4fc9-9e48-954558575482","Type":"ContainerStarted","Data":"16b6293f39e0ce49bd79a33b5517e2c6288e9e0b0b4c5137486b1baccfc709e2"}
Jul 14 23:36:20.937946 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:20.937892    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-638b58/auth-test-model-638b58-kserve-54b66cd79-p8knm" podStartSLOduration=1.9378732159999998 podStartE2EDuration="1.937873216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:36:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:36:20.937469984 +0000 UTC m=+2368.240460776" watchObservedRunningTime="2026-07-14 23:36:20.937873216 +0000 UTC m=+2368.240864008"
Jul 14 23:36:30.013354 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:30.013319    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-638b58/auth-test-model-638b58-kserve-54b66cd79-p8knm"
Jul 14 23:36:30.013812 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:30.013368    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-638b58/auth-test-model-638b58-kserve-54b66cd79-p8knm"
Jul 14 23:36:30.025847 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:30.025822    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-638b58/auth-test-model-638b58-kserve-54b66cd79-p8knm"
Jul 14 23:36:30.982541 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:36:30.982505    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-638b58/auth-test-model-638b58-kserve-54b66cd79-p8knm"
Jul 14 23:38:17.815208 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:17.815160    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d62786/auth-test-model-d62786-kserve-6f86689575-gxsrr"]
Jul 14 23:38:18.695717 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:18.695683    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-638b58/auth-test-model-638b58-kserve-54b66cd79-p8knm"]
Jul 14 23:38:32.835854 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:32.835800    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-d62786/auth-test-model-d62786-kserve-6f86689575-gxsrr" podUID="adaa4248-0b75-41b3-9d9e-48c8f1026162" containerName="main" containerID="cri-o://a205a2fca30593783dacc3d7d8a6b2deffa4122508444070c0d55bd97fcdb067" gracePeriod=45
Jul 14 23:38:33.082685 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:33.082655    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d62786/auth-test-model-d62786-kserve-6f86689575-gxsrr"
Jul 14 23:38:33.212946 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:33.212849    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adaa4248-0b75-41b3-9d9e-48c8f1026162-tmp-dir\") pod \"adaa4248-0b75-41b3-9d9e-48c8f1026162\" (UID: \"adaa4248-0b75-41b3-9d9e-48c8f1026162\") "
Jul 14 23:38:33.212946 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:33.212916    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhf8g\" (UniqueName: \"kubernetes.io/projected/adaa4248-0b75-41b3-9d9e-48c8f1026162-kube-api-access-bhf8g\") pod \"adaa4248-0b75-41b3-9d9e-48c8f1026162\" (UID: \"adaa4248-0b75-41b3-9d9e-48c8f1026162\") "
Jul 14 23:38:33.213187 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:33.212952    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adaa4248-0b75-41b3-9d9e-48c8f1026162-home\") pod \"adaa4248-0b75-41b3-9d9e-48c8f1026162\" (UID: \"adaa4248-0b75-41b3-9d9e-48c8f1026162\") "
Jul 14 23:38:33.213187 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:33.212980    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adaa4248-0b75-41b3-9d9e-48c8f1026162-tls-certs\") pod \"adaa4248-0b75-41b3-9d9e-48c8f1026162\" (UID: \"adaa4248-0b75-41b3-9d9e-48c8f1026162\") "
Jul 14 23:38:33.213187 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:33.213135    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adaa4248-0b75-41b3-9d9e-48c8f1026162-dshm\") pod \"adaa4248-0b75-41b3-9d9e-48c8f1026162\" (UID: \"adaa4248-0b75-41b3-9d9e-48c8f1026162\") "
Jul 14 23:38:33.213389 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:33.213208    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adaa4248-0b75-41b3-9d9e-48c8f1026162-model-cache\") pod \"adaa4248-0b75-41b3-9d9e-48c8f1026162\" (UID: \"adaa4248-0b75-41b3-9d9e-48c8f1026162\") "
Jul 14 23:38:33.213389 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:33.213129    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adaa4248-0b75-41b3-9d9e-48c8f1026162-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "adaa4248-0b75-41b3-9d9e-48c8f1026162" (UID: "adaa4248-0b75-41b3-9d9e-48c8f1026162"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:38:33.213389 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:33.213228    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adaa4248-0b75-41b3-9d9e-48c8f1026162-home" (OuterVolumeSpecName: "home") pod "adaa4248-0b75-41b3-9d9e-48c8f1026162" (UID: "adaa4248-0b75-41b3-9d9e-48c8f1026162"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:38:33.213539 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:33.213461    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adaa4248-0b75-41b3-9d9e-48c8f1026162-model-cache" (OuterVolumeSpecName: "model-cache") pod "adaa4248-0b75-41b3-9d9e-48c8f1026162" (UID: "adaa4248-0b75-41b3-9d9e-48c8f1026162"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:38:33.213539 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:33.213518    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adaa4248-0b75-41b3-9d9e-48c8f1026162-home\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:38:33.213539 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:33.213532    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adaa4248-0b75-41b3-9d9e-48c8f1026162-tmp-dir\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:38:33.215365 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:33.215340    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adaa4248-0b75-41b3-9d9e-48c8f1026162-dshm" (OuterVolumeSpecName: "dshm") pod "adaa4248-0b75-41b3-9d9e-48c8f1026162" (UID: "adaa4248-0b75-41b3-9d9e-48c8f1026162"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:38:33.215474 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:33.215448    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adaa4248-0b75-41b3-9d9e-48c8f1026162-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "adaa4248-0b75-41b3-9d9e-48c8f1026162" (UID: "adaa4248-0b75-41b3-9d9e-48c8f1026162"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 23:38:33.215474 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:33.215454    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adaa4248-0b75-41b3-9d9e-48c8f1026162-kube-api-access-bhf8g" (OuterVolumeSpecName: "kube-api-access-bhf8g") pod "adaa4248-0b75-41b3-9d9e-48c8f1026162" (UID: "adaa4248-0b75-41b3-9d9e-48c8f1026162"). InnerVolumeSpecName "kube-api-access-bhf8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:38:33.314133 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:33.314101    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adaa4248-0b75-41b3-9d9e-48c8f1026162-model-cache\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:38:33.314133 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:33.314130    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhf8g\" (UniqueName: \"kubernetes.io/projected/adaa4248-0b75-41b3-9d9e-48c8f1026162-kube-api-access-bhf8g\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:38:33.314133 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:33.314140    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adaa4248-0b75-41b3-9d9e-48c8f1026162-tls-certs\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:38:33.314375 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:33.314149    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adaa4248-0b75-41b3-9d9e-48c8f1026162-dshm\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:38:33.568164 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:33.568127    2563 generic.go:356] "Generic (PLEG): container finished" podID="adaa4248-0b75-41b3-9d9e-48c8f1026162" containerID="a205a2fca30593783dacc3d7d8a6b2deffa4122508444070c0d55bd97fcdb067" exitCode=0
Jul 14 23:38:33.568371 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:33.568170    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d62786/auth-test-model-d62786-kserve-6f86689575-gxsrr"
Jul 14 23:38:33.568371 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:33.568218    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d62786/auth-test-model-d62786-kserve-6f86689575-gxsrr" event={"ID":"adaa4248-0b75-41b3-9d9e-48c8f1026162","Type":"ContainerDied","Data":"a205a2fca30593783dacc3d7d8a6b2deffa4122508444070c0d55bd97fcdb067"}
Jul 14 23:38:33.568371 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:33.568260    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d62786/auth-test-model-d62786-kserve-6f86689575-gxsrr" event={"ID":"adaa4248-0b75-41b3-9d9e-48c8f1026162","Type":"ContainerDied","Data":"c0c79580d518bbf15658773fba3e2e1757dcfd9fca496fa416fd1232586dd4e3"}
Jul 14 23:38:33.568371 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:33.568302    2563 scope.go:117] "RemoveContainer" containerID="a205a2fca30593783dacc3d7d8a6b2deffa4122508444070c0d55bd97fcdb067"
Jul 14 23:38:33.576885 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:33.576871    2563 scope.go:117] "RemoveContainer" containerID="a205a2fca30593783dacc3d7d8a6b2deffa4122508444070c0d55bd97fcdb067"
Jul 14 23:38:33.577170 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:38:33.577152    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a205a2fca30593783dacc3d7d8a6b2deffa4122508444070c0d55bd97fcdb067\": container with ID starting with a205a2fca30593783dacc3d7d8a6b2deffa4122508444070c0d55bd97fcdb067 not found: ID does not exist" containerID="a205a2fca30593783dacc3d7d8a6b2deffa4122508444070c0d55bd97fcdb067"
Jul 14 23:38:33.577211 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:33.577183    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a205a2fca30593783dacc3d7d8a6b2deffa4122508444070c0d55bd97fcdb067"} err="failed to get container status \"a205a2fca30593783dacc3d7d8a6b2deffa4122508444070c0d55bd97fcdb067\": rpc error: code = NotFound desc = could not find container \"a205a2fca30593783dacc3d7d8a6b2deffa4122508444070c0d55bd97fcdb067\": container with ID starting with a205a2fca30593783dacc3d7d8a6b2deffa4122508444070c0d55bd97fcdb067 not found: ID does not exist"
Jul 14 23:38:33.584710 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:33.584684    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d62786/auth-test-model-d62786-kserve-6f86689575-gxsrr"]
Jul 14 23:38:33.587776 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:33.587755    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-d62786/auth-test-model-d62786-kserve-6f86689575-gxsrr"]
Jul 14 23:38:33.718419 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:33.718378    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-638b58/auth-test-model-638b58-kserve-54b66cd79-p8knm" podUID="20f50c93-9a09-4fc9-9e48-954558575482" containerName="main" containerID="cri-o://78a8a95124018226c5af4652d1444e8d888f02eb06eeb184d98035509c43c528" gracePeriod=45
Jul 14 23:38:33.969094 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:33.969067    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-638b58/auth-test-model-638b58-kserve-54b66cd79-p8knm"
Jul 14 23:38:34.020894 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:34.020864    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20f50c93-9a09-4fc9-9e48-954558575482-model-cache\") pod \"20f50c93-9a09-4fc9-9e48-954558575482\" (UID: \"20f50c93-9a09-4fc9-9e48-954558575482\") "
Jul 14 23:38:34.020894 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:34.020898    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8zwf\" (UniqueName: \"kubernetes.io/projected/20f50c93-9a09-4fc9-9e48-954558575482-kube-api-access-v8zwf\") pod \"20f50c93-9a09-4fc9-9e48-954558575482\" (UID: \"20f50c93-9a09-4fc9-9e48-954558575482\") "
Jul 14 23:38:34.021078 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:34.020936    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20f50c93-9a09-4fc9-9e48-954558575482-tls-certs\") pod \"20f50c93-9a09-4fc9-9e48-954558575482\" (UID: \"20f50c93-9a09-4fc9-9e48-954558575482\") "
Jul 14 23:38:34.021078 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:34.020971    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20f50c93-9a09-4fc9-9e48-954558575482-home\") pod \"20f50c93-9a09-4fc9-9e48-954558575482\" (UID: \"20f50c93-9a09-4fc9-9e48-954558575482\") "
Jul 14 23:38:34.021078 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:34.021021    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20f50c93-9a09-4fc9-9e48-954558575482-dshm\") pod \"20f50c93-9a09-4fc9-9e48-954558575482\" (UID: \"20f50c93-9a09-4fc9-9e48-954558575482\") "
Jul 14 23:38:34.021078 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:34.021050    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20f50c93-9a09-4fc9-9e48-954558575482-tmp-dir\") pod \"20f50c93-9a09-4fc9-9e48-954558575482\" (UID: \"20f50c93-9a09-4fc9-9e48-954558575482\") "
Jul 14 23:38:34.021274 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:34.021169    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20f50c93-9a09-4fc9-9e48-954558575482-model-cache" (OuterVolumeSpecName: "model-cache") pod "20f50c93-9a09-4fc9-9e48-954558575482" (UID: "20f50c93-9a09-4fc9-9e48-954558575482"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:38:34.021274 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:34.021263    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20f50c93-9a09-4fc9-9e48-954558575482-home" (OuterVolumeSpecName: "home") pod "20f50c93-9a09-4fc9-9e48-954558575482" (UID: "20f50c93-9a09-4fc9-9e48-954558575482"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:38:34.021421 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:34.021371    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20f50c93-9a09-4fc9-9e48-954558575482-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "20f50c93-9a09-4fc9-9e48-954558575482" (UID: "20f50c93-9a09-4fc9-9e48-954558575482"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:38:34.021421 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:34.021400    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20f50c93-9a09-4fc9-9e48-954558575482-model-cache\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:38:34.021421 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:34.021418    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20f50c93-9a09-4fc9-9e48-954558575482-home\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:38:34.023384 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:34.023355    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20f50c93-9a09-4fc9-9e48-954558575482-dshm" (OuterVolumeSpecName: "dshm") pod "20f50c93-9a09-4fc9-9e48-954558575482" (UID: "20f50c93-9a09-4fc9-9e48-954558575482"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:38:34.023604 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:34.023587    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20f50c93-9a09-4fc9-9e48-954558575482-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "20f50c93-9a09-4fc9-9e48-954558575482" (UID: "20f50c93-9a09-4fc9-9e48-954558575482"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 23:38:34.023671 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:34.023656    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20f50c93-9a09-4fc9-9e48-954558575482-kube-api-access-v8zwf" (OuterVolumeSpecName: "kube-api-access-v8zwf") pod "20f50c93-9a09-4fc9-9e48-954558575482" (UID: "20f50c93-9a09-4fc9-9e48-954558575482"). InnerVolumeSpecName "kube-api-access-v8zwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:38:34.122374 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:34.122250    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8zwf\" (UniqueName: \"kubernetes.io/projected/20f50c93-9a09-4fc9-9e48-954558575482-kube-api-access-v8zwf\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:38:34.122374 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:34.122305    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20f50c93-9a09-4fc9-9e48-954558575482-tls-certs\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:38:34.122374 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:34.122319    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20f50c93-9a09-4fc9-9e48-954558575482-dshm\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:38:34.122374 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:34.122331    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20f50c93-9a09-4fc9-9e48-954558575482-tmp-dir\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:38:34.574566 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:34.574531    2563 generic.go:356] "Generic (PLEG): container finished" podID="20f50c93-9a09-4fc9-9e48-954558575482" containerID="78a8a95124018226c5af4652d1444e8d888f02eb06eeb184d98035509c43c528" exitCode=0
Jul 14 23:38:34.574769 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:34.574575    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-638b58/auth-test-model-638b58-kserve-54b66cd79-p8knm"
Jul 14 23:38:34.574769 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:34.574575    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-638b58/auth-test-model-638b58-kserve-54b66cd79-p8knm" event={"ID":"20f50c93-9a09-4fc9-9e48-954558575482","Type":"ContainerDied","Data":"78a8a95124018226c5af4652d1444e8d888f02eb06eeb184d98035509c43c528"}
Jul 14 23:38:34.574769 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:34.574714    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-638b58/auth-test-model-638b58-kserve-54b66cd79-p8knm" event={"ID":"20f50c93-9a09-4fc9-9e48-954558575482","Type":"ContainerDied","Data":"16b6293f39e0ce49bd79a33b5517e2c6288e9e0b0b4c5137486b1baccfc709e2"}
Jul 14 23:38:34.574769 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:34.574744    2563 scope.go:117] "RemoveContainer" containerID="78a8a95124018226c5af4652d1444e8d888f02eb06eeb184d98035509c43c528"
Jul 14 23:38:34.583871 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:34.583855    2563 scope.go:117] "RemoveContainer" containerID="78a8a95124018226c5af4652d1444e8d888f02eb06eeb184d98035509c43c528"
Jul 14 23:38:34.584144 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:38:34.584123    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78a8a95124018226c5af4652d1444e8d888f02eb06eeb184d98035509c43c528\": container with ID starting with 78a8a95124018226c5af4652d1444e8d888f02eb06eeb184d98035509c43c528 not found: ID does not exist" containerID="78a8a95124018226c5af4652d1444e8d888f02eb06eeb184d98035509c43c528"
Jul 14 23:38:34.584235 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:34.584151    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78a8a95124018226c5af4652d1444e8d888f02eb06eeb184d98035509c43c528"} err="failed to get container status \"78a8a95124018226c5af4652d1444e8d888f02eb06eeb184d98035509c43c528\": rpc error: code = NotFound desc = could not find container \"78a8a95124018226c5af4652d1444e8d888f02eb06eeb184d98035509c43c528\": container with ID starting with 78a8a95124018226c5af4652d1444e8d888f02eb06eeb184d98035509c43c528 not found: ID does not exist"
Jul 14 23:38:34.596837 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:34.596815    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-638b58/auth-test-model-638b58-kserve-54b66cd79-p8knm"]
Jul 14 23:38:34.600725 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:34.600702    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-638b58/auth-test-model-638b58-kserve-54b66cd79-p8knm"]
Jul 14 23:38:35.229209 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:35.229175    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20f50c93-9a09-4fc9-9e48-954558575482" path="/var/lib/kubelet/pods/20f50c93-9a09-4fc9-9e48-954558575482/volumes"
Jul 14 23:38:35.229620 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:35.229587    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adaa4248-0b75-41b3-9d9e-48c8f1026162" path="/var/lib/kubelet/pods/adaa4248-0b75-41b3-9d9e-48c8f1026162/volumes"
Jul 14 23:38:45.596795 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:45.596718    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-638b58/sub-test-model-638b58-kserve-f8f9fb955-hlb94"]
Jul 14 23:38:45.597253 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:45.597075    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adaa4248-0b75-41b3-9d9e-48c8f1026162" containerName="main"
Jul 14 23:38:45.597253 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:45.597087    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="adaa4248-0b75-41b3-9d9e-48c8f1026162" containerName="main"
Jul 14 23:38:45.597253 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:45.597112    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20f50c93-9a09-4fc9-9e48-954558575482" containerName="main"
Jul 14 23:38:45.597253 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:45.597118    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="20f50c93-9a09-4fc9-9e48-954558575482" containerName="main"
Jul 14 23:38:45.597253 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:45.597178    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="20f50c93-9a09-4fc9-9e48-954558575482" containerName="main"
Jul 14 23:38:45.597253 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:45.597193    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="adaa4248-0b75-41b3-9d9e-48c8f1026162" containerName="main"
Jul 14 23:38:45.601401 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:45.601381    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-638b58/sub-test-model-638b58-kserve-f8f9fb955-hlb94"
Jul 14 23:38:45.604472 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:45.604448    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-638b58\"/\"kube-root-ca.crt\""
Jul 14 23:38:45.604595 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:45.604531    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-638b58\"/\"openshift-service-ca.crt\""
Jul 14 23:38:45.606121 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:45.606103    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-638b58\"/\"sub-test-model-638b58-kserve-self-signed-certs\""
Jul 14 23:38:45.606215 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:45.606174    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-638b58\"/\"default-dockercfg-xvhb8\""
Jul 14 23:38:45.613677 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:45.613655    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-638b58/sub-test-model-638b58-kserve-f8f9fb955-hlb94"]
Jul 14 23:38:45.727048 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:45.727001    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d55a4e35-54f8-4439-aa23-1f3b8e1c44c0-tls-certs\") pod \"sub-test-model-638b58-kserve-f8f9fb955-hlb94\" (UID: \"d55a4e35-54f8-4439-aa23-1f3b8e1c44c0\") " pod="ai-tenant-e2e-shared-b-638b58/sub-test-model-638b58-kserve-f8f9fb955-hlb94"
Jul 14 23:38:45.727048 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:45.727047    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d55a4e35-54f8-4439-aa23-1f3b8e1c44c0-home\") pod \"sub-test-model-638b58-kserve-f8f9fb955-hlb94\" (UID: \"d55a4e35-54f8-4439-aa23-1f3b8e1c44c0\") " pod="ai-tenant-e2e-shared-b-638b58/sub-test-model-638b58-kserve-f8f9fb955-hlb94"
Jul 14 23:38:45.727322 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:45.727091    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d55a4e35-54f8-4439-aa23-1f3b8e1c44c0-dshm\") pod \"sub-test-model-638b58-kserve-f8f9fb955-hlb94\" (UID: \"d55a4e35-54f8-4439-aa23-1f3b8e1c44c0\") " pod="ai-tenant-e2e-shared-b-638b58/sub-test-model-638b58-kserve-f8f9fb955-hlb94"
Jul 14 23:38:45.727322 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:45.727111    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d55a4e35-54f8-4439-aa23-1f3b8e1c44c0-tmp-dir\") pod \"sub-test-model-638b58-kserve-f8f9fb955-hlb94\" (UID: \"d55a4e35-54f8-4439-aa23-1f3b8e1c44c0\") " pod="ai-tenant-e2e-shared-b-638b58/sub-test-model-638b58-kserve-f8f9fb955-hlb94"
Jul 14 23:38:45.727322 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:45.727201    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4d8g\" (UniqueName: \"kubernetes.io/projected/d55a4e35-54f8-4439-aa23-1f3b8e1c44c0-kube-api-access-b4d8g\") pod \"sub-test-model-638b58-kserve-f8f9fb955-hlb94\" (UID: \"d55a4e35-54f8-4439-aa23-1f3b8e1c44c0\") " pod="ai-tenant-e2e-shared-b-638b58/sub-test-model-638b58-kserve-f8f9fb955-hlb94"
Jul 14 23:38:45.727322 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:45.727249    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d55a4e35-54f8-4439-aa23-1f3b8e1c44c0-model-cache\") pod \"sub-test-model-638b58-kserve-f8f9fb955-hlb94\" (UID: \"d55a4e35-54f8-4439-aa23-1f3b8e1c44c0\") " pod="ai-tenant-e2e-shared-b-638b58/sub-test-model-638b58-kserve-f8f9fb955-hlb94"
Jul 14 23:38:45.828031 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:45.827991    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d55a4e35-54f8-4439-aa23-1f3b8e1c44c0-home\") pod \"sub-test-model-638b58-kserve-f8f9fb955-hlb94\" (UID: \"d55a4e35-54f8-4439-aa23-1f3b8e1c44c0\") " pod="ai-tenant-e2e-shared-b-638b58/sub-test-model-638b58-kserve-f8f9fb955-hlb94"
Jul 14 23:38:45.828321 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:45.828061    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d55a4e35-54f8-4439-aa23-1f3b8e1c44c0-dshm\") pod \"sub-test-model-638b58-kserve-f8f9fb955-hlb94\" (UID: \"d55a4e35-54f8-4439-aa23-1f3b8e1c44c0\") " pod="ai-tenant-e2e-shared-b-638b58/sub-test-model-638b58-kserve-f8f9fb955-hlb94"
Jul 14 23:38:45.828321 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:45.828091    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d55a4e35-54f8-4439-aa23-1f3b8e1c44c0-tmp-dir\") pod \"sub-test-model-638b58-kserve-f8f9fb955-hlb94\" (UID: \"d55a4e35-54f8-4439-aa23-1f3b8e1c44c0\") " pod="ai-tenant-e2e-shared-b-638b58/sub-test-model-638b58-kserve-f8f9fb955-hlb94"
Jul 14 23:38:45.828321 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:45.828133    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4d8g\" (UniqueName: \"kubernetes.io/projected/d55a4e35-54f8-4439-aa23-1f3b8e1c44c0-kube-api-access-b4d8g\") pod \"sub-test-model-638b58-kserve-f8f9fb955-hlb94\" (UID: \"d55a4e35-54f8-4439-aa23-1f3b8e1c44c0\") " pod="ai-tenant-e2e-shared-b-638b58/sub-test-model-638b58-kserve-f8f9fb955-hlb94"
Jul 14 23:38:45.828321 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:45.828180    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d55a4e35-54f8-4439-aa23-1f3b8e1c44c0-model-cache\") pod \"sub-test-model-638b58-kserve-f8f9fb955-hlb94\" (UID: \"d55a4e35-54f8-4439-aa23-1f3b8e1c44c0\") " pod="ai-tenant-e2e-shared-b-638b58/sub-test-model-638b58-kserve-f8f9fb955-hlb94"
Jul 14 23:38:45.828321 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:45.828238    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d55a4e35-54f8-4439-aa23-1f3b8e1c44c0-tls-certs\") pod \"sub-test-model-638b58-kserve-f8f9fb955-hlb94\" (UID: \"d55a4e35-54f8-4439-aa23-1f3b8e1c44c0\") " pod="ai-tenant-e2e-shared-b-638b58/sub-test-model-638b58-kserve-f8f9fb955-hlb94"
Jul 14 23:38:45.828604 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:45.828548    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d55a4e35-54f8-4439-aa23-1f3b8e1c44c0-home\") pod \"sub-test-model-638b58-kserve-f8f9fb955-hlb94\" (UID: \"d55a4e35-54f8-4439-aa23-1f3b8e1c44c0\") " pod="ai-tenant-e2e-shared-b-638b58/sub-test-model-638b58-kserve-f8f9fb955-hlb94"
Jul 14 23:38:45.828604 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:45.828579    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d55a4e35-54f8-4439-aa23-1f3b8e1c44c0-tmp-dir\") pod \"sub-test-model-638b58-kserve-f8f9fb955-hlb94\" (UID: \"d55a4e35-54f8-4439-aa23-1f3b8e1c44c0\") " pod="ai-tenant-e2e-shared-b-638b58/sub-test-model-638b58-kserve-f8f9fb955-hlb94"
Jul 14 23:38:45.828673 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:45.828613    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d55a4e35-54f8-4439-aa23-1f3b8e1c44c0-model-cache\") pod \"sub-test-model-638b58-kserve-f8f9fb955-hlb94\" (UID: \"d55a4e35-54f8-4439-aa23-1f3b8e1c44c0\") " pod="ai-tenant-e2e-shared-b-638b58/sub-test-model-638b58-kserve-f8f9fb955-hlb94"
Jul 14 23:38:45.830682 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:45.830656    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d55a4e35-54f8-4439-aa23-1f3b8e1c44c0-dshm\") pod \"sub-test-model-638b58-kserve-f8f9fb955-hlb94\" (UID: \"d55a4e35-54f8-4439-aa23-1f3b8e1c44c0\") " pod="ai-tenant-e2e-shared-b-638b58/sub-test-model-638b58-kserve-f8f9fb955-hlb94"
Jul 14 23:38:45.830791 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:45.830777    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d55a4e35-54f8-4439-aa23-1f3b8e1c44c0-tls-certs\") pod \"sub-test-model-638b58-kserve-f8f9fb955-hlb94\" (UID: \"d55a4e35-54f8-4439-aa23-1f3b8e1c44c0\") " pod="ai-tenant-e2e-shared-b-638b58/sub-test-model-638b58-kserve-f8f9fb955-hlb94"
Jul 14 23:38:45.839629 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:45.839609    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4d8g\" (UniqueName: \"kubernetes.io/projected/d55a4e35-54f8-4439-aa23-1f3b8e1c44c0-kube-api-access-b4d8g\") pod \"sub-test-model-638b58-kserve-f8f9fb955-hlb94\" (UID: \"d55a4e35-54f8-4439-aa23-1f3b8e1c44c0\") " pod="ai-tenant-e2e-shared-b-638b58/sub-test-model-638b58-kserve-f8f9fb955-hlb94"
Jul 14 23:38:45.910955 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:45.910863    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-638b58/sub-test-model-638b58-kserve-f8f9fb955-hlb94"
Jul 14 23:38:46.048241 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:46.048200    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-638b58/sub-test-model-638b58-kserve-f8f9fb955-hlb94"]
Jul 14 23:38:46.050559 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:38:46.050532    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd55a4e35_54f8_4439_aa23_1f3b8e1c44c0.slice/crio-d0ddf5cd96d83a16b736b60a609f4261695a2aa3507a7a44aa1dd96b01a48e0d WatchSource:0}: Error finding container d0ddf5cd96d83a16b736b60a609f4261695a2aa3507a7a44aa1dd96b01a48e0d: Status 404 returned error can't find the container with id d0ddf5cd96d83a16b736b60a609f4261695a2aa3507a7a44aa1dd96b01a48e0d
Jul 14 23:38:46.633363 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:46.633325    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-638b58/sub-test-model-638b58-kserve-f8f9fb955-hlb94" event={"ID":"d55a4e35-54f8-4439-aa23-1f3b8e1c44c0","Type":"ContainerStarted","Data":"b99c1db7aefbea43eba40d292e35544c652f607f9e59f0fa379f6204e10e7927"}
Jul 14 23:38:46.633363 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:46.633370    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-638b58/sub-test-model-638b58-kserve-f8f9fb955-hlb94" event={"ID":"d55a4e35-54f8-4439-aa23-1f3b8e1c44c0","Type":"ContainerStarted","Data":"d0ddf5cd96d83a16b736b60a609f4261695a2aa3507a7a44aa1dd96b01a48e0d"}
Jul 14 23:38:46.652472 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:46.652423    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-638b58/sub-test-model-638b58-kserve-f8f9fb955-hlb94" podStartSLOduration=1.652407888 podStartE2EDuration="1.652407888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:38:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:38:46.650421216 +0000 UTC m=+2513.953412009" watchObservedRunningTime="2026-07-14 23:38:46.652407888 +0000 UTC m=+2513.955398680"
Jul 14 23:38:55.911323 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:55.911254    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-638b58/sub-test-model-638b58-kserve-f8f9fb955-hlb94"
Jul 14 23:38:55.911808 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:55.911344    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-638b58/sub-test-model-638b58-kserve-f8f9fb955-hlb94"
Jul 14 23:38:55.924229 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:55.924201    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-638b58/sub-test-model-638b58-kserve-f8f9fb955-hlb94"
Jul 14 23:38:56.690917 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:38:56.690888    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-638b58/sub-test-model-638b58-kserve-f8f9fb955-hlb94"
Jul 14 23:39:33.503396 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:33.503360    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-638b58/sub-test-model-638b58-kserve-f8f9fb955-hlb94"]
Jul 14 23:39:41.391668 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:41.391624    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-d62786/rate-test-model-d62786-kserve-86c954d8f9-nbmbm"]
Jul 14 23:39:41.394982 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:41.394964    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d62786/rate-test-model-d62786-kserve-86c954d8f9-nbmbm"
Jul 14 23:39:41.398174 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:41.398152    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-d62786\"/\"openshift-service-ca.crt\""
Jul 14 23:39:41.398322 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:41.398191    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-d62786\"/\"kube-root-ca.crt\""
Jul 14 23:39:41.399503 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:41.399484    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d62786\"/\"default-dockercfg-mvxpj\""
Jul 14 23:39:41.399571 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:41.399526    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d62786\"/\"rate-test-model-d62786-kserve-self-signed-certs\""
Jul 14 23:39:41.404481 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:41.404460    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d62786/rate-test-model-d62786-kserve-86c954d8f9-nbmbm"]
Jul 14 23:39:41.431874 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:41.431844    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/492a9ea3-4908-4fb0-be0d-7eeac57acf57-dshm\") pod \"rate-test-model-d62786-kserve-86c954d8f9-nbmbm\" (UID: \"492a9ea3-4908-4fb0-be0d-7eeac57acf57\") " pod="ai-tenant-e2e-shared-a-d62786/rate-test-model-d62786-kserve-86c954d8f9-nbmbm"
Jul 14 23:39:41.432033 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:41.431901    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/492a9ea3-4908-4fb0-be0d-7eeac57acf57-home\") pod \"rate-test-model-d62786-kserve-86c954d8f9-nbmbm\" (UID: \"492a9ea3-4908-4fb0-be0d-7eeac57acf57\") " pod="ai-tenant-e2e-shared-a-d62786/rate-test-model-d62786-kserve-86c954d8f9-nbmbm"
Jul 14 23:39:41.432033 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:41.431945    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/492a9ea3-4908-4fb0-be0d-7eeac57acf57-tmp-dir\") pod \"rate-test-model-d62786-kserve-86c954d8f9-nbmbm\" (UID: \"492a9ea3-4908-4fb0-be0d-7eeac57acf57\") " pod="ai-tenant-e2e-shared-a-d62786/rate-test-model-d62786-kserve-86c954d8f9-nbmbm"
Jul 14 23:39:41.432033 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:41.431991    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/492a9ea3-4908-4fb0-be0d-7eeac57acf57-model-cache\") pod \"rate-test-model-d62786-kserve-86c954d8f9-nbmbm\" (UID: \"492a9ea3-4908-4fb0-be0d-7eeac57acf57\") " pod="ai-tenant-e2e-shared-a-d62786/rate-test-model-d62786-kserve-86c954d8f9-nbmbm"
Jul 14 23:39:41.432168 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:41.432042    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrf7b\" (UniqueName: \"kubernetes.io/projected/492a9ea3-4908-4fb0-be0d-7eeac57acf57-kube-api-access-vrf7b\") pod \"rate-test-model-d62786-kserve-86c954d8f9-nbmbm\" (UID: \"492a9ea3-4908-4fb0-be0d-7eeac57acf57\") " pod="ai-tenant-e2e-shared-a-d62786/rate-test-model-d62786-kserve-86c954d8f9-nbmbm"
Jul 14 23:39:41.432168 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:41.432095    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/492a9ea3-4908-4fb0-be0d-7eeac57acf57-tls-certs\") pod \"rate-test-model-d62786-kserve-86c954d8f9-nbmbm\" (UID: \"492a9ea3-4908-4fb0-be0d-7eeac57acf57\") " pod="ai-tenant-e2e-shared-a-d62786/rate-test-model-d62786-kserve-86c954d8f9-nbmbm"
Jul 14 23:39:41.533372 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:41.533336    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrf7b\" (UniqueName: \"kubernetes.io/projected/492a9ea3-4908-4fb0-be0d-7eeac57acf57-kube-api-access-vrf7b\") pod \"rate-test-model-d62786-kserve-86c954d8f9-nbmbm\" (UID: \"492a9ea3-4908-4fb0-be0d-7eeac57acf57\") " pod="ai-tenant-e2e-shared-a-d62786/rate-test-model-d62786-kserve-86c954d8f9-nbmbm"
Jul 14 23:39:41.533372 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:41.533379    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/492a9ea3-4908-4fb0-be0d-7eeac57acf57-tls-certs\") pod \"rate-test-model-d62786-kserve-86c954d8f9-nbmbm\" (UID: \"492a9ea3-4908-4fb0-be0d-7eeac57acf57\") " pod="ai-tenant-e2e-shared-a-d62786/rate-test-model-d62786-kserve-86c954d8f9-nbmbm"
Jul 14 23:39:41.533621 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:41.533455    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/492a9ea3-4908-4fb0-be0d-7eeac57acf57-dshm\") pod \"rate-test-model-d62786-kserve-86c954d8f9-nbmbm\" (UID: \"492a9ea3-4908-4fb0-be0d-7eeac57acf57\") " pod="ai-tenant-e2e-shared-a-d62786/rate-test-model-d62786-kserve-86c954d8f9-nbmbm"
Jul 14 23:39:41.533621 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:41.533494    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/492a9ea3-4908-4fb0-be0d-7eeac57acf57-home\") pod \"rate-test-model-d62786-kserve-86c954d8f9-nbmbm\" (UID: \"492a9ea3-4908-4fb0-be0d-7eeac57acf57\") " pod="ai-tenant-e2e-shared-a-d62786/rate-test-model-d62786-kserve-86c954d8f9-nbmbm"
Jul 14 23:39:41.533621 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:41.533517    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/492a9ea3-4908-4fb0-be0d-7eeac57acf57-tmp-dir\") pod \"rate-test-model-d62786-kserve-86c954d8f9-nbmbm\" (UID: \"492a9ea3-4908-4fb0-be0d-7eeac57acf57\") " pod="ai-tenant-e2e-shared-a-d62786/rate-test-model-d62786-kserve-86c954d8f9-nbmbm"
Jul 14 23:39:41.533621 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:41.533549    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/492a9ea3-4908-4fb0-be0d-7eeac57acf57-model-cache\") pod \"rate-test-model-d62786-kserve-86c954d8f9-nbmbm\" (UID: \"492a9ea3-4908-4fb0-be0d-7eeac57acf57\") " pod="ai-tenant-e2e-shared-a-d62786/rate-test-model-d62786-kserve-86c954d8f9-nbmbm"
Jul 14 23:39:41.533996 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:41.533965    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/492a9ea3-4908-4fb0-be0d-7eeac57acf57-tmp-dir\") pod \"rate-test-model-d62786-kserve-86c954d8f9-nbmbm\" (UID: \"492a9ea3-4908-4fb0-be0d-7eeac57acf57\") " pod="ai-tenant-e2e-shared-a-d62786/rate-test-model-d62786-kserve-86c954d8f9-nbmbm"
Jul 14 23:39:41.534076 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:41.533991    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/492a9ea3-4908-4fb0-be0d-7eeac57acf57-model-cache\") pod \"rate-test-model-d62786-kserve-86c954d8f9-nbmbm\" (UID: \"492a9ea3-4908-4fb0-be0d-7eeac57acf57\") " pod="ai-tenant-e2e-shared-a-d62786/rate-test-model-d62786-kserve-86c954d8f9-nbmbm"
Jul 14 23:39:41.534076 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:41.534032    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/492a9ea3-4908-4fb0-be0d-7eeac57acf57-home\") pod \"rate-test-model-d62786-kserve-86c954d8f9-nbmbm\" (UID: \"492a9ea3-4908-4fb0-be0d-7eeac57acf57\") " pod="ai-tenant-e2e-shared-a-d62786/rate-test-model-d62786-kserve-86c954d8f9-nbmbm"
Jul 14 23:39:41.535998 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:41.535968    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/492a9ea3-4908-4fb0-be0d-7eeac57acf57-dshm\") pod \"rate-test-model-d62786-kserve-86c954d8f9-nbmbm\" (UID: \"492a9ea3-4908-4fb0-be0d-7eeac57acf57\") " pod="ai-tenant-e2e-shared-a-d62786/rate-test-model-d62786-kserve-86c954d8f9-nbmbm"
Jul 14 23:39:41.536167 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:41.536150    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/492a9ea3-4908-4fb0-be0d-7eeac57acf57-tls-certs\") pod \"rate-test-model-d62786-kserve-86c954d8f9-nbmbm\" (UID: \"492a9ea3-4908-4fb0-be0d-7eeac57acf57\") " pod="ai-tenant-e2e-shared-a-d62786/rate-test-model-d62786-kserve-86c954d8f9-nbmbm"
Jul 14 23:39:41.543202 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:41.543180    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrf7b\" (UniqueName: \"kubernetes.io/projected/492a9ea3-4908-4fb0-be0d-7eeac57acf57-kube-api-access-vrf7b\") pod \"rate-test-model-d62786-kserve-86c954d8f9-nbmbm\" (UID: \"492a9ea3-4908-4fb0-be0d-7eeac57acf57\") " pod="ai-tenant-e2e-shared-a-d62786/rate-test-model-d62786-kserve-86c954d8f9-nbmbm"
Jul 14 23:39:41.705219 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:41.705114    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d62786/rate-test-model-d62786-kserve-86c954d8f9-nbmbm"
Jul 14 23:39:41.844195 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:41.844170    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d62786/rate-test-model-d62786-kserve-86c954d8f9-nbmbm"]
Jul 14 23:39:41.846605 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:39:41.846580    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod492a9ea3_4908_4fb0_be0d_7eeac57acf57.slice/crio-a6f6b5ac31d454dcc4d1d4ed67e5c581d8eece74787b5170a575b063aca466d4 WatchSource:0}: Error finding container a6f6b5ac31d454dcc4d1d4ed67e5c581d8eece74787b5170a575b063aca466d4: Status 404 returned error can't find the container with id a6f6b5ac31d454dcc4d1d4ed67e5c581d8eece74787b5170a575b063aca466d4
Jul 14 23:39:41.900148 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:41.900122    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d62786/rate-test-model-d62786-kserve-86c954d8f9-nbmbm" event={"ID":"492a9ea3-4908-4fb0-be0d-7eeac57acf57","Type":"ContainerStarted","Data":"a6f6b5ac31d454dcc4d1d4ed67e5c581d8eece74787b5170a575b063aca466d4"}
Jul 14 23:39:42.906844 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:42.906802    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d62786/rate-test-model-d62786-kserve-86c954d8f9-nbmbm" event={"ID":"492a9ea3-4908-4fb0-be0d-7eeac57acf57","Type":"ContainerStarted","Data":"3eaddda261c09bbfc0eaade5b4cf3376c6288ef07992d77393f2050edc016498"}
Jul 14 23:39:42.924902 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:42.924851    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-d62786/rate-test-model-d62786-kserve-86c954d8f9-nbmbm" podStartSLOduration=1.924836736 podStartE2EDuration="1.924836736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:39:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:39:42.923849212 +0000 UTC m=+2570.226840015" watchObservedRunningTime="2026-07-14 23:39:42.924836736 +0000 UTC m=+2570.227827568"
Jul 14 23:39:48.523568 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:48.523527    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-638b58/sub-test-model-638b58-kserve-f8f9fb955-hlb94" podUID="d55a4e35-54f8-4439-aa23-1f3b8e1c44c0" containerName="main" containerID="cri-o://b99c1db7aefbea43eba40d292e35544c652f607f9e59f0fa379f6204e10e7927" gracePeriod=45
Jul 14 23:39:48.775832 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:48.775772    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-638b58/sub-test-model-638b58-kserve-f8f9fb955-hlb94"
Jul 14 23:39:48.799494 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:48.799466    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4d8g\" (UniqueName: \"kubernetes.io/projected/d55a4e35-54f8-4439-aa23-1f3b8e1c44c0-kube-api-access-b4d8g\") pod \"d55a4e35-54f8-4439-aa23-1f3b8e1c44c0\" (UID: \"d55a4e35-54f8-4439-aa23-1f3b8e1c44c0\") "
Jul 14 23:39:48.799633 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:48.799513    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d55a4e35-54f8-4439-aa23-1f3b8e1c44c0-home\") pod \"d55a4e35-54f8-4439-aa23-1f3b8e1c44c0\" (UID: \"d55a4e35-54f8-4439-aa23-1f3b8e1c44c0\") "
Jul 14 23:39:48.799633 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:48.799585    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d55a4e35-54f8-4439-aa23-1f3b8e1c44c0-dshm\") pod \"d55a4e35-54f8-4439-aa23-1f3b8e1c44c0\" (UID: \"d55a4e35-54f8-4439-aa23-1f3b8e1c44c0\") "
Jul 14 23:39:48.799633 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:48.799621    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d55a4e35-54f8-4439-aa23-1f3b8e1c44c0-tmp-dir\") pod \"d55a4e35-54f8-4439-aa23-1f3b8e1c44c0\" (UID: \"d55a4e35-54f8-4439-aa23-1f3b8e1c44c0\") "
Jul 14 23:39:48.799805 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:48.799635    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d55a4e35-54f8-4439-aa23-1f3b8e1c44c0-model-cache\") pod \"d55a4e35-54f8-4439-aa23-1f3b8e1c44c0\" (UID: \"d55a4e35-54f8-4439-aa23-1f3b8e1c44c0\") "
Jul 14 23:39:48.799805 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:48.799663    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d55a4e35-54f8-4439-aa23-1f3b8e1c44c0-tls-certs\") pod \"d55a4e35-54f8-4439-aa23-1f3b8e1c44c0\" (UID: \"d55a4e35-54f8-4439-aa23-1f3b8e1c44c0\") "
Jul 14 23:39:48.799805 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:48.799717    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d55a4e35-54f8-4439-aa23-1f3b8e1c44c0-home" (OuterVolumeSpecName: "home") pod "d55a4e35-54f8-4439-aa23-1f3b8e1c44c0" (UID: "d55a4e35-54f8-4439-aa23-1f3b8e1c44c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:39:48.799945 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:48.799869    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d55a4e35-54f8-4439-aa23-1f3b8e1c44c0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d55a4e35-54f8-4439-aa23-1f3b8e1c44c0" (UID: "d55a4e35-54f8-4439-aa23-1f3b8e1c44c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:39:48.800011 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:48.799988    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d55a4e35-54f8-4439-aa23-1f3b8e1c44c0-model-cache" (OuterVolumeSpecName: "model-cache") pod "d55a4e35-54f8-4439-aa23-1f3b8e1c44c0" (UID: "d55a4e35-54f8-4439-aa23-1f3b8e1c44c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:39:48.800070 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:48.800006    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d55a4e35-54f8-4439-aa23-1f3b8e1c44c0-home\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:39:48.800070 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:48.800025    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d55a4e35-54f8-4439-aa23-1f3b8e1c44c0-tmp-dir\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:39:48.801789 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:48.801761    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d55a4e35-54f8-4439-aa23-1f3b8e1c44c0-kube-api-access-b4d8g" (OuterVolumeSpecName: "kube-api-access-b4d8g") pod "d55a4e35-54f8-4439-aa23-1f3b8e1c44c0" (UID: "d55a4e35-54f8-4439-aa23-1f3b8e1c44c0"). InnerVolumeSpecName "kube-api-access-b4d8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:39:48.801927 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:48.801801    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d55a4e35-54f8-4439-aa23-1f3b8e1c44c0-dshm" (OuterVolumeSpecName: "dshm") pod "d55a4e35-54f8-4439-aa23-1f3b8e1c44c0" (UID: "d55a4e35-54f8-4439-aa23-1f3b8e1c44c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:39:48.801927 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:48.801866    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d55a4e35-54f8-4439-aa23-1f3b8e1c44c0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d55a4e35-54f8-4439-aa23-1f3b8e1c44c0" (UID: "d55a4e35-54f8-4439-aa23-1f3b8e1c44c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 23:39:48.901074 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:48.901033    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d55a4e35-54f8-4439-aa23-1f3b8e1c44c0-tls-certs\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:39:48.901074 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:48.901070    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4d8g\" (UniqueName: \"kubernetes.io/projected/d55a4e35-54f8-4439-aa23-1f3b8e1c44c0-kube-api-access-b4d8g\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:39:48.901074 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:48.901081    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d55a4e35-54f8-4439-aa23-1f3b8e1c44c0-dshm\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:39:48.901074 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:48.901090    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d55a4e35-54f8-4439-aa23-1f3b8e1c44c0-model-cache\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:39:48.936416 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:48.936388    2563 generic.go:356] "Generic (PLEG): container finished" podID="d55a4e35-54f8-4439-aa23-1f3b8e1c44c0" containerID="b99c1db7aefbea43eba40d292e35544c652f607f9e59f0fa379f6204e10e7927" exitCode=0
Jul 14 23:39:48.936592 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:48.936432    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-638b58/sub-test-model-638b58-kserve-f8f9fb955-hlb94"
Jul 14 23:39:48.936592 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:48.936478    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-638b58/sub-test-model-638b58-kserve-f8f9fb955-hlb94" event={"ID":"d55a4e35-54f8-4439-aa23-1f3b8e1c44c0","Type":"ContainerDied","Data":"b99c1db7aefbea43eba40d292e35544c652f607f9e59f0fa379f6204e10e7927"}
Jul 14 23:39:48.936592 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:48.936520    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-638b58/sub-test-model-638b58-kserve-f8f9fb955-hlb94" event={"ID":"d55a4e35-54f8-4439-aa23-1f3b8e1c44c0","Type":"ContainerDied","Data":"d0ddf5cd96d83a16b736b60a609f4261695a2aa3507a7a44aa1dd96b01a48e0d"}
Jul 14 23:39:48.936592 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:48.936542    2563 scope.go:117] "RemoveContainer" containerID="b99c1db7aefbea43eba40d292e35544c652f607f9e59f0fa379f6204e10e7927"
Jul 14 23:39:48.945766 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:48.945750    2563 scope.go:117] "RemoveContainer" containerID="b99c1db7aefbea43eba40d292e35544c652f607f9e59f0fa379f6204e10e7927"
Jul 14 23:39:48.946000 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:39:48.945982    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b99c1db7aefbea43eba40d292e35544c652f607f9e59f0fa379f6204e10e7927\": container with ID starting with b99c1db7aefbea43eba40d292e35544c652f607f9e59f0fa379f6204e10e7927 not found: ID does not exist" containerID="b99c1db7aefbea43eba40d292e35544c652f607f9e59f0fa379f6204e10e7927"
Jul 14 23:39:48.946054 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:48.946009    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b99c1db7aefbea43eba40d292e35544c652f607f9e59f0fa379f6204e10e7927"} err="failed to get container status \"b99c1db7aefbea43eba40d292e35544c652f607f9e59f0fa379f6204e10e7927\": rpc error: code = NotFound desc = could not find container \"b99c1db7aefbea43eba40d292e35544c652f607f9e59f0fa379f6204e10e7927\": container with ID starting with b99c1db7aefbea43eba40d292e35544c652f607f9e59f0fa379f6204e10e7927 not found: ID does not exist"
Jul 14 23:39:48.958642 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:48.958621    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-638b58/sub-test-model-638b58-kserve-f8f9fb955-hlb94"]
Jul 14 23:39:48.964108 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:48.964087    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-638b58/sub-test-model-638b58-kserve-f8f9fb955-hlb94"]
Jul 14 23:39:49.234918 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:49.234833    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d55a4e35-54f8-4439-aa23-1f3b8e1c44c0" path="/var/lib/kubelet/pods/d55a4e35-54f8-4439-aa23-1f3b8e1c44c0/volumes"
Jul 14 23:39:51.706119 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:51.706085    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-d62786/rate-test-model-d62786-kserve-86c954d8f9-nbmbm"
Jul 14 23:39:51.706578 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:51.706139    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-d62786/rate-test-model-d62786-kserve-86c954d8f9-nbmbm"
Jul 14 23:39:51.718372 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:51.718343    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-d62786/rate-test-model-d62786-kserve-86c954d8f9-nbmbm"
Jul 14 23:39:51.962199 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:39:51.962114    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-d62786/rate-test-model-d62786-kserve-86c954d8f9-nbmbm"
Jul 14 23:41:10.430239 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:10.430206    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d62786/rate-test-model-d62786-kserve-86c954d8f9-nbmbm"]
Jul 14 23:41:12.272119 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:12.272086    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7800-140280926443696"]
Jul 14 23:41:12.272512 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:12.272420    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d55a4e35-54f8-4439-aa23-1f3b8e1c44c0" containerName="main"
Jul 14 23:41:12.272512 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:12.272430    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d55a4e35-54f8-4439-aa23-1f3b8e1c44c0" containerName="main"
Jul 14 23:41:12.272512 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:12.272496    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d55a4e35-54f8-4439-aa23-1f3b8e1c44c0" containerName="main"
Jul 14 23:41:12.275464 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:12.275446    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7800-140280926443696"
Jul 14 23:41:12.278671 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:12.278650    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-n6p75\""
Jul 14 23:41:12.281539 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:12.281518    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7800-140280926443696"]
Jul 14 23:41:12.360026 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:12.359991    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skllh\" (UniqueName: \"kubernetes.io/projected/70df58df-d9ef-42ad-9afa-c1f024b4d956-kube-api-access-skllh\") pod \"test-curl-7800-140280926443696\" (UID: \"70df58df-d9ef-42ad-9afa-c1f024b4d956\") " pod="opendatahub/test-curl-7800-140280926443696"
Jul 14 23:41:12.460629 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:12.460593    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skllh\" (UniqueName: \"kubernetes.io/projected/70df58df-d9ef-42ad-9afa-c1f024b4d956-kube-api-access-skllh\") pod \"test-curl-7800-140280926443696\" (UID: \"70df58df-d9ef-42ad-9afa-c1f024b4d956\") " pod="opendatahub/test-curl-7800-140280926443696"
Jul 14 23:41:12.469969 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:12.469941    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skllh\" (UniqueName: \"kubernetes.io/projected/70df58df-d9ef-42ad-9afa-c1f024b4d956-kube-api-access-skllh\") pod \"test-curl-7800-140280926443696\" (UID: \"70df58df-d9ef-42ad-9afa-c1f024b4d956\") " pod="opendatahub/test-curl-7800-140280926443696"
Jul 14 23:41:12.586173 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:12.586090    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7800-140280926443696"
Jul 14 23:41:12.720533 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:12.720507    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7800-140280926443696"]
Jul 14 23:41:12.722592 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:41:12.722562    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70df58df_d9ef_42ad_9afa_c1f024b4d956.slice/crio-ed5ae710fc06bd84115c11dfcb7389f505f67bb14b740d6b61837a0a20b6d862 WatchSource:0}: Error finding container ed5ae710fc06bd84115c11dfcb7389f505f67bb14b740d6b61837a0a20b6d862: Status 404 returned error can't find the container with id ed5ae710fc06bd84115c11dfcb7389f505f67bb14b740d6b61837a0a20b6d862
Jul 14 23:41:12.723817 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:12.723800    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 23:41:13.341898 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:13.341857    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7800-140280926443696" event={"ID":"70df58df-d9ef-42ad-9afa-c1f024b4d956","Type":"ContainerStarted","Data":"ed5ae710fc06bd84115c11dfcb7389f505f67bb14b740d6b61837a0a20b6d862"}
Jul 14 23:41:14.348480 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:14.348446    2563 generic.go:356] "Generic (PLEG): container finished" podID="70df58df-d9ef-42ad-9afa-c1f024b4d956" containerID="a9700fd6a3f2d6dbcc3e0644c297890d33eb23e8e13d180767d69392937681d6" exitCode=0
Jul 14 23:41:14.348894 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:14.348532    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7800-140280926443696" event={"ID":"70df58df-d9ef-42ad-9afa-c1f024b4d956","Type":"ContainerDied","Data":"a9700fd6a3f2d6dbcc3e0644c297890d33eb23e8e13d180767d69392937681d6"}
Jul 14 23:41:15.491962 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:15.491934    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7800-140280926443696"
Jul 14 23:41:15.531408 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:15.531381    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7800-140280926443696_70df58df-d9ef-42ad-9afa-c1f024b4d956/test-curl-7800-140280926443696/0.log"
Jul 14 23:41:15.558331 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:15.558301    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7800-140280926443696"]
Jul 14 23:41:15.561446 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:15.561424    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7800-140280926443696"]
Jul 14 23:41:15.582546 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:15.582527    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-skllh\" (UniqueName: \"kubernetes.io/projected/70df58df-d9ef-42ad-9afa-c1f024b4d956-kube-api-access-skllh\") pod \"70df58df-d9ef-42ad-9afa-c1f024b4d956\" (UID: \"70df58df-d9ef-42ad-9afa-c1f024b4d956\") "
Jul 14 23:41:15.584811 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:15.584785    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70df58df-d9ef-42ad-9afa-c1f024b4d956-kube-api-access-skllh" (OuterVolumeSpecName: "kube-api-access-skllh") pod "70df58df-d9ef-42ad-9afa-c1f024b4d956" (UID: "70df58df-d9ef-42ad-9afa-c1f024b4d956"). InnerVolumeSpecName "kube-api-access-skllh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:41:15.683447 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:15.683366    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skllh\" (UniqueName: \"kubernetes.io/projected/70df58df-d9ef-42ad-9afa-c1f024b4d956-kube-api-access-skllh\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:41:16.359124 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:16.359091    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7800-140280926443696"
Jul 14 23:41:16.359124 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:16.359108    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed5ae710fc06bd84115c11dfcb7389f505f67bb14b740d6b61837a0a20b6d862"
Jul 14 23:41:16.375416 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:41:16.375382    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7800-140280926443696\" is forbidden: User \"system:node:ip-10-0-134-8.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-8.ec2.internal' and this object" podUID="70df58df-d9ef-42ad-9afa-c1f024b4d956" pod="opendatahub/test-curl-7800-140280926443696"
Jul 14 23:41:17.229894 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:17.229858    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70df58df-d9ef-42ad-9afa-c1f024b4d956" path="/var/lib/kubelet/pods/70df58df-d9ef-42ad-9afa-c1f024b4d956/volumes"
Jul 14 23:41:22.147887 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:22.147847    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7800-140280926457904"]
Jul 14 23:41:22.148310 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:22.148201    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70df58df-d9ef-42ad-9afa-c1f024b4d956" containerName="test-curl-7800-140280926443696"
Jul 14 23:41:22.148310 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:22.148212    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="70df58df-d9ef-42ad-9afa-c1f024b4d956" containerName="test-curl-7800-140280926443696"
Jul 14 23:41:22.148310 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:22.148272    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="70df58df-d9ef-42ad-9afa-c1f024b4d956" containerName="test-curl-7800-140280926443696"
Jul 14 23:41:22.151503 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:22.151485    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7800-140280926457904"
Jul 14 23:41:22.154828 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:22.154795    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-n6p75\""
Jul 14 23:41:22.155851 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:22.155824    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7800-140280926457904"]
Jul 14 23:41:22.236900 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:22.236863    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb7qv\" (UniqueName: \"kubernetes.io/projected/d0b01514-2ba5-4d15-9a37-679896d9c52d-kube-api-access-vb7qv\") pod \"test-curl-7800-140280926457904\" (UID: \"d0b01514-2ba5-4d15-9a37-679896d9c52d\") " pod="opendatahub/test-curl-7800-140280926457904"
Jul 14 23:41:22.337662 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:22.337614    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb7qv\" (UniqueName: \"kubernetes.io/projected/d0b01514-2ba5-4d15-9a37-679896d9c52d-kube-api-access-vb7qv\") pod \"test-curl-7800-140280926457904\" (UID: \"d0b01514-2ba5-4d15-9a37-679896d9c52d\") " pod="opendatahub/test-curl-7800-140280926457904"
Jul 14 23:41:22.346752 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:22.346724    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb7qv\" (UniqueName: \"kubernetes.io/projected/d0b01514-2ba5-4d15-9a37-679896d9c52d-kube-api-access-vb7qv\") pod \"test-curl-7800-140280926457904\" (UID: \"d0b01514-2ba5-4d15-9a37-679896d9c52d\") " pod="opendatahub/test-curl-7800-140280926457904"
Jul 14 23:41:22.462125 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:22.462033    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7800-140280926457904"
Jul 14 23:41:22.602624 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:22.602596    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7800-140280926457904"]
Jul 14 23:41:22.605349 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:41:22.605316    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0b01514_2ba5_4d15_9a37_679896d9c52d.slice/crio-86e2dd80bf1fdc6ef1908bb13f2871113c508f1c12ab722f83baced79455b783 WatchSource:0}: Error finding container 86e2dd80bf1fdc6ef1908bb13f2871113c508f1c12ab722f83baced79455b783: Status 404 returned error can't find the container with id 86e2dd80bf1fdc6ef1908bb13f2871113c508f1c12ab722f83baced79455b783
Jul 14 23:41:23.400749 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:23.400709    2563 generic.go:356] "Generic (PLEG): container finished" podID="d0b01514-2ba5-4d15-9a37-679896d9c52d" containerID="22433efbf0ff132938c7ae3bd25814d0e69811ba576ca0c9febe641050223644" exitCode=0
Jul 14 23:41:23.401167 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:23.400799    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7800-140280926457904" event={"ID":"d0b01514-2ba5-4d15-9a37-679896d9c52d","Type":"ContainerDied","Data":"22433efbf0ff132938c7ae3bd25814d0e69811ba576ca0c9febe641050223644"}
Jul 14 23:41:23.401167 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:23.400849    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7800-140280926457904" event={"ID":"d0b01514-2ba5-4d15-9a37-679896d9c52d","Type":"ContainerStarted","Data":"86e2dd80bf1fdc6ef1908bb13f2871113c508f1c12ab722f83baced79455b783"}
Jul 14 23:41:24.550541 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:24.550514    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7800-140280926457904"
Jul 14 23:41:24.573773 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:24.573749    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7800-140280926457904_d0b01514-2ba5-4d15-9a37-679896d9c52d/test-curl-7800-140280926457904/0.log"
Jul 14 23:41:24.600475 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:24.600432    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7800-140280926457904"]
Jul 14 23:41:24.604372 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:24.604341    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7800-140280926457904"]
Jul 14 23:41:24.657092 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:24.656989    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vb7qv\" (UniqueName: \"kubernetes.io/projected/d0b01514-2ba5-4d15-9a37-679896d9c52d-kube-api-access-vb7qv\") pod \"d0b01514-2ba5-4d15-9a37-679896d9c52d\" (UID: \"d0b01514-2ba5-4d15-9a37-679896d9c52d\") "
Jul 14 23:41:24.659350 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:24.659313    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0b01514-2ba5-4d15-9a37-679896d9c52d-kube-api-access-vb7qv" (OuterVolumeSpecName: "kube-api-access-vb7qv") pod "d0b01514-2ba5-4d15-9a37-679896d9c52d" (UID: "d0b01514-2ba5-4d15-9a37-679896d9c52d"). InnerVolumeSpecName "kube-api-access-vb7qv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:41:24.724789 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:24.724755    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7800-140280926458912"]
Jul 14 23:41:24.725116 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:24.725104    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0b01514-2ba5-4d15-9a37-679896d9c52d" containerName="test-curl-7800-140280926457904"
Jul 14 23:41:24.725179 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:24.725118    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0b01514-2ba5-4d15-9a37-679896d9c52d" containerName="test-curl-7800-140280926457904"
Jul 14 23:41:24.725179 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:24.725178    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0b01514-2ba5-4d15-9a37-679896d9c52d" containerName="test-curl-7800-140280926457904"
Jul 14 23:41:24.728133 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:24.728114    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7800-140280926458912"
Jul 14 23:41:24.734231 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:24.734204    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7800-140280926458912"]
Jul 14 23:41:24.757900 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:24.757866    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fp2m\" (UniqueName: \"kubernetes.io/projected/e16fb9ab-6c84-4412-b05c-2b990e6fa853-kube-api-access-2fp2m\") pod \"test-curl-7800-140280926458912\" (UID: \"e16fb9ab-6c84-4412-b05c-2b990e6fa853\") " pod="opendatahub/test-curl-7800-140280926458912"
Jul 14 23:41:24.758083 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:24.757935    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb7qv\" (UniqueName: \"kubernetes.io/projected/d0b01514-2ba5-4d15-9a37-679896d9c52d-kube-api-access-vb7qv\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:41:24.859220 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:24.859185    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fp2m\" (UniqueName: \"kubernetes.io/projected/e16fb9ab-6c84-4412-b05c-2b990e6fa853-kube-api-access-2fp2m\") pod \"test-curl-7800-140280926458912\" (UID: \"e16fb9ab-6c84-4412-b05c-2b990e6fa853\") " pod="opendatahub/test-curl-7800-140280926458912"
Jul 14 23:41:24.868463 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:24.868429    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fp2m\" (UniqueName: \"kubernetes.io/projected/e16fb9ab-6c84-4412-b05c-2b990e6fa853-kube-api-access-2fp2m\") pod \"test-curl-7800-140280926458912\" (UID: \"e16fb9ab-6c84-4412-b05c-2b990e6fa853\") " pod="opendatahub/test-curl-7800-140280926458912"
Jul 14 23:41:25.038822 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:25.038784    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7800-140280926458912"
Jul 14 23:41:25.173896 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:25.173789    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7800-140280926458912"]
Jul 14 23:41:25.176554 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:41:25.176515    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode16fb9ab_6c84_4412_b05c_2b990e6fa853.slice/crio-6e16c46d5442dac885b3f7291b574b7b4aa09506caee38f8f182d4720f5317a4 WatchSource:0}: Error finding container 6e16c46d5442dac885b3f7291b574b7b4aa09506caee38f8f182d4720f5317a4: Status 404 returned error can't find the container with id 6e16c46d5442dac885b3f7291b574b7b4aa09506caee38f8f182d4720f5317a4
Jul 14 23:41:25.231193 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:25.231157    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0b01514-2ba5-4d15-9a37-679896d9c52d" path="/var/lib/kubelet/pods/d0b01514-2ba5-4d15-9a37-679896d9c52d/volumes"
Jul 14 23:41:25.411319 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:25.411200    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7800-140280926458912" event={"ID":"e16fb9ab-6c84-4412-b05c-2b990e6fa853","Type":"ContainerStarted","Data":"6e16c46d5442dac885b3f7291b574b7b4aa09506caee38f8f182d4720f5317a4"}
Jul 14 23:41:25.412228 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:25.412210    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7800-140280926457904"
Jul 14 23:41:25.412304 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:25.412262    2563 scope.go:117] "RemoveContainer" containerID="22433efbf0ff132938c7ae3bd25814d0e69811ba576ca0c9febe641050223644"
Jul 14 23:41:25.450335 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:25.450291    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-d62786/rate-test-model-d62786-kserve-86c954d8f9-nbmbm" podUID="492a9ea3-4908-4fb0-be0d-7eeac57acf57" containerName="main" containerID="cri-o://3eaddda261c09bbfc0eaade5b4cf3376c6288ef07992d77393f2050edc016498" gracePeriod=45
Jul 14 23:41:25.716190 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:25.716159    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d62786/rate-test-model-d62786-kserve-86c954d8f9-nbmbm"
Jul 14 23:41:25.766784 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:25.766752    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/492a9ea3-4908-4fb0-be0d-7eeac57acf57-model-cache\") pod \"492a9ea3-4908-4fb0-be0d-7eeac57acf57\" (UID: \"492a9ea3-4908-4fb0-be0d-7eeac57acf57\") "
Jul 14 23:41:25.766989 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:25.766824    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/492a9ea3-4908-4fb0-be0d-7eeac57acf57-dshm\") pod \"492a9ea3-4908-4fb0-be0d-7eeac57acf57\" (UID: \"492a9ea3-4908-4fb0-be0d-7eeac57acf57\") "
Jul 14 23:41:25.766989 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:25.766860    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/492a9ea3-4908-4fb0-be0d-7eeac57acf57-tls-certs\") pod \"492a9ea3-4908-4fb0-be0d-7eeac57acf57\" (UID: \"492a9ea3-4908-4fb0-be0d-7eeac57acf57\") "
Jul 14 23:41:25.766989 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:25.766893    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrf7b\" (UniqueName: \"kubernetes.io/projected/492a9ea3-4908-4fb0-be0d-7eeac57acf57-kube-api-access-vrf7b\") pod \"492a9ea3-4908-4fb0-be0d-7eeac57acf57\" (UID: \"492a9ea3-4908-4fb0-be0d-7eeac57acf57\") "
Jul 14 23:41:25.766989 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:25.766932    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/492a9ea3-4908-4fb0-be0d-7eeac57acf57-home\") pod \"492a9ea3-4908-4fb0-be0d-7eeac57acf57\" (UID: \"492a9ea3-4908-4fb0-be0d-7eeac57acf57\") "
Jul 14 23:41:25.766989 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:25.766951    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/492a9ea3-4908-4fb0-be0d-7eeac57acf57-tmp-dir\") pod \"492a9ea3-4908-4fb0-be0d-7eeac57acf57\" (UID: \"492a9ea3-4908-4fb0-be0d-7eeac57acf57\") "
Jul 14 23:41:25.767241 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:25.767101    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492a9ea3-4908-4fb0-be0d-7eeac57acf57-model-cache" (OuterVolumeSpecName: "model-cache") pod "492a9ea3-4908-4fb0-be0d-7eeac57acf57" (UID: "492a9ea3-4908-4fb0-be0d-7eeac57acf57"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:41:25.767329 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:25.767262    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/492a9ea3-4908-4fb0-be0d-7eeac57acf57-model-cache\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:41:25.767329 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:25.767274    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492a9ea3-4908-4fb0-be0d-7eeac57acf57-home" (OuterVolumeSpecName: "home") pod "492a9ea3-4908-4fb0-be0d-7eeac57acf57" (UID: "492a9ea3-4908-4fb0-be0d-7eeac57acf57"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:41:25.767329 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:25.767314    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492a9ea3-4908-4fb0-be0d-7eeac57acf57-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "492a9ea3-4908-4fb0-be0d-7eeac57acf57" (UID: "492a9ea3-4908-4fb0-be0d-7eeac57acf57"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:41:25.769174 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:25.769146    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492a9ea3-4908-4fb0-be0d-7eeac57acf57-dshm" (OuterVolumeSpecName: "dshm") pod "492a9ea3-4908-4fb0-be0d-7eeac57acf57" (UID: "492a9ea3-4908-4fb0-be0d-7eeac57acf57"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:41:25.769174 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:25.769165    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/492a9ea3-4908-4fb0-be0d-7eeac57acf57-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "492a9ea3-4908-4fb0-be0d-7eeac57acf57" (UID: "492a9ea3-4908-4fb0-be0d-7eeac57acf57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 23:41:25.769354 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:25.769176    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/492a9ea3-4908-4fb0-be0d-7eeac57acf57-kube-api-access-vrf7b" (OuterVolumeSpecName: "kube-api-access-vrf7b") pod "492a9ea3-4908-4fb0-be0d-7eeac57acf57" (UID: "492a9ea3-4908-4fb0-be0d-7eeac57acf57"). InnerVolumeSpecName "kube-api-access-vrf7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:41:25.868235 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:25.868196    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/492a9ea3-4908-4fb0-be0d-7eeac57acf57-home\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:41:25.868235 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:25.868229    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/492a9ea3-4908-4fb0-be0d-7eeac57acf57-tmp-dir\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:41:25.868235 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:25.868238    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/492a9ea3-4908-4fb0-be0d-7eeac57acf57-dshm\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:41:25.868235 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:25.868245    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/492a9ea3-4908-4fb0-be0d-7eeac57acf57-tls-certs\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:41:25.868543 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:25.868253    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrf7b\" (UniqueName: \"kubernetes.io/projected/492a9ea3-4908-4fb0-be0d-7eeac57acf57-kube-api-access-vrf7b\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:41:26.418170 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:26.418130    2563 generic.go:356] "Generic (PLEG): container finished" podID="492a9ea3-4908-4fb0-be0d-7eeac57acf57" containerID="3eaddda261c09bbfc0eaade5b4cf3376c6288ef07992d77393f2050edc016498" exitCode=0
Jul 14 23:41:26.418411 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:26.418175    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d62786/rate-test-model-d62786-kserve-86c954d8f9-nbmbm"
Jul 14 23:41:26.418411 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:26.418222    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d62786/rate-test-model-d62786-kserve-86c954d8f9-nbmbm" event={"ID":"492a9ea3-4908-4fb0-be0d-7eeac57acf57","Type":"ContainerDied","Data":"3eaddda261c09bbfc0eaade5b4cf3376c6288ef07992d77393f2050edc016498"}
Jul 14 23:41:26.418411 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:26.418267    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d62786/rate-test-model-d62786-kserve-86c954d8f9-nbmbm" event={"ID":"492a9ea3-4908-4fb0-be0d-7eeac57acf57","Type":"ContainerDied","Data":"a6f6b5ac31d454dcc4d1d4ed67e5c581d8eece74787b5170a575b063aca466d4"}
Jul 14 23:41:26.418411 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:26.418309    2563 scope.go:117] "RemoveContainer" containerID="3eaddda261c09bbfc0eaade5b4cf3376c6288ef07992d77393f2050edc016498"
Jul 14 23:41:26.419921 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:26.419894    2563 generic.go:356] "Generic (PLEG): container finished" podID="e16fb9ab-6c84-4412-b05c-2b990e6fa853" containerID="ceb079a5ebc485b83b5a4f2e08c5922e54ec4bd0d020316584e215adc41f006c" exitCode=0
Jul 14 23:41:26.420035 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:26.419986    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7800-140280926458912" event={"ID":"e16fb9ab-6c84-4412-b05c-2b990e6fa853","Type":"ContainerDied","Data":"ceb079a5ebc485b83b5a4f2e08c5922e54ec4bd0d020316584e215adc41f006c"}
Jul 14 23:41:26.427858 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:26.427840    2563 scope.go:117] "RemoveContainer" containerID="3eaddda261c09bbfc0eaade5b4cf3376c6288ef07992d77393f2050edc016498"
Jul 14 23:41:26.428107 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:41:26.428090    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eaddda261c09bbfc0eaade5b4cf3376c6288ef07992d77393f2050edc016498\": container with ID starting with 3eaddda261c09bbfc0eaade5b4cf3376c6288ef07992d77393f2050edc016498 not found: ID does not exist" containerID="3eaddda261c09bbfc0eaade5b4cf3376c6288ef07992d77393f2050edc016498"
Jul 14 23:41:26.428143 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:26.428116    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eaddda261c09bbfc0eaade5b4cf3376c6288ef07992d77393f2050edc016498"} err="failed to get container status \"3eaddda261c09bbfc0eaade5b4cf3376c6288ef07992d77393f2050edc016498\": rpc error: code = NotFound desc = could not find container \"3eaddda261c09bbfc0eaade5b4cf3376c6288ef07992d77393f2050edc016498\": container with ID starting with 3eaddda261c09bbfc0eaade5b4cf3376c6288ef07992d77393f2050edc016498 not found: ID does not exist"
Jul 14 23:41:26.449979 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:26.449948    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d62786/rate-test-model-d62786-kserve-86c954d8f9-nbmbm"]
Jul 14 23:41:26.451966 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:26.451942    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-d62786/rate-test-model-d62786-kserve-86c954d8f9-nbmbm"]
Jul 14 23:41:27.230017 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:27.229977    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="492a9ea3-4908-4fb0-be0d-7eeac57acf57" path="/var/lib/kubelet/pods/492a9ea3-4908-4fb0-be0d-7eeac57acf57/volumes"
Jul 14 23:41:27.563108 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:27.563076    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7800-140280926458912"
Jul 14 23:41:27.583962 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:27.583932    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fp2m\" (UniqueName: \"kubernetes.io/projected/e16fb9ab-6c84-4412-b05c-2b990e6fa853-kube-api-access-2fp2m\") pod \"e16fb9ab-6c84-4412-b05c-2b990e6fa853\" (UID: \"e16fb9ab-6c84-4412-b05c-2b990e6fa853\") "
Jul 14 23:41:27.585011 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:27.584981    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7800-140280926458912_e16fb9ab-6c84-4412-b05c-2b990e6fa853/test-curl-7800-140280926458912/0.log"
Jul 14 23:41:27.586779 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:27.586748    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e16fb9ab-6c84-4412-b05c-2b990e6fa853-kube-api-access-2fp2m" (OuterVolumeSpecName: "kube-api-access-2fp2m") pod "e16fb9ab-6c84-4412-b05c-2b990e6fa853" (UID: "e16fb9ab-6c84-4412-b05c-2b990e6fa853"). InnerVolumeSpecName "kube-api-access-2fp2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:41:27.605675 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:27.605645    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7800-140280926458912"]
Jul 14 23:41:27.610925 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:27.610892    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7800-140280926458912"]
Jul 14 23:41:27.685550 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:27.685509    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fp2m\" (UniqueName: \"kubernetes.io/projected/e16fb9ab-6c84-4412-b05c-2b990e6fa853-kube-api-access-2fp2m\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:41:27.725697 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:27.725662    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7800-140280926458912"]
Jul 14 23:41:27.725997 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:27.725986    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="492a9ea3-4908-4fb0-be0d-7eeac57acf57" containerName="main"
Jul 14 23:41:27.726037 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:27.726000    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="492a9ea3-4908-4fb0-be0d-7eeac57acf57" containerName="main"
Jul 14 23:41:27.726037 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:27.726024    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e16fb9ab-6c84-4412-b05c-2b990e6fa853" containerName="test-curl-7800-140280926458912"
Jul 14 23:41:27.726037 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:27.726030    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e16fb9ab-6c84-4412-b05c-2b990e6fa853" containerName="test-curl-7800-140280926458912"
Jul 14 23:41:27.726125 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:27.726092    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e16fb9ab-6c84-4412-b05c-2b990e6fa853" containerName="test-curl-7800-140280926458912"
Jul 14 23:41:27.726125 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:27.726101    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="492a9ea3-4908-4fb0-be0d-7eeac57acf57" containerName="main"
Jul 14 23:41:27.729323 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:27.729302    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7800-140280926458912"
Jul 14 23:41:27.736191 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:27.736069    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7800-140280926458912"]
Jul 14 23:41:27.786515 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:27.786471    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqsrf\" (UniqueName: \"kubernetes.io/projected/e413a5ab-be3e-4b71-b3a6-64abf7a1ecff-kube-api-access-rqsrf\") pod \"test-curl-7800-140280926458912\" (UID: \"e413a5ab-be3e-4b71-b3a6-64abf7a1ecff\") " pod="opendatahub/test-curl-7800-140280926458912"
Jul 14 23:41:27.887353 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:27.887247    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqsrf\" (UniqueName: \"kubernetes.io/projected/e413a5ab-be3e-4b71-b3a6-64abf7a1ecff-kube-api-access-rqsrf\") pod \"test-curl-7800-140280926458912\" (UID: \"e413a5ab-be3e-4b71-b3a6-64abf7a1ecff\") " pod="opendatahub/test-curl-7800-140280926458912"
Jul 14 23:41:27.895959 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:27.895923    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqsrf\" (UniqueName: \"kubernetes.io/projected/e413a5ab-be3e-4b71-b3a6-64abf7a1ecff-kube-api-access-rqsrf\") pod \"test-curl-7800-140280926458912\" (UID: \"e413a5ab-be3e-4b71-b3a6-64abf7a1ecff\") " pod="opendatahub/test-curl-7800-140280926458912"
Jul 14 23:41:28.039413 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:28.039363    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7800-140280926458912"
Jul 14 23:41:28.178562 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:28.178511    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7800-140280926458912"]
Jul 14 23:41:28.180739 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:41:28.180710    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode413a5ab_be3e_4b71_b3a6_64abf7a1ecff.slice/crio-0363915354d2fa4da4f3f106c2002ba33b9293ef0a82288c02769add613c931c WatchSource:0}: Error finding container 0363915354d2fa4da4f3f106c2002ba33b9293ef0a82288c02769add613c931c: Status 404 returned error can't find the container with id 0363915354d2fa4da4f3f106c2002ba33b9293ef0a82288c02769add613c931c
Jul 14 23:41:28.433166 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:28.433074    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7800-140280926458912"
Jul 14 23:41:28.433166 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:28.433119    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e16c46d5442dac885b3f7291b574b7b4aa09506caee38f8f182d4720f5317a4"
Jul 14 23:41:28.434387 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:28.434348    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7800-140280926458912" event={"ID":"e413a5ab-be3e-4b71-b3a6-64abf7a1ecff","Type":"ContainerStarted","Data":"0363915354d2fa4da4f3f106c2002ba33b9293ef0a82288c02769add613c931c"}
Jul 14 23:41:28.446381 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:28.446336    2563 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7800-140280926458912" oldPodUID="e16fb9ab-6c84-4412-b05c-2b990e6fa853" podUID="e413a5ab-be3e-4b71-b3a6-64abf7a1ecff"
Jul 14 23:41:29.231277 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:29.231242    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e16fb9ab-6c84-4412-b05c-2b990e6fa853" path="/var/lib/kubelet/pods/e16fb9ab-6c84-4412-b05c-2b990e6fa853/volumes"
Jul 14 23:41:29.440622 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:29.440584    2563 generic.go:356] "Generic (PLEG): container finished" podID="e413a5ab-be3e-4b71-b3a6-64abf7a1ecff" containerID="690e0c4c1bba6ea3fbd4e6aede443da1ac4f83f2771b25a107ca0b7b0baf310f" exitCode=0
Jul 14 23:41:29.440980 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:29.440652    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7800-140280926458912" event={"ID":"e413a5ab-be3e-4b71-b3a6-64abf7a1ecff","Type":"ContainerDied","Data":"690e0c4c1bba6ea3fbd4e6aede443da1ac4f83f2771b25a107ca0b7b0baf310f"}
Jul 14 23:41:30.586466 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:30.586440    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7800-140280926458912"
Jul 14 23:41:30.608114 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:30.608083    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7800-140280926458912_e413a5ab-be3e-4b71-b3a6-64abf7a1ecff/test-curl-7800-140280926458912/0.log"
Jul 14 23:41:30.610248 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:30.610222    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqsrf\" (UniqueName: \"kubernetes.io/projected/e413a5ab-be3e-4b71-b3a6-64abf7a1ecff-kube-api-access-rqsrf\") pod \"e413a5ab-be3e-4b71-b3a6-64abf7a1ecff\" (UID: \"e413a5ab-be3e-4b71-b3a6-64abf7a1ecff\") "
Jul 14 23:41:30.612400 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:30.612376    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e413a5ab-be3e-4b71-b3a6-64abf7a1ecff-kube-api-access-rqsrf" (OuterVolumeSpecName: "kube-api-access-rqsrf") pod "e413a5ab-be3e-4b71-b3a6-64abf7a1ecff" (UID: "e413a5ab-be3e-4b71-b3a6-64abf7a1ecff"). InnerVolumeSpecName "kube-api-access-rqsrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:41:30.640248 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:30.639052    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7800-140280926458912"]
Jul 14 23:41:30.640248 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:30.639122    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7800-140280926458912"]
Jul 14 23:41:30.711005 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:30.710915    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqsrf\" (UniqueName: \"kubernetes.io/projected/e413a5ab-be3e-4b71-b3a6-64abf7a1ecff-kube-api-access-rqsrf\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:41:31.230128 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:31.230092    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e413a5ab-be3e-4b71-b3a6-64abf7a1ecff" path="/var/lib/kubelet/pods/e413a5ab-be3e-4b71-b3a6-64abf7a1ecff/volumes"
Jul 14 23:41:31.451346 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:31.451317    2563 scope.go:117] "RemoveContainer" containerID="690e0c4c1bba6ea3fbd4e6aede443da1ac4f83f2771b25a107ca0b7b0baf310f"
Jul 14 23:41:31.451346 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:31.451347    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7800-140280926458912"
Jul 14 23:41:41.245973 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:41.245933    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d62786-74788c9dc9-lfs6q"]
Jul 14 23:41:41.246566 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:41.246175    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d62786-74788c9dc9-lfs6q" podUID="c933a07f-bc78-48ac-ba27-9c59fe97a35f" containerName="maas-api" containerID="cri-o://0ec911c5bd3dd3097339692fcf8a2b4e121dbd290467a8fdd61aa814dcaf6d1d" gracePeriod=30
Jul 14 23:41:41.717429 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:41.717402    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d62786-74788c9dc9-lfs6q"
Jul 14 23:41:41.819875 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:41.819762    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c933a07f-bc78-48ac-ba27-9c59fe97a35f-openshift-service-ca\") pod \"c933a07f-bc78-48ac-ba27-9c59fe97a35f\" (UID: \"c933a07f-bc78-48ac-ba27-9c59fe97a35f\") "
Jul 14 23:41:41.819875 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:41.819838    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbc7x\" (UniqueName: \"kubernetes.io/projected/c933a07f-bc78-48ac-ba27-9c59fe97a35f-kube-api-access-nbc7x\") pod \"c933a07f-bc78-48ac-ba27-9c59fe97a35f\" (UID: \"c933a07f-bc78-48ac-ba27-9c59fe97a35f\") "
Jul 14 23:41:41.820116 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:41.819904    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c933a07f-bc78-48ac-ba27-9c59fe97a35f-maas-api-tls\") pod \"c933a07f-bc78-48ac-ba27-9c59fe97a35f\" (UID: \"c933a07f-bc78-48ac-ba27-9c59fe97a35f\") "
Jul 14 23:41:41.820173 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:41.820149    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c933a07f-bc78-48ac-ba27-9c59fe97a35f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c933a07f-bc78-48ac-ba27-9c59fe97a35f" (UID: "c933a07f-bc78-48ac-ba27-9c59fe97a35f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 23:41:41.822121 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:41.822090    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c933a07f-bc78-48ac-ba27-9c59fe97a35f-kube-api-access-nbc7x" (OuterVolumeSpecName: "kube-api-access-nbc7x") pod "c933a07f-bc78-48ac-ba27-9c59fe97a35f" (UID: "c933a07f-bc78-48ac-ba27-9c59fe97a35f"). InnerVolumeSpecName "kube-api-access-nbc7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:41:41.822240 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:41.822181    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c933a07f-bc78-48ac-ba27-9c59fe97a35f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c933a07f-bc78-48ac-ba27-9c59fe97a35f" (UID: "c933a07f-bc78-48ac-ba27-9c59fe97a35f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 23:41:41.920827 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:41.920786    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c933a07f-bc78-48ac-ba27-9c59fe97a35f-maas-api-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:41:41.920827 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:41.920820    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c933a07f-bc78-48ac-ba27-9c59fe97a35f-openshift-service-ca\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:41:41.920827 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:41.920832    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbc7x\" (UniqueName: \"kubernetes.io/projected/c933a07f-bc78-48ac-ba27-9c59fe97a35f-kube-api-access-nbc7x\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:41:42.514657 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:42.514624    2563 generic.go:356] "Generic (PLEG): container finished" podID="c933a07f-bc78-48ac-ba27-9c59fe97a35f" containerID="0ec911c5bd3dd3097339692fcf8a2b4e121dbd290467a8fdd61aa814dcaf6d1d" exitCode=0
Jul 14 23:41:42.515117 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:42.514668    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d62786-74788c9dc9-lfs6q"
Jul 14 23:41:42.515117 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:42.514697    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d62786-74788c9dc9-lfs6q" event={"ID":"c933a07f-bc78-48ac-ba27-9c59fe97a35f","Type":"ContainerDied","Data":"0ec911c5bd3dd3097339692fcf8a2b4e121dbd290467a8fdd61aa814dcaf6d1d"}
Jul 14 23:41:42.515117 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:42.514738    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d62786-74788c9dc9-lfs6q" event={"ID":"c933a07f-bc78-48ac-ba27-9c59fe97a35f","Type":"ContainerDied","Data":"3b7b3dd91d106645b41a73a68d478a251bcf89adc89743aa4962d1434b04e849"}
Jul 14 23:41:42.515117 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:42.514760    2563 scope.go:117] "RemoveContainer" containerID="0ec911c5bd3dd3097339692fcf8a2b4e121dbd290467a8fdd61aa814dcaf6d1d"
Jul 14 23:41:42.524424 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:42.524403    2563 scope.go:117] "RemoveContainer" containerID="0ec911c5bd3dd3097339692fcf8a2b4e121dbd290467a8fdd61aa814dcaf6d1d"
Jul 14 23:41:42.524716 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:41:42.524695    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ec911c5bd3dd3097339692fcf8a2b4e121dbd290467a8fdd61aa814dcaf6d1d\": container with ID starting with 0ec911c5bd3dd3097339692fcf8a2b4e121dbd290467a8fdd61aa814dcaf6d1d not found: ID does not exist" containerID="0ec911c5bd3dd3097339692fcf8a2b4e121dbd290467a8fdd61aa814dcaf6d1d"
Jul 14 23:41:42.524783 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:42.524728    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ec911c5bd3dd3097339692fcf8a2b4e121dbd290467a8fdd61aa814dcaf6d1d"} err="failed to get container status \"0ec911c5bd3dd3097339692fcf8a2b4e121dbd290467a8fdd61aa814dcaf6d1d\": rpc error: code = NotFound desc = could not find container \"0ec911c5bd3dd3097339692fcf8a2b4e121dbd290467a8fdd61aa814dcaf6d1d\": container with ID starting with 0ec911c5bd3dd3097339692fcf8a2b4e121dbd290467a8fdd61aa814dcaf6d1d not found: ID does not exist"
Jul 14 23:41:42.536918 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:42.536885    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d62786-74788c9dc9-lfs6q"]
Jul 14 23:41:42.540471 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:42.540447    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d62786-74788c9dc9-lfs6q"]
Jul 14 23:41:43.230123 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:43.230084    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c933a07f-bc78-48ac-ba27-9c59fe97a35f" path="/var/lib/kubelet/pods/c933a07f-bc78-48ac-ba27-9c59fe97a35f/volumes"
Jul 14 23:41:56.600099 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:56.600066    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"]
Jul 14 23:41:56.600524 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:41:56.600303    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq" podUID="0718434c-4640-407f-8215-872ae7c6a09d" containerName="istio-proxy" containerID="cri-o://51f058db4ec441ad4d7294b630dc33707af3d41b95b27db7ab0c300c676eaef5" gracePeriod=30
Jul 14 23:42:00.987951 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:00.987915    2563 patch_prober.go:28] interesting pod/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 14 23:42:00.988391 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:00.987973    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq" podUID="0718434c-4640-407f-8215-872ae7c6a09d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 14 23:42:01.861559 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:01.861494    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"
Jul 14 23:42:01.900198 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:01.900163    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0718434c-4640-407f-8215-872ae7c6a09d-workload-certs\") pod \"0718434c-4640-407f-8215-872ae7c6a09d\" (UID: \"0718434c-4640-407f-8215-872ae7c6a09d\") "
Jul 14 23:42:01.900407 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:01.900206    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0718434c-4640-407f-8215-872ae7c6a09d-istiod-ca-cert\") pod \"0718434c-4640-407f-8215-872ae7c6a09d\" (UID: \"0718434c-4640-407f-8215-872ae7c6a09d\") "
Jul 14 23:42:01.900407 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:01.900237    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0718434c-4640-407f-8215-872ae7c6a09d-istio-token\") pod \"0718434c-4640-407f-8215-872ae7c6a09d\" (UID: \"0718434c-4640-407f-8215-872ae7c6a09d\") "
Jul 14 23:42:01.900407 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:01.900304    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0718434c-4640-407f-8215-872ae7c6a09d-istio-envoy\") pod \"0718434c-4640-407f-8215-872ae7c6a09d\" (UID: \"0718434c-4640-407f-8215-872ae7c6a09d\") "
Jul 14 23:42:01.900407 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:01.900344    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0718434c-4640-407f-8215-872ae7c6a09d-istio-podinfo\") pod \"0718434c-4640-407f-8215-872ae7c6a09d\" (UID: \"0718434c-4640-407f-8215-872ae7c6a09d\") "
Jul 14 23:42:01.900407 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:01.900389    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0718434c-4640-407f-8215-872ae7c6a09d-istio-data\") pod \"0718434c-4640-407f-8215-872ae7c6a09d\" (UID: \"0718434c-4640-407f-8215-872ae7c6a09d\") "
Jul 14 23:42:01.900691 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:01.900421    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0718434c-4640-407f-8215-872ae7c6a09d-credential-socket\") pod \"0718434c-4640-407f-8215-872ae7c6a09d\" (UID: \"0718434c-4640-407f-8215-872ae7c6a09d\") "
Jul 14 23:42:01.900691 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:01.900467    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0718434c-4640-407f-8215-872ae7c6a09d-workload-socket\") pod \"0718434c-4640-407f-8215-872ae7c6a09d\" (UID: \"0718434c-4640-407f-8215-872ae7c6a09d\") "
Jul 14 23:42:01.900691 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:01.900498    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pst6\" (UniqueName: \"kubernetes.io/projected/0718434c-4640-407f-8215-872ae7c6a09d-kube-api-access-7pst6\") pod \"0718434c-4640-407f-8215-872ae7c6a09d\" (UID: \"0718434c-4640-407f-8215-872ae7c6a09d\") "
Jul 14 23:42:01.900691 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:01.900519    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0718434c-4640-407f-8215-872ae7c6a09d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0718434c-4640-407f-8215-872ae7c6a09d" (UID: "0718434c-4640-407f-8215-872ae7c6a09d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:42:01.900887 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:01.900728    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0718434c-4640-407f-8215-872ae7c6a09d-workload-certs\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:42:01.900887 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:01.900741    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0718434c-4640-407f-8215-872ae7c6a09d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0718434c-4640-407f-8215-872ae7c6a09d" (UID: "0718434c-4640-407f-8215-872ae7c6a09d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:42:01.900887 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:01.900762    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0718434c-4640-407f-8215-872ae7c6a09d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0718434c-4640-407f-8215-872ae7c6a09d" (UID: "0718434c-4640-407f-8215-872ae7c6a09d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:42:01.900887 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:01.900769    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0718434c-4640-407f-8215-872ae7c6a09d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0718434c-4640-407f-8215-872ae7c6a09d" (UID: "0718434c-4640-407f-8215-872ae7c6a09d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 23:42:01.901658 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:01.901628    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0718434c-4640-407f-8215-872ae7c6a09d-istio-data" (OuterVolumeSpecName: "istio-data") pod "0718434c-4640-407f-8215-872ae7c6a09d" (UID: "0718434c-4640-407f-8215-872ae7c6a09d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:42:01.902794 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:01.902764    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0718434c-4640-407f-8215-872ae7c6a09d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0718434c-4640-407f-8215-872ae7c6a09d" (UID: "0718434c-4640-407f-8215-872ae7c6a09d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 23:42:01.902917 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:01.902897    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0718434c-4640-407f-8215-872ae7c6a09d-istio-token" (OuterVolumeSpecName: "istio-token") pod "0718434c-4640-407f-8215-872ae7c6a09d" (UID: "0718434c-4640-407f-8215-872ae7c6a09d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:42:01.903421 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:01.903401    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0718434c-4640-407f-8215-872ae7c6a09d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0718434c-4640-407f-8215-872ae7c6a09d" (UID: "0718434c-4640-407f-8215-872ae7c6a09d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:42:01.903421 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:01.903408    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0718434c-4640-407f-8215-872ae7c6a09d-kube-api-access-7pst6" (OuterVolumeSpecName: "kube-api-access-7pst6") pod "0718434c-4640-407f-8215-872ae7c6a09d" (UID: "0718434c-4640-407f-8215-872ae7c6a09d"). InnerVolumeSpecName "kube-api-access-7pst6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:42:02.001209 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:02.001161    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0718434c-4640-407f-8215-872ae7c6a09d-workload-socket\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:42:02.001209 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:02.001204    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pst6\" (UniqueName: \"kubernetes.io/projected/0718434c-4640-407f-8215-872ae7c6a09d-kube-api-access-7pst6\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:42:02.001209 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:02.001214    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0718434c-4640-407f-8215-872ae7c6a09d-istiod-ca-cert\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:42:02.001209 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:02.001223    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0718434c-4640-407f-8215-872ae7c6a09d-istio-token\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:42:02.001209 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:02.001230    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0718434c-4640-407f-8215-872ae7c6a09d-istio-envoy\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:42:02.001754 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:02.001237    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0718434c-4640-407f-8215-872ae7c6a09d-istio-podinfo\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:42:02.001754 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:02.001245    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0718434c-4640-407f-8215-872ae7c6a09d-istio-data\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:42:02.001754 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:02.001252    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0718434c-4640-407f-8215-872ae7c6a09d-credential-socket\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:42:02.620699 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:02.620659    2563 generic.go:356] "Generic (PLEG): container finished" podID="0718434c-4640-407f-8215-872ae7c6a09d" containerID="51f058db4ec441ad4d7294b630dc33707af3d41b95b27db7ab0c300c676eaef5" exitCode=0
Jul 14 23:42:02.620883 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:02.620705    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq" event={"ID":"0718434c-4640-407f-8215-872ae7c6a09d","Type":"ContainerDied","Data":"51f058db4ec441ad4d7294b630dc33707af3d41b95b27db7ab0c300c676eaef5"}
Jul 14 23:42:02.620883 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:02.620710    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"
Jul 14 23:42:02.620883 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:02.620749    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq" event={"ID":"0718434c-4640-407f-8215-872ae7c6a09d","Type":"ContainerDied","Data":"a9deb7c538db9b02538170bdbea124d0edcca2a8c28a0d0e9152a50d174fd142"}
Jul 14 23:42:02.620883 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:02.620772    2563 scope.go:117] "RemoveContainer" containerID="51f058db4ec441ad4d7294b630dc33707af3d41b95b27db7ab0c300c676eaef5"
Jul 14 23:42:02.629766 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:02.629745    2563 scope.go:117] "RemoveContainer" containerID="eab5b6d53d10b6a1d502e072fe892c84a94db2a70aa1bdef1139620af96ba024"
Jul 14 23:42:02.637649 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:02.637627    2563 scope.go:117] "RemoveContainer" containerID="51f058db4ec441ad4d7294b630dc33707af3d41b95b27db7ab0c300c676eaef5"
Jul 14 23:42:02.637903 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:42:02.637883    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51f058db4ec441ad4d7294b630dc33707af3d41b95b27db7ab0c300c676eaef5\": container with ID starting with 51f058db4ec441ad4d7294b630dc33707af3d41b95b27db7ab0c300c676eaef5 not found: ID does not exist" containerID="51f058db4ec441ad4d7294b630dc33707af3d41b95b27db7ab0c300c676eaef5"
Jul 14 23:42:02.637981 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:02.637911    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51f058db4ec441ad4d7294b630dc33707af3d41b95b27db7ab0c300c676eaef5"} err="failed to get container status \"51f058db4ec441ad4d7294b630dc33707af3d41b95b27db7ab0c300c676eaef5\": rpc error: code = NotFound desc = could not find container \"51f058db4ec441ad4d7294b630dc33707af3d41b95b27db7ab0c300c676eaef5\": container with ID starting with 51f058db4ec441ad4d7294b630dc33707af3d41b95b27db7ab0c300c676eaef5 not found: ID does not exist"
Jul 14 23:42:02.637981 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:02.637927    2563 scope.go:117] "RemoveContainer" containerID="eab5b6d53d10b6a1d502e072fe892c84a94db2a70aa1bdef1139620af96ba024"
Jul 14 23:42:02.638155 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:42:02.638139    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eab5b6d53d10b6a1d502e072fe892c84a94db2a70aa1bdef1139620af96ba024\": container with ID starting with eab5b6d53d10b6a1d502e072fe892c84a94db2a70aa1bdef1139620af96ba024 not found: ID does not exist" containerID="eab5b6d53d10b6a1d502e072fe892c84a94db2a70aa1bdef1139620af96ba024"
Jul 14 23:42:02.638216 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:02.638157    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eab5b6d53d10b6a1d502e072fe892c84a94db2a70aa1bdef1139620af96ba024"} err="failed to get container status \"eab5b6d53d10b6a1d502e072fe892c84a94db2a70aa1bdef1139620af96ba024\": rpc error: code = NotFound desc = could not find container \"eab5b6d53d10b6a1d502e072fe892c84a94db2a70aa1bdef1139620af96ba024\": container with ID starting with eab5b6d53d10b6a1d502e072fe892c84a94db2a70aa1bdef1139620af96ba024 not found: ID does not exist"
Jul 14 23:42:02.643686 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:02.643660    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"]
Jul 14 23:42:02.646252 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:02.646229    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-d62786-openshift-default-74dcb56575-9fkbq"]
Jul 14 23:42:03.231013 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:03.230973    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0718434c-4640-407f-8215-872ae7c6a09d" path="/var/lib/kubelet/pods/0718434c-4640-407f-8215-872ae7c6a09d/volumes"
Jul 14 23:42:07.392159 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:07.392121    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-638b58-5c7d44b967-qc9dv"]
Jul 14 23:42:07.392667 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:07.392402    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-638b58-5c7d44b967-qc9dv" podUID="4fcb78ef-281c-451f-9cd9-64e932f5cd29" containerName="maas-api" containerID="cri-o://d206d25ab25a3d2009347851e59f0c2a62cf0eb5122335d00d916b326c80c81e" gracePeriod=30
Jul 14 23:42:07.838265 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:07.838238    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-638b58-5c7d44b967-qc9dv"
Jul 14 23:42:07.853843 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:07.853814    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fcb78ef-281c-451f-9cd9-64e932f5cd29-openshift-service-ca\") pod \"4fcb78ef-281c-451f-9cd9-64e932f5cd29\" (UID: \"4fcb78ef-281c-451f-9cd9-64e932f5cd29\") "
Jul 14 23:42:07.853955 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:07.853891    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2f2k9\" (UniqueName: \"kubernetes.io/projected/4fcb78ef-281c-451f-9cd9-64e932f5cd29-kube-api-access-2f2k9\") pod \"4fcb78ef-281c-451f-9cd9-64e932f5cd29\" (UID: \"4fcb78ef-281c-451f-9cd9-64e932f5cd29\") "
Jul 14 23:42:07.853955 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:07.853930    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fcb78ef-281c-451f-9cd9-64e932f5cd29-maas-api-tls\") pod \"4fcb78ef-281c-451f-9cd9-64e932f5cd29\" (UID: \"4fcb78ef-281c-451f-9cd9-64e932f5cd29\") "
Jul 14 23:42:07.854231 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:07.854194    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fcb78ef-281c-451f-9cd9-64e932f5cd29-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4fcb78ef-281c-451f-9cd9-64e932f5cd29" (UID: "4fcb78ef-281c-451f-9cd9-64e932f5cd29"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 23:42:07.857877 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:07.857822    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fcb78ef-281c-451f-9cd9-64e932f5cd29-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4fcb78ef-281c-451f-9cd9-64e932f5cd29" (UID: "4fcb78ef-281c-451f-9cd9-64e932f5cd29"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 23:42:07.858833 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:07.858802    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fcb78ef-281c-451f-9cd9-64e932f5cd29-kube-api-access-2f2k9" (OuterVolumeSpecName: "kube-api-access-2f2k9") pod "4fcb78ef-281c-451f-9cd9-64e932f5cd29" (UID: "4fcb78ef-281c-451f-9cd9-64e932f5cd29"). InnerVolumeSpecName "kube-api-access-2f2k9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:42:07.954659 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:07.954560    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fcb78ef-281c-451f-9cd9-64e932f5cd29-maas-api-tls\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:42:07.954659 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:07.954597    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fcb78ef-281c-451f-9cd9-64e932f5cd29-openshift-service-ca\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:42:07.954659 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:07.954611    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f2k9\" (UniqueName: \"kubernetes.io/projected/4fcb78ef-281c-451f-9cd9-64e932f5cd29-kube-api-access-2f2k9\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:42:08.652626 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:08.652589    2563 generic.go:356] "Generic (PLEG): container finished" podID="4fcb78ef-281c-451f-9cd9-64e932f5cd29" containerID="d206d25ab25a3d2009347851e59f0c2a62cf0eb5122335d00d916b326c80c81e" exitCode=0
Jul 14 23:42:08.653068 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:08.652634    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-638b58-5c7d44b967-qc9dv"
Jul 14 23:42:08.653068 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:08.652677    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-638b58-5c7d44b967-qc9dv" event={"ID":"4fcb78ef-281c-451f-9cd9-64e932f5cd29","Type":"ContainerDied","Data":"d206d25ab25a3d2009347851e59f0c2a62cf0eb5122335d00d916b326c80c81e"}
Jul 14 23:42:08.653068 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:08.652733    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-638b58-5c7d44b967-qc9dv" event={"ID":"4fcb78ef-281c-451f-9cd9-64e932f5cd29","Type":"ContainerDied","Data":"9adf229b71d10185e7cf2b78e8aae6158f581d99a766b0f6c6d699e4cee57041"}
Jul 14 23:42:08.653068 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:08.652760    2563 scope.go:117] "RemoveContainer" containerID="d206d25ab25a3d2009347851e59f0c2a62cf0eb5122335d00d916b326c80c81e"
Jul 14 23:42:08.662153 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:08.662135    2563 scope.go:117] "RemoveContainer" containerID="d206d25ab25a3d2009347851e59f0c2a62cf0eb5122335d00d916b326c80c81e"
Jul 14 23:42:08.662456 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:42:08.662435    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d206d25ab25a3d2009347851e59f0c2a62cf0eb5122335d00d916b326c80c81e\": container with ID starting with d206d25ab25a3d2009347851e59f0c2a62cf0eb5122335d00d916b326c80c81e not found: ID does not exist" containerID="d206d25ab25a3d2009347851e59f0c2a62cf0eb5122335d00d916b326c80c81e"
Jul 14 23:42:08.662523 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:08.662463    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d206d25ab25a3d2009347851e59f0c2a62cf0eb5122335d00d916b326c80c81e"} err="failed to get container status \"d206d25ab25a3d2009347851e59f0c2a62cf0eb5122335d00d916b326c80c81e\": rpc error: code = NotFound desc = could not find container \"d206d25ab25a3d2009347851e59f0c2a62cf0eb5122335d00d916b326c80c81e\": container with ID starting with d206d25ab25a3d2009347851e59f0c2a62cf0eb5122335d00d916b326c80c81e not found: ID does not exist"
Jul 14 23:42:08.675955 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:08.675935    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-638b58-5c7d44b967-qc9dv"]
Jul 14 23:42:08.682153 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:08.682132    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-638b58-5c7d44b967-qc9dv"]
Jul 14 23:42:09.232313 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:09.230731    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fcb78ef-281c-451f-9cd9-64e932f5cd29" path="/var/lib/kubelet/pods/4fcb78ef-281c-451f-9cd9-64e932f5cd29/volumes"
Jul 14 23:42:22.740586 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:22.740550    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"]
Jul 14 23:42:22.741206 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:22.740774    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm" podUID="c2a753c8-3cdc-418b-b490-ec0fe6fd4a30" containerName="istio-proxy" containerID="cri-o://1ab11a47375f21ce93184457994e965149e8694fe55fba2f658dcad1dc3189c0" gracePeriod=30
Jul 14 23:42:23.425111 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:23.425074    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5d685bdbbd-6vx8d_eda3da51-cec1-4290-965f-e8162f9079b2/authorino/0.log"
Jul 14 23:42:24.096693 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:24.096653    2563 patch_prober.go:28] interesting pod/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 14 23:42:24.097067 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:24.096715    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm" podUID="c2a753c8-3cdc-418b-b490-ec0fe6fd4a30" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 14 23:42:27.994868 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:27.994838    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"
Jul 14 23:42:28.136782 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:28.136674    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-istiod-ca-cert\") pod \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\" (UID: \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\") "
Jul 14 23:42:28.136782 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:28.136746    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-istio-token\") pod \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\" (UID: \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\") "
Jul 14 23:42:28.136782 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:28.136783    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-workload-certs\") pod \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\" (UID: \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\") "
Jul 14 23:42:28.137063 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:28.136877    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-workload-socket\") pod \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\" (UID: \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\") "
Jul 14 23:42:28.137063 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:28.136936    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-istio-podinfo\") pod \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\" (UID: \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\") "
Jul 14 23:42:28.137063 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:28.136961    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frtxf\" (UniqueName: \"kubernetes.io/projected/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-kube-api-access-frtxf\") pod \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\" (UID: \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\") "
Jul 14 23:42:28.137063 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:28.136980    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-credential-socket\") pod \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\" (UID: \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\") "
Jul 14 23:42:28.137063 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:28.137026    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-istio-data\") pod \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\" (UID: \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\") "
Jul 14 23:42:28.137353 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:28.137094    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c2a753c8-3cdc-418b-b490-ec0fe6fd4a30" (UID: "c2a753c8-3cdc-418b-b490-ec0fe6fd4a30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:42:28.137353 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:28.137139    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-istio-envoy\") pod \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\" (UID: \"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30\") "
Jul 14 23:42:28.137786 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:28.137183    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c2a753c8-3cdc-418b-b490-ec0fe6fd4a30" (UID: "c2a753c8-3cdc-418b-b490-ec0fe6fd4a30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 23:42:28.137875 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:28.137367    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c2a753c8-3cdc-418b-b490-ec0fe6fd4a30" (UID: "c2a753c8-3cdc-418b-b490-ec0fe6fd4a30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:42:28.137875 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:28.137397    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c2a753c8-3cdc-418b-b490-ec0fe6fd4a30" (UID: "c2a753c8-3cdc-418b-b490-ec0fe6fd4a30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:42:28.137875 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:28.137497    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-workload-certs\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:42:28.138161 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:28.138098    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-istio-data" (OuterVolumeSpecName: "istio-data") pod "c2a753c8-3cdc-418b-b490-ec0fe6fd4a30" (UID: "c2a753c8-3cdc-418b-b490-ec0fe6fd4a30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:42:28.139201 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:28.139178    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-istio-token" (OuterVolumeSpecName: "istio-token") pod "c2a753c8-3cdc-418b-b490-ec0fe6fd4a30" (UID: "c2a753c8-3cdc-418b-b490-ec0fe6fd4a30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:42:28.139344 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:28.139324    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c2a753c8-3cdc-418b-b490-ec0fe6fd4a30" (UID: "c2a753c8-3cdc-418b-b490-ec0fe6fd4a30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 23:42:28.139407 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:28.139366    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-kube-api-access-frtxf" (OuterVolumeSpecName: "kube-api-access-frtxf") pod "c2a753c8-3cdc-418b-b490-ec0fe6fd4a30" (UID: "c2a753c8-3cdc-418b-b490-ec0fe6fd4a30"). InnerVolumeSpecName "kube-api-access-frtxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 23:42:28.139575 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:28.139561    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c2a753c8-3cdc-418b-b490-ec0fe6fd4a30" (UID: "c2a753c8-3cdc-418b-b490-ec0fe6fd4a30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 23:42:28.239224 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:28.239187    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-istio-token\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:42:28.239224 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:28.239221    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-workload-socket\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:42:28.239224 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:28.239230    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-istio-podinfo\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:42:28.239489 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:28.239238    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frtxf\" (UniqueName: \"kubernetes.io/projected/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-kube-api-access-frtxf\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:42:28.239489 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:28.239247    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-credential-socket\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:42:28.239489 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:28.239255    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-istio-data\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:42:28.239489 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:28.239263    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-istio-envoy\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:42:28.239489 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:28.239270    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30-istiod-ca-cert\") on node \"ip-10-0-134-8.ec2.internal\" DevicePath \"\""
Jul 14 23:42:28.746479 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:28.746444    2563 generic.go:356] "Generic (PLEG): container finished" podID="c2a753c8-3cdc-418b-b490-ec0fe6fd4a30" containerID="1ab11a47375f21ce93184457994e965149e8694fe55fba2f658dcad1dc3189c0" exitCode=0
Jul 14 23:42:28.746658 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:28.746492    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"
Jul 14 23:42:28.746658 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:28.746520    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm" event={"ID":"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30","Type":"ContainerDied","Data":"1ab11a47375f21ce93184457994e965149e8694fe55fba2f658dcad1dc3189c0"}
Jul 14 23:42:28.746658 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:28.746565    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm" event={"ID":"c2a753c8-3cdc-418b-b490-ec0fe6fd4a30","Type":"ContainerDied","Data":"9f2bf5f0daf7d320398633cb5c117ca4996e51a9b96749bb28c61aa79ecd0866"}
Jul 14 23:42:28.746658 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:28.746589    2563 scope.go:117] "RemoveContainer" containerID="1ab11a47375f21ce93184457994e965149e8694fe55fba2f658dcad1dc3189c0"
Jul 14 23:42:28.755805 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:28.755786    2563 scope.go:117] "RemoveContainer" containerID="f52d1466e684ae5faea8fbe5f21260467af0c7c9387b2c5129c21d5a895b63a3"
Jul 14 23:42:28.764106 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:28.764087    2563 scope.go:117] "RemoveContainer" containerID="1ab11a47375f21ce93184457994e965149e8694fe55fba2f658dcad1dc3189c0"
Jul 14 23:42:28.764345 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:42:28.764325    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ab11a47375f21ce93184457994e965149e8694fe55fba2f658dcad1dc3189c0\": container with ID starting with 1ab11a47375f21ce93184457994e965149e8694fe55fba2f658dcad1dc3189c0 not found: ID does not exist" containerID="1ab11a47375f21ce93184457994e965149e8694fe55fba2f658dcad1dc3189c0"
Jul 14 23:42:28.764463 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:28.764351    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ab11a47375f21ce93184457994e965149e8694fe55fba2f658dcad1dc3189c0"} err="failed to get container status \"1ab11a47375f21ce93184457994e965149e8694fe55fba2f658dcad1dc3189c0\": rpc error: code = NotFound desc = could not find container \"1ab11a47375f21ce93184457994e965149e8694fe55fba2f658dcad1dc3189c0\": container with ID starting with 1ab11a47375f21ce93184457994e965149e8694fe55fba2f658dcad1dc3189c0 not found: ID does not exist"
Jul 14 23:42:28.764463 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:28.764367    2563 scope.go:117] "RemoveContainer" containerID="f52d1466e684ae5faea8fbe5f21260467af0c7c9387b2c5129c21d5a895b63a3"
Jul 14 23:42:28.764608 ip-10-0-134-8 kubenswrapper[2563]: E0714 23:42:28.764594    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f52d1466e684ae5faea8fbe5f21260467af0c7c9387b2c5129c21d5a895b63a3\": container with ID starting with f52d1466e684ae5faea8fbe5f21260467af0c7c9387b2c5129c21d5a895b63a3 not found: ID does not exist" containerID="f52d1466e684ae5faea8fbe5f21260467af0c7c9387b2c5129c21d5a895b63a3"
Jul 14 23:42:28.764656 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:28.764612    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f52d1466e684ae5faea8fbe5f21260467af0c7c9387b2c5129c21d5a895b63a3"} err="failed to get container status \"f52d1466e684ae5faea8fbe5f21260467af0c7c9387b2c5129c21d5a895b63a3\": rpc error: code = NotFound desc = could not find container \"f52d1466e684ae5faea8fbe5f21260467af0c7c9387b2c5129c21d5a895b63a3\": container with ID starting with f52d1466e684ae5faea8fbe5f21260467af0c7c9387b2c5129c21d5a895b63a3 not found: ID does not exist"
Jul 14 23:42:28.768967 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:28.768946    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"]
Jul 14 23:42:28.773216 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:28.773190    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-638b58-openshift-default-5759f68487-r4qmm"]
Jul 14 23:42:29.178380 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:29.178266    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-9w2wb_435e7622-b867-42ea-a6e6-f7e6f857972b/server/0.log"
Jul 14 23:42:29.229863 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:29.229826    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2a753c8-3cdc-418b-b490-ec0fe6fd4a30" path="/var/lib/kubelet/pods/c2a753c8-3cdc-418b-b490-ec0fe6fd4a30/volumes"
Jul 14 23:42:29.298950 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:29.298914    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-drvt2_e6034bd0-b905-40c7-8380-eff83fd5522d/manager/0.log"
Jul 14 23:42:29.677077 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:29.677044    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d756c8cff-xt4p6_fc1f564c-8810-45db-9764-2cce76d67e29/copy-manifests/0.log"
Jul 14 23:42:29.685173 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:29.685145    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d756c8cff-xt4p6_fc1f564c-8810-45db-9764-2cce76d67e29/manager/0.log"
Jul 14 23:42:30.165508 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:30.165480    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29734515-tf6xf_b91367bb-5189-4278-9e1e-90f33f2343d3/cleanup/0.log"
Jul 14 23:42:30.397908 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:30.397876    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-37a8657d-7cm86_c82f60e2-39e5-4c61-8fcc-331db1e25e24/revoke-keys/0.log"
Jul 14 23:42:30.517907 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:30.517877    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-c1ba7d12-hkc4q_2a76d6a7-015a-4458-a14a-a03a015231e3/revoke-keys/0.log"
Jul 14 23:42:31.427544 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:31.427514    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-9084746a-nxbxh_2730bbc7-5e83-418c-90b1-5c95a0a8a022/revoke-keys/0.log"
Jul 14 23:42:33.121530 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:33.121497    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5d685bdbbd-6vx8d_eda3da51-cec1-4290-965f-e8162f9079b2/authorino/0.log"
Jul 14 23:42:33.469134 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:33.469050    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-8f4gh_cdf2efa0-3d6a-4168-ba16-fb365f021f1c/kuadrant-console-plugin/0.log"
Jul 14 23:42:34.552072 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:34.552039    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-946bb8597-h8jmp_4e061977-cf57-46d7-9804-ed6f91ce7e60/kube-auth-proxy/0.log"
Jul 14 23:42:34.903458 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:34.903379    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-x84b2_b89eeddb-0012-4f87-9ec6-baafc6fb8013/payload-pre-processing/0.log"
Jul 14 23:42:35.579876 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:35.579848    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-4jthh_dac78e62-49ce-446f-960b-f1c5ab165acc/main/0.log"
Jul 14 23:42:42.115408 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:42.115376    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hkhmk/must-gather-mnr7p"]
Jul 14 23:42:42.115785 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:42.115675    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0718434c-4640-407f-8215-872ae7c6a09d" containerName="istio-proxy"
Jul 14 23:42:42.115785 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:42.115685    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0718434c-4640-407f-8215-872ae7c6a09d" containerName="istio-proxy"
Jul 14 23:42:42.115785 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:42.115693    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2a753c8-3cdc-418b-b490-ec0fe6fd4a30" containerName="istio-proxy"
Jul 14 23:42:42.115785 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:42.115700    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2a753c8-3cdc-418b-b490-ec0fe6fd4a30" containerName="istio-proxy"
Jul 14 23:42:42.115785 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:42.115712    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c933a07f-bc78-48ac-ba27-9c59fe97a35f" containerName="maas-api"
Jul 14 23:42:42.115785 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:42.115717    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c933a07f-bc78-48ac-ba27-9c59fe97a35f" containerName="maas-api"
Jul 14 23:42:42.115785 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:42.115723    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2a753c8-3cdc-418b-b490-ec0fe6fd4a30" containerName="istio-proxy"
Jul 14 23:42:42.115785 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:42.115728    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2a753c8-3cdc-418b-b490-ec0fe6fd4a30" containerName="istio-proxy"
Jul 14 23:42:42.115785 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:42.115740    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e413a5ab-be3e-4b71-b3a6-64abf7a1ecff" containerName="test-curl-7800-140280926458912"
Jul 14 23:42:42.115785 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:42.115746    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e413a5ab-be3e-4b71-b3a6-64abf7a1ecff" containerName="test-curl-7800-140280926458912"
Jul 14 23:42:42.115785 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:42.115757    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fcb78ef-281c-451f-9cd9-64e932f5cd29" containerName="maas-api"
Jul 14 23:42:42.115785 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:42.115762    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fcb78ef-281c-451f-9cd9-64e932f5cd29" containerName="maas-api"
Jul 14 23:42:42.116150 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:42.115813    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fcb78ef-281c-451f-9cd9-64e932f5cd29" containerName="maas-api"
Jul 14 23:42:42.116150 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:42.115823    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0718434c-4640-407f-8215-872ae7c6a09d" containerName="istio-proxy"
Jul 14 23:42:42.116150 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:42.115829    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2a753c8-3cdc-418b-b490-ec0fe6fd4a30" containerName="istio-proxy"
Jul 14 23:42:42.116150 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:42.115835    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e413a5ab-be3e-4b71-b3a6-64abf7a1ecff" containerName="test-curl-7800-140280926458912"
Jul 14 23:42:42.116150 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:42.115842    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c933a07f-bc78-48ac-ba27-9c59fe97a35f" containerName="maas-api"
Jul 14 23:42:42.116150 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:42.115848    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0718434c-4640-407f-8215-872ae7c6a09d" containerName="istio-proxy"
Jul 14 23:42:42.116150 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:42.115853    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2a753c8-3cdc-418b-b490-ec0fe6fd4a30" containerName="istio-proxy"
Jul 14 23:42:42.116150 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:42.115895    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0718434c-4640-407f-8215-872ae7c6a09d" containerName="istio-proxy"
Jul 14 23:42:42.116150 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:42.115900    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0718434c-4640-407f-8215-872ae7c6a09d" containerName="istio-proxy"
Jul 14 23:42:42.118880 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:42.118860    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hkhmk/must-gather-mnr7p"
Jul 14 23:42:42.122836 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:42.122809    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hkhmk\"/\"default-dockercfg-jtt7v\""
Jul 14 23:42:42.122961 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:42.122841    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hkhmk\"/\"openshift-service-ca.crt\""
Jul 14 23:42:42.124300 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:42.124267    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hkhmk\"/\"kube-root-ca.crt\""
Jul 14 23:42:42.129687 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:42.129663    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hkhmk/must-gather-mnr7p"]
Jul 14 23:42:42.261723 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:42.261685    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjbn2\" (UniqueName: \"kubernetes.io/projected/cef5d22a-3a68-4e3a-aff3-a93945fac61b-kube-api-access-tjbn2\") pod \"must-gather-mnr7p\" (UID: \"cef5d22a-3a68-4e3a-aff3-a93945fac61b\") " pod="openshift-must-gather-hkhmk/must-gather-mnr7p"
Jul 14 23:42:42.261915 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:42.261750    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cef5d22a-3a68-4e3a-aff3-a93945fac61b-must-gather-output\") pod \"must-gather-mnr7p\" (UID: \"cef5d22a-3a68-4e3a-aff3-a93945fac61b\") " pod="openshift-must-gather-hkhmk/must-gather-mnr7p"
Jul 14 23:42:42.362727 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:42.362696    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cef5d22a-3a68-4e3a-aff3-a93945fac61b-must-gather-output\") pod \"must-gather-mnr7p\" (UID: \"cef5d22a-3a68-4e3a-aff3-a93945fac61b\") " pod="openshift-must-gather-hkhmk/must-gather-mnr7p"
Jul 14 23:42:42.362920 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:42.362775    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjbn2\" (UniqueName: \"kubernetes.io/projected/cef5d22a-3a68-4e3a-aff3-a93945fac61b-kube-api-access-tjbn2\") pod \"must-gather-mnr7p\" (UID: \"cef5d22a-3a68-4e3a-aff3-a93945fac61b\") " pod="openshift-must-gather-hkhmk/must-gather-mnr7p"
Jul 14 23:42:42.363044 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:42.363024    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cef5d22a-3a68-4e3a-aff3-a93945fac61b-must-gather-output\") pod \"must-gather-mnr7p\" (UID: \"cef5d22a-3a68-4e3a-aff3-a93945fac61b\") " pod="openshift-must-gather-hkhmk/must-gather-mnr7p"
Jul 14 23:42:42.374604 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:42.374535    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjbn2\" (UniqueName: \"kubernetes.io/projected/cef5d22a-3a68-4e3a-aff3-a93945fac61b-kube-api-access-tjbn2\") pod \"must-gather-mnr7p\" (UID: \"cef5d22a-3a68-4e3a-aff3-a93945fac61b\") " pod="openshift-must-gather-hkhmk/must-gather-mnr7p"
Jul 14 23:42:42.429052 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:42.429018    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hkhmk/must-gather-mnr7p"
Jul 14 23:42:42.568149 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:42.568033    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hkhmk/must-gather-mnr7p"]
Jul 14 23:42:42.570783 ip-10-0-134-8 kubenswrapper[2563]: W0714 23:42:42.570748    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcef5d22a_3a68_4e3a_aff3_a93945fac61b.slice/crio-6ed2d0dca0f7f3323c0e5a9233cc747d5dc211e90eca2cdbcabc6e4e706523a8 WatchSource:0}: Error finding container 6ed2d0dca0f7f3323c0e5a9233cc747d5dc211e90eca2cdbcabc6e4e706523a8: Status 404 returned error can't find the container with id 6ed2d0dca0f7f3323c0e5a9233cc747d5dc211e90eca2cdbcabc6e4e706523a8
Jul 14 23:42:42.811171 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:42.811129    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hkhmk/must-gather-mnr7p" event={"ID":"cef5d22a-3a68-4e3a-aff3-a93945fac61b","Type":"ContainerStarted","Data":"6ed2d0dca0f7f3323c0e5a9233cc747d5dc211e90eca2cdbcabc6e4e706523a8"}
Jul 14 23:42:44.822896 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:44.822854    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hkhmk/must-gather-mnr7p" event={"ID":"cef5d22a-3a68-4e3a-aff3-a93945fac61b","Type":"ContainerStarted","Data":"59c8ed985b688990b37eaa5ee8b21ce908800496426c0a82f8a3190c1ae83897"}
Jul 14 23:42:44.823270 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:44.822915    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hkhmk/must-gather-mnr7p" event={"ID":"cef5d22a-3a68-4e3a-aff3-a93945fac61b","Type":"ContainerStarted","Data":"58d6efdbdd2dde660aa499ba5fbb3f50a6e107c6671b0f812e0c3f4b7f3eaddf"}
Jul 14 23:42:44.841038 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:44.840973    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hkhmk/must-gather-mnr7p" podStartSLOduration=1.446079649 podStartE2EDuration="2.840952446s" totalImagesPullingTime="1.394872797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 23:42:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:42:44.838527469 +0000 UTC m=+2752.141518262" watchObservedRunningTime="2026-07-14 23:42:44.840952446 +0000 UTC m=+2752.143943239"
Jul 14 23:42:45.552352 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:45.552318    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xjw4z_7e3e444e-1753-442e-8430-24b0022f9418/global-pull-secret-syncer/0.log"
Jul 14 23:42:45.677700 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:45.677664    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pthrz_186005d6-1e47-4ea0-816c-469b21cc77d4/konnectivity-agent/0.log"
Jul 14 23:42:45.724437 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:45.724407    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-8.ec2.internal_3aa82a5123efb70e24ea524a0aeee7ed/haproxy/0.log"
Jul 14 23:42:51.307225 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:51.307022    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5d685bdbbd-6vx8d_eda3da51-cec1-4290-965f-e8162f9079b2/authorino/0.log"
Jul 14 23:42:51.399496 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:51.399463    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-8f4gh_cdf2efa0-3d6a-4168-ba16-fb365f021f1c/kuadrant-console-plugin/0.log"
Jul 14 23:42:53.719455 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:53.719424    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-74qb8_a998b1ac-c2d1-4c50-bfcd-4a700894beaa/node-exporter/0.log"
Jul 14 23:42:53.744226 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:53.744200    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-74qb8_a998b1ac-c2d1-4c50-bfcd-4a700894beaa/kube-rbac-proxy/0.log"
Jul 14 23:42:53.767859 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:53.767831    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-74qb8_a998b1ac-c2d1-4c50-bfcd-4a700894beaa/init-textfile/0.log"
Jul 14 23:42:55.350034 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:55.349989    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hkhmk/perf-node-gather-daemonset-2p9px"]
Jul 14 23:42:55.356094 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:55.356066    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hkhmk/perf-node-gather-daemonset-2p9px"
Jul 14 23:42:55.369564 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:55.369151    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hkhmk/perf-node-gather-daemonset-2p9px"]
Jul 14 23:42:55.428019 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:55.427983    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1607b96f-4b83-41c7-b9d6-19a41405f851-proc\") pod \"perf-node-gather-daemonset-2p9px\" (UID: \"1607b96f-4b83-41c7-b9d6-19a41405f851\") " pod="openshift-must-gather-hkhmk/perf-node-gather-daemonset-2p9px"
Jul 14 23:42:55.428217 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:55.428033    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fp66\" (UniqueName: \"kubernetes.io/projected/1607b96f-4b83-41c7-b9d6-19a41405f851-kube-api-access-2fp66\") pod \"perf-node-gather-daemonset-2p9px\" (UID: \"1607b96f-4b83-41c7-b9d6-19a41405f851\") " pod="openshift-must-gather-hkhmk/perf-node-gather-daemonset-2p9px"
Jul 14 23:42:55.428217 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:55.428097    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1607b96f-4b83-41c7-b9d6-19a41405f851-lib-modules\") pod \"perf-node-gather-daemonset-2p9px\" (UID: \"1607b96f-4b83-41c7-b9d6-19a41405f851\") " pod="openshift-must-gather-hkhmk/perf-node-gather-daemonset-2p9px"
Jul 14 23:42:55.428217 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:55.428164    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1607b96f-4b83-41c7-b9d6-19a41405f851-sys\") pod \"perf-node-gather-daemonset-2p9px\" (UID: \"1607b96f-4b83-41c7-b9d6-19a41405f851\") " pod="openshift-must-gather-hkhmk/perf-node-gather-daemonset-2p9px"
Jul 14 23:42:55.428217 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:55.428188    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1607b96f-4b83-41c7-b9d6-19a41405f851-podres\") pod \"perf-node-gather-daemonset-2p9px\" (UID: \"1607b96f-4b83-41c7-b9d6-19a41405f851\") " pod="openshift-must-gather-hkhmk/perf-node-gather-daemonset-2p9px"
Jul 14 23:42:55.529016 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:55.528978    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1607b96f-4b83-41c7-b9d6-19a41405f851-sys\") pod \"perf-node-gather-daemonset-2p9px\" (UID: \"1607b96f-4b83-41c7-b9d6-19a41405f851\") " pod="openshift-must-gather-hkhmk/perf-node-gather-daemonset-2p9px"
Jul 14 23:42:55.529016 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:55.529019    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1607b96f-4b83-41c7-b9d6-19a41405f851-podres\") pod \"perf-node-gather-daemonset-2p9px\" (UID: \"1607b96f-4b83-41c7-b9d6-19a41405f851\") " pod="openshift-must-gather-hkhmk/perf-node-gather-daemonset-2p9px"
Jul 14 23:42:55.529317 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:55.529043    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1607b96f-4b83-41c7-b9d6-19a41405f851-proc\") pod \"perf-node-gather-daemonset-2p9px\" (UID: \"1607b96f-4b83-41c7-b9d6-19a41405f851\") " pod="openshift-must-gather-hkhmk/perf-node-gather-daemonset-2p9px"
Jul 14 23:42:55.529317 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:55.529063    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fp66\" (UniqueName: \"kubernetes.io/projected/1607b96f-4b83-41c7-b9d6-19a41405f851-kube-api-access-2fp66\") pod \"perf-node-gather-daemonset-2p9px\" (UID: \"1607b96f-4b83-41c7-b9d6-19a41405f851\") " pod="openshift-must-gather-hkhmk/perf-node-gather-daemonset-2p9px"
Jul 14 23:42:55.529317 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:55.529100    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1607b96f-4b83-41c7-b9d6-19a41405f851-sys\") pod \"perf-node-gather-daemonset-2p9px\" (UID: \"1607b96f-4b83-41c7-b9d6-19a41405f851\") " pod="openshift-must-gather-hkhmk/perf-node-gather-daemonset-2p9px"
Jul 14 23:42:55.529317 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:55.529107    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1607b96f-4b83-41c7-b9d6-19a41405f851-lib-modules\") pod \"perf-node-gather-daemonset-2p9px\" (UID: \"1607b96f-4b83-41c7-b9d6-19a41405f851\") " pod="openshift-must-gather-hkhmk/perf-node-gather-daemonset-2p9px"
Jul 14 23:42:55.529317 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:55.529176    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1607b96f-4b83-41c7-b9d6-19a41405f851-proc\") pod \"perf-node-gather-daemonset-2p9px\" (UID: \"1607b96f-4b83-41c7-b9d6-19a41405f851\") " pod="openshift-must-gather-hkhmk/perf-node-gather-daemonset-2p9px"
Jul 14 23:42:55.529317 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:55.529200    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1607b96f-4b83-41c7-b9d6-19a41405f851-lib-modules\") pod \"perf-node-gather-daemonset-2p9px\" (UID: \"1607b96f-4b83-41c7-b9d6-19a41405f851\") " pod="openshift-must-gather-hkhmk/perf-node-gather-daemonset-2p9px"
Jul 14 23:42:55.529317 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:55.529203    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1607b96f-4b83-41c7-b9d6-19a41405f851-podres\") pod \"perf-node-gather-daemonset-2p9px\" (UID: \"1607b96f-4b83-41c7-b9d6-19a41405f851\") " pod="openshift-must-gather-hkhmk/perf-node-gather-daemonset-2p9px"
Jul 14 23:42:55.539803 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:55.539770    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fp66\" (UniqueName: \"kubernetes.io/projected/1607b96f-4b83-41c7-b9d6-19a41405f851-kube-api-access-2fp66\") pod \"perf-node-gather-daemonset-2p9px\" (UID: \"1607b96f-4b83-41c7-b9d6-19a41405f851\") " pod="openshift-must-gather-hkhmk/perf-node-gather-daemonset-2p9px"
Jul 14 23:42:55.657319 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:55.657185    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-gzz8f_c8c0fd62-6e13-4d36-b83e-c2a7542cf8b5/networking-console-plugin/0.log"
Jul 14 23:42:55.670255 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:55.670222    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hkhmk/perf-node-gather-daemonset-2p9px"
Jul 14 23:42:55.866345 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:55.865935    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hkhmk/perf-node-gather-daemonset-2p9px"]
Jul 14 23:42:55.919355 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:55.905785    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hkhmk/perf-node-gather-daemonset-2p9px" event={"ID":"1607b96f-4b83-41c7-b9d6-19a41405f851","Type":"ContainerStarted","Data":"5ef1abd3be686b7377ac079c7ac8dee36a6205680d3922ba987d0fb22eba26d9"}
Jul 14 23:42:56.778342 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:56.778304    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-c59575677-kj6sb_4512e88b-99f9-440f-b7d4-78a8f5dfed1a/console/0.log"
Jul 14 23:42:56.834816 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:56.834785    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-58gwx_f136e031-4413-4c00-ade9-a01658d10755/download-server/0.log"
Jul 14 23:42:56.914241 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:56.914194    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hkhmk/perf-node-gather-daemonset-2p9px" event={"ID":"1607b96f-4b83-41c7-b9d6-19a41405f851","Type":"ContainerStarted","Data":"a8242aa8aafcc0d215b6934d9b990a81d306417263f1915ead6fac76d5ffabc7"}
Jul 14 23:42:56.914709 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:56.914679    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-hkhmk/perf-node-gather-daemonset-2p9px"
Jul 14 23:42:56.933114 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:56.933041    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hkhmk/perf-node-gather-daemonset-2p9px" podStartSLOduration=1.933016924 podStartE2EDuration="1.933016924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 23:42:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 23:42:56.929726668 +0000 UTC m=+2764.232717462" watchObservedRunningTime="2026-07-14 23:42:56.933016924 +0000 UTC m=+2764.236007717"
Jul 14 23:42:58.501094 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:58.501056    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-q4982_ffb5cdb7-2bde-418f-884d-0962fc26414e/dns/0.log"
Jul 14 23:42:58.522986 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:58.522931    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-q4982_ffb5cdb7-2bde-418f-884d-0962fc26414e/kube-rbac-proxy/0.log"
Jul 14 23:42:58.625473 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:58.625444    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-s4zgk_59f4f5d3-9650-40e0-969a-e912429a2481/dns-node-resolver/0.log"
Jul 14 23:42:59.183880 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:59.183843    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-548c9cffcc-2qm89_13b8983f-4071-41c4-9946-7a48c932d4ca/registry/0.log"
Jul 14 23:42:59.231537 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:42:59.231500    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lzt25_1ef7f81d-618b-4d9a-b018-9e3346c27bd1/node-ca/0.log"
Jul 14 23:43:00.283093 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:43:00.283066    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-946bb8597-h8jmp_4e061977-cf57-46d7-9804-ed6f91ce7e60/kube-auth-proxy/0.log"
Jul 14 23:43:00.428040 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:43:00.427998    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-x84b2_b89eeddb-0012-4f87-9ec6-baafc6fb8013/payload-pre-processing/0.log"
Jul 14 23:43:01.158126 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:43:01.158050    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vdd85_4be0471d-f724-4147-b1b0-c5ff7a82b7d1/serve-healthcheck-canary/0.log"
Jul 14 23:43:01.651953 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:43:01.651924    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-28rrp_341a042f-b19e-418e-958a-f42c097be024/kube-rbac-proxy/0.log"
Jul 14 23:43:01.673763 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:43:01.673739    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-28rrp_341a042f-b19e-418e-958a-f42c097be024/exporter/0.log"
Jul 14 23:43:01.694147 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:43:01.694107    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-28rrp_341a042f-b19e-418e-958a-f42c097be024/extractor/0.log"
Jul 14 23:43:02.934096 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:43:02.934063    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-hkhmk/perf-node-gather-daemonset-2p9px"
Jul 14 23:43:04.423024 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:43:04.422984    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-9w2wb_435e7622-b867-42ea-a6e6-f7e6f857972b/server/0.log"
Jul 14 23:43:04.519168 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:43:04.519124    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-drvt2_e6034bd0-b905-40c7-8380-eff83fd5522d/manager/0.log"
Jul 14 23:43:04.696562 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:43:04.696483    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d756c8cff-xt4p6_fc1f564c-8810-45db-9764-2cce76d67e29/manager/0.log"
Jul 14 23:43:04.721682 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:43:04.721642    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d756c8cff-xt4p6_fc1f564c-8810-45db-9764-2cce76d67e29/copy-manifests/0.log"
Jul 14 23:43:06.065425 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:43:06.065384    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-86f675f86-94m4p_8a7c1c58-c882-400e-9d2b-a764c20bbbcb/manager/0.log"
Jul 14 23:43:06.146307 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:43:06.146260    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-946bb8597-h8jmp_4e061977-cf57-46d7-9804-ed6f91ce7e60/kube-auth-proxy/0.log"
Jul 14 23:43:06.269397 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:43:06.269365    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-x84b2_b89eeddb-0012-4f87-9ec6-baafc6fb8013/payload-pre-processing/0.log"
Jul 14 23:43:07.674250 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:43:07.674169    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-946bb8597-h8jmp_4e061977-cf57-46d7-9804-ed6f91ce7e60/kube-auth-proxy/0.log"
Jul 14 23:43:07.801804 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:43:07.801772    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-x84b2_b89eeddb-0012-4f87-9ec6-baafc6fb8013/payload-pre-processing/0.log"
Jul 14 23:43:10.765473 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:43:10.765425    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-nn4qg_acd21154-5243-4aeb-bd93-acd9da0ca28c/migrator/0.log"
Jul 14 23:43:10.785683 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:43:10.785657    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-nn4qg_acd21154-5243-4aeb-bd93-acd9da0ca28c/graceful-termination/0.log"
Jul 14 23:43:12.136368 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:43:12.136339    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r5s76_da7d977f-1dd0-42e6-b3a7-455490097670/kube-multus-additional-cni-plugins/0.log"
Jul 14 23:43:12.156992 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:43:12.156967    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r5s76_da7d977f-1dd0-42e6-b3a7-455490097670/egress-router-binary-copy/0.log"
Jul 14 23:43:12.176788 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:43:12.176761    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r5s76_da7d977f-1dd0-42e6-b3a7-455490097670/cni-plugins/0.log"
Jul 14 23:43:12.198777 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:43:12.198748    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r5s76_da7d977f-1dd0-42e6-b3a7-455490097670/bond-cni-plugin/0.log"
Jul 14 23:43:12.218407 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:43:12.218371    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r5s76_da7d977f-1dd0-42e6-b3a7-455490097670/routeoverride-cni/0.log"
Jul 14 23:43:12.238351 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:43:12.238327    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r5s76_da7d977f-1dd0-42e6-b3a7-455490097670/whereabouts-cni-bincopy/0.log"
Jul 14 23:43:12.257800 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:43:12.257772    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r5s76_da7d977f-1dd0-42e6-b3a7-455490097670/whereabouts-cni/0.log"
Jul 14 23:43:12.646644 ip-10-0-134-8 kubenswrapper[2563]: I0714 23:43:12.646603    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-vlbvv_cb9705e3-b1b7-44a7-b4b8-49db0e928da9/kube-multus/0.log"