Jul 12 22:10:26.382093 ip-10-0-135-47 systemd[1]: Starting Kubernetes Kubelet...
Jul 12 22:10:26.853870 ip-10-0-135-47 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 12 22:10:26.853870 ip-10-0-135-47 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 12 22:10:26.853870 ip-10-0-135-47 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 12 22:10:26.853870 ip-10-0-135-47 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 12 22:10:26.853870 ip-10-0-135-47 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 12 22:10:26.856817 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.856711    2560 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 12 22:10:26.862478 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862461    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 12 22:10:26.862478 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862478    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 12 22:10:26.862549 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862481    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 12 22:10:26.862549 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862485    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 12 22:10:26.862549 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862489    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 12 22:10:26.862549 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862492    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 12 22:10:26.862549 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862494    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 12 22:10:26.862549 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862498    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 12 22:10:26.862549 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862502    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 12 22:10:26.862549 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862505    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 12 22:10:26.862549 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862507    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 12 22:10:26.862549 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862510    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 12 22:10:26.862549 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862512    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 12 22:10:26.862549 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862515    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 12 22:10:26.862549 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862518    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 12 22:10:26.862549 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862521    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 12 22:10:26.862549 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862523    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 12 22:10:26.862549 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862532    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 12 22:10:26.862549 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862534    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 12 22:10:26.862549 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862537    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 12 22:10:26.862549 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862539    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 12 22:10:26.862549 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862541    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 12 22:10:26.863047 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862544    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 12 22:10:26.863047 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862546    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 12 22:10:26.863047 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862549    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 12 22:10:26.863047 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862552    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 12 22:10:26.863047 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862555    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 12 22:10:26.863047 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862557    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 12 22:10:26.863047 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862560    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 12 22:10:26.863047 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862563    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 12 22:10:26.863047 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862566    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 12 22:10:26.863047 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862568    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 12 22:10:26.863047 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862571    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 12 22:10:26.863047 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862575    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 12 22:10:26.863047 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862578    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 12 22:10:26.863047 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862581    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 12 22:10:26.863047 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862584    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 12 22:10:26.863047 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862588    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 12 22:10:26.863047 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862591    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 12 22:10:26.863047 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862593    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 12 22:10:26.863047 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862596    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 12 22:10:26.863512 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862598    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 12 22:10:26.863512 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862601    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 12 22:10:26.863512 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862603    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 12 22:10:26.863512 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862606    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 12 22:10:26.863512 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862609    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 12 22:10:26.863512 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862611    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 12 22:10:26.863512 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862615    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 12 22:10:26.863512 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862619    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 12 22:10:26.863512 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862622    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 12 22:10:26.863512 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862624    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 12 22:10:26.863512 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862627    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 12 22:10:26.863512 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862629    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 12 22:10:26.863512 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862631    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 12 22:10:26.863512 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862633    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 12 22:10:26.863512 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862637    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 12 22:10:26.863512 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862640    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 12 22:10:26.863512 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862643    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 12 22:10:26.863512 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862645    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 12 22:10:26.863512 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862648    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 12 22:10:26.863512 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862651    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 12 22:10:26.864036 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862653    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 12 22:10:26.864036 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862656    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 12 22:10:26.864036 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862658    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 12 22:10:26.864036 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862662    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 12 22:10:26.864036 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862665    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 12 22:10:26.864036 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862667    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 12 22:10:26.864036 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862670    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 12 22:10:26.864036 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862672    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 12 22:10:26.864036 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862674    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 12 22:10:26.864036 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862677    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 12 22:10:26.864036 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862680    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 12 22:10:26.864036 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862682    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 12 22:10:26.864036 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862686    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 12 22:10:26.864036 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862689    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 12 22:10:26.864036 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862693    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 12 22:10:26.864036 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862696    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 12 22:10:26.864036 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862699    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 12 22:10:26.864036 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862701    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 12 22:10:26.864036 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862704    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 12 22:10:26.864489 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862707    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 12 22:10:26.864489 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862709    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 12 22:10:26.864489 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862712    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 12 22:10:26.864489 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862714    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 12 22:10:26.864489 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862716    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 12 22:10:26.864489 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862719    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 12 22:10:26.864489 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862722    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 12 22:10:26.864489 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862725    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 12 22:10:26.864489 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862728    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 12 22:10:26.864489 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862730    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 12 22:10:26.864489 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862733    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 12 22:10:26.864489 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862735    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 12 22:10:26.864489 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862749    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 12 22:10:26.864489 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862752    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 12 22:10:26.864489 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862754    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 12 22:10:26.864489 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862757    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 12 22:10:26.864489 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862760    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 12 22:10:26.864489 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862762    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 12 22:10:26.864489 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.862765    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 12 22:10:26.864489 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863466    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 12 22:10:26.865014 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863473    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 12 22:10:26.865014 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863476    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 12 22:10:26.865014 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863481    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 12 22:10:26.865014 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863485    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 12 22:10:26.865014 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863488    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 12 22:10:26.865014 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863490    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 12 22:10:26.865014 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863493    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 12 22:10:26.865014 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863496    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 12 22:10:26.865014 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863499    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 12 22:10:26.865014 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863502    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 12 22:10:26.865014 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863504    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 12 22:10:26.865014 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863507    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 12 22:10:26.865014 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863509    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 12 22:10:26.865014 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863513    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 12 22:10:26.865014 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863515    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 12 22:10:26.865014 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863518    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 12 22:10:26.865014 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863521    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 12 22:10:26.865014 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863524    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 12 22:10:26.865014 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863526    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 12 22:10:26.865509 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863529    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 12 22:10:26.865509 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863532    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 12 22:10:26.865509 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863535    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 12 22:10:26.865509 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863537    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 12 22:10:26.865509 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863540    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 12 22:10:26.865509 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863542    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 12 22:10:26.865509 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863545    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 12 22:10:26.865509 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863548    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 12 22:10:26.865509 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863550    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 12 22:10:26.865509 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863553    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 12 22:10:26.865509 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863555    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 12 22:10:26.865509 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863557    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 12 22:10:26.865509 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863560    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 12 22:10:26.865509 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863562    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 12 22:10:26.865509 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863565    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 12 22:10:26.865509 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863567    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 12 22:10:26.865509 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863570    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 12 22:10:26.865509 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863572    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 12 22:10:26.865509 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863574    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 12 22:10:26.865509 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863577    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 12 22:10:26.866100 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863579    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 12 22:10:26.866100 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863582    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 12 22:10:26.866100 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863585    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 12 22:10:26.866100 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863588    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 12 22:10:26.866100 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863590    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 12 22:10:26.866100 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863593    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 12 22:10:26.866100 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863596    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 12 22:10:26.866100 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863598    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 12 22:10:26.866100 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863600    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 12 22:10:26.866100 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863603    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 12 22:10:26.866100 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863605    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 12 22:10:26.866100 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863608    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 12 22:10:26.866100 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863610    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 12 22:10:26.866100 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863613    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 12 22:10:26.866100 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863615    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 12 22:10:26.866100 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863618    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 12 22:10:26.866100 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863620    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 12 22:10:26.866100 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863623    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 12 22:10:26.866100 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863625    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 12 22:10:26.866100 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863628    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 12 22:10:26.866593 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863630    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 12 22:10:26.866593 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863633    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 12 22:10:26.866593 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863635    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 12 22:10:26.866593 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863638    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 12 22:10:26.866593 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863640    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 12 22:10:26.866593 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863644    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 12 22:10:26.866593 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863647    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 12 22:10:26.866593 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863650    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 12 22:10:26.866593 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863655    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 12 22:10:26.866593 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863658    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 12 22:10:26.866593 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863660    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 12 22:10:26.866593 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863663    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 12 22:10:26.866593 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863665    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 12 22:10:26.866593 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863668    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 12 22:10:26.866593 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863670    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 12 22:10:26.866593 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863673    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 12 22:10:26.866593 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863675    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 12 22:10:26.866593 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863679    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 12 22:10:26.867049 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863681    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 12 22:10:26.867049 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863684    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 12 22:10:26.867049 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863686    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 12 22:10:26.867049 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863688    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 12 22:10:26.867049 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863691    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 12 22:10:26.867049 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863694    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 12 22:10:26.867049 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863696    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 12 22:10:26.867049 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863698    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 12 22:10:26.867049 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863701    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 12 22:10:26.867049 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863704    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 12 22:10:26.867049 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863706    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 12 22:10:26.867049 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863709    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 12 22:10:26.867049 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863711    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 12 22:10:26.867049 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863714    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 12 22:10:26.867049 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863717    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 12 22:10:26.867049 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863720    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 12 22:10:26.867049 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863723    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 12 22:10:26.867049 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863725    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 12 22:10:26.867049 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863728    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 12 22:10:26.867049 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863730    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 12 22:10:26.867548 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.863733    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 12 22:10:26.867548 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864186    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 12 22:10:26.867548 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864191    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 12 22:10:26.867548 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864193    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 12 22:10:26.867548 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864196    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 12 22:10:26.867548 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864198    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 12 22:10:26.867548 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864201    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 12 22:10:26.867548 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864204    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 12 22:10:26.867548 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864206    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 12 22:10:26.867548 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864215    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 12 22:10:26.867548 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864218    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 12 22:10:26.867548 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864221    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 12 22:10:26.867548 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864223    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 12 22:10:26.867548 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864226    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 12 22:10:26.867548 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864228    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 12 22:10:26.867548 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864231    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 12 22:10:26.867548 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864234    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 12 22:10:26.867548 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864236    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 12 22:10:26.867548 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864239    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 12 22:10:26.867548 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864241    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 12 22:10:26.868038 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864244    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 12 22:10:26.868038 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864247    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 12 22:10:26.868038 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864249    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 12 22:10:26.868038 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864252    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 12 22:10:26.868038 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864254    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 12 22:10:26.868038 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864258    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 12 22:10:26.868038 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864262    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 12 22:10:26.868038 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864266    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 12 22:10:26.868038 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864269    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 12 22:10:26.868038 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864271    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 12 22:10:26.868038 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864274    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 12 22:10:26.868038 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864277    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 12 22:10:26.868038 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864280    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 12 22:10:26.868038 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864283    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 12 22:10:26.868038 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864286    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 12 22:10:26.868038 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864288    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 12 22:10:26.868038 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864290    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 12 22:10:26.868038 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864293    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 12 22:10:26.868038 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864295    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 12 22:10:26.868495 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864298    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 12 22:10:26.868495 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864300    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 12 22:10:26.868495 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864302    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 12 22:10:26.868495 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864311    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 12 22:10:26.868495 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864314    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 12 22:10:26.868495 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864319    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 12 22:10:26.868495 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864321    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 12 22:10:26.868495 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864324    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 12 22:10:26.868495 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864327    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 12 22:10:26.868495 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864329    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 12 22:10:26.868495 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864331    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 12 22:10:26.868495 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864334    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 12 22:10:26.868495 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864336    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 12 22:10:26.868495 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864339    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 12 22:10:26.868495 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864341    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 12 22:10:26.868495 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864343    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 12 22:10:26.868495 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864346    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 12 22:10:26.868495 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864349    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 12 22:10:26.868495 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864351    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 12 22:10:26.868495 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864354    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 12 22:10:26.868989 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864356    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 12 22:10:26.868989 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864359    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 12 22:10:26.868989 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864361    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 12 22:10:26.868989 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864364    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 12 22:10:26.868989 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864366    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 12 22:10:26.868989 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864369    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 12 22:10:26.868989 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864373    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 12 22:10:26.868989 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864375    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 12 22:10:26.868989 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864378    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 12 22:10:26.868989 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864381    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 12 22:10:26.868989 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864384    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 12 22:10:26.868989 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864386    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 12 22:10:26.868989 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864388    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 12 22:10:26.868989 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864391    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 12 22:10:26.868989 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864393    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 12 22:10:26.868989 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864396    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 12 22:10:26.868989 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864404    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 12 22:10:26.868989 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864408    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 12 22:10:26.868989 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864410    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 12 22:10:26.869436 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864413    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 12 22:10:26.869436 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864415    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 12 22:10:26.869436 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864418    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 12 22:10:26.869436 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864420    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 12 22:10:26.869436 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864423    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 12 22:10:26.869436 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864426    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 12 22:10:26.869436 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864428    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 12 22:10:26.869436 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864431    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 12 22:10:26.869436 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864433    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 12 22:10:26.869436 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864436    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 12 22:10:26.869436 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864438    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 12 22:10:26.869436 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864440    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 12 22:10:26.869436 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864443    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 12 22:10:26.869436 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864445    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 12 22:10:26.869436 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864447    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 12 22:10:26.869436 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864450    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 12 22:10:26.869436 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864453    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 12 22:10:26.869436 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864455    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 12 22:10:26.869436 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864457    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 12 22:10:26.869436 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864460    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 12 22:10:26.869913 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864462    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 12 22:10:26.869913 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.864465    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 12 22:10:26.869913 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864545    2560 flags.go:64] FLAG: --address="0.0.0.0"
Jul 12 22:10:26.869913 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864555    2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 12 22:10:26.869913 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864574    2560 flags.go:64] FLAG: --anonymous-auth="true"
Jul 12 22:10:26.869913 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864580    2560 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 12 22:10:26.869913 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864586    2560 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 12 22:10:26.869913 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864591    2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 12 22:10:26.869913 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864597    2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 12 22:10:26.869913 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864602    2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 12 22:10:26.869913 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864615    2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 12 22:10:26.869913 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864618    2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 12 22:10:26.869913 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864622    2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 12 22:10:26.869913 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864626    2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 12 22:10:26.869913 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864628    2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 12 22:10:26.869913 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864631    2560 flags.go:64] FLAG: --cgroup-root=""
Jul 12 22:10:26.869913 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864636    2560 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 12 22:10:26.869913 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864639    2560 flags.go:64] FLAG: --client-ca-file=""
Jul 12 22:10:26.869913 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864641    2560 flags.go:64] FLAG: --cloud-provider="external"
Jul 12 22:10:26.869913 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864644    2560 flags.go:64] FLAG: --cluster-dns="[]"
Jul 12 22:10:26.869913 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864651    2560 flags.go:64] FLAG: --cluster-domain=""
Jul 12 22:10:26.869913 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864653    2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 12 22:10:26.869913 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864656    2560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 12 22:10:26.870472 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864659    2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 12 22:10:26.870472 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864662    2560 flags.go:64] FLAG: --container-log-max-files="5"
Jul 12 22:10:26.870472 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864666    2560 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 12 22:10:26.870472 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864669    2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 12 22:10:26.870472 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864672    2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 12 22:10:26.870472 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864675    2560 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 12 22:10:26.870472 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864678    2560 flags.go:64] FLAG: --contention-profiling="false"
Jul 12 22:10:26.870472 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864681    2560 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 12 22:10:26.870472 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864684    2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 12 22:10:26.870472 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864687    2560 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 12 22:10:26.870472 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864689    2560 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 12 22:10:26.870472 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864693    2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 12 22:10:26.870472 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864696    2560 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 12 22:10:26.870472 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864699    2560 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 12 22:10:26.870472 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864702    2560 flags.go:64] FLAG: --enable-load-reader="false"
Jul 12 22:10:26.870472 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864705    2560 flags.go:64] FLAG: --enable-server="true"
Jul 12 22:10:26.870472 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864707    2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 12 22:10:26.870472 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864714    2560 flags.go:64] FLAG: --event-burst="100"
Jul 12 22:10:26.870472 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864717    2560 flags.go:64] FLAG: --event-qps="50"
Jul 12 22:10:26.870472 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864721    2560 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 12 22:10:26.870472 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864730    2560 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 12 22:10:26.870472 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864733    2560 flags.go:64] FLAG: --eviction-hard=""
Jul 12 22:10:26.870472 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864749    2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 12 22:10:26.870472 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864752    2560 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 12 22:10:26.871125 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864756    2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 12 22:10:26.871125 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864760    2560 flags.go:64] FLAG: --eviction-soft=""
Jul 12 22:10:26.871125 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864763    2560 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 12 22:10:26.871125 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864766    2560 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 12 22:10:26.871125 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864769    2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 12 22:10:26.871125 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864771    2560 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 12 22:10:26.871125 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864774    2560 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 12 22:10:26.871125 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864777    2560 flags.go:64] FLAG: --fail-swap-on="true"
Jul 12 22:10:26.871125 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864780    2560 flags.go:64] FLAG: --feature-gates=""
Jul 12 22:10:26.871125 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864783    2560 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 12 22:10:26.871125 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864786    2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 12 22:10:26.871125 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864789    2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 12 22:10:26.871125 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864792    2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 12 22:10:26.871125 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864795    2560 flags.go:64] FLAG: --healthz-port="10248"
Jul 12 22:10:26.871125 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864798    2560 flags.go:64] FLAG: --help="false"
Jul 12 22:10:26.871125 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864801    2560 flags.go:64] FLAG: --hostname-override="ip-10-0-135-47.ec2.internal"
Jul 12 22:10:26.871125 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864804    2560 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 12 22:10:26.871125 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864807    2560 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 12 22:10:26.871125 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864810    2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 12 22:10:26.871125 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864813    2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 12 22:10:26.871125 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864817    2560 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 12 22:10:26.871125 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864819    2560 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 12 22:10:26.871125 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864822    2560 flags.go:64] FLAG: --image-service-endpoint=""
Jul 12 22:10:26.871700 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864825    2560 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 12 22:10:26.871700 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864828    2560 flags.go:64] FLAG: --kube-api-burst="100"
Jul 12 22:10:26.871700 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864830    2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 12 22:10:26.871700 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864833    2560 flags.go:64] FLAG: --kube-api-qps="50"
Jul 12 22:10:26.871700 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864837    2560 flags.go:64] FLAG: --kube-reserved=""
Jul 12 22:10:26.871700 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864840    2560 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 12 22:10:26.871700 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864848    2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 12 22:10:26.871700 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864852    2560 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 12 22:10:26.871700 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864854    2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 12 22:10:26.871700 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864857    2560 flags.go:64] FLAG: --lock-file=""
Jul 12 22:10:26.871700 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864861    2560 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 12 22:10:26.871700 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864864    2560 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 12 22:10:26.871700 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864867    2560 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 12 22:10:26.871700 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864876    2560 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 12 22:10:26.871700 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864878    2560 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 12 22:10:26.871700 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864881    2560 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 12 22:10:26.871700 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864884    2560 flags.go:64] FLAG: --logging-format="text"
Jul 12 22:10:26.871700 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864887    2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 12 22:10:26.871700 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864890    2560 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 12 22:10:26.871700 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864893    2560 flags.go:64] FLAG: --manifest-url=""
Jul 12 22:10:26.871700 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864896    2560 flags.go:64] FLAG: --manifest-url-header=""
Jul 12 22:10:26.871700 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864900    2560 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 12 22:10:26.871700 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864903    2560 flags.go:64] FLAG: --max-open-files="1000000"
Jul 12 22:10:26.871700 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864907    2560 flags.go:64] FLAG: --max-pods="110"
Jul 12 22:10:26.871700 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864910    2560 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 12 22:10:26.872341 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864913    2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 12 22:10:26.872341 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864916    2560 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 12 22:10:26.872341 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864919    2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 12 22:10:26.872341 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864922    2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 12 22:10:26.872341 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864924    2560 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 12 22:10:26.872341 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864927    2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 12 22:10:26.872341 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864934    2560 flags.go:64] FLAG: --node-status-max-images="50"
Jul 12 22:10:26.872341 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864937    2560 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 12 22:10:26.872341 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864940    2560 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 12 22:10:26.872341 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864943    2560 flags.go:64] FLAG: --pod-cidr=""
Jul 12 22:10:26.872341 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864946    2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 12 22:10:26.872341 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864951    2560 flags.go:64] FLAG: --pod-manifest-path=""
Jul 12 22:10:26.872341 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864956    2560 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 12 22:10:26.872341 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864959    2560 flags.go:64] FLAG: --pods-per-core="0"
Jul 12 22:10:26.872341 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864967    2560 flags.go:64] FLAG: --port="10250"
Jul 12 22:10:26.872341 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864970    2560 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 12 22:10:26.872341 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864973    2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f24b7b254f73e598"
Jul 12 22:10:26.872341 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864977    2560 flags.go:64] FLAG: --qos-reserved=""
Jul 12 22:10:26.872341 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864980    2560 flags.go:64] FLAG: --read-only-port="10255"
Jul 12 22:10:26.872341 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864983    2560 flags.go:64] FLAG: --register-node="true"
Jul 12 22:10:26.872341 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864986    2560 flags.go:64] FLAG: --register-with-taints=""
Jul 12 22:10:26.872341 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864989    2560 flags.go:64] FLAG: --registry-burst="10"
Jul 12 22:10:26.872341 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864992    2560 flags.go:64] FLAG: --registry-qps="5"
Jul 12 22:10:26.872341 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864995    2560 flags.go:64] FLAG: --reserved-cpus=""
Jul 12 22:10:26.873353 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.864998    2560 flags.go:64] FLAG: --reserved-memory=""
Jul 12 22:10:26.873353 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.865002    2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 12 22:10:26.873353 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.865004    2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 12 22:10:26.873353 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.865007    2560 flags.go:64] FLAG: --rotate-certificates="false"
Jul 12 22:10:26.873353 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.865010    2560 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 12 22:10:26.873353 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.865013    2560 flags.go:64] FLAG: --runonce="false"
Jul 12 22:10:26.873353 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.865016    2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 12 22:10:26.873353 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.865019    2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 12 22:10:26.873353 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.865022    2560 flags.go:64] FLAG: --seccomp-default="false"
Jul 12 22:10:26.873353 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.865024    2560 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 12 22:10:26.873353 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.865027    2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 12 22:10:26.873353 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.865030    2560 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 12 22:10:26.873353 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.865033    2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 12 22:10:26.873353 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.865036    2560 flags.go:64] FLAG: --storage-driver-password="root"
Jul 12 22:10:26.873353 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.865039    2560 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 12 22:10:26.873353 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.865042    2560 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 12 22:10:26.873353 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.865044    2560 flags.go:64] FLAG: --storage-driver-user="root"
Jul 12 22:10:26.873353 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.865047    2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 12 22:10:26.873353 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.865050    2560 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 12 22:10:26.873353 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.865053    2560 flags.go:64] FLAG: --system-cgroups=""
Jul 12 22:10:26.873353 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.865056    2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 12 22:10:26.873353 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.865061    2560 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 12 22:10:26.873353 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.865064    2560 flags.go:64] FLAG: --tls-cert-file=""
Jul 12 22:10:26.873353 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.865072    2560 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 12 22:10:26.873353 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.865078    2560 flags.go:64] FLAG: --tls-min-version=""
Jul 12 22:10:26.874409 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.865082    2560 flags.go:64] FLAG: --tls-private-key-file=""
Jul 12 22:10:26.874409 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.865084    2560 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 12 22:10:26.874409 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.865087    2560 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 12 22:10:26.874409 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.865090    2560 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 12 22:10:26.874409 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.865093    2560 flags.go:64] FLAG: --v="2"
Jul 12 22:10:26.874409 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.865096    2560 flags.go:64] FLAG: --version="false"
Jul 12 22:10:26.874409 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.865100    2560 flags.go:64] FLAG: --vmodule=""
Jul 12 22:10:26.874409 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.865103    2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 12 22:10:26.874409 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.865106    2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 12 22:10:26.874409 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865291    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 12 22:10:26.874409 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865295    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 12 22:10:26.874409 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865298    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 12 22:10:26.874409 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865301    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 12 22:10:26.874409 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865304    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 12 22:10:26.874409 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865307    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 12 22:10:26.874409 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865310    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 12 22:10:26.874409 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865318    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 12 22:10:26.874409 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865321    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 12 22:10:26.874409 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865323    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 12 22:10:26.874409 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865326    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 12 22:10:26.874409 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865328    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 12 22:10:26.874409 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865330    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 12 22:10:26.875368 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865333    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 12 22:10:26.875368 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865335    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 12 22:10:26.875368 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865338    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 12 22:10:26.875368 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865340    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 12 22:10:26.875368 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865342    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 12 22:10:26.875368 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865345    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 12 22:10:26.875368 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865348    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 12 22:10:26.875368 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865350    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 12 22:10:26.875368 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865353    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 12 22:10:26.875368 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865361    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 12 22:10:26.875368 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865365    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 12 22:10:26.875368 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865367    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 12 22:10:26.875368 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865370    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 12 22:10:26.875368 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865372    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 12 22:10:26.875368 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865374    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 12 22:10:26.875368 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865377    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 12 22:10:26.875368 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865379    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 12 22:10:26.875368 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865382    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 12 22:10:26.875368 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865384    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 12 22:10:26.875368 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865387    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 12 22:10:26.876296 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865389    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 12 22:10:26.876296 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865391    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 12 22:10:26.876296 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865394    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 12 22:10:26.876296 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865396    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 12 22:10:26.876296 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865399    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 12 22:10:26.876296 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865401    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 12 22:10:26.876296 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865403    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 12 22:10:26.876296 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865406    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 12 22:10:26.876296 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865408    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 12 22:10:26.876296 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865411    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 12 22:10:26.876296 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865413    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 12 22:10:26.876296 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865415    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 12 22:10:26.876296 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865418    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 12 22:10:26.876296 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865420    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 12 22:10:26.876296 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865422    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 12 22:10:26.876296 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865425    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 12 22:10:26.876296 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865427    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 12 22:10:26.876296 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865430    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 12 22:10:26.876296 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865433    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 12 22:10:26.876789 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865435    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 12 22:10:26.876789 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865438    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 12 22:10:26.876789 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865440    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 12 22:10:26.876789 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865450    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 12 22:10:26.876789 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865455    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 12 22:10:26.876789 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865458    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 12 22:10:26.876789 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865460    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 12 22:10:26.876789 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865463    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 12 22:10:26.876789 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865465    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 12 22:10:26.876789 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865468    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 12 22:10:26.876789 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865470    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 12 22:10:26.876789 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865473    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 12 22:10:26.876789 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865475    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 12 22:10:26.876789 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865477    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 12 22:10:26.876789 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865480    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 12 22:10:26.876789 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865482    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 12 22:10:26.876789 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865486    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 12 22:10:26.876789 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865489    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 12 22:10:26.876789 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865492    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 12 22:10:26.877520 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865494    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 12 22:10:26.877520 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865497    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 12 22:10:26.877520 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865501    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 12 22:10:26.877520 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865504    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 12 22:10:26.877520 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865507    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 12 22:10:26.877520 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865510    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 12 22:10:26.877520 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865513    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 12 22:10:26.877520 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865516    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 12 22:10:26.877520 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865518    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 12 22:10:26.877520 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865521    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 12 22:10:26.877520 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865523    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 12 22:10:26.877520 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865526    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 12 22:10:26.877520 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865528    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 12 22:10:26.877520 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865531    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 12 22:10:26.877520 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865533    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 12 22:10:26.877520 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865535    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 12 22:10:26.877520 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865539    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 12 22:10:26.877520 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865547    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 12 22:10:26.877520 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865549    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 12 22:10:26.877520 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865552    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 12 22:10:26.878342 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865554    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 12 22:10:26.878342 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865557    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 12 22:10:26.878342 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865559    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 12 22:10:26.878342 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865562    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 12 22:10:26.878342 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865564    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 12 22:10:26.878342 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865567    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 12 22:10:26.878342 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865569    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 12 22:10:26.878342 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.865572    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 12 22:10:26.878342 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.866402    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 12 22:10:26.878342 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.874624    2560 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 12 22:10:26.878342 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.874644    2560 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 12 22:10:26.878342 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.874672    2560 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 12 22:10:26.878342 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.874681    2560 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 12 22:10:26.878342 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874788    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 12 22:10:26.878342 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874799    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 12 22:10:26.878782 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874804    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 12 22:10:26.878782 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874809    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 12 22:10:26.878782 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874814    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 12 22:10:26.878782 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874819    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 12 22:10:26.878782 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874823    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 12 22:10:26.878782 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874827    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 12 22:10:26.878782 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874831    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 12 22:10:26.878782 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874835    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 12 22:10:26.878782 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874840    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 12 22:10:26.878782 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874844    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 12 22:10:26.878782 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874849    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 12 22:10:26.878782 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874853    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 12 22:10:26.878782 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874857    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 12 22:10:26.878782 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874861    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 12 22:10:26.878782 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874866    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 12 22:10:26.878782 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874870    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 12 22:10:26.878782 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874874    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 12 22:10:26.878782 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874878    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 12 22:10:26.878782 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874884    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 12 22:10:26.878782 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874889    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 12 22:10:26.879387 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874893    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 12 22:10:26.879387 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874897    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 12 22:10:26.879387 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874902    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 12 22:10:26.879387 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874906    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 12 22:10:26.879387 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874910    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 12 22:10:26.879387 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874914    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 12 22:10:26.879387 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874918    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 12 22:10:26.879387 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874925    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 12 22:10:26.879387 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874930    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 12 22:10:26.879387 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874934    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 12 22:10:26.879387 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874938    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 12 22:10:26.879387 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874942    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 12 22:10:26.879387 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874946    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 12 22:10:26.879387 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874950    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 12 22:10:26.879387 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874957    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 12 22:10:26.879387 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874962    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 12 22:10:26.879387 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874967    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 12 22:10:26.879387 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874972    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 12 22:10:26.879387 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874977    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 12 22:10:26.879387 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874981    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 12 22:10:26.879919 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874985    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 12 22:10:26.879919 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874989    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 12 22:10:26.879919 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874994    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 12 22:10:26.879919 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.874998    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 12 22:10:26.879919 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875003    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 12 22:10:26.879919 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875007    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 12 22:10:26.879919 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875011    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 12 22:10:26.879919 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875015    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 12 22:10:26.879919 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875019    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 12 22:10:26.879919 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875024    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 12 22:10:26.879919 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875028    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 12 22:10:26.879919 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875032    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 12 22:10:26.879919 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875036    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 12 22:10:26.879919 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875040    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 12 22:10:26.879919 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875044    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 12 22:10:26.879919 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875049    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 12 22:10:26.879919 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875061    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 12 22:10:26.879919 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875067    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 12 22:10:26.879919 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875074    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 12 22:10:26.880402 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875079    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 12 22:10:26.880402 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875084    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 12 22:10:26.880402 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875088    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 12 22:10:26.880402 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875093    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 12 22:10:26.880402 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875097    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 12 22:10:26.880402 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875101    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 12 22:10:26.880402 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875105    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 12 22:10:26.880402 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875110    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 12 22:10:26.880402 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875114    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 12 22:10:26.880402 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875118    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 12 22:10:26.880402 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875122    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 12 22:10:26.880402 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875126    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 12 22:10:26.880402 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875130    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 12 22:10:26.880402 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875134    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 12 22:10:26.880402 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875138    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 12 22:10:26.880402 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875143    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 12 22:10:26.880402 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875147    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 12 22:10:26.880402 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875151    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 12 22:10:26.880402 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875155    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 12 22:10:26.880402 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875159    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 12 22:10:26.880940 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875163    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 12 22:10:26.880940 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875167    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 12 22:10:26.880940 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875171    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 12 22:10:26.880940 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875175    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 12 22:10:26.880940 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875180    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 12 22:10:26.880940 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875184    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 12 22:10:26.880940 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875188    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 12 22:10:26.880940 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875192    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 12 22:10:26.880940 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875196    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 12 22:10:26.880940 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875200    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 12 22:10:26.880940 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875205    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 12 22:10:26.880940 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875210    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 12 22:10:26.880940 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875215    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 12 22:10:26.880940 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875219    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 12 22:10:26.880940 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875224    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 12 22:10:26.880940 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875228    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 12 22:10:26.880940 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875233    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 12 22:10:26.880940 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875237    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 12 22:10:26.881437 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.875245    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 12 22:10:26.881437 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875423    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 12 22:10:26.881437 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875432    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 12 22:10:26.881437 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875437    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 12 22:10:26.881437 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875442    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 12 22:10:26.881437 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875447    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 12 22:10:26.881437 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875451    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 12 22:10:26.881437 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875456    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 12 22:10:26.881437 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875460    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 12 22:10:26.881437 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875464    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 12 22:10:26.881437 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875468    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 12 22:10:26.881437 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875472    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 12 22:10:26.881437 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875476    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 12 22:10:26.881437 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875481    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 12 22:10:26.881437 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875485    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 12 22:10:26.881921 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875489    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 12 22:10:26.881921 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875493    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 12 22:10:26.881921 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875497    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 12 22:10:26.881921 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875501    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 12 22:10:26.881921 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875505    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 12 22:10:26.881921 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875509    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 12 22:10:26.881921 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875513    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 12 22:10:26.881921 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875517    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 12 22:10:26.881921 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875521    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 12 22:10:26.881921 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875526    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 12 22:10:26.881921 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875531    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 12 22:10:26.881921 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875534    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 12 22:10:26.881921 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875539    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 12 22:10:26.881921 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875544    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 12 22:10:26.881921 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875549    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 12 22:10:26.881921 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875553    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 12 22:10:26.881921 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875557    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 12 22:10:26.881921 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875561    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 12 22:10:26.881921 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875565    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 12 22:10:26.881921 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875569    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 12 22:10:26.882691 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875573    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 12 22:10:26.882691 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875577    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 12 22:10:26.882691 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875581    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 12 22:10:26.882691 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875585    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 12 22:10:26.882691 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875589    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 12 22:10:26.882691 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875593    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 12 22:10:26.882691 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875597    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 12 22:10:26.882691 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875601    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 12 22:10:26.882691 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875605    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 12 22:10:26.882691 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875609    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 12 22:10:26.882691 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875613    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 12 22:10:26.882691 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875617    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 12 22:10:26.882691 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875621    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 12 22:10:26.882691 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875626    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 12 22:10:26.882691 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875630    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 12 22:10:26.882691 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875634    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 12 22:10:26.882691 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875638    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 12 22:10:26.882691 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875642    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 12 22:10:26.882691 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875646    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 12 22:10:26.882691 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875650    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 12 22:10:26.883464 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875654    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 12 22:10:26.883464 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875659    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 12 22:10:26.883464 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875666    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 12 22:10:26.883464 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875673    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 12 22:10:26.883464 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875679    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 12 22:10:26.883464 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875684    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 12 22:10:26.883464 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875689    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 12 22:10:26.883464 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875694    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 12 22:10:26.883464 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875698    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 12 22:10:26.883464 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875703    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 12 22:10:26.883464 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875707    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 12 22:10:26.883464 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875712    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 12 22:10:26.883464 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875716    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 12 22:10:26.883464 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875719    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 12 22:10:26.883464 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875724    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 12 22:10:26.883464 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875728    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 12 22:10:26.883464 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875733    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 12 22:10:26.883464 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875756    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 12 22:10:26.883464 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875761    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 12 22:10:26.883958 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875765    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 12 22:10:26.883958 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875768    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 12 22:10:26.883958 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875772    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 12 22:10:26.883958 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875776    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 12 22:10:26.883958 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875779    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 12 22:10:26.883958 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875783    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 12 22:10:26.883958 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875787    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 12 22:10:26.883958 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875792    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 12 22:10:26.883958 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875795    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 12 22:10:26.883958 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875799    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 12 22:10:26.883958 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875803    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 12 22:10:26.883958 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875807    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 12 22:10:26.883958 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875811    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 12 22:10:26.883958 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875815    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 12 22:10:26.883958 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875818    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 12 22:10:26.883958 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875822    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 12 22:10:26.883958 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875826    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 12 22:10:26.883958 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875830    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 12 22:10:26.883958 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875834    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 12 22:10:26.883958 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875838    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 12 22:10:26.884477 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875842    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 12 22:10:26.884477 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875847    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 12 22:10:26.884477 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875851    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 12 22:10:26.884477 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875855    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 12 22:10:26.884477 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875859    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 12 22:10:26.884477 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:26.875863    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 12 22:10:26.884477 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.875871    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 12 22:10:26.884477 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.876042    2560 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 12 22:10:26.884477 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.879212    2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 12 22:10:26.884477 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.881092    2560 server.go:1019] "Starting client certificate rotation"
Jul 12 22:10:26.884477 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.881196    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 12 22:10:26.884477 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.881239    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 12 22:10:26.885822 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.885805    2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 12 22:10:26.886931 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.886915    2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 12 22:10:26.901973 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.901950    2560 log.go:25] "Validated CRI v1 runtime API"
Jul 12 22:10:26.904505 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.904491    2560 log.go:25] "Validated CRI v1 image API"
Jul 12 22:10:26.905373 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.905354    2560 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 12 22:10:26.909031 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.909009    2560 fs.go:135] Filesystem UUIDs: map[33b60b4d-c3b2-4b98-a1e9-a99747838b7f:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ce1fd8b9-a7be-4eb4-ac76-403f36eabdb2:/dev/nvme0n1p3]
Jul 12 22:10:26.909090 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.909031    2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 12 22:10:26.911914 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.911894    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 12 22:10:26.915175 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.915063    2560 manager.go:216] Machine: {Timestamp:2026-07-12 22:10:26.913621269 +0000 UTC m=+0.391915447 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099576 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22c492a8caeb94e23ba7bf92951537 SystemUUID:ec22c492-a8ca-eb94-e23b-a7bf92951537 BootID:ed4433e7-bf91-4e6e-8486-64e5a63c05f4 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:bd:26:20:19:cf Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:bd:26:20:19:cf Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d6:95:8f:a2:47:69 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 12 22:10:26.915175 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.915165    2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 12 22:10:26.915293 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.915248    2560 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 12 22:10:26.917847 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.917820    2560 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 12 22:10:26.917981 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.917850    2560 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-47.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 12 22:10:26.918729 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.918719    2560 topology_manager.go:138] "Creating topology manager with none policy"
Jul 12 22:10:26.918779 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.918732    2560 container_manager_linux.go:309] "Creating device plugin manager"
Jul 12 22:10:26.918779 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.918757    2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 12 22:10:26.918779 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.918776    2560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 12 22:10:26.920672 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.920662    2560 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 12 22:10:26.925951 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.925938    2560 state_mem.go:36] "Initialized new in-memory state store"
Jul 12 22:10:26.926075 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.926066    2560 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 12 22:10:26.926113 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.926097    2560 kubelet.go:486] "Attempting to sync node with API server"
Jul 12 22:10:26.926113 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.926112    2560 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 12 22:10:26.926170 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.926127    2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 12 22:10:26.926170 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.926136    2560 kubelet.go:398] "Adding apiserver pod source"
Jul 12 22:10:26.926170 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.926145    2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 12 22:10:26.926512 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.926496    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 12 22:10:26.926579 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.926517    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 12 22:10:26.928814 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.928796    2560 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 12 22:10:26.932010 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.931062    2560 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 12 22:10:26.932139 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.932120    2560 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 12 22:10:26.932402 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.932387    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 12 22:10:26.932468 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.932410    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 12 22:10:26.932468 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.932420    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 12 22:10:26.932468 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.932437    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 12 22:10:26.932468 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.932449    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 12 22:10:26.932468 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.932458    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 12 22:10:26.933494 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.932982    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 12 22:10:26.933494 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.932997    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 12 22:10:26.933494 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.933010    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 12 22:10:26.933494 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.933018    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 12 22:10:26.933494 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.933053    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 12 22:10:26.933494 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.933071    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 12 22:10:26.933494 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.933106    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 12 22:10:26.933494 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.933114    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 12 22:10:26.936866 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.936850    2560 server.go:1268] "Started kubelet"
Jul 12 22:10:26.937009 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.936960    2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 12 22:10:26.937095 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.936991    2560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 12 22:10:26.937095 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.937060    2560 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 12 22:10:26.937701 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.937680    2560 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-47.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 12 22:10:26.937784 ip-10-0-135-47 systemd[1]: Started Kubernetes Kubelet.
Jul 12 22:10:26.937978 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:26.937956    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-47.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 12 22:10:26.938116 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:26.938033    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 12 22:10:26.938269 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.938249    2560 server.go:310] "Adding debug handlers to kubelet server"
Jul 12 22:10:26.938759 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.938589    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-s6jmd"
Jul 12 22:10:26.941130 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.941111    2560 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 12 22:10:26.942098 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:26.942077    2560 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 12 22:10:26.943729 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:26.942593    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-47.ec2.internal.18c1aa5911706580  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-47.ec2.internal,UID:ip-10-0-135-47.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-47.ec2.internal,},FirstTimestamp:2026-07-12 22:10:26.936817024 +0000 UTC m=+0.415111202,LastTimestamp:2026-07-12 22:10:26.936817024 +0000 UTC m=+0.415111202,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-47.ec2.internal,}"
Jul 12 22:10:26.948159 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.948138    2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 12 22:10:26.948159 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.948153    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 12 22:10:26.948307 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.948206    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-s6jmd"
Jul 12 22:10:26.948388 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:26.948367    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-47.ec2.internal\" not found"
Jul 12 22:10:26.948956 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.948925    2560 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 12 22:10:26.949027 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.948957    2560 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 12 22:10:26.949027 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.948982    2560 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 12 22:10:26.949120 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.949056    2560 reconstruct.go:97] "Volume reconstruction finished"
Jul 12 22:10:26.949120 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.949066    2560 reconciler.go:29] "Reconciler: start to sync state"
Jul 12 22:10:26.949210 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.949167    2560 factory.go:55] Registering systemd factory
Jul 12 22:10:26.949210 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.949204    2560 factory.go:223] Registration of the systemd container factory successfully
Jul 12 22:10:26.950778 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.950735    2560 factory.go:153] Registering CRI-O factory
Jul 12 22:10:26.950778 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.950778    2560 factory.go:223] Registration of the crio container factory successfully
Jul 12 22:10:26.950914 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.950847    2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 12 22:10:26.950914 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.950872    2560 factory.go:103] Registering Raw factory
Jul 12 22:10:26.950914 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:26.950877    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 12 22:10:26.950914 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.950887    2560 manager.go:1196] Started watching for new ooms in manager
Jul 12 22:10:26.951099 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:26.950922    2560 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-47.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 12 22:10:26.951371 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.951357    2560 manager.go:349] Starting recovery of all containers
Jul 12 22:10:26.958551 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.958439    2560 manager.go:354] Recovery completed
Jul 12 22:10:26.964820 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.964804    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 12 22:10:26.966769 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.966733    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-47.ec2.internal" event="NodeHasSufficientMemory"
Jul 12 22:10:26.966859 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.966785    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-47.ec2.internal" event="NodeHasNoDiskPressure"
Jul 12 22:10:26.966859 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.966798    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-47.ec2.internal" event="NodeHasSufficientPID"
Jul 12 22:10:26.967415 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.967386    2560 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 12 22:10:26.967415 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.967403    2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 12 22:10:26.967415 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.967412    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 12 22:10:26.967577 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.967432    2560 state_mem.go:36] "Initialized new in-memory state store"
Jul 12 22:10:26.968123 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.968108    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-47.ec2.internal" event="NodeHasSufficientMemory"
Jul 12 22:10:26.968202 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.968134    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-47.ec2.internal" event="NodeHasNoDiskPressure"
Jul 12 22:10:26.968202 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.968144    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-47.ec2.internal" event="NodeHasSufficientPID"
Jul 12 22:10:26.969574 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.969558    2560 policy_none.go:49] "None policy: Start"
Jul 12 22:10:26.969638 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.969580    2560 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 12 22:10:26.969638 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.969594    2560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 12 22:10:26.971617 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:26.971606    2560 policy_none.go:47] "Start"
Jul 12 22:10:27.013367 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.013345    2560 manager.go:337] "Starting Device Plugin manager"
Jul 12 22:10:27.013509 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:27.013378    2560 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 12 22:10:27.013509 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.013388    2560 server.go:87] "Starting device plugin registration server"
Jul 12 22:10:27.013720 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.013702    2560 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 12 22:10:27.014021 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.013721    2560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 12 22:10:27.014021 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.013839    2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 12 22:10:27.014021 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.013932    2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 12 22:10:27.014021 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.013943    2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 12 22:10:27.014469 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:27.014448    2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 12 22:10:27.014562 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:27.014496    2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-47.ec2.internal\" not found"
Jul 12 22:10:27.072907 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.072864    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 12 22:10:27.075186 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.074306    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 12 22:10:27.075186 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.074344    2560 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 12 22:10:27.075186 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.074366    2560 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 12 22:10:27.075186 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:27.074400    2560 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 12 22:10:27.076516 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.076491    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 12 22:10:27.114659 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.114565    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 12 22:10:27.116580 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.116563    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-47.ec2.internal" event="NodeHasSufficientMemory"
Jul 12 22:10:27.116653 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.116595    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-47.ec2.internal" event="NodeHasNoDiskPressure"
Jul 12 22:10:27.116653 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.116605    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-47.ec2.internal" event="NodeHasSufficientPID"
Jul 12 22:10:27.116653 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.116627    2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-47.ec2.internal"
Jul 12 22:10:27.127379 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.127359    2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-47.ec2.internal"
Jul 12 22:10:27.127476 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:27.127385    2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-47.ec2.internal\": node \"ip-10-0-135-47.ec2.internal\" not found"
Jul 12 22:10:27.150313 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:27.150281    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-47.ec2.internal\" not found"
Jul 12 22:10:27.175208 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.175174    2560 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-47.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-47.ec2.internal"]
Jul 12 22:10:27.175315 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.175264    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 12 22:10:27.176886 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.176870    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-47.ec2.internal" event="NodeHasSufficientMemory"
Jul 12 22:10:27.176971 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.176902    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-47.ec2.internal" event="NodeHasNoDiskPressure"
Jul 12 22:10:27.176971 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.176912    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-47.ec2.internal" event="NodeHasSufficientPID"
Jul 12 22:10:27.179352 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.179339    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 12 22:10:27.179457 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.179441    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-47.ec2.internal"
Jul 12 22:10:27.179506 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.179477    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 12 22:10:27.180209 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.180195    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-47.ec2.internal" event="NodeHasSufficientMemory"
Jul 12 22:10:27.180281 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.180221    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-47.ec2.internal" event="NodeHasNoDiskPressure"
Jul 12 22:10:27.180281 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.180233    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-47.ec2.internal" event="NodeHasSufficientPID"
Jul 12 22:10:27.180281 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.180237    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-47.ec2.internal" event="NodeHasSufficientMemory"
Jul 12 22:10:27.180281 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.180255    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-47.ec2.internal" event="NodeHasNoDiskPressure"
Jul 12 22:10:27.180281 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.180265    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-47.ec2.internal" event="NodeHasSufficientPID"
Jul 12 22:10:27.182587 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.182571    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-47.ec2.internal"
Jul 12 22:10:27.182672 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.182602    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 12 22:10:27.183304 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.183288    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-47.ec2.internal" event="NodeHasSufficientMemory"
Jul 12 22:10:27.183392 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.183315    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-47.ec2.internal" event="NodeHasNoDiskPressure"
Jul 12 22:10:27.183392 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.183325    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-47.ec2.internal" event="NodeHasSufficientPID"
Jul 12 22:10:27.204571 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:27.204540    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-47.ec2.internal\" not found" node="ip-10-0-135-47.ec2.internal"
Jul 12 22:10:27.209034 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:27.209015    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-47.ec2.internal\" not found" node="ip-10-0-135-47.ec2.internal"
Jul 12 22:10:27.250928 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:27.250896    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-47.ec2.internal\" not found"
Jul 12 22:10:27.251082 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.250947    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3c0dae55120c29fa0e8b16a46a0faece-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-47.ec2.internal\" (UID: \"3c0dae55120c29fa0e8b16a46a0faece\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-47.ec2.internal"
Jul 12 22:10:27.251082 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.250971    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c0dae55120c29fa0e8b16a46a0faece-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-47.ec2.internal\" (UID: \"3c0dae55120c29fa0e8b16a46a0faece\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-47.ec2.internal"
Jul 12 22:10:27.251082 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.250991    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2f0d6da932fad75070caf7c5d75748da-config\") pod \"kube-apiserver-proxy-ip-10-0-135-47.ec2.internal\" (UID: \"2f0d6da932fad75070caf7c5d75748da\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-47.ec2.internal"
Jul 12 22:10:27.351902 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:27.351866    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-47.ec2.internal\" not found"
Jul 12 22:10:27.352058 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.351922    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2f0d6da932fad75070caf7c5d75748da-config\") pod \"kube-apiserver-proxy-ip-10-0-135-47.ec2.internal\" (UID: \"2f0d6da932fad75070caf7c5d75748da\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-47.ec2.internal"
Jul 12 22:10:27.352058 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.351946    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3c0dae55120c29fa0e8b16a46a0faece-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-47.ec2.internal\" (UID: \"3c0dae55120c29fa0e8b16a46a0faece\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-47.ec2.internal"
Jul 12 22:10:27.352058 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.351963    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c0dae55120c29fa0e8b16a46a0faece-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-47.ec2.internal\" (UID: \"3c0dae55120c29fa0e8b16a46a0faece\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-47.ec2.internal"
Jul 12 22:10:27.352058 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.352002    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c0dae55120c29fa0e8b16a46a0faece-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-47.ec2.internal\" (UID: \"3c0dae55120c29fa0e8b16a46a0faece\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-47.ec2.internal"
Jul 12 22:10:27.352058 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.352040    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2f0d6da932fad75070caf7c5d75748da-config\") pod \"kube-apiserver-proxy-ip-10-0-135-47.ec2.internal\" (UID: \"2f0d6da932fad75070caf7c5d75748da\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-47.ec2.internal"
Jul 12 22:10:27.352212 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.352040    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3c0dae55120c29fa0e8b16a46a0faece-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-47.ec2.internal\" (UID: \"3c0dae55120c29fa0e8b16a46a0faece\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-47.ec2.internal"
Jul 12 22:10:27.452621 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:27.452544    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-47.ec2.internal\" not found"
Jul 12 22:10:27.506781 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.506731    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-47.ec2.internal"
Jul 12 22:10:27.511963 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.511793    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-47.ec2.internal"
Jul 12 22:10:27.553022 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:27.552995    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-47.ec2.internal\" not found"
Jul 12 22:10:27.653445 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:27.653404    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-47.ec2.internal\" not found"
Jul 12 22:10:27.753954 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:27.753878    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-47.ec2.internal\" not found"
Jul 12 22:10:27.854368 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:27.854335    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-47.ec2.internal\" not found"
Jul 12 22:10:27.880547 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.880522    2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 12 22:10:27.880684 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.880663    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 12 22:10:27.954996 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:27.954970    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-47.ec2.internal\" not found"
Jul 12 22:10:27.954996 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.954971    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-11 22:05:28 +0000 UTC" deadline="2027-12-19 12:08:50.465567923 +0000 UTC"
Jul 12 22:10:27.954996 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:27.954999    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12589h58m22.510572771s"
Jul 12 22:10:28.055270 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:28.055232    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-47.ec2.internal\" not found"
Jul 12 22:10:28.092330 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:28.092306    2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 12 22:10:28.124629 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:28.124611    2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 12 22:10:28.130780 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:28.130724    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f0d6da932fad75070caf7c5d75748da.slice/crio-bd52414b265ba5a666092183c048569da006633ea4e27ae4914c41f723c109e5 WatchSource:0}: Error finding container bd52414b265ba5a666092183c048569da006633ea4e27ae4914c41f723c109e5: Status 404 returned error can't find the container with id bd52414b265ba5a666092183c048569da006633ea4e27ae4914c41f723c109e5
Jul 12 22:10:28.134656 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:28.134641    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 12 22:10:28.148781 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:28.148755    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-47.ec2.internal"
Jul 12 22:10:28.160394 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:28.160370    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 12 22:10:28.161818 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:28.161804    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-47.ec2.internal"
Jul 12 22:10:28.170818 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:28.170797    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 12 22:10:29.471580 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:29.471545    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c0dae55120c29fa0e8b16a46a0faece.slice/crio-1059feb34be938771ea20a08515ce8132b2b4b54c040cbaf9ea40cb0e8709de5 WatchSource:0}: Error finding container 1059feb34be938771ea20a08515ce8132b2b4b54c040cbaf9ea40cb0e8709de5: Status 404 returned error can't find the container with id 1059feb34be938771ea20a08515ce8132b2b4b54c040cbaf9ea40cb0e8709de5
Jul 12 22:10:29.679211 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:29.679186    2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 12 22:10:30.073065 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.073035    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 12 22:10:30.092549 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.092514    2560 apiserver.go:52] "Watching apiserver"
Jul 12 22:10:30.101499 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.101456    2560 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 12 22:10:30.103252 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.103223    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-9fpmt","openshift-image-registry/node-ca-zkk8b","openshift-multus/multus-2t5r9","openshift-multus/multus-additional-cni-plugins-kvvgp","openshift-multus/network-metrics-daemon-st67z","kube-system/kube-apiserver-proxy-ip-10-0-135-47.ec2.internal","openshift-dns/node-resolver-g4qw4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-47.ec2.internal","openshift-network-diagnostics/network-check-target-cn4d2","openshift-network-operator/iptables-alerter-6qrng","openshift-ovn-kubernetes/ovnkube-node-52pbf","kube-system/global-pull-secret-syncer-b7t5r","kube-system/konnectivity-agent-lbvpn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9c7h"]
Jul 12 22:10:30.105844 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.105820    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g4qw4"
Jul 12 22:10:30.108022 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.107999    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lbvpn"
Jul 12 22:10:30.108967 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.108947    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 12 22:10:30.109076 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.108997    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pbwjv\""
Jul 12 22:10:30.109142 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.109101    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 12 22:10:30.110719 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.110651    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 12 22:10:30.110719 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.110660    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.111133 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.111116    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 12 22:10:30.111244 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.111116    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qhtpw\""
Jul 12 22:10:30.113120 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.113104    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kvvgp"
Jul 12 22:10:30.113429 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.113408    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-pk4nv\""
Jul 12 22:10:30.113556 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.113448    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 12 22:10:30.113556 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.113465    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 12 22:10:30.113556 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.113411    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 12 22:10:30.113763 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.113746    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 12 22:10:30.113991 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.113976    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 12 22:10:30.115935 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.115918    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-st67z"
Jul 12 22:10:30.116046 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:30.116009    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-st67z" podUID="551bf86c-bd1a-40be-b977-41205da830c2"
Jul 12 22:10:30.116293 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.116277    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8k662\""
Jul 12 22:10:30.116375 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.116305    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 12 22:10:30.116375 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.116277    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 12 22:10:30.117930 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.117911    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9c7h"
Jul 12 22:10:30.120358 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.120337    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 12 22:10:30.120554 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.120337    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-t65pq\""
Jul 12 22:10:30.120677 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.120656    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 12 22:10:30.120778 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.120755    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 12 22:10:30.120871 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.120842    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cn4d2"
Jul 12 22:10:30.120956 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:30.120917    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cn4d2" podUID="bcf4ee57-809d-4e8b-be7f-593b88689c16"
Jul 12 22:10:30.125196 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.124397    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/00bab1f2-a13f-47ff-b0c7-c04484a4c85e-konnectivity-ca\") pod \"konnectivity-agent-lbvpn\" (UID: \"00bab1f2-a13f-47ff-b0c7-c04484a4c85e\") " pod="kube-system/konnectivity-agent-lbvpn"
Jul 12 22:10:30.125196 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.124431    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-host-run-netns\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.125196 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.124471    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-multus-daemon-config\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.125196 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.124485    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9fpmt"
Jul 12 22:10:30.125196 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.124497    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/912f085a-8942-4512-98f9-c8ba189509c2-cni-binary-copy\") pod \"multus-additional-cni-plugins-kvvgp\" (UID: \"912f085a-8942-4512-98f9-c8ba189509c2\") " pod="openshift-multus/multus-additional-cni-plugins-kvvgp"
Jul 12 22:10:30.125196 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.124543    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n454l\" (UniqueName: \"kubernetes.io/projected/551bf86c-bd1a-40be-b977-41205da830c2-kube-api-access-n454l\") pod \"network-metrics-daemon-st67z\" (UID: \"551bf86c-bd1a-40be-b977-41205da830c2\") " pod="openshift-multus/network-metrics-daemon-st67z"
Jul 12 22:10:30.125196 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.124581    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-os-release\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.125196 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.124604    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-host-run-k8s-cni-cncf-io\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.125196 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.124627    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9jkf\" (UniqueName: \"kubernetes.io/projected/912f085a-8942-4512-98f9-c8ba189509c2-kube-api-access-k9jkf\") pod \"multus-additional-cni-plugins-kvvgp\" (UID: \"912f085a-8942-4512-98f9-c8ba189509c2\") " pod="openshift-multus/multus-additional-cni-plugins-kvvgp"
Jul 12 22:10:30.125196 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.124652    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-host-run-multus-certs\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.125196 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.124674    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/05ebf682-f7b0-4267-8a22-46d0c41ecba9-hosts-file\") pod \"node-resolver-g4qw4\" (UID: \"05ebf682-f7b0-4267-8a22-46d0c41ecba9\") " pod="openshift-dns/node-resolver-g4qw4"
Jul 12 22:10:30.125196 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.124696    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-cnibin\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.125196 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.124718    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-multus-socket-dir-parent\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.125196 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.124741    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-multus-conf-dir\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.125196 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.124768    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/912f085a-8942-4512-98f9-c8ba189509c2-cnibin\") pod \"multus-additional-cni-plugins-kvvgp\" (UID: \"912f085a-8942-4512-98f9-c8ba189509c2\") " pod="openshift-multus/multus-additional-cni-plugins-kvvgp"
Jul 12 22:10:30.125196 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.124821    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/912f085a-8942-4512-98f9-c8ba189509c2-os-release\") pod \"multus-additional-cni-plugins-kvvgp\" (UID: \"912f085a-8942-4512-98f9-c8ba189509c2\") " pod="openshift-multus/multus-additional-cni-plugins-kvvgp"
Jul 12 22:10:30.125196 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.124837    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/00bab1f2-a13f-47ff-b0c7-c04484a4c85e-agent-certs\") pod \"konnectivity-agent-lbvpn\" (UID: \"00bab1f2-a13f-47ff-b0c7-c04484a4c85e\") " pod="kube-system/konnectivity-agent-lbvpn"
Jul 12 22:10:30.125896 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.124851    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-cni-binary-copy\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.125896 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.124877    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-host-var-lib-cni-multus\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.125896 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.124896    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/912f085a-8942-4512-98f9-c8ba189509c2-system-cni-dir\") pod \"multus-additional-cni-plugins-kvvgp\" (UID: \"912f085a-8942-4512-98f9-c8ba189509c2\") " pod="openshift-multus/multus-additional-cni-plugins-kvvgp"
Jul 12 22:10:30.125896 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.124911    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/912f085a-8942-4512-98f9-c8ba189509c2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kvvgp\" (UID: \"912f085a-8942-4512-98f9-c8ba189509c2\") " pod="openshift-multus/multus-additional-cni-plugins-kvvgp"
Jul 12 22:10:30.125896 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.124928    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-host-var-lib-kubelet\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.125896 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.124942    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-etc-kubernetes\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.125896 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.124958    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/912f085a-8942-4512-98f9-c8ba189509c2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kvvgp\" (UID: \"912f085a-8942-4512-98f9-c8ba189509c2\") " pod="openshift-multus/multus-additional-cni-plugins-kvvgp"
Jul 12 22:10:30.125896 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.124974    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05ebf682-f7b0-4267-8a22-46d0c41ecba9-tmp-dir\") pod \"node-resolver-g4qw4\" (UID: \"05ebf682-f7b0-4267-8a22-46d0c41ecba9\") " pod="openshift-dns/node-resolver-g4qw4"
Jul 12 22:10:30.125896 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.124988    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8tjb\" (UniqueName: \"kubernetes.io/projected/05ebf682-f7b0-4267-8a22-46d0c41ecba9-kube-api-access-w8tjb\") pod \"node-resolver-g4qw4\" (UID: \"05ebf682-f7b0-4267-8a22-46d0c41ecba9\") " pod="openshift-dns/node-resolver-g4qw4"
Jul 12 22:10:30.125896 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.125002    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-system-cni-dir\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.125896 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.125046    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-hostroot\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.125896 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.125131    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-multus-cni-dir\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.125896 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.125150    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-host-var-lib-cni-bin\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.125896 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.125177    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxh26\" (UniqueName: \"kubernetes.io/projected/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-kube-api-access-dxh26\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.125896 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.125195    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/912f085a-8942-4512-98f9-c8ba189509c2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kvvgp\" (UID: \"912f085a-8942-4512-98f9-c8ba189509c2\") " pod="openshift-multus/multus-additional-cni-plugins-kvvgp"
Jul 12 22:10:30.125896 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.125219    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/551bf86c-bd1a-40be-b977-41205da830c2-metrics-certs\") pod \"network-metrics-daemon-st67z\" (UID: \"551bf86c-bd1a-40be-b977-41205da830c2\") " pod="openshift-multus/network-metrics-daemon-st67z"
Jul 12 22:10:30.126906 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.126888    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zkk8b"
Jul 12 22:10:30.127415 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.127395    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-69srw\""
Jul 12 22:10:30.127518 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.127402    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 12 22:10:30.127761 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.127740    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 12 22:10:30.128204 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.128187    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 12 22:10:30.130026 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.130005    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 12 22:10:30.130260 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.130246    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zcx4x\""
Jul 12 22:10:30.130336 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.130329    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 12 22:10:30.130392 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.130246    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 12 22:10:30.130596 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.130582    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6qrng"
Jul 12 22:10:30.133774 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.132724    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.134134 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.134117    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pm2bv\""
Jul 12 22:10:30.134224 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.134196    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 12 22:10:30.134320 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.134303    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 12 22:10:30.134771 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.134745    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 12 22:10:30.135182 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.135163    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7t5r"
Jul 12 22:10:30.135638 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:30.135443    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7t5r" podUID="700bcb2a-4c93-4abe-b403-9b404b3731e3"
Jul 12 22:10:30.136432 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.136223    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 12 22:10:30.136544 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.136498    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 12 22:10:30.136845 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.136822    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 12 22:10:30.137038 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.137024    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 12 22:10:30.137714 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.137691    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 12 22:10:30.138130 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.138058    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8sjhs\""
Jul 12 22:10:30.138130 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.138072    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 12 22:10:30.150190 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.150163    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-b2m79"
Jul 12 22:10:30.158152 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.158127    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-b2m79"
Jul 12 22:10:30.215549 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.215518    2560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 12 22:10:30.225991 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.225946    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/34560420-a512-498c-b8f8-159906bb8638-etc-tuned\") pod \"tuned-9fpmt\" (UID: \"34560420-a512-498c-b8f8-159906bb8638\") " pod="openshift-cluster-node-tuning-operator/tuned-9fpmt"
Jul 12 22:10:30.226153 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.226004    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-host-run-netns\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.226153 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.226028    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-multus-daemon-config\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.226153 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.226050    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f1ab5f7c-025e-43c0-893d-039e9d41b889-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t9c7h\" (UID: \"f1ab5f7c-025e-43c0-893d-039e9d41b889\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9c7h"
Jul 12 22:10:30.226153 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.226069    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34560420-a512-498c-b8f8-159906bb8638-sys\") pod \"tuned-9fpmt\" (UID: \"34560420-a512-498c-b8f8-159906bb8638\") " pod="openshift-cluster-node-tuning-operator/tuned-9fpmt"
Jul 12 22:10:30.226153 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.226090    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c946a409-971a-4760-b311-5278608e31f9-serviceca\") pod \"node-ca-zkk8b\" (UID: \"c946a409-971a-4760-b311-5278608e31f9\") " pod="openshift-image-registry/node-ca-zkk8b"
Jul 12 22:10:30.226153 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.226088    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-host-run-netns\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.226153 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.226112    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-ovnkube-config\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.226475 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.226162    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-ovn-node-metrics-cert\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.226475 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.226278    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-os-release\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.226475 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.226325    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-host-run-k8s-cni-cncf-io\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.226475 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.226352    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9jkf\" (UniqueName: \"kubernetes.io/projected/912f085a-8942-4512-98f9-c8ba189509c2-kube-api-access-k9jkf\") pod \"multus-additional-cni-plugins-kvvgp\" (UID: \"912f085a-8942-4512-98f9-c8ba189509c2\") " pod="openshift-multus/multus-additional-cni-plugins-kvvgp"
Jul 12 22:10:30.226475 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.226377    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f1ab5f7c-025e-43c0-893d-039e9d41b889-etc-selinux\") pod \"aws-ebs-csi-driver-node-t9c7h\" (UID: \"f1ab5f7c-025e-43c0-893d-039e9d41b889\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9c7h"
Jul 12 22:10:30.226475 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.226425    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/34560420-a512-498c-b8f8-159906bb8638-lib-modules\") pod \"tuned-9fpmt\" (UID: \"34560420-a512-498c-b8f8-159906bb8638\") " pod="openshift-cluster-node-tuning-operator/tuned-9fpmt"
Jul 12 22:10:30.226475 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.226443    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/34560420-a512-498c-b8f8-159906bb8638-etc-sysctl-conf\") pod \"tuned-9fpmt\" (UID: \"34560420-a512-498c-b8f8-159906bb8638\") " pod="openshift-cluster-node-tuning-operator/tuned-9fpmt"
Jul 12 22:10:30.226475 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.226457    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-host-kubelet\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.226475 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.226475    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-systemd-units\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.226911 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.226503    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l5hk\" (UniqueName: \"kubernetes.io/projected/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-kube-api-access-7l5hk\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.226911 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.226560    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/05ebf682-f7b0-4267-8a22-46d0c41ecba9-hosts-file\") pod \"node-resolver-g4qw4\" (UID: \"05ebf682-f7b0-4267-8a22-46d0c41ecba9\") " pod="openshift-dns/node-resolver-g4qw4"
Jul 12 22:10:30.226911 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.226625    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/912f085a-8942-4512-98f9-c8ba189509c2-os-release\") pod \"multus-additional-cni-plugins-kvvgp\" (UID: \"912f085a-8942-4512-98f9-c8ba189509c2\") " pod="openshift-multus/multus-additional-cni-plugins-kvvgp"
Jul 12 22:10:30.226911 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.226650    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f1ab5f7c-025e-43c0-893d-039e9d41b889-socket-dir\") pod \"aws-ebs-csi-driver-node-t9c7h\" (UID: \"f1ab5f7c-025e-43c0-893d-039e9d41b889\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9c7h"
Jul 12 22:10:30.226911 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.226674    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-cni-binary-copy\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.226911 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.226709    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-var-lib-openvswitch\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.226911 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.226733    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-run-ovn\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.226911 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.226774    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-env-overrides\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.226911 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.226801    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/912f085a-8942-4512-98f9-c8ba189509c2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kvvgp\" (UID: \"912f085a-8942-4512-98f9-c8ba189509c2\") " pod="openshift-multus/multus-additional-cni-plugins-kvvgp"
Jul 12 22:10:30.226911 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.226802    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-multus-daemon-config\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.226911 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.226825    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/34560420-a512-498c-b8f8-159906bb8638-etc-modprobe-d\") pod \"tuned-9fpmt\" (UID: \"34560420-a512-498c-b8f8-159906bb8638\") " pod="openshift-cluster-node-tuning-operator/tuned-9fpmt"
Jul 12 22:10:30.226911 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.226848    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/34560420-a512-498c-b8f8-159906bb8638-etc-sysctl-d\") pod \"tuned-9fpmt\" (UID: \"34560420-a512-498c-b8f8-159906bb8638\") " pod="openshift-cluster-node-tuning-operator/tuned-9fpmt"
Jul 12 22:10:30.226911 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.226873    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34560420-a512-498c-b8f8-159906bb8638-var-lib-kubelet\") pod \"tuned-9fpmt\" (UID: \"34560420-a512-498c-b8f8-159906bb8638\") " pod="openshift-cluster-node-tuning-operator/tuned-9fpmt"
Jul 12 22:10:30.226911 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.226897    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05ebf682-f7b0-4267-8a22-46d0c41ecba9-tmp-dir\") pod \"node-resolver-g4qw4\" (UID: \"05ebf682-f7b0-4267-8a22-46d0c41ecba9\") " pod="openshift-dns/node-resolver-g4qw4"
Jul 12 22:10:30.227583 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.226923    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-system-cni-dir\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.227583 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.226945    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/34560420-a512-498c-b8f8-159906bb8638-host\") pod \"tuned-9fpmt\" (UID: \"34560420-a512-498c-b8f8-159906bb8638\") " pod="openshift-cluster-node-tuning-operator/tuned-9fpmt"
Jul 12 22:10:30.227583 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.226984    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-run-openvswitch\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.227583 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.226952    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-os-release\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.227583 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.227302    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05ebf682-f7b0-4267-8a22-46d0c41ecba9-tmp-dir\") pod \"node-resolver-g4qw4\" (UID: \"05ebf682-f7b0-4267-8a22-46d0c41ecba9\") " pod="openshift-dns/node-resolver-g4qw4"
Jul 12 22:10:30.227583 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.226897    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/05ebf682-f7b0-4267-8a22-46d0c41ecba9-hosts-file\") pod \"node-resolver-g4qw4\" (UID: \"05ebf682-f7b0-4267-8a22-46d0c41ecba9\") " pod="openshift-dns/node-resolver-g4qw4"
Jul 12 22:10:30.227583 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.227377    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-system-cni-dir\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.227583 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.227420    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-host-run-k8s-cni-cncf-io\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.227958 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.227564    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/912f085a-8942-4512-98f9-c8ba189509c2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kvvgp\" (UID: \"912f085a-8942-4512-98f9-c8ba189509c2\") " pod="openshift-multus/multus-additional-cni-plugins-kvvgp"
Jul 12 22:10:30.227958 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.227757    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/912f085a-8942-4512-98f9-c8ba189509c2-os-release\") pod \"multus-additional-cni-plugins-kvvgp\" (UID: \"912f085a-8942-4512-98f9-c8ba189509c2\") " pod="openshift-multus/multus-additional-cni-plugins-kvvgp"
Jul 12 22:10:30.227958 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.227798    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxh26\" (UniqueName: \"kubernetes.io/projected/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-kube-api-access-dxh26\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.227958 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.227825    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/551bf86c-bd1a-40be-b977-41205da830c2-metrics-certs\") pod \"network-metrics-daemon-st67z\" (UID: \"551bf86c-bd1a-40be-b977-41205da830c2\") " pod="openshift-multus/network-metrics-daemon-st67z"
Jul 12 22:10:30.227958 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.227865    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpv7p\" (UniqueName: \"kubernetes.io/projected/bcf4ee57-809d-4e8b-be7f-593b88689c16-kube-api-access-bpv7p\") pod \"network-check-target-cn4d2\" (UID: \"bcf4ee57-809d-4e8b-be7f-593b88689c16\") " pod="openshift-network-diagnostics/network-check-target-cn4d2"
Jul 12 22:10:30.227958 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.227891    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-log-socket\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.227958 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.227917    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-host-run-ovn-kubernetes\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.227958 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.227940    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-cni-binary-copy\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.228321 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:30.228050    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 12 22:10:30.228321 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:30.228123    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/551bf86c-bd1a-40be-b977-41205da830c2-metrics-certs podName:551bf86c-bd1a-40be-b977-41205da830c2 nodeName:}" failed. No retries permitted until 2026-07-12 22:10:30.728102894 +0000 UTC m=+3.041606231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/551bf86c-bd1a-40be-b977-41205da830c2-metrics-certs") pod "network-metrics-daemon-st67z" (UID: "551bf86c-bd1a-40be-b977-41205da830c2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 12 22:10:30.228321 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.228173    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-host-cni-bin\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.228321 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.228223    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/00bab1f2-a13f-47ff-b0c7-c04484a4c85e-konnectivity-ca\") pod \"konnectivity-agent-lbvpn\" (UID: \"00bab1f2-a13f-47ff-b0c7-c04484a4c85e\") " pod="kube-system/konnectivity-agent-lbvpn"
Jul 12 22:10:30.228321 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.228252    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/912f085a-8942-4512-98f9-c8ba189509c2-cni-binary-copy\") pod \"multus-additional-cni-plugins-kvvgp\" (UID: \"912f085a-8942-4512-98f9-c8ba189509c2\") " pod="openshift-multus/multus-additional-cni-plugins-kvvgp"
Jul 12 22:10:30.228321 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.228288    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n454l\" (UniqueName: \"kubernetes.io/projected/551bf86c-bd1a-40be-b977-41205da830c2-kube-api-access-n454l\") pod \"network-metrics-daemon-st67z\" (UID: \"551bf86c-bd1a-40be-b977-41205da830c2\") " pod="openshift-multus/network-metrics-daemon-st67z"
Jul 12 22:10:30.228636 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.228357    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f1ab5f7c-025e-43c0-893d-039e9d41b889-device-dir\") pod \"aws-ebs-csi-driver-node-t9c7h\" (UID: \"f1ab5f7c-025e-43c0-893d-039e9d41b889\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9c7h"
Jul 12 22:10:30.228636 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.228386    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f1ab5f7c-025e-43c0-893d-039e9d41b889-sys-fs\") pod \"aws-ebs-csi-driver-node-t9c7h\" (UID: \"f1ab5f7c-025e-43c0-893d-039e9d41b889\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9c7h"
Jul 12 22:10:30.228636 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.228410    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/34560420-a512-498c-b8f8-159906bb8638-etc-sysconfig\") pod \"tuned-9fpmt\" (UID: \"34560420-a512-498c-b8f8-159906bb8638\") " pod="openshift-cluster-node-tuning-operator/tuned-9fpmt"
Jul 12 22:10:30.228636 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.228457    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34560420-a512-498c-b8f8-159906bb8638-etc-kubernetes\") pod \"tuned-9fpmt\" (UID: \"34560420-a512-498c-b8f8-159906bb8638\") " pod="openshift-cluster-node-tuning-operator/tuned-9fpmt"
Jul 12 22:10:30.228636 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.228481    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/34560420-a512-498c-b8f8-159906bb8638-etc-systemd\") pod \"tuned-9fpmt\" (UID: \"34560420-a512-498c-b8f8-159906bb8638\") " pod="openshift-cluster-node-tuning-operator/tuned-9fpmt"
Jul 12 22:10:30.228636 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.228505    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpgdb\" (UniqueName: \"kubernetes.io/projected/34560420-a512-498c-b8f8-159906bb8638-kube-api-access-cpgdb\") pod \"tuned-9fpmt\" (UID: \"34560420-a512-498c-b8f8-159906bb8638\") " pod="openshift-cluster-node-tuning-operator/tuned-9fpmt"
Jul 12 22:10:30.228636 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.228521    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34560420-a512-498c-b8f8-159906bb8638-tmp\") pod \"tuned-9fpmt\" (UID: \"34560420-a512-498c-b8f8-159906bb8638\") " pod="openshift-cluster-node-tuning-operator/tuned-9fpmt"
Jul 12 22:10:30.228636 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.228570    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-run-systemd\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.228636 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.228589    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-etc-openvswitch\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.228636 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.228638    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-ovnkube-script-lib\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.229095 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.228661    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-host-run-multus-certs\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.229095 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.228676    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-host-slash\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.229095 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.228691    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-cnibin\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.229095 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.228714    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-multus-socket-dir-parent\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.229095 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.228714    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/912f085a-8942-4512-98f9-c8ba189509c2-cni-binary-copy\") pod \"multus-additional-cni-plugins-kvvgp\" (UID: \"912f085a-8942-4512-98f9-c8ba189509c2\") " pod="openshift-multus/multus-additional-cni-plugins-kvvgp"
Jul 12 22:10:30.229095 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.228732    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-multus-conf-dir\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.229095 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.228768    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-multus-conf-dir\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.229095 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.228806    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-host-run-multus-certs\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.229095 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.228842    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-cnibin\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.229095 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.228904    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-multus-socket-dir-parent\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.229095 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.229096    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/912f085a-8942-4512-98f9-c8ba189509c2-cnibin\") pod \"multus-additional-cni-plugins-kvvgp\" (UID: \"912f085a-8942-4512-98f9-c8ba189509c2\") " pod="openshift-multus/multus-additional-cni-plugins-kvvgp"
Jul 12 22:10:30.229637 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.229113    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c946a409-971a-4760-b311-5278608e31f9-host\") pod \"node-ca-zkk8b\" (UID: \"c946a409-971a-4760-b311-5278608e31f9\") " pod="openshift-image-registry/node-ca-zkk8b"
Jul 12 22:10:30.229637 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.229129    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-host-cni-netd\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.229637 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.229152    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/00bab1f2-a13f-47ff-b0c7-c04484a4c85e-agent-certs\") pod \"konnectivity-agent-lbvpn\" (UID: \"00bab1f2-a13f-47ff-b0c7-c04484a4c85e\") " pod="kube-system/konnectivity-agent-lbvpn"
Jul 12 22:10:30.229637 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.229175    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-host-var-lib-cni-multus\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.229637 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.229199    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/912f085a-8942-4512-98f9-c8ba189509c2-system-cni-dir\") pod \"multus-additional-cni-plugins-kvvgp\" (UID: \"912f085a-8942-4512-98f9-c8ba189509c2\") " pod="openshift-multus/multus-additional-cni-plugins-kvvgp"
Jul 12 22:10:30.229637 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.229224    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/912f085a-8942-4512-98f9-c8ba189509c2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kvvgp\" (UID: \"912f085a-8942-4512-98f9-c8ba189509c2\") " pod="openshift-multus/multus-additional-cni-plugins-kvvgp"
Jul 12 22:10:30.229637 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.229259    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f1ab5f7c-025e-43c0-893d-039e9d41b889-registration-dir\") pod \"aws-ebs-csi-driver-node-t9c7h\" (UID: \"f1ab5f7c-025e-43c0-893d-039e9d41b889\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9c7h"
Jul 12 22:10:30.229637 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.229282    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxvsx\" (UniqueName: \"kubernetes.io/projected/f1ab5f7c-025e-43c0-893d-039e9d41b889-kube-api-access-mxvsx\") pod \"aws-ebs-csi-driver-node-t9c7h\" (UID: \"f1ab5f7c-025e-43c0-893d-039e9d41b889\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9c7h"
Jul 12 22:10:30.229637 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.229304    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/34560420-a512-498c-b8f8-159906bb8638-run\") pod \"tuned-9fpmt\" (UID: \"34560420-a512-498c-b8f8-159906bb8638\") " pod="openshift-cluster-node-tuning-operator/tuned-9fpmt"
Jul 12 22:10:30.229637 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.229325    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/37f0637a-26e3-4126-b33e-8bd8096369d2-host-slash\") pod \"iptables-alerter-6qrng\" (UID: \"37f0637a-26e3-4126-b33e-8bd8096369d2\") " pod="openshift-network-operator/iptables-alerter-6qrng"
Jul 12 22:10:30.229637 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.229341    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-host-var-lib-kubelet\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.229637 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.229354    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-etc-kubernetes\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.229637 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.229370    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/00bab1f2-a13f-47ff-b0c7-c04484a4c85e-konnectivity-ca\") pod \"konnectivity-agent-lbvpn\" (UID: \"00bab1f2-a13f-47ff-b0c7-c04484a4c85e\") " pod="kube-system/konnectivity-agent-lbvpn"
Jul 12 22:10:30.229637 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.229420    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-host-var-lib-kubelet\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.229637 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.229431    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/912f085a-8942-4512-98f9-c8ba189509c2-system-cni-dir\") pod \"multus-additional-cni-plugins-kvvgp\" (UID: \"912f085a-8942-4512-98f9-c8ba189509c2\") " pod="openshift-multus/multus-additional-cni-plugins-kvvgp"
Jul 12 22:10:30.229637 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.229440    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-host-var-lib-cni-multus\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.229637 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.229373    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh4pv\" (UniqueName: \"kubernetes.io/projected/37f0637a-26e3-4126-b33e-8bd8096369d2-kube-api-access-fh4pv\") pod \"iptables-alerter-6qrng\" (UID: \"37f0637a-26e3-4126-b33e-8bd8096369d2\") " pod="openshift-network-operator/iptables-alerter-6qrng"
Jul 12 22:10:30.230439 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.229460    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-etc-kubernetes\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.230439 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.229483    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-host-run-netns\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.230439 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.229498    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-node-log\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.230439 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.229519    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.230439 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.229564    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/700bcb2a-4c93-4abe-b403-9b404b3731e3-dbus\") pod \"global-pull-secret-syncer-b7t5r\" (UID: \"700bcb2a-4c93-4abe-b403-9b404b3731e3\") " pod="kube-system/global-pull-secret-syncer-b7t5r"
Jul 12 22:10:30.230439 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.229608    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/912f085a-8942-4512-98f9-c8ba189509c2-cnibin\") pod \"multus-additional-cni-plugins-kvvgp\" (UID: \"912f085a-8942-4512-98f9-c8ba189509c2\") " pod="openshift-multus/multus-additional-cni-plugins-kvvgp"
Jul 12 22:10:30.230439 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.229813    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/912f085a-8942-4512-98f9-c8ba189509c2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kvvgp\" (UID: \"912f085a-8942-4512-98f9-c8ba189509c2\") " pod="openshift-multus/multus-additional-cni-plugins-kvvgp"
Jul 12 22:10:30.230439 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.229867    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/700bcb2a-4c93-4abe-b403-9b404b3731e3-original-pull-secret\") pod \"global-pull-secret-syncer-b7t5r\" (UID: \"700bcb2a-4c93-4abe-b403-9b404b3731e3\") " pod="kube-system/global-pull-secret-syncer-b7t5r"
Jul 12 22:10:30.230439 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.229898    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8tjb\" (UniqueName: \"kubernetes.io/projected/05ebf682-f7b0-4267-8a22-46d0c41ecba9-kube-api-access-w8tjb\") pod \"node-resolver-g4qw4\" (UID: \"05ebf682-f7b0-4267-8a22-46d0c41ecba9\") " pod="openshift-dns/node-resolver-g4qw4"
Jul 12 22:10:30.230439 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.229930    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-hostroot\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.230439 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.229927    2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 12 22:10:30.230439 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.230004    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6nmw\" (UniqueName: \"kubernetes.io/projected/c946a409-971a-4760-b311-5278608e31f9-kube-api-access-w6nmw\") pod \"node-ca-zkk8b\" (UID: \"c946a409-971a-4760-b311-5278608e31f9\") " pod="openshift-image-registry/node-ca-zkk8b"
Jul 12 22:10:30.230439 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.230031    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/37f0637a-26e3-4126-b33e-8bd8096369d2-iptables-alerter-script\") pod \"iptables-alerter-6qrng\" (UID: \"37f0637a-26e3-4126-b33e-8bd8096369d2\") " pod="openshift-network-operator/iptables-alerter-6qrng"
Jul 12 22:10:30.230439 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.230059    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/700bcb2a-4c93-4abe-b403-9b404b3731e3-kubelet-config\") pod \"global-pull-secret-syncer-b7t5r\" (UID: \"700bcb2a-4c93-4abe-b403-9b404b3731e3\") " pod="kube-system/global-pull-secret-syncer-b7t5r"
Jul 12 22:10:30.230439 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.230087    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-multus-cni-dir\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.230439 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.230112    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-host-var-lib-cni-bin\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.230439 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.230183    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/912f085a-8942-4512-98f9-c8ba189509c2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kvvgp\" (UID: \"912f085a-8942-4512-98f9-c8ba189509c2\") " pod="openshift-multus/multus-additional-cni-plugins-kvvgp"
Jul 12 22:10:30.231022 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.230293    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-multus-cni-dir\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.231022 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.230301    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-hostroot\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.231022 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.230349    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-host-var-lib-cni-bin\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.231022 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.230404    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/912f085a-8942-4512-98f9-c8ba189509c2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kvvgp\" (UID: \"912f085a-8942-4512-98f9-c8ba189509c2\") " pod="openshift-multus/multus-additional-cni-plugins-kvvgp"
Jul 12 22:10:30.234269 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.234243    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/00bab1f2-a13f-47ff-b0c7-c04484a4c85e-agent-certs\") pod \"konnectivity-agent-lbvpn\" (UID: \"00bab1f2-a13f-47ff-b0c7-c04484a4c85e\") " pod="kube-system/konnectivity-agent-lbvpn"
Jul 12 22:10:30.239087 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.239060    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxh26\" (UniqueName: \"kubernetes.io/projected/669e3fb7-aeeb-4746-b4ee-84c83cd5a972-kube-api-access-dxh26\") pod \"multus-2t5r9\" (UID: \"669e3fb7-aeeb-4746-b4ee-84c83cd5a972\") " pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.239087 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.239071    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9jkf\" (UniqueName: \"kubernetes.io/projected/912f085a-8942-4512-98f9-c8ba189509c2-kube-api-access-k9jkf\") pod \"multus-additional-cni-plugins-kvvgp\" (UID: \"912f085a-8942-4512-98f9-c8ba189509c2\") " pod="openshift-multus/multus-additional-cni-plugins-kvvgp"
Jul 12 22:10:30.239832 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.239807    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n454l\" (UniqueName: \"kubernetes.io/projected/551bf86c-bd1a-40be-b977-41205da830c2-kube-api-access-n454l\") pod \"network-metrics-daemon-st67z\" (UID: \"551bf86c-bd1a-40be-b977-41205da830c2\") " pod="openshift-multus/network-metrics-daemon-st67z"
Jul 12 22:10:30.242075 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.242051    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8tjb\" (UniqueName: \"kubernetes.io/projected/05ebf682-f7b0-4267-8a22-46d0c41ecba9-kube-api-access-w8tjb\") pod \"node-resolver-g4qw4\" (UID: \"05ebf682-f7b0-4267-8a22-46d0c41ecba9\") " pod="openshift-dns/node-resolver-g4qw4"
Jul 12 22:10:30.246222 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.246166    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-47.ec2.internal" event={"ID":"3c0dae55120c29fa0e8b16a46a0faece","Type":"ContainerStarted","Data":"1059feb34be938771ea20a08515ce8132b2b4b54c040cbaf9ea40cb0e8709de5"}
Jul 12 22:10:30.247497 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.247470    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-47.ec2.internal" event={"ID":"2f0d6da932fad75070caf7c5d75748da","Type":"ContainerStarted","Data":"bd52414b265ba5a666092183c048569da006633ea4e27ae4914c41f723c109e5"}
Jul 12 22:10:30.331154 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.331069    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f1ab5f7c-025e-43c0-893d-039e9d41b889-registration-dir\") pod \"aws-ebs-csi-driver-node-t9c7h\" (UID: \"f1ab5f7c-025e-43c0-893d-039e9d41b889\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9c7h"
Jul 12 22:10:30.331154 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.331117    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxvsx\" (UniqueName: \"kubernetes.io/projected/f1ab5f7c-025e-43c0-893d-039e9d41b889-kube-api-access-mxvsx\") pod \"aws-ebs-csi-driver-node-t9c7h\" (UID: \"f1ab5f7c-025e-43c0-893d-039e9d41b889\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9c7h"
Jul 12 22:10:30.331154 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.331148    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/34560420-a512-498c-b8f8-159906bb8638-run\") pod \"tuned-9fpmt\" (UID: \"34560420-a512-498c-b8f8-159906bb8638\") " pod="openshift-cluster-node-tuning-operator/tuned-9fpmt"
Jul 12 22:10:30.331370 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.331172    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/37f0637a-26e3-4126-b33e-8bd8096369d2-host-slash\") pod \"iptables-alerter-6qrng\" (UID: \"37f0637a-26e3-4126-b33e-8bd8096369d2\") " pod="openshift-network-operator/iptables-alerter-6qrng"
Jul 12 22:10:30.331370 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.331202    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh4pv\" (UniqueName: \"kubernetes.io/projected/37f0637a-26e3-4126-b33e-8bd8096369d2-kube-api-access-fh4pv\") pod \"iptables-alerter-6qrng\" (UID: \"37f0637a-26e3-4126-b33e-8bd8096369d2\") " pod="openshift-network-operator/iptables-alerter-6qrng"
Jul 12 22:10:30.331370 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.331225    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-host-run-netns\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.331370 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.331228    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f1ab5f7c-025e-43c0-893d-039e9d41b889-registration-dir\") pod \"aws-ebs-csi-driver-node-t9c7h\" (UID: \"f1ab5f7c-025e-43c0-893d-039e9d41b889\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9c7h"
Jul 12 22:10:30.331370 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.331247    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-node-log\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.331370 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.331271    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.331370 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.331279    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/37f0637a-26e3-4126-b33e-8bd8096369d2-host-slash\") pod \"iptables-alerter-6qrng\" (UID: \"37f0637a-26e3-4126-b33e-8bd8096369d2\") " pod="openshift-network-operator/iptables-alerter-6qrng"
Jul 12 22:10:30.331370 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.331297    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/700bcb2a-4c93-4abe-b403-9b404b3731e3-dbus\") pod \"global-pull-secret-syncer-b7t5r\" (UID: \"700bcb2a-4c93-4abe-b403-9b404b3731e3\") " pod="kube-system/global-pull-secret-syncer-b7t5r"
Jul 12 22:10:30.331370 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.331334    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/700bcb2a-4c93-4abe-b403-9b404b3731e3-original-pull-secret\") pod \"global-pull-secret-syncer-b7t5r\" (UID: \"700bcb2a-4c93-4abe-b403-9b404b3731e3\") " pod="kube-system/global-pull-secret-syncer-b7t5r"
Jul 12 22:10:30.331801 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.331365    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6nmw\" (UniqueName: \"kubernetes.io/projected/c946a409-971a-4760-b311-5278608e31f9-kube-api-access-w6nmw\") pod \"node-ca-zkk8b\" (UID: \"c946a409-971a-4760-b311-5278608e31f9\") " pod="openshift-image-registry/node-ca-zkk8b"
Jul 12 22:10:30.331801 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.331403    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/37f0637a-26e3-4126-b33e-8bd8096369d2-iptables-alerter-script\") pod \"iptables-alerter-6qrng\" (UID: \"37f0637a-26e3-4126-b33e-8bd8096369d2\") " pod="openshift-network-operator/iptables-alerter-6qrng"
Jul 12 22:10:30.331801 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.331425    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/700bcb2a-4c93-4abe-b403-9b404b3731e3-kubelet-config\") pod \"global-pull-secret-syncer-b7t5r\" (UID: \"700bcb2a-4c93-4abe-b403-9b404b3731e3\") " pod="kube-system/global-pull-secret-syncer-b7t5r"
Jul 12 22:10:30.331801 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.331449    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/34560420-a512-498c-b8f8-159906bb8638-etc-tuned\") pod \"tuned-9fpmt\" (UID: \"34560420-a512-498c-b8f8-159906bb8638\") " pod="openshift-cluster-node-tuning-operator/tuned-9fpmt"
Jul 12 22:10:30.331801 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.331472    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f1ab5f7c-025e-43c0-893d-039e9d41b889-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t9c7h\" (UID: \"f1ab5f7c-025e-43c0-893d-039e9d41b889\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9c7h"
Jul 12 22:10:30.331801 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.331492    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34560420-a512-498c-b8f8-159906bb8638-sys\") pod \"tuned-9fpmt\" (UID: \"34560420-a512-498c-b8f8-159906bb8638\") " pod="openshift-cluster-node-tuning-operator/tuned-9fpmt"
Jul 12 22:10:30.331801 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.331513    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c946a409-971a-4760-b311-5278608e31f9-serviceca\") pod \"node-ca-zkk8b\" (UID: \"c946a409-971a-4760-b311-5278608e31f9\") " pod="openshift-image-registry/node-ca-zkk8b"
Jul 12 22:10:30.331801 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.331559    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-ovnkube-config\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.331801 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.331595    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/34560420-a512-498c-b8f8-159906bb8638-run\") pod \"tuned-9fpmt\" (UID: \"34560420-a512-498c-b8f8-159906bb8638\") " pod="openshift-cluster-node-tuning-operator/tuned-9fpmt"
Jul 12 22:10:30.332224 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.331584    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-ovn-node-metrics-cert\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.332224 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.331943    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f1ab5f7c-025e-43c0-893d-039e9d41b889-etc-selinux\") pod \"aws-ebs-csi-driver-node-t9c7h\" (UID: \"f1ab5f7c-025e-43c0-893d-039e9d41b889\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9c7h"
Jul 12 22:10:30.332224 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.331982    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/34560420-a512-498c-b8f8-159906bb8638-lib-modules\") pod \"tuned-9fpmt\" (UID: \"34560420-a512-498c-b8f8-159906bb8638\") " pod="openshift-cluster-node-tuning-operator/tuned-9fpmt"
Jul 12 22:10:30.336053 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.336021    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/34560420-a512-498c-b8f8-159906bb8638-etc-sysctl-conf\") pod \"tuned-9fpmt\" (UID: \"34560420-a512-498c-b8f8-159906bb8638\") " pod="openshift-cluster-node-tuning-operator/tuned-9fpmt"
Jul 12 22:10:30.336190 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.336075    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-host-kubelet\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.336190 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.336106    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-systemd-units\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.336190 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.336131    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l5hk\" (UniqueName: \"kubernetes.io/projected/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-kube-api-access-7l5hk\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.336190 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.336164    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f1ab5f7c-025e-43c0-893d-039e9d41b889-socket-dir\") pod \"aws-ebs-csi-driver-node-t9c7h\" (UID: \"f1ab5f7c-025e-43c0-893d-039e9d41b889\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9c7h"
Jul 12 22:10:30.336412 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.336193    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-var-lib-openvswitch\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.336412 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.336239    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-run-ovn\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.336412 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.336281    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-env-overrides\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.336412 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.336312    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/34560420-a512-498c-b8f8-159906bb8638-etc-modprobe-d\") pod \"tuned-9fpmt\" (UID: \"34560420-a512-498c-b8f8-159906bb8638\") " pod="openshift-cluster-node-tuning-operator/tuned-9fpmt"
Jul 12 22:10:30.336412 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.336344    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/34560420-a512-498c-b8f8-159906bb8638-etc-sysctl-d\") pod \"tuned-9fpmt\" (UID: \"34560420-a512-498c-b8f8-159906bb8638\") " pod="openshift-cluster-node-tuning-operator/tuned-9fpmt"
Jul 12 22:10:30.336412 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.336376    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34560420-a512-498c-b8f8-159906bb8638-var-lib-kubelet\") pod \"tuned-9fpmt\" (UID: \"34560420-a512-498c-b8f8-159906bb8638\") " pod="openshift-cluster-node-tuning-operator/tuned-9fpmt"
Jul 12 22:10:30.336726 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.336413    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/34560420-a512-498c-b8f8-159906bb8638-host\") pod \"tuned-9fpmt\" (UID: \"34560420-a512-498c-b8f8-159906bb8638\") " pod="openshift-cluster-node-tuning-operator/tuned-9fpmt"
Jul 12 22:10:30.336726 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.336444    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-run-openvswitch\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.336726 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.336497    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpv7p\" (UniqueName: \"kubernetes.io/projected/bcf4ee57-809d-4e8b-be7f-593b88689c16-kube-api-access-bpv7p\") pod \"network-check-target-cn4d2\" (UID: \"bcf4ee57-809d-4e8b-be7f-593b88689c16\") " pod="openshift-network-diagnostics/network-check-target-cn4d2"
Jul 12 22:10:30.336726 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.336550    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-log-socket\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.336726 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.336580    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-host-run-ovn-kubernetes\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.336726 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.336612    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-host-cni-bin\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.336726 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.336648    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f1ab5f7c-025e-43c0-893d-039e9d41b889-device-dir\") pod \"aws-ebs-csi-driver-node-t9c7h\" (UID: \"f1ab5f7c-025e-43c0-893d-039e9d41b889\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9c7h"
Jul 12 22:10:30.336726 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.336681    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f1ab5f7c-025e-43c0-893d-039e9d41b889-sys-fs\") pod \"aws-ebs-csi-driver-node-t9c7h\" (UID: \"f1ab5f7c-025e-43c0-893d-039e9d41b889\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9c7h"
Jul 12 22:10:30.336726 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.336713    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/34560420-a512-498c-b8f8-159906bb8638-etc-sysconfig\") pod \"tuned-9fpmt\" (UID: \"34560420-a512-498c-b8f8-159906bb8638\") " pod="openshift-cluster-node-tuning-operator/tuned-9fpmt"
Jul 12 22:10:30.337149 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.336738    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34560420-a512-498c-b8f8-159906bb8638-etc-kubernetes\") pod \"tuned-9fpmt\" (UID: \"34560420-a512-498c-b8f8-159906bb8638\") " pod="openshift-cluster-node-tuning-operator/tuned-9fpmt"
Jul 12 22:10:30.337149 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.336770    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/34560420-a512-498c-b8f8-159906bb8638-etc-systemd\") pod \"tuned-9fpmt\" (UID: \"34560420-a512-498c-b8f8-159906bb8638\") " pod="openshift-cluster-node-tuning-operator/tuned-9fpmt"
Jul 12 22:10:30.337149 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.336802    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpgdb\" (UniqueName: \"kubernetes.io/projected/34560420-a512-498c-b8f8-159906bb8638-kube-api-access-cpgdb\") pod \"tuned-9fpmt\" (UID: \"34560420-a512-498c-b8f8-159906bb8638\") " pod="openshift-cluster-node-tuning-operator/tuned-9fpmt"
Jul 12 22:10:30.337149 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.336833    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34560420-a512-498c-b8f8-159906bb8638-tmp\") pod \"tuned-9fpmt\" (UID: \"34560420-a512-498c-b8f8-159906bb8638\") " pod="openshift-cluster-node-tuning-operator/tuned-9fpmt"
Jul 12 22:10:30.337149 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.336864    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-run-systemd\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.337149 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.336890    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-etc-openvswitch\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.337149 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.336923    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-ovnkube-script-lib\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.337149 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.336956    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-host-slash\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.337149 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.336994    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c946a409-971a-4760-b311-5278608e31f9-host\") pod \"node-ca-zkk8b\" (UID: \"c946a409-971a-4760-b311-5278608e31f9\") " pod="openshift-image-registry/node-ca-zkk8b"
Jul 12 22:10:30.337149 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.337022    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-host-cni-netd\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.337636 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.337148    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-host-cni-netd\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.337636 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.337307    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/34560420-a512-498c-b8f8-159906bb8638-etc-sysctl-conf\") pod \"tuned-9fpmt\" (UID: \"34560420-a512-498c-b8f8-159906bb8638\") " pod="openshift-cluster-node-tuning-operator/tuned-9fpmt"
Jul 12 22:10:30.337636 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.337358    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-host-kubelet\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.337636 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.337412    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-systemd-units\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.337907 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.337885    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f1ab5f7c-025e-43c0-893d-039e9d41b889-socket-dir\") pod \"aws-ebs-csi-driver-node-t9c7h\" (UID: \"f1ab5f7c-025e-43c0-893d-039e9d41b889\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9c7h"
Jul 12 22:10:30.337987 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.337958    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-var-lib-openvswitch\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.338047 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.338011    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-run-ovn\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.338547 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.338489    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-env-overrides\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.338736 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.338660    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/34560420-a512-498c-b8f8-159906bb8638-etc-modprobe-d\") pod \"tuned-9fpmt\" (UID: \"34560420-a512-498c-b8f8-159906bb8638\") " pod="openshift-cluster-node-tuning-operator/tuned-9fpmt"
Jul 12 22:10:30.338819 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.338785    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/34560420-a512-498c-b8f8-159906bb8638-etc-sysctl-d\") pod \"tuned-9fpmt\" (UID: \"34560420-a512-498c-b8f8-159906bb8638\") " pod="openshift-cluster-node-tuning-operator/tuned-9fpmt"
Jul 12 22:10:30.338875 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.338855    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34560420-a512-498c-b8f8-159906bb8638-var-lib-kubelet\") pod \"tuned-9fpmt\" (UID: \"34560420-a512-498c-b8f8-159906bb8638\") " pod="openshift-cluster-node-tuning-operator/tuned-9fpmt"
Jul 12 22:10:30.338937 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.338915    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/34560420-a512-498c-b8f8-159906bb8638-host\") pod \"tuned-9fpmt\" (UID: \"34560420-a512-498c-b8f8-159906bb8638\") " pod="openshift-cluster-node-tuning-operator/tuned-9fpmt"
Jul 12 22:10:30.338993 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.338969    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-run-openvswitch\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.339177 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.339156    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-log-socket\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.339433 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.339407    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c946a409-971a-4760-b311-5278608e31f9-host\") pod \"node-ca-zkk8b\" (UID: \"c946a409-971a-4760-b311-5278608e31f9\") " pod="openshift-image-registry/node-ca-zkk8b"
Jul 12 22:10:30.339509 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.339454    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-host-run-ovn-kubernetes\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.339509 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.339496    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-host-cni-bin\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.339672 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.339564    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f1ab5f7c-025e-43c0-893d-039e9d41b889-device-dir\") pod \"aws-ebs-csi-driver-node-t9c7h\" (UID: \"f1ab5f7c-025e-43c0-893d-039e9d41b889\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9c7h"
Jul 12 22:10:30.339672 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.339613    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f1ab5f7c-025e-43c0-893d-039e9d41b889-sys-fs\") pod \"aws-ebs-csi-driver-node-t9c7h\" (UID: \"f1ab5f7c-025e-43c0-893d-039e9d41b889\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9c7h"
Jul 12 22:10:30.339778 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.339695    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34560420-a512-498c-b8f8-159906bb8638-etc-kubernetes\") pod \"tuned-9fpmt\" (UID: \"34560420-a512-498c-b8f8-159906bb8638\") " pod="openshift-cluster-node-tuning-operator/tuned-9fpmt"
Jul 12 22:10:30.339778 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.339738    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/34560420-a512-498c-b8f8-159906bb8638-etc-systemd\") pod \"tuned-9fpmt\" (UID: \"34560420-a512-498c-b8f8-159906bb8638\") " pod="openshift-cluster-node-tuning-operator/tuned-9fpmt"
Jul 12 22:10:30.339886 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.339790    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/34560420-a512-498c-b8f8-159906bb8638-etc-sysconfig\") pod \"tuned-9fpmt\" (UID: \"34560420-a512-498c-b8f8-159906bb8638\") " pod="openshift-cluster-node-tuning-operator/tuned-9fpmt"
Jul 12 22:10:30.339886 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.339835    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/700bcb2a-4c93-4abe-b403-9b404b3731e3-kubelet-config\") pod \"global-pull-secret-syncer-b7t5r\" (UID: \"700bcb2a-4c93-4abe-b403-9b404b3731e3\") " pod="kube-system/global-pull-secret-syncer-b7t5r"
Jul 12 22:10:30.340328 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.340306    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-ovnkube-config\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.342910 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.340617    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-host-run-netns\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.342910 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.340678    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-node-log\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.342910 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.340726    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.342910 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.340718    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/34560420-a512-498c-b8f8-159906bb8638-lib-modules\") pod \"tuned-9fpmt\" (UID: \"34560420-a512-498c-b8f8-159906bb8638\") " pod="openshift-cluster-node-tuning-operator/tuned-9fpmt"
Jul 12 22:10:30.342910 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.340847    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/700bcb2a-4c93-4abe-b403-9b404b3731e3-dbus\") pod \"global-pull-secret-syncer-b7t5r\" (UID: \"700bcb2a-4c93-4abe-b403-9b404b3731e3\") " pod="kube-system/global-pull-secret-syncer-b7t5r"
Jul 12 22:10:30.342910 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.341166    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c946a409-971a-4760-b311-5278608e31f9-serviceca\") pod \"node-ca-zkk8b\" (UID: \"c946a409-971a-4760-b311-5278608e31f9\") " pod="openshift-image-registry/node-ca-zkk8b"
Jul 12 22:10:30.342910 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:30.341267    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 12 22:10:30.342910 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:30.341342    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/700bcb2a-4c93-4abe-b403-9b404b3731e3-original-pull-secret podName:700bcb2a-4c93-4abe-b403-9b404b3731e3 nodeName:}" failed. No retries permitted until 2026-07-12 22:10:30.841315059 +0000 UTC m=+3.154818396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/700bcb2a-4c93-4abe-b403-9b404b3731e3-original-pull-secret") pod "global-pull-secret-syncer-b7t5r" (UID: "700bcb2a-4c93-4abe-b403-9b404b3731e3") : object "kube-system"/"original-pull-secret" not registered
Jul 12 22:10:30.342910 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.341611    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-run-systemd\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.342910 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.341643    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f1ab5f7c-025e-43c0-893d-039e9d41b889-etc-selinux\") pod \"aws-ebs-csi-driver-node-t9c7h\" (UID: \"f1ab5f7c-025e-43c0-893d-039e9d41b889\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9c7h"
Jul 12 22:10:30.342910 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.342220    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-ovnkube-script-lib\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.342910 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.342336    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/37f0637a-26e3-4126-b33e-8bd8096369d2-iptables-alerter-script\") pod \"iptables-alerter-6qrng\" (UID: \"37f0637a-26e3-4126-b33e-8bd8096369d2\") " pod="openshift-network-operator/iptables-alerter-6qrng"
Jul 12 22:10:30.343551 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.343159    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34560420-a512-498c-b8f8-159906bb8638-tmp\") pod \"tuned-9fpmt\" (UID: \"34560420-a512-498c-b8f8-159906bb8638\") " pod="openshift-cluster-node-tuning-operator/tuned-9fpmt"
Jul 12 22:10:30.343661 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.343636    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-etc-openvswitch\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.344764 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.343730    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34560420-a512-498c-b8f8-159906bb8638-sys\") pod \"tuned-9fpmt\" (UID: \"34560420-a512-498c-b8f8-159906bb8638\") " pod="openshift-cluster-node-tuning-operator/tuned-9fpmt"
Jul 12 22:10:30.344764 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.343759    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f1ab5f7c-025e-43c0-893d-039e9d41b889-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t9c7h\" (UID: \"f1ab5f7c-025e-43c0-893d-039e9d41b889\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9c7h"
Jul 12 22:10:30.344764 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.343832    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-host-slash\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.344764 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.344257    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-ovn-node-metrics-cert\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.344987 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.344827    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/34560420-a512-498c-b8f8-159906bb8638-etc-tuned\") pod \"tuned-9fpmt\" (UID: \"34560420-a512-498c-b8f8-159906bb8638\") " pod="openshift-cluster-node-tuning-operator/tuned-9fpmt"
Jul 12 22:10:30.347060 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.346654    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxvsx\" (UniqueName: \"kubernetes.io/projected/f1ab5f7c-025e-43c0-893d-039e9d41b889-kube-api-access-mxvsx\") pod \"aws-ebs-csi-driver-node-t9c7h\" (UID: \"f1ab5f7c-025e-43c0-893d-039e9d41b889\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9c7h"
Jul 12 22:10:30.347207 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.347168    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l5hk\" (UniqueName: \"kubernetes.io/projected/1b1f17bc-a992-4c72-8fdc-a33a3c3649c4-kube-api-access-7l5hk\") pod \"ovnkube-node-52pbf\" (UID: \"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4\") " pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.350702 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:30.350680    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 12 22:10:30.350702 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:30.350703    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 12 22:10:30.350940 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:30.350714    2560 projected.go:196] Error preparing data for projected volume kube-api-access-bpv7p for pod openshift-network-diagnostics/network-check-target-cn4d2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 12 22:10:30.350940 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:30.350773    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcf4ee57-809d-4e8b-be7f-593b88689c16-kube-api-access-bpv7p podName:bcf4ee57-809d-4e8b-be7f-593b88689c16 nodeName:}" failed. No retries permitted until 2026-07-12 22:10:30.850758654 +0000 UTC m=+3.164261986 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bpv7p" (UniqueName: "kubernetes.io/projected/bcf4ee57-809d-4e8b-be7f-593b88689c16-kube-api-access-bpv7p") pod "network-check-target-cn4d2" (UID: "bcf4ee57-809d-4e8b-be7f-593b88689c16") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 12 22:10:30.353586 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.353444    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpgdb\" (UniqueName: \"kubernetes.io/projected/34560420-a512-498c-b8f8-159906bb8638-kube-api-access-cpgdb\") pod \"tuned-9fpmt\" (UID: \"34560420-a512-498c-b8f8-159906bb8638\") " pod="openshift-cluster-node-tuning-operator/tuned-9fpmt"
Jul 12 22:10:30.353678 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.353611    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6nmw\" (UniqueName: \"kubernetes.io/projected/c946a409-971a-4760-b311-5278608e31f9-kube-api-access-w6nmw\") pod \"node-ca-zkk8b\" (UID: \"c946a409-971a-4760-b311-5278608e31f9\") " pod="openshift-image-registry/node-ca-zkk8b"
Jul 12 22:10:30.360019 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.359984    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh4pv\" (UniqueName: \"kubernetes.io/projected/37f0637a-26e3-4126-b33e-8bd8096369d2-kube-api-access-fh4pv\") pod \"iptables-alerter-6qrng\" (UID: \"37f0637a-26e3-4126-b33e-8bd8096369d2\") " pod="openshift-network-operator/iptables-alerter-6qrng"
Jul 12 22:10:30.417967 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.417927    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g4qw4"
Jul 12 22:10:30.426467 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.426443    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lbvpn"
Jul 12 22:10:30.434430 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.434399    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2t5r9"
Jul 12 22:10:30.441460 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.441436    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kvvgp"
Jul 12 22:10:30.449652 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.449628    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9c7h"
Jul 12 22:10:30.460729 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.460695    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9fpmt"
Jul 12 22:10:30.468732 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.468543    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zkk8b"
Jul 12 22:10:30.476938 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.476909    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6qrng"
Jul 12 22:10:30.481918 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.481897    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:30.739674 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.739588    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/551bf86c-bd1a-40be-b977-41205da830c2-metrics-certs\") pod \"network-metrics-daemon-st67z\" (UID: \"551bf86c-bd1a-40be-b977-41205da830c2\") " pod="openshift-multus/network-metrics-daemon-st67z"
Jul 12 22:10:30.739856 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:30.739728    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 12 22:10:30.739856 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:30.739791    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/551bf86c-bd1a-40be-b977-41205da830c2-metrics-certs podName:551bf86c-bd1a-40be-b977-41205da830c2 nodeName:}" failed. No retries permitted until 2026-07-12 22:10:31.73977664 +0000 UTC m=+4.053279971 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/551bf86c-bd1a-40be-b977-41205da830c2-metrics-certs") pod "network-metrics-daemon-st67z" (UID: "551bf86c-bd1a-40be-b977-41205da830c2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 12 22:10:30.940609 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.940559    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/700bcb2a-4c93-4abe-b403-9b404b3731e3-original-pull-secret\") pod \"global-pull-secret-syncer-b7t5r\" (UID: \"700bcb2a-4c93-4abe-b403-9b404b3731e3\") " pod="kube-system/global-pull-secret-syncer-b7t5r"
Jul 12 22:10:30.940783 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:30.940651    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpv7p\" (UniqueName: \"kubernetes.io/projected/bcf4ee57-809d-4e8b-be7f-593b88689c16-kube-api-access-bpv7p\") pod \"network-check-target-cn4d2\" (UID: \"bcf4ee57-809d-4e8b-be7f-593b88689c16\") " pod="openshift-network-diagnostics/network-check-target-cn4d2"
Jul 12 22:10:30.940783 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:30.940724    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 12 22:10:30.940888 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:30.940782    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 12 22:10:30.940888 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:30.940795    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/700bcb2a-4c93-4abe-b403-9b404b3731e3-original-pull-secret podName:700bcb2a-4c93-4abe-b403-9b404b3731e3 nodeName:}" failed. No retries permitted until 2026-07-12 22:10:31.940780329 +0000 UTC m=+4.254283665 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/700bcb2a-4c93-4abe-b403-9b404b3731e3-original-pull-secret") pod "global-pull-secret-syncer-b7t5r" (UID: "700bcb2a-4c93-4abe-b403-9b404b3731e3") : object "kube-system"/"original-pull-secret" not registered
Jul 12 22:10:30.940888 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:30.940806    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 12 22:10:30.940888 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:30.940820    2560 projected.go:196] Error preparing data for projected volume kube-api-access-bpv7p for pod openshift-network-diagnostics/network-check-target-cn4d2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 12 22:10:30.940888 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:30.940870    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcf4ee57-809d-4e8b-be7f-593b88689c16-kube-api-access-bpv7p podName:bcf4ee57-809d-4e8b-be7f-593b88689c16 nodeName:}" failed. No retries permitted until 2026-07-12 22:10:31.940854248 +0000 UTC m=+4.254357595 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bpv7p" (UniqueName: "kubernetes.io/projected/bcf4ee57-809d-4e8b-be7f-593b88689c16-kube-api-access-bpv7p") pod "network-check-target-cn4d2" (UID: "bcf4ee57-809d-4e8b-be7f-593b88689c16") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 12 22:10:31.161442 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:31.161365    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-11 22:05:30 +0000 UTC" deadline="2027-12-09 15:53:03.430827916 +0000 UTC"
Jul 12 22:10:31.161442 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:31.161400    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12353h42m32.269433106s"
Jul 12 22:10:31.381692 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:31.381459    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod669e3fb7_aeeb_4746_b4ee_84c83cd5a972.slice/crio-1df3d4e6478e606dff62b2304054e83ce6ef3ff720cfb8197445b7863ac7b811 WatchSource:0}: Error finding container 1df3d4e6478e606dff62b2304054e83ce6ef3ff720cfb8197445b7863ac7b811: Status 404 returned error can't find the container with id 1df3d4e6478e606dff62b2304054e83ce6ef3ff720cfb8197445b7863ac7b811
Jul 12 22:10:31.382556 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:31.382506    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00bab1f2_a13f_47ff_b0c7_c04484a4c85e.slice/crio-6ba60e57f15ffdde05d24c873b4c68e4869a218af28a6585565a136241816ac4 WatchSource:0}: Error finding container 6ba60e57f15ffdde05d24c873b4c68e4869a218af28a6585565a136241816ac4: Status 404 returned error can't find the container with id 6ba60e57f15ffdde05d24c873b4c68e4869a218af28a6585565a136241816ac4
Jul 12 22:10:31.383134 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:31.383111    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37f0637a_26e3_4126_b33e_8bd8096369d2.slice/crio-b9d57b30cbed8b385c01058cda33586d17eac453d02e48966491b34a0c8fa34e WatchSource:0}: Error finding container b9d57b30cbed8b385c01058cda33586d17eac453d02e48966491b34a0c8fa34e: Status 404 returned error can't find the container with id b9d57b30cbed8b385c01058cda33586d17eac453d02e48966491b34a0c8fa34e
Jul 12 22:10:31.387562 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:31.387505    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod912f085a_8942_4512_98f9_c8ba189509c2.slice/crio-a5829bf8664bc4199c427af83d37a44af2ed615f7fa63286976c3eade6d2790e WatchSource:0}: Error finding container a5829bf8664bc4199c427af83d37a44af2ed615f7fa63286976c3eade6d2790e: Status 404 returned error can't find the container with id a5829bf8664bc4199c427af83d37a44af2ed615f7fa63286976c3eade6d2790e
Jul 12 22:10:31.389171 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:31.389152    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34560420_a512_498c_b8f8_159906bb8638.slice/crio-41edbeaea1dd50eba81dc89da215105a13c483a9aa9fbf62b0160459d1c92493 WatchSource:0}: Error finding container 41edbeaea1dd50eba81dc89da215105a13c483a9aa9fbf62b0160459d1c92493: Status 404 returned error can't find the container with id 41edbeaea1dd50eba81dc89da215105a13c483a9aa9fbf62b0160459d1c92493
Jul 12 22:10:31.390044 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:31.390025    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b1f17bc_a992_4c72_8fdc_a33a3c3649c4.slice/crio-166f2780372cb2325320ef173d43a4d47fb15eb63634f4b03fba80b404a7db7c WatchSource:0}: Error finding container 166f2780372cb2325320ef173d43a4d47fb15eb63634f4b03fba80b404a7db7c: Status 404 returned error can't find the container with id 166f2780372cb2325320ef173d43a4d47fb15eb63634f4b03fba80b404a7db7c
Jul 12 22:10:31.391866 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:31.391834    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ab5f7c_025e_43c0_893d_039e9d41b889.slice/crio-db95ff758e85afff1a1a8a1e122c2f0329cc7779a26a4b5e97f1f02d57e667fe WatchSource:0}: Error finding container db95ff758e85afff1a1a8a1e122c2f0329cc7779a26a4b5e97f1f02d57e667fe: Status 404 returned error can't find the container with id db95ff758e85afff1a1a8a1e122c2f0329cc7779a26a4b5e97f1f02d57e667fe
Jul 12 22:10:31.392863 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:31.392843    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc946a409_971a_4760_b311_5278608e31f9.slice/crio-71d022f548f912762fe9beb6a4e02ffb0996da0fd59b0e1535e28a378dd8ec95 WatchSource:0}: Error finding container 71d022f548f912762fe9beb6a4e02ffb0996da0fd59b0e1535e28a378dd8ec95: Status 404 returned error can't find the container with id 71d022f548f912762fe9beb6a4e02ffb0996da0fd59b0e1535e28a378dd8ec95
Jul 12 22:10:31.393746 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:10:31.393728    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05ebf682_f7b0_4267_8a22_46d0c41ecba9.slice/crio-35236fc16e31a57e0f924e6eaf3e6034d202f106515179d53b257cb5a42968e4 WatchSource:0}: Error finding container 35236fc16e31a57e0f924e6eaf3e6034d202f106515179d53b257cb5a42968e4: Status 404 returned error can't find the container with id 35236fc16e31a57e0f924e6eaf3e6034d202f106515179d53b257cb5a42968e4
Jul 12 22:10:31.747963 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:31.747867    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/551bf86c-bd1a-40be-b977-41205da830c2-metrics-certs\") pod \"network-metrics-daemon-st67z\" (UID: \"551bf86c-bd1a-40be-b977-41205da830c2\") " pod="openshift-multus/network-metrics-daemon-st67z"
Jul 12 22:10:31.748125 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:31.748037    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 12 22:10:31.748125 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:31.748098    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/551bf86c-bd1a-40be-b977-41205da830c2-metrics-certs podName:551bf86c-bd1a-40be-b977-41205da830c2 nodeName:}" failed. No retries permitted until 2026-07-12 22:10:33.748084234 +0000 UTC m=+6.061587565 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/551bf86c-bd1a-40be-b977-41205da830c2-metrics-certs") pod "network-metrics-daemon-st67z" (UID: "551bf86c-bd1a-40be-b977-41205da830c2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 12 22:10:31.949795 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:31.949751    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpv7p\" (UniqueName: \"kubernetes.io/projected/bcf4ee57-809d-4e8b-be7f-593b88689c16-kube-api-access-bpv7p\") pod \"network-check-target-cn4d2\" (UID: \"bcf4ee57-809d-4e8b-be7f-593b88689c16\") " pod="openshift-network-diagnostics/network-check-target-cn4d2"
Jul 12 22:10:31.949989 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:31.949826    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/700bcb2a-4c93-4abe-b403-9b404b3731e3-original-pull-secret\") pod \"global-pull-secret-syncer-b7t5r\" (UID: \"700bcb2a-4c93-4abe-b403-9b404b3731e3\") " pod="kube-system/global-pull-secret-syncer-b7t5r"
Jul 12 22:10:31.949989 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:31.949957    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 12 22:10:31.950112 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:31.950022    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/700bcb2a-4c93-4abe-b403-9b404b3731e3-original-pull-secret podName:700bcb2a-4c93-4abe-b403-9b404b3731e3 nodeName:}" failed. No retries permitted until 2026-07-12 22:10:33.950003488 +0000 UTC m=+6.263506824 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/700bcb2a-4c93-4abe-b403-9b404b3731e3-original-pull-secret") pod "global-pull-secret-syncer-b7t5r" (UID: "700bcb2a-4c93-4abe-b403-9b404b3731e3") : object "kube-system"/"original-pull-secret" not registered
Jul 12 22:10:31.950437 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:31.950420    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 12 22:10:31.950516 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:31.950445    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 12 22:10:31.950516 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:31.950458    2560 projected.go:196] Error preparing data for projected volume kube-api-access-bpv7p for pod openshift-network-diagnostics/network-check-target-cn4d2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 12 22:10:31.950516 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:31.950506    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcf4ee57-809d-4e8b-be7f-593b88689c16-kube-api-access-bpv7p podName:bcf4ee57-809d-4e8b-be7f-593b88689c16 nodeName:}" failed. No retries permitted until 2026-07-12 22:10:33.950490487 +0000 UTC m=+6.263993827 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bpv7p" (UniqueName: "kubernetes.io/projected/bcf4ee57-809d-4e8b-be7f-593b88689c16-kube-api-access-bpv7p") pod "network-check-target-cn4d2" (UID: "bcf4ee57-809d-4e8b-be7f-593b88689c16") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 12 22:10:32.162051 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:32.161959    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-11 22:05:30 +0000 UTC" deadline="2027-12-12 19:28:02.817499872 +0000 UTC"
Jul 12 22:10:32.162051 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:32.162003    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12429h17m30.655501573s"
Jul 12 22:10:32.241851 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:32.241763    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7t5r"
Jul 12 22:10:32.242031 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:32.241891    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7t5r" podUID="700bcb2a-4c93-4abe-b403-9b404b3731e3"
Jul 12 22:10:32.242568 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:32.242266    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-st67z"
Jul 12 22:10:32.242568 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:32.242366    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-st67z" podUID="551bf86c-bd1a-40be-b977-41205da830c2"
Jul 12 22:10:32.242568 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:32.242421    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cn4d2"
Jul 12 22:10:32.242568 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:32.242483    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cn4d2" podUID="bcf4ee57-809d-4e8b-be7f-593b88689c16"
Jul 12 22:10:32.285910 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:32.285875    2560 generic.go:356] "Generic (PLEG): container finished" podID="3c0dae55120c29fa0e8b16a46a0faece" containerID="682de31f32eeee71973196779200a329899f6d897e3b00eb092635d9f44737e3" exitCode=0
Jul 12 22:10:32.286093 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:32.285972    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-47.ec2.internal" event={"ID":"3c0dae55120c29fa0e8b16a46a0faece","Type":"ContainerDied","Data":"682de31f32eeee71973196779200a329899f6d897e3b00eb092635d9f44737e3"}
Jul 12 22:10:32.302587 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:32.302051    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g4qw4" event={"ID":"05ebf682-f7b0-4267-8a22-46d0c41ecba9","Type":"ContainerStarted","Data":"35236fc16e31a57e0f924e6eaf3e6034d202f106515179d53b257cb5a42968e4"}
Jul 12 22:10:32.306994 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:32.306925    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9fpmt" event={"ID":"34560420-a512-498c-b8f8-159906bb8638","Type":"ContainerStarted","Data":"41edbeaea1dd50eba81dc89da215105a13c483a9aa9fbf62b0160459d1c92493"}
Jul 12 22:10:32.317337 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:32.317244    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6qrng" event={"ID":"37f0637a-26e3-4126-b33e-8bd8096369d2","Type":"ContainerStarted","Data":"b9d57b30cbed8b385c01058cda33586d17eac453d02e48966491b34a0c8fa34e"}
Jul 12 22:10:32.322881 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:32.322845    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lbvpn" event={"ID":"00bab1f2-a13f-47ff-b0c7-c04484a4c85e","Type":"ContainerStarted","Data":"6ba60e57f15ffdde05d24c873b4c68e4869a218af28a6585565a136241816ac4"}
Jul 12 22:10:32.334747 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:32.334674    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-47.ec2.internal" event={"ID":"2f0d6da932fad75070caf7c5d75748da","Type":"ContainerStarted","Data":"d79bb0cfc6b4df2d31e1466cd04d5bf736e7f2daa7a215ce1f75699d610793a8"}
Jul 12 22:10:32.361764 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:32.361725    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zkk8b" event={"ID":"c946a409-971a-4760-b311-5278608e31f9","Type":"ContainerStarted","Data":"71d022f548f912762fe9beb6a4e02ffb0996da0fd59b0e1535e28a378dd8ec95"}
Jul 12 22:10:32.381999 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:32.381956    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-52pbf" event={"ID":"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4","Type":"ContainerStarted","Data":"166f2780372cb2325320ef173d43a4d47fb15eb63634f4b03fba80b404a7db7c"}
Jul 12 22:10:32.400954 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:32.400867    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9c7h" event={"ID":"f1ab5f7c-025e-43c0-893d-039e9d41b889","Type":"ContainerStarted","Data":"db95ff758e85afff1a1a8a1e122c2f0329cc7779a26a4b5e97f1f02d57e667fe"}
Jul 12 22:10:32.413902 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:32.413819    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kvvgp" event={"ID":"912f085a-8942-4512-98f9-c8ba189509c2","Type":"ContainerStarted","Data":"a5829bf8664bc4199c427af83d37a44af2ed615f7fa63286976c3eade6d2790e"}
Jul 12 22:10:32.426195 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:32.426155    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2t5r9" event={"ID":"669e3fb7-aeeb-4746-b4ee-84c83cd5a972","Type":"ContainerStarted","Data":"1df3d4e6478e606dff62b2304054e83ce6ef3ff720cfb8197445b7863ac7b811"}
Jul 12 22:10:33.444220 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:33.443150    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-47.ec2.internal" event={"ID":"3c0dae55120c29fa0e8b16a46a0faece","Type":"ContainerStarted","Data":"5490d78e2894e13d7a915c455651c56e3f90c40f22fc920ef30a7224eb8bf60d"}
Jul 12 22:10:33.461584 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:33.461501    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-47.ec2.internal" podStartSLOduration=4.461480044 podStartE2EDuration="4.461480044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 22:10:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:10:32.359718928 +0000 UTC m=+4.673222283" watchObservedRunningTime="2026-07-12 22:10:33.461480044 +0000 UTC m=+5.774983399"
Jul 12 22:10:33.766316 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:33.765566    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/551bf86c-bd1a-40be-b977-41205da830c2-metrics-certs\") pod \"network-metrics-daemon-st67z\" (UID: \"551bf86c-bd1a-40be-b977-41205da830c2\") " pod="openshift-multus/network-metrics-daemon-st67z"
Jul 12 22:10:33.766316 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:33.765730    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 12 22:10:33.766316 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:33.765799    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/551bf86c-bd1a-40be-b977-41205da830c2-metrics-certs podName:551bf86c-bd1a-40be-b977-41205da830c2 nodeName:}" failed. No retries permitted until 2026-07-12 22:10:37.765780241 +0000 UTC m=+10.079283588 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/551bf86c-bd1a-40be-b977-41205da830c2-metrics-certs") pod "network-metrics-daemon-st67z" (UID: "551bf86c-bd1a-40be-b977-41205da830c2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 12 22:10:33.967182 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:33.967132    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpv7p\" (UniqueName: \"kubernetes.io/projected/bcf4ee57-809d-4e8b-be7f-593b88689c16-kube-api-access-bpv7p\") pod \"network-check-target-cn4d2\" (UID: \"bcf4ee57-809d-4e8b-be7f-593b88689c16\") " pod="openshift-network-diagnostics/network-check-target-cn4d2"
Jul 12 22:10:33.967367 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:33.967210    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/700bcb2a-4c93-4abe-b403-9b404b3731e3-original-pull-secret\") pod \"global-pull-secret-syncer-b7t5r\" (UID: \"700bcb2a-4c93-4abe-b403-9b404b3731e3\") " pod="kube-system/global-pull-secret-syncer-b7t5r"
Jul 12 22:10:33.967367 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:33.967349    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 12 22:10:33.967479 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:33.967410    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/700bcb2a-4c93-4abe-b403-9b404b3731e3-original-pull-secret podName:700bcb2a-4c93-4abe-b403-9b404b3731e3 nodeName:}" failed. No retries permitted until 2026-07-12 22:10:37.967392093 +0000 UTC m=+10.280895441 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/700bcb2a-4c93-4abe-b403-9b404b3731e3-original-pull-secret") pod "global-pull-secret-syncer-b7t5r" (UID: "700bcb2a-4c93-4abe-b403-9b404b3731e3") : object "kube-system"/"original-pull-secret" not registered
Jul 12 22:10:33.967788 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:33.967756    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 12 22:10:33.967895 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:33.967792    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 12 22:10:33.967895 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:33.967805    2560 projected.go:196] Error preparing data for projected volume kube-api-access-bpv7p for pod openshift-network-diagnostics/network-check-target-cn4d2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 12 22:10:33.967895 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:33.967862    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcf4ee57-809d-4e8b-be7f-593b88689c16-kube-api-access-bpv7p podName:bcf4ee57-809d-4e8b-be7f-593b88689c16 nodeName:}" failed. No retries permitted until 2026-07-12 22:10:37.967838266 +0000 UTC m=+10.281341622 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bpv7p" (UniqueName: "kubernetes.io/projected/bcf4ee57-809d-4e8b-be7f-593b88689c16-kube-api-access-bpv7p") pod "network-check-target-cn4d2" (UID: "bcf4ee57-809d-4e8b-be7f-593b88689c16") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 12 22:10:34.242114 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:34.242037    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-st67z"
Jul 12 22:10:34.242264 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:34.242182    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-st67z" podUID="551bf86c-bd1a-40be-b977-41205da830c2"
Jul 12 22:10:34.242719 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:34.242629    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7t5r"
Jul 12 22:10:34.242836 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:34.242743    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7t5r" podUID="700bcb2a-4c93-4abe-b403-9b404b3731e3"
Jul 12 22:10:34.242836 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:34.242803    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cn4d2"
Jul 12 22:10:34.242943 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:34.242870    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cn4d2" podUID="bcf4ee57-809d-4e8b-be7f-593b88689c16"
Jul 12 22:10:36.240662 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:36.240634    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-st67z"
Jul 12 22:10:36.241028 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:36.240761    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-st67z" podUID="551bf86c-bd1a-40be-b977-41205da830c2"
Jul 12 22:10:36.241028 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:36.240634    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cn4d2"
Jul 12 22:10:36.241028 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:36.240873    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cn4d2" podUID="bcf4ee57-809d-4e8b-be7f-593b88689c16"
Jul 12 22:10:36.241028 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:36.240912    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7t5r"
Jul 12 22:10:36.241028 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:36.240993    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7t5r" podUID="700bcb2a-4c93-4abe-b403-9b404b3731e3"
Jul 12 22:10:37.802817 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:37.802769    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/551bf86c-bd1a-40be-b977-41205da830c2-metrics-certs\") pod \"network-metrics-daemon-st67z\" (UID: \"551bf86c-bd1a-40be-b977-41205da830c2\") " pod="openshift-multus/network-metrics-daemon-st67z"
Jul 12 22:10:37.803284 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:37.802961    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 12 22:10:37.803284 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:37.803029    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/551bf86c-bd1a-40be-b977-41205da830c2-metrics-certs podName:551bf86c-bd1a-40be-b977-41205da830c2 nodeName:}" failed. No retries permitted until 2026-07-12 22:10:45.803009267 +0000 UTC m=+18.116512619 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/551bf86c-bd1a-40be-b977-41205da830c2-metrics-certs") pod "network-metrics-daemon-st67z" (UID: "551bf86c-bd1a-40be-b977-41205da830c2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 12 22:10:38.004828 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:38.004796    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/700bcb2a-4c93-4abe-b403-9b404b3731e3-original-pull-secret\") pod \"global-pull-secret-syncer-b7t5r\" (UID: \"700bcb2a-4c93-4abe-b403-9b404b3731e3\") " pod="kube-system/global-pull-secret-syncer-b7t5r"
Jul 12 22:10:38.005005 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:38.004863    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpv7p\" (UniqueName: \"kubernetes.io/projected/bcf4ee57-809d-4e8b-be7f-593b88689c16-kube-api-access-bpv7p\") pod \"network-check-target-cn4d2\" (UID: \"bcf4ee57-809d-4e8b-be7f-593b88689c16\") " pod="openshift-network-diagnostics/network-check-target-cn4d2"
Jul 12 22:10:38.005069 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:38.005017    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 12 22:10:38.005069 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:38.005037    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 12 22:10:38.005069 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:38.005049    2560 projected.go:196] Error preparing data for projected volume kube-api-access-bpv7p for pod openshift-network-diagnostics/network-check-target-cn4d2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 12 22:10:38.005234 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:38.005107    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcf4ee57-809d-4e8b-be7f-593b88689c16-kube-api-access-bpv7p podName:bcf4ee57-809d-4e8b-be7f-593b88689c16 nodeName:}" failed. No retries permitted until 2026-07-12 22:10:46.00508906 +0000 UTC m=+18.318592407 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bpv7p" (UniqueName: "kubernetes.io/projected/bcf4ee57-809d-4e8b-be7f-593b88689c16-kube-api-access-bpv7p") pod "network-check-target-cn4d2" (UID: "bcf4ee57-809d-4e8b-be7f-593b88689c16") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 12 22:10:38.005471 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:38.005370    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 12 22:10:38.005471 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:38.005421    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/700bcb2a-4c93-4abe-b403-9b404b3731e3-original-pull-secret podName:700bcb2a-4c93-4abe-b403-9b404b3731e3 nodeName:}" failed. No retries permitted until 2026-07-12 22:10:46.005407966 +0000 UTC m=+18.318911298 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/700bcb2a-4c93-4abe-b403-9b404b3731e3-original-pull-secret") pod "global-pull-secret-syncer-b7t5r" (UID: "700bcb2a-4c93-4abe-b403-9b404b3731e3") : object "kube-system"/"original-pull-secret" not registered
Jul 12 22:10:38.240815 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:38.240741    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-st67z"
Jul 12 22:10:38.240988 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:38.240874    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-st67z" podUID="551bf86c-bd1a-40be-b977-41205da830c2"
Jul 12 22:10:38.241276 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:38.241257    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cn4d2"
Jul 12 22:10:38.241379 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:38.241358    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cn4d2" podUID="bcf4ee57-809d-4e8b-be7f-593b88689c16"
Jul 12 22:10:38.241558 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:38.241519    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7t5r"
Jul 12 22:10:38.241685 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:38.241646    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7t5r" podUID="700bcb2a-4c93-4abe-b403-9b404b3731e3"
Jul 12 22:10:40.242728 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:40.242687    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-st67z"
Jul 12 22:10:40.243167 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:40.242824    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-st67z" podUID="551bf86c-bd1a-40be-b977-41205da830c2"
Jul 12 22:10:40.243167 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:40.243102    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cn4d2"
Jul 12 22:10:40.243167 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:40.243160    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cn4d2" podUID="bcf4ee57-809d-4e8b-be7f-593b88689c16"
Jul 12 22:10:40.243331 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:40.243190    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7t5r"
Jul 12 22:10:40.243331 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:40.243238    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7t5r" podUID="700bcb2a-4c93-4abe-b403-9b404b3731e3"
Jul 12 22:10:42.240086 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:42.240047    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7t5r"
Jul 12 22:10:42.240086 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:42.240065    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cn4d2"
Jul 12 22:10:42.240508 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:42.240047    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-st67z"
Jul 12 22:10:42.240508 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:42.240163    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7t5r" podUID="700bcb2a-4c93-4abe-b403-9b404b3731e3"
Jul 12 22:10:42.240508 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:42.240245    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-st67z" podUID="551bf86c-bd1a-40be-b977-41205da830c2"
Jul 12 22:10:42.240508 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:42.240305    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cn4d2" podUID="bcf4ee57-809d-4e8b-be7f-593b88689c16"
Jul 12 22:10:44.240118 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:44.240082    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cn4d2"
Jul 12 22:10:44.240583 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:44.240082    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-st67z"
Jul 12 22:10:44.240583 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:44.240212    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cn4d2" podUID="bcf4ee57-809d-4e8b-be7f-593b88689c16"
Jul 12 22:10:44.240583 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:44.240084    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7t5r"
Jul 12 22:10:44.240583 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:44.240271    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-st67z" podUID="551bf86c-bd1a-40be-b977-41205da830c2"
Jul 12 22:10:44.240583 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:44.240341    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7t5r" podUID="700bcb2a-4c93-4abe-b403-9b404b3731e3"
Jul 12 22:10:45.871409 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:45.871368    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/551bf86c-bd1a-40be-b977-41205da830c2-metrics-certs\") pod \"network-metrics-daemon-st67z\" (UID: \"551bf86c-bd1a-40be-b977-41205da830c2\") " pod="openshift-multus/network-metrics-daemon-st67z"
Jul 12 22:10:45.871925 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:45.871570    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 12 22:10:45.871925 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:45.871653    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/551bf86c-bd1a-40be-b977-41205da830c2-metrics-certs podName:551bf86c-bd1a-40be-b977-41205da830c2 nodeName:}" failed. No retries permitted until 2026-07-12 22:11:01.871630617 +0000 UTC m=+34.185133970 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/551bf86c-bd1a-40be-b977-41205da830c2-metrics-certs") pod "network-metrics-daemon-st67z" (UID: "551bf86c-bd1a-40be-b977-41205da830c2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 12 22:10:46.072670 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:46.072626    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpv7p\" (UniqueName: \"kubernetes.io/projected/bcf4ee57-809d-4e8b-be7f-593b88689c16-kube-api-access-bpv7p\") pod \"network-check-target-cn4d2\" (UID: \"bcf4ee57-809d-4e8b-be7f-593b88689c16\") " pod="openshift-network-diagnostics/network-check-target-cn4d2"
Jul 12 22:10:46.072670 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:46.072674    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/700bcb2a-4c93-4abe-b403-9b404b3731e3-original-pull-secret\") pod \"global-pull-secret-syncer-b7t5r\" (UID: \"700bcb2a-4c93-4abe-b403-9b404b3731e3\") " pod="kube-system/global-pull-secret-syncer-b7t5r"
Jul 12 22:10:46.072843 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:46.072772    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 12 22:10:46.072843 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:46.072797    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 12 22:10:46.072843 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:46.072821    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 12 22:10:46.072843 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:46.072826    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/700bcb2a-4c93-4abe-b403-9b404b3731e3-original-pull-secret podName:700bcb2a-4c93-4abe-b403-9b404b3731e3 nodeName:}" failed. No retries permitted until 2026-07-12 22:11:02.072812974 +0000 UTC m=+34.386316308 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/700bcb2a-4c93-4abe-b403-9b404b3731e3-original-pull-secret") pod "global-pull-secret-syncer-b7t5r" (UID: "700bcb2a-4c93-4abe-b403-9b404b3731e3") : object "kube-system"/"original-pull-secret" not registered
Jul 12 22:10:46.072843 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:46.072830    2560 projected.go:196] Error preparing data for projected volume kube-api-access-bpv7p for pod openshift-network-diagnostics/network-check-target-cn4d2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 12 22:10:46.073050 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:46.072869    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcf4ee57-809d-4e8b-be7f-593b88689c16-kube-api-access-bpv7p podName:bcf4ee57-809d-4e8b-be7f-593b88689c16 nodeName:}" failed. No retries permitted until 2026-07-12 22:11:02.072858192 +0000 UTC m=+34.386361524 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bpv7p" (UniqueName: "kubernetes.io/projected/bcf4ee57-809d-4e8b-be7f-593b88689c16-kube-api-access-bpv7p") pod "network-check-target-cn4d2" (UID: "bcf4ee57-809d-4e8b-be7f-593b88689c16") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 12 22:10:46.240306 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:46.240221    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-st67z"
Jul 12 22:10:46.240464 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:46.240221    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cn4d2"
Jul 12 22:10:46.240464 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:46.240356    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-st67z" podUID="551bf86c-bd1a-40be-b977-41205da830c2"
Jul 12 22:10:46.240464 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:46.240363    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7t5r"
Jul 12 22:10:46.240700 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:46.240483    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cn4d2" podUID="bcf4ee57-809d-4e8b-be7f-593b88689c16"
Jul 12 22:10:46.240700 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:46.240591    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7t5r" podUID="700bcb2a-4c93-4abe-b403-9b404b3731e3"
Jul 12 22:10:48.240389 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:48.240354    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-st67z"
Jul 12 22:10:48.240883 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:48.240478    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-st67z" podUID="551bf86c-bd1a-40be-b977-41205da830c2"
Jul 12 22:10:48.240883 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:48.240495    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cn4d2"
Jul 12 22:10:48.240883 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:48.240664    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cn4d2" podUID="bcf4ee57-809d-4e8b-be7f-593b88689c16"
Jul 12 22:10:48.240883 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:48.240705    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7t5r"
Jul 12 22:10:48.240883 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:48.240769    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7t5r" podUID="700bcb2a-4c93-4abe-b403-9b404b3731e3"
Jul 12 22:10:50.240375 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:50.240332    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-st67z"
Jul 12 22:10:50.240923 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:50.240332    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cn4d2"
Jul 12 22:10:50.240923 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:50.240474    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-st67z" podUID="551bf86c-bd1a-40be-b977-41205da830c2"
Jul 12 22:10:50.240923 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:50.240334    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7t5r"
Jul 12 22:10:50.240923 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:50.240582    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cn4d2" podUID="bcf4ee57-809d-4e8b-be7f-593b88689c16"
Jul 12 22:10:50.240923 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:50.240633    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7t5r" podUID="700bcb2a-4c93-4abe-b403-9b404b3731e3"
Jul 12 22:10:51.483226 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:51.482798    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9c7h" event={"ID":"f1ab5f7c-025e-43c0-893d-039e9d41b889","Type":"ContainerStarted","Data":"532c92737a5a419293fe4cee3f7f5e531470c38cf80e2328dc910285f1e383ab"}
Jul 12 22:10:51.484634 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:51.484613    2560 generic.go:356] "Generic (PLEG): container finished" podID="912f085a-8942-4512-98f9-c8ba189509c2" containerID="7b4b9d51a66414a07d4731a5d64f88dd6a702ca71f186ee0a40dae2fed801dcc" exitCode=0
Jul 12 22:10:51.484732 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:51.484684    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kvvgp" event={"ID":"912f085a-8942-4512-98f9-c8ba189509c2","Type":"ContainerDied","Data":"7b4b9d51a66414a07d4731a5d64f88dd6a702ca71f186ee0a40dae2fed801dcc"}
Jul 12 22:10:51.486388 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:51.486366    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2t5r9" event={"ID":"669e3fb7-aeeb-4746-b4ee-84c83cd5a972","Type":"ContainerStarted","Data":"6ef32bc624c0f7f36630d8cbfd627dbe010e1576b193d161e85708e2ed55d00f"}
Jul 12 22:10:51.488398 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:51.488375    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g4qw4" event={"ID":"05ebf682-f7b0-4267-8a22-46d0c41ecba9","Type":"ContainerStarted","Data":"3c345a80465f915b847b999f3081e762000fcd8fc93d3ae2e0a5191b3770b26f"}
Jul 12 22:10:51.490244 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:51.490224    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9fpmt" event={"ID":"34560420-a512-498c-b8f8-159906bb8638","Type":"ContainerStarted","Data":"a81d4799509fe06896b685198ebcecc256d40f795f2f7aaba1d48e9683dfac45"}
Jul 12 22:10:51.491819 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:51.491796    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lbvpn" event={"ID":"00bab1f2-a13f-47ff-b0c7-c04484a4c85e","Type":"ContainerStarted","Data":"af095f783d643ba8c29d6017f1bc81a312a1322d692f692030d536f3da6e8388"}
Jul 12 22:10:51.493496 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:51.493472    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zkk8b" event={"ID":"c946a409-971a-4760-b311-5278608e31f9","Type":"ContainerStarted","Data":"20fec9451a4ce39e7529f475601505198db71adf7dade1dba3bbeb746ae1593b"}
Jul 12 22:10:51.496644 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:51.496628    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52pbf_1b1f17bc-a992-4c72-8fdc-a33a3c3649c4/ovn-acl-logging/0.log"
Jul 12 22:10:51.497001 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:51.496977    2560 generic.go:356] "Generic (PLEG): container finished" podID="1b1f17bc-a992-4c72-8fdc-a33a3c3649c4" containerID="cd654a09f1d3bc38a2541a813351cedae696f072ac3e5e97000d538f43eb3b6c" exitCode=1
Jul 12 22:10:51.497105 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:51.497023    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-52pbf" event={"ID":"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4","Type":"ContainerStarted","Data":"684ded19ef02d97cbf72ccbe0724c473e0dd4d0403ba1d077f04aa46532d28ce"}
Jul 12 22:10:51.497105 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:51.497078    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-52pbf" event={"ID":"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4","Type":"ContainerStarted","Data":"a7b595a34b6f426730e5c797c4b6119b2896b71d6ac987147f2597248afe65bb"}
Jul 12 22:10:51.497217 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:51.497118    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-52pbf" event={"ID":"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4","Type":"ContainerStarted","Data":"d549edf02a985e66d3fa33732a42ad47e9cf3f37deef4d9853c4dd6d90e11cf3"}
Jul 12 22:10:51.497217 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:51.497162    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-52pbf" event={"ID":"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4","Type":"ContainerStarted","Data":"774f7ac503331576e4057f80070e454e4b746f28c222794d6dcf7e65ceddc657"}
Jul 12 22:10:51.497217 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:51.497199    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-52pbf" event={"ID":"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4","Type":"ContainerDied","Data":"cd654a09f1d3bc38a2541a813351cedae696f072ac3e5e97000d538f43eb3b6c"}
Jul 12 22:10:51.497365 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:51.497237    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-52pbf" event={"ID":"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4","Type":"ContainerStarted","Data":"e29d6d6d7d46944b2e6c3546aa24dc6ddd33830f9c647b27a729b752498cdc34"}
Jul 12 22:10:51.512427 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:51.512370    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-47.ec2.internal" podStartSLOduration=22.512351486 podStartE2EDuration="22.512351486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 22:10:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:10:33.462040498 +0000 UTC m=+5.775543853" watchObservedRunningTime="2026-07-12 22:10:51.512351486 +0000 UTC m=+23.825854844"
Jul 12 22:10:51.528422 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:51.528379    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-g4qw4" podStartSLOduration=4.427263792 podStartE2EDuration="23.528363047s" totalImagesPullingTime="19.101099255s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 22:10:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:10:51.527955465 +0000 UTC m=+23.841458818" watchObservedRunningTime="2026-07-12 22:10:51.528363047 +0000 UTC m=+23.841866401"
Jul 12 22:10:51.543416 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:51.543361    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lbvpn" podStartSLOduration=4.44375213 podStartE2EDuration="23.543345823s" totalImagesPullingTime="19.099593693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:10:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:10:51.542945057 +0000 UTC m=+23.856448411" watchObservedRunningTime="2026-07-12 22:10:51.543345823 +0000 UTC m=+23.856849177"
Jul 12 22:10:51.561081 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:51.561035    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-2t5r9" podStartSLOduration=4.378302752 podStartE2EDuration="23.561015638s" totalImagesPullingTime="19.182712886s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 22:10:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:10:51.560393373 +0000 UTC m=+23.873896727" watchObservedRunningTime="2026-07-12 22:10:51.561015638 +0000 UTC m=+23.874519025"
Jul 12 22:10:51.577174 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:51.577118    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zkk8b" podStartSLOduration=4.484587008 podStartE2EDuration="23.577094468s" totalImagesPullingTime="19.09250746s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 22:10:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:10:51.57673598 +0000 UTC m=+23.890239335" watchObservedRunningTime="2026-07-12 22:10:51.577094468 +0000 UTC m=+23.890597829"
Jul 12 22:10:51.598681 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:51.598635    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9fpmt" podStartSLOduration=4.490591025 podStartE2EDuration="23.598621687s" totalImagesPullingTime="19.108030662s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 22:10:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:10:51.59843628 +0000 UTC m=+23.911939634" watchObservedRunningTime="2026-07-12 22:10:51.598621687 +0000 UTC m=+23.912125074"
Jul 12 22:10:52.117673 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:52.117486    2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 12 22:10:52.191214 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:52.191094    2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-12T22:10:52.117666499Z","UUID":"6a36ea28-a11a-401f-9547-f9d6d8b36f2c","Handler":null,"Name":"","Endpoint":""}
Jul 12 22:10:52.194664 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:52.194633    2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 12 22:10:52.194664 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:52.194667    2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 12 22:10:52.240329 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:52.240295    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cn4d2"
Jul 12 22:10:52.240548 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:52.240419    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-st67z"
Jul 12 22:10:52.240548 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:52.240444    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cn4d2" podUID="bcf4ee57-809d-4e8b-be7f-593b88689c16"
Jul 12 22:10:52.243611 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:52.240916    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7t5r"
Jul 12 22:10:52.243611 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:52.240973    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-st67z" podUID="551bf86c-bd1a-40be-b977-41205da830c2"
Jul 12 22:10:52.243611 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:52.241116    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7t5r" podUID="700bcb2a-4c93-4abe-b403-9b404b3731e3"
Jul 12 22:10:52.501325 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:52.501240    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9c7h" event={"ID":"f1ab5f7c-025e-43c0-893d-039e9d41b889","Type":"ContainerStarted","Data":"20557819d6b44095b15ffa88435943e9fcb412deb2099f9352e25c0c49fafedc"}
Jul 12 22:10:52.502844 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:52.502815    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6qrng" event={"ID":"37f0637a-26e3-4126-b33e-8bd8096369d2","Type":"ContainerStarted","Data":"c8c45ef7aa344ecd3646d70a77d7abd3d4d4e18403c688d354d771b2e69541f6"}
Jul 12 22:10:52.517884 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:52.517833    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6qrng" podStartSLOduration=5.41838591 podStartE2EDuration="24.517820172s" totalImagesPullingTime="19.099434262s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 22:10:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:10:52.51749912 +0000 UTC m=+24.831002473" watchObservedRunningTime="2026-07-12 22:10:52.517820172 +0000 UTC m=+24.831323525"
Jul 12 22:10:53.508406 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:53.508381    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52pbf_1b1f17bc-a992-4c72-8fdc-a33a3c3649c4/ovn-acl-logging/0.log"
Jul 12 22:10:53.509147 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:53.508802    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-52pbf" event={"ID":"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4","Type":"ContainerStarted","Data":"7e68e00cf209a596ba939ced949e80f5d69322d2382ea2490cc7a077e3bc9582"}
Jul 12 22:10:53.510906 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:53.510882    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9c7h" event={"ID":"f1ab5f7c-025e-43c0-893d-039e9d41b889","Type":"ContainerStarted","Data":"b1dd92682a17d9e9428db5ae7ed3e374a7299bee157a352dbc222713e781c170"}
Jul 12 22:10:54.240002 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:54.239972    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cn4d2"
Jul 12 22:10:54.240288 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:54.239972    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7t5r"
Jul 12 22:10:54.240288 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:54.240107    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cn4d2" podUID="bcf4ee57-809d-4e8b-be7f-593b88689c16"
Jul 12 22:10:54.240288 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:54.239972    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-st67z"
Jul 12 22:10:54.240288 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:54.240181    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7t5r" podUID="700bcb2a-4c93-4abe-b403-9b404b3731e3"
Jul 12 22:10:54.240288 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:54.240244    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-st67z" podUID="551bf86c-bd1a-40be-b977-41205da830c2"
Jul 12 22:10:55.656022 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:55.655990    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lbvpn"
Jul 12 22:10:55.656950 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:55.656930    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lbvpn"
Jul 12 22:10:55.690073 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:55.690011    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t9c7h" podStartSLOduration=6.28468714 podStartE2EDuration="27.689989768s" totalImagesPullingTime="21.405302628s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 22:10:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:10:53.551212206 +0000 UTC m=+25.864715561" watchObservedRunningTime="2026-07-12 22:10:55.689989768 +0000 UTC m=+28.003493124"
Jul 12 22:10:56.239960 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:56.239926    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-st67z"
Jul 12 22:10:56.240147 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:56.239927    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7t5r"
Jul 12 22:10:56.240147 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:56.240058    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-st67z" podUID="551bf86c-bd1a-40be-b977-41205da830c2"
Jul 12 22:10:56.240147 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:56.240115    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7t5r" podUID="700bcb2a-4c93-4abe-b403-9b404b3731e3"
Jul 12 22:10:56.240147 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:56.239927    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cn4d2"
Jul 12 22:10:56.240365 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:56.240221    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cn4d2" podUID="bcf4ee57-809d-4e8b-be7f-593b88689c16"
Jul 12 22:10:56.517224 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:56.517062    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lbvpn"
Jul 12 22:10:56.517607 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:56.517594    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lbvpn"
Jul 12 22:10:57.522600 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:57.522575    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52pbf_1b1f17bc-a992-4c72-8fdc-a33a3c3649c4/ovn-acl-logging/0.log"
Jul 12 22:10:57.523201 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:57.522940    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-52pbf" event={"ID":"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4","Type":"ContainerStarted","Data":"3beaa8b311a45258881d7c9b2431ea5918da65f899dbbdfc9b34540f049b2138"}
Jul 12 22:10:57.523269 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:57.523207    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:57.523433 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:57.523415    2560 scope.go:117] "RemoveContainer" containerID="cd654a09f1d3bc38a2541a813351cedae696f072ac3e5e97000d538f43eb3b6c"
Jul 12 22:10:57.524955 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:57.524933    2560 generic.go:356] "Generic (PLEG): container finished" podID="912f085a-8942-4512-98f9-c8ba189509c2" containerID="065dc3b3c928437e65d37fa6f904037fd5b5faf1879463f77b3f083aaa352607" exitCode=0
Jul 12 22:10:57.525057 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:57.525013    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kvvgp" event={"ID":"912f085a-8942-4512-98f9-c8ba189509c2","Type":"ContainerDied","Data":"065dc3b3c928437e65d37fa6f904037fd5b5faf1879463f77b3f083aaa352607"}
Jul 12 22:10:57.541230 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:57.541206    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:58.241447 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:58.241266    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cn4d2"
Jul 12 22:10:58.241747 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:58.241312    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-st67z"
Jul 12 22:10:58.241747 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:58.241519    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cn4d2" podUID="bcf4ee57-809d-4e8b-be7f-593b88689c16"
Jul 12 22:10:58.241747 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:58.241655    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-st67z" podUID="551bf86c-bd1a-40be-b977-41205da830c2"
Jul 12 22:10:58.241747 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:58.241337    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7t5r"
Jul 12 22:10:58.241747 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:58.241732    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7t5r" podUID="700bcb2a-4c93-4abe-b403-9b404b3731e3"
Jul 12 22:10:58.532193 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:58.532163    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52pbf_1b1f17bc-a992-4c72-8fdc-a33a3c3649c4/ovn-acl-logging/0.log"
Jul 12 22:10:58.532710 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:58.532683    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-52pbf" event={"ID":"1b1f17bc-a992-4c72-8fdc-a33a3c3649c4","Type":"ContainerStarted","Data":"886264ee0d40c052c0b8906e5359099b95cb985acef6407559a9ef5b44417ded"}
Jul 12 22:10:58.532835 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:58.532802    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:58.532901 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:58.532850    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:58.553820 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:58.553793    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:10:58.568319 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:58.568267    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-52pbf" podStartSLOduration=11.398151579 podStartE2EDuration="30.568251974s" totalImagesPullingTime="19.170100395s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 22:10:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:10:58.566687913 +0000 UTC m=+30.880191268" watchObservedRunningTime="2026-07-12 22:10:58.568251974 +0000 UTC m=+30.881755327"
Jul 12 22:10:58.851610 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:58.851499    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cn4d2"]
Jul 12 22:10:58.851773 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:58.851680    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cn4d2"
Jul 12 22:10:58.851838 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:58.851793    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cn4d2" podUID="bcf4ee57-809d-4e8b-be7f-593b88689c16"
Jul 12 22:10:58.857129 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:58.857104    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b7t5r"]
Jul 12 22:10:58.857316 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:58.857291    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7t5r"
Jul 12 22:10:58.857420 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:58.857399    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7t5r" podUID="700bcb2a-4c93-4abe-b403-9b404b3731e3"
Jul 12 22:10:58.883019 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:58.882989    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-st67z"]
Jul 12 22:10:58.883182 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:58.883110    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-st67z"
Jul 12 22:10:58.883247 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:10:58.883204    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-st67z" podUID="551bf86c-bd1a-40be-b977-41205da830c2"
Jul 12 22:10:59.538191 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:59.538144    2560 generic.go:356] "Generic (PLEG): container finished" podID="912f085a-8942-4512-98f9-c8ba189509c2" containerID="3083e3980c5ca52d878d715097c406ede25c5acd9669f30e0a4c6f792a28a748" exitCode=0
Jul 12 22:10:59.538739 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:10:59.538220    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kvvgp" event={"ID":"912f085a-8942-4512-98f9-c8ba189509c2","Type":"ContainerDied","Data":"3083e3980c5ca52d878d715097c406ede25c5acd9669f30e0a4c6f792a28a748"}
Jul 12 22:11:00.240150 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:00.240119    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-st67z"
Jul 12 22:11:00.240293 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:00.240161    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7t5r"
Jul 12 22:11:00.240293 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:00.240254    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-st67z" podUID="551bf86c-bd1a-40be-b977-41205da830c2"
Jul 12 22:11:00.240373 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:00.240322    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7t5r" podUID="700bcb2a-4c93-4abe-b403-9b404b3731e3"
Jul 12 22:11:01.239896 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:01.239704    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cn4d2"
Jul 12 22:11:01.240382 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:01.239974    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cn4d2" podUID="bcf4ee57-809d-4e8b-be7f-593b88689c16"
Jul 12 22:11:01.544760 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:01.544674    2560 generic.go:356] "Generic (PLEG): container finished" podID="912f085a-8942-4512-98f9-c8ba189509c2" containerID="f1c07a0553192306e9565df31a1132ab2f3fa4aeb2912d973e266fb8a290c8a1" exitCode=0
Jul 12 22:11:01.544760 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:01.544731    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kvvgp" event={"ID":"912f085a-8942-4512-98f9-c8ba189509c2","Type":"ContainerDied","Data":"f1c07a0553192306e9565df31a1132ab2f3fa4aeb2912d973e266fb8a290c8a1"}
Jul 12 22:11:01.890727 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:01.890599    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/551bf86c-bd1a-40be-b977-41205da830c2-metrics-certs\") pod \"network-metrics-daemon-st67z\" (UID: \"551bf86c-bd1a-40be-b977-41205da830c2\") " pod="openshift-multus/network-metrics-daemon-st67z"
Jul 12 22:11:01.890906 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:01.890768    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 12 22:11:01.890906 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:01.890849    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/551bf86c-bd1a-40be-b977-41205da830c2-metrics-certs podName:551bf86c-bd1a-40be-b977-41205da830c2 nodeName:}" failed. No retries permitted until 2026-07-12 22:11:33.890828393 +0000 UTC m=+66.204331731 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/551bf86c-bd1a-40be-b977-41205da830c2-metrics-certs") pod "network-metrics-daemon-st67z" (UID: "551bf86c-bd1a-40be-b977-41205da830c2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 12 22:11:02.092210 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:02.092168    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpv7p\" (UniqueName: \"kubernetes.io/projected/bcf4ee57-809d-4e8b-be7f-593b88689c16-kube-api-access-bpv7p\") pod \"network-check-target-cn4d2\" (UID: \"bcf4ee57-809d-4e8b-be7f-593b88689c16\") " pod="openshift-network-diagnostics/network-check-target-cn4d2"
Jul 12 22:11:02.092374 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:02.092222    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/700bcb2a-4c93-4abe-b403-9b404b3731e3-original-pull-secret\") pod \"global-pull-secret-syncer-b7t5r\" (UID: \"700bcb2a-4c93-4abe-b403-9b404b3731e3\") " pod="kube-system/global-pull-secret-syncer-b7t5r"
Jul 12 22:11:02.092374 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:02.092318    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 12 22:11:02.092374 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:02.092336    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 12 22:11:02.092374 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:02.092361    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 12 22:11:02.092374 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:02.092371    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/700bcb2a-4c93-4abe-b403-9b404b3731e3-original-pull-secret podName:700bcb2a-4c93-4abe-b403-9b404b3731e3 nodeName:}" failed. No retries permitted until 2026-07-12 22:11:34.092357835 +0000 UTC m=+66.405861171 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/700bcb2a-4c93-4abe-b403-9b404b3731e3-original-pull-secret") pod "global-pull-secret-syncer-b7t5r" (UID: "700bcb2a-4c93-4abe-b403-9b404b3731e3") : object "kube-system"/"original-pull-secret" not registered
Jul 12 22:11:02.092374 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:02.092373    2560 projected.go:196] Error preparing data for projected volume kube-api-access-bpv7p for pod openshift-network-diagnostics/network-check-target-cn4d2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 12 22:11:02.092596 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:02.092417    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bcf4ee57-809d-4e8b-be7f-593b88689c16-kube-api-access-bpv7p podName:bcf4ee57-809d-4e8b-be7f-593b88689c16 nodeName:}" failed. No retries permitted until 2026-07-12 22:11:34.092404326 +0000 UTC m=+66.405907659 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bpv7p" (UniqueName: "kubernetes.io/projected/bcf4ee57-809d-4e8b-be7f-593b88689c16-kube-api-access-bpv7p") pod "network-check-target-cn4d2" (UID: "bcf4ee57-809d-4e8b-be7f-593b88689c16") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 12 22:11:02.240350 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:02.240268    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-st67z"
Jul 12 22:11:02.240905 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:02.240365    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7t5r"
Jul 12 22:11:02.241110 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:02.241076    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-st67z" podUID="551bf86c-bd1a-40be-b977-41205da830c2"
Jul 12 22:11:02.241401 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:02.241371    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7t5r" podUID="700bcb2a-4c93-4abe-b403-9b404b3731e3"
Jul 12 22:11:03.239381 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:03.239347    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cn4d2"
Jul 12 22:11:03.239597 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:03.239466    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cn4d2" podUID="bcf4ee57-809d-4e8b-be7f-593b88689c16"
Jul 12 22:11:04.239388 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:04.239362    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-st67z"
Jul 12 22:11:04.239826 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:04.239494    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-st67z" podUID="551bf86c-bd1a-40be-b977-41205da830c2"
Jul 12 22:11:04.239826 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:04.239559    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7t5r"
Jul 12 22:11:04.239826 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:04.239671    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b7t5r" podUID="700bcb2a-4c93-4abe-b403-9b404b3731e3"
Jul 12 22:11:04.440729 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:04.440647    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-47.ec2.internal" event="NodeReady"
Jul 12 22:11:04.440912 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:04.440800    2560 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 12 22:11:04.490051 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:04.490018    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jhqcn"]
Jul 12 22:11:04.493979 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:04.493952    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jhqcn"
Jul 12 22:11:04.497156 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:04.497128    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8965j"]
Jul 12 22:11:04.500367 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:04.499627    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 12 22:11:04.500367 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:04.500108    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9r9hf\""
Jul 12 22:11:04.500751 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:04.500728    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 12 22:11:04.504479 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:04.504451    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8965j"
Jul 12 22:11:04.507322 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:04.507302    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 12 22:11:04.507457 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:04.507437    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 12 22:11:04.507630 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:04.507387    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-nwvjz\""
Jul 12 22:11:04.507734 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:04.507318    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 12 22:11:04.513626 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:04.513523    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jhqcn"]
Jul 12 22:11:04.514398 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:04.514375    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8965j"]
Jul 12 22:11:04.609008 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:04.608969    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n6qt\" (UniqueName: \"kubernetes.io/projected/fd201c7d-6b11-430a-b079-9a16e001c434-kube-api-access-4n6qt\") pod \"ingress-canary-8965j\" (UID: \"fd201c7d-6b11-430a-b079-9a16e001c434\") " pod="openshift-ingress-canary/ingress-canary-8965j"
Jul 12 22:11:04.609193 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:04.609014    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e3e0d19f-0837-473f-a4eb-89df81150888-config-volume\") pod \"dns-default-jhqcn\" (UID: \"e3e0d19f-0837-473f-a4eb-89df81150888\") " pod="openshift-dns/dns-default-jhqcn"
Jul 12 22:11:04.609193 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:04.609053    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqg64\" (UniqueName: \"kubernetes.io/projected/e3e0d19f-0837-473f-a4eb-89df81150888-kube-api-access-dqg64\") pod \"dns-default-jhqcn\" (UID: \"e3e0d19f-0837-473f-a4eb-89df81150888\") " pod="openshift-dns/dns-default-jhqcn"
Jul 12 22:11:04.609193 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:04.609085    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3e0d19f-0837-473f-a4eb-89df81150888-metrics-tls\") pod \"dns-default-jhqcn\" (UID: \"e3e0d19f-0837-473f-a4eb-89df81150888\") " pod="openshift-dns/dns-default-jhqcn"
Jul 12 22:11:04.609193 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:04.609111    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3e0d19f-0837-473f-a4eb-89df81150888-tmp-dir\") pod \"dns-default-jhqcn\" (UID: \"e3e0d19f-0837-473f-a4eb-89df81150888\") " pod="openshift-dns/dns-default-jhqcn"
Jul 12 22:11:04.609193 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:04.609147    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd201c7d-6b11-430a-b079-9a16e001c434-cert\") pod \"ingress-canary-8965j\" (UID: \"fd201c7d-6b11-430a-b079-9a16e001c434\") " pod="openshift-ingress-canary/ingress-canary-8965j"
Jul 12 22:11:04.709985 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:04.709896    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n6qt\" (UniqueName: \"kubernetes.io/projected/fd201c7d-6b11-430a-b079-9a16e001c434-kube-api-access-4n6qt\") pod \"ingress-canary-8965j\" (UID: \"fd201c7d-6b11-430a-b079-9a16e001c434\") " pod="openshift-ingress-canary/ingress-canary-8965j"
Jul 12 22:11:04.709985 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:04.709937    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e3e0d19f-0837-473f-a4eb-89df81150888-config-volume\") pod \"dns-default-jhqcn\" (UID: \"e3e0d19f-0837-473f-a4eb-89df81150888\") " pod="openshift-dns/dns-default-jhqcn"
Jul 12 22:11:04.709985 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:04.709958    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqg64\" (UniqueName: \"kubernetes.io/projected/e3e0d19f-0837-473f-a4eb-89df81150888-kube-api-access-dqg64\") pod \"dns-default-jhqcn\" (UID: \"e3e0d19f-0837-473f-a4eb-89df81150888\") " pod="openshift-dns/dns-default-jhqcn"
Jul 12 22:11:04.710246 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:04.710189    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3e0d19f-0837-473f-a4eb-89df81150888-metrics-tls\") pod \"dns-default-jhqcn\" (UID: \"e3e0d19f-0837-473f-a4eb-89df81150888\") " pod="openshift-dns/dns-default-jhqcn"
Jul 12 22:11:04.710246 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:04.710243    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3e0d19f-0837-473f-a4eb-89df81150888-tmp-dir\") pod \"dns-default-jhqcn\" (UID: \"e3e0d19f-0837-473f-a4eb-89df81150888\") " pod="openshift-dns/dns-default-jhqcn"
Jul 12 22:11:04.710349 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:04.710287    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd201c7d-6b11-430a-b079-9a16e001c434-cert\") pod \"ingress-canary-8965j\" (UID: \"fd201c7d-6b11-430a-b079-9a16e001c434\") " pod="openshift-ingress-canary/ingress-canary-8965j"
Jul 12 22:11:04.710349 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:04.710330    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 12 22:11:04.710443 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:04.710386    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3e0d19f-0837-473f-a4eb-89df81150888-metrics-tls podName:e3e0d19f-0837-473f-a4eb-89df81150888 nodeName:}" failed. No retries permitted until 2026-07-12 22:11:05.210372175 +0000 UTC m=+37.523875507 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e3e0d19f-0837-473f-a4eb-89df81150888-metrics-tls") pod "dns-default-jhqcn" (UID: "e3e0d19f-0837-473f-a4eb-89df81150888") : secret "dns-default-metrics-tls" not found
Jul 12 22:11:04.710605 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:04.710577    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3e0d19f-0837-473f-a4eb-89df81150888-tmp-dir\") pod \"dns-default-jhqcn\" (UID: \"e3e0d19f-0837-473f-a4eb-89df81150888\") " pod="openshift-dns/dns-default-jhqcn"
Jul 12 22:11:04.710745 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:04.710644    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e3e0d19f-0837-473f-a4eb-89df81150888-config-volume\") pod \"dns-default-jhqcn\" (UID: \"e3e0d19f-0837-473f-a4eb-89df81150888\") " pod="openshift-dns/dns-default-jhqcn"
Jul 12 22:11:04.715382 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:04.715359    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd201c7d-6b11-430a-b079-9a16e001c434-cert\") pod \"ingress-canary-8965j\" (UID: \"fd201c7d-6b11-430a-b079-9a16e001c434\") " pod="openshift-ingress-canary/ingress-canary-8965j"
Jul 12 22:11:04.728304 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:04.728282    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n6qt\" (UniqueName: \"kubernetes.io/projected/fd201c7d-6b11-430a-b079-9a16e001c434-kube-api-access-4n6qt\") pod \"ingress-canary-8965j\" (UID: \"fd201c7d-6b11-430a-b079-9a16e001c434\") " pod="openshift-ingress-canary/ingress-canary-8965j"
Jul 12 22:11:04.734739 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:04.734713    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqg64\" (UniqueName: \"kubernetes.io/projected/e3e0d19f-0837-473f-a4eb-89df81150888-kube-api-access-dqg64\") pod \"dns-default-jhqcn\" (UID: \"e3e0d19f-0837-473f-a4eb-89df81150888\") " pod="openshift-dns/dns-default-jhqcn"
Jul 12 22:11:04.817818 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:04.817784    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8965j"
Jul 12 22:11:05.214346 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.214304    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3e0d19f-0837-473f-a4eb-89df81150888-metrics-tls\") pod \"dns-default-jhqcn\" (UID: \"e3e0d19f-0837-473f-a4eb-89df81150888\") " pod="openshift-dns/dns-default-jhqcn"
Jul 12 22:11:05.214526 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:05.214486    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 12 22:11:05.214634 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:05.214620    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3e0d19f-0837-473f-a4eb-89df81150888-metrics-tls podName:e3e0d19f-0837-473f-a4eb-89df81150888 nodeName:}" failed. No retries permitted until 2026-07-12 22:11:06.214597498 +0000 UTC m=+38.528100832 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e3e0d19f-0837-473f-a4eb-89df81150888-metrics-tls") pod "dns-default-jhqcn" (UID: "e3e0d19f-0837-473f-a4eb-89df81150888") : secret "dns-default-metrics-tls" not found
Jul 12 22:11:05.239580 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.239543    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cn4d2"
Jul 12 22:11:05.242549 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.242501    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bgjrf\""
Jul 12 22:11:05.242693 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.242582    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 12 22:11:05.243709 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.243688    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 12 22:11:05.777194 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.777166    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gf2x8"]
Jul 12 22:11:05.780570 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.780548    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gf2x8"
Jul 12 22:11:05.783415 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.783393    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-tghfc\""
Jul 12 22:11:05.783514 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.783419    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 12 22:11:05.784438 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.784413    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 12 22:11:05.791515 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.791494    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gf2x8"]
Jul 12 22:11:05.815867 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.815837    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-c9nrk"]
Jul 12 22:11:05.818691 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.818664    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-c9nrk"
Jul 12 22:11:05.823294 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.822731    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-fkzdt\""
Jul 12 22:11:05.827134 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.827109    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dqc2v"]
Jul 12 22:11:05.830053 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.830032    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dqc2v"
Jul 12 22:11:05.833380 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.833286    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 12 22:11:05.834905 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.833750    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 12 22:11:05.834905 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.833976    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 12 22:11:05.834905 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.834416    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 12 22:11:05.834905 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.834725    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-8s529\""
Jul 12 22:11:05.834905 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.834829    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-c9nrk"]
Jul 12 22:11:05.835874 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.835835    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-248f2"]
Jul 12 22:11:05.840005 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.839984    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7cb4cc88b6-fscqr"]
Jul 12 22:11:05.843560 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.843501    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-248f2"
Jul 12 22:11:05.846122 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.845282    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7cb4cc88b6-fscqr"
Jul 12 22:11:05.849358 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.849294    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dqc2v"]
Jul 12 22:11:05.852065 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.851461    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 12 22:11:05.852065 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.851725    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 12 22:11:05.852065 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.851940    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 12 22:11:05.853390 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.853335    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 12 22:11:05.854111 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.853819    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-wdpmv\""
Jul 12 22:11:05.855081 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.854878    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-c7pws\""
Jul 12 22:11:05.855559 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.855246    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 12 22:11:05.855647 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.855558    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 12 22:11:05.855708 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.855659    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 12 22:11:05.856515 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.855846    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 12 22:11:05.857049 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.856523    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 12 22:11:05.860431 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.860404    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 12 22:11:05.864275 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.864254    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7cb4cc88b6-fscqr"]
Jul 12 22:11:05.865481 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.865462    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-248f2"]
Jul 12 22:11:05.920165 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.920135    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjx4d\" (UniqueName: \"kubernetes.io/projected/772a2a08-9010-4063-ba53-040e1e5881d1-kube-api-access-vjx4d\") pod \"volume-data-source-validator-78495597f6-gf2x8\" (UID: \"772a2a08-9010-4063-ba53-040e1e5881d1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gf2x8"
Jul 12 22:11:05.920268 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.920210    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f91f26b5-1055-4423-a442-da2443b2d44e-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-dqc2v\" (UID: \"f91f26b5-1055-4423-a442-da2443b2d44e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dqc2v"
Jul 12 22:11:05.920268 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.920240    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f91f26b5-1055-4423-a442-da2443b2d44e-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-dqc2v\" (UID: \"f91f26b5-1055-4423-a442-da2443b2d44e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dqc2v"
Jul 12 22:11:05.920373 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.920320    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx9tp\" (UniqueName: \"kubernetes.io/projected/f91f26b5-1055-4423-a442-da2443b2d44e-kube-api-access-mx9tp\") pod \"kube-storage-version-migrator-operator-799cb7f958-dqc2v\" (UID: \"f91f26b5-1055-4423-a442-da2443b2d44e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dqc2v"
Jul 12 22:11:05.920423 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.920393    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xt77\" (UniqueName: \"kubernetes.io/projected/aa72db1a-a0ab-45e1-b5e4-c4ebb5de7640-kube-api-access-7xt77\") pod \"network-check-source-869dcfcdcd-c9nrk\" (UID: \"aa72db1a-a0ab-45e1-b5e4-c4ebb5de7640\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-c9nrk"
Jul 12 22:11:05.925288 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.925263    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-6zwcc"]
Jul 12 22:11:05.930744 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.930717    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-6zwcc"
Jul 12 22:11:05.932996 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.932973    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8965j"]
Jul 12 22:11:05.934213 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.934013    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 12 22:11:05.934213 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.934066    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 12 22:11:05.934213 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.934088    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-kcpnw\""
Jul 12 22:11:05.934391 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.934224    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 12 22:11:05.938524 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:11:05.938503    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd201c7d_6b11_430a_b079_9a16e001c434.slice/crio-0b667fa8c904a56d6f135137810cf88a23194eb2bd3b4b31e3c93fbee97d2dee WatchSource:0}: Error finding container 0b667fa8c904a56d6f135137810cf88a23194eb2bd3b4b31e3c93fbee97d2dee: Status 404 returned error can't find the container with id 0b667fa8c904a56d6f135137810cf88a23194eb2bd3b4b31e3c93fbee97d2dee
Jul 12 22:11:05.945555 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:05.945501    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-6zwcc"]
Jul 12 22:11:06.021776 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.021551    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/43823729-01b0-480f-83bd-faf0b86abcdc-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-6zwcc\" (UID: \"43823729-01b0-480f-83bd-faf0b86abcdc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-6zwcc"
Jul 12 22:11:06.021776 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.021763    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjx4d\" (UniqueName: \"kubernetes.io/projected/772a2a08-9010-4063-ba53-040e1e5881d1-kube-api-access-vjx4d\") pod \"volume-data-source-validator-78495597f6-gf2x8\" (UID: \"772a2a08-9010-4063-ba53-040e1e5881d1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gf2x8"
Jul 12 22:11:06.021983 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.021787    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc370348-fbc2-4e67-96ce-a1129336bf21-metrics-certs\") pod \"router-default-7cb4cc88b6-fscqr\" (UID: \"bc370348-fbc2-4e67-96ce-a1129336bf21\") " pod="openshift-ingress/router-default-7cb4cc88b6-fscqr"
Jul 12 22:11:06.021983 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.021819    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f91f26b5-1055-4423-a442-da2443b2d44e-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-dqc2v\" (UID: \"f91f26b5-1055-4423-a442-da2443b2d44e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dqc2v"
Jul 12 22:11:06.021983 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.021836    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-248f2\" (UID: \"25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-248f2"
Jul 12 22:11:06.021983 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.021900    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f91f26b5-1055-4423-a442-da2443b2d44e-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-dqc2v\" (UID: \"f91f26b5-1055-4423-a442-da2443b2d44e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dqc2v"
Jul 12 22:11:06.021983 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.021933    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bc370348-fbc2-4e67-96ce-a1129336bf21-stats-auth\") pod \"router-default-7cb4cc88b6-fscqr\" (UID: \"bc370348-fbc2-4e67-96ce-a1129336bf21\") " pod="openshift-ingress/router-default-7cb4cc88b6-fscqr"
Jul 12 22:11:06.021983 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.021955    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmzwm\" (UniqueName: \"kubernetes.io/projected/43823729-01b0-480f-83bd-faf0b86abcdc-kube-api-access-tmzwm\") pod \"cluster-samples-operator-65db858894-6zwcc\" (UID: \"43823729-01b0-480f-83bd-faf0b86abcdc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-6zwcc"
Jul 12 22:11:06.021983 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.021980    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-248f2\" (UID: \"25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-248f2"
Jul 12 22:11:06.022311 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.022014    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkwc7\" (UniqueName: \"kubernetes.io/projected/25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa-kube-api-access-wkwc7\") pod \"cluster-monitoring-operator-5f7f494dd4-248f2\" (UID: \"25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-248f2"
Jul 12 22:11:06.022311 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.022052    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs8zt\" (UniqueName: \"kubernetes.io/projected/bc370348-fbc2-4e67-96ce-a1129336bf21-kube-api-access-rs8zt\") pod \"router-default-7cb4cc88b6-fscqr\" (UID: \"bc370348-fbc2-4e67-96ce-a1129336bf21\") " pod="openshift-ingress/router-default-7cb4cc88b6-fscqr"
Jul 12 22:11:06.022311 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.022120    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bc370348-fbc2-4e67-96ce-a1129336bf21-default-certificate\") pod \"router-default-7cb4cc88b6-fscqr\" (UID: \"bc370348-fbc2-4e67-96ce-a1129336bf21\") " pod="openshift-ingress/router-default-7cb4cc88b6-fscqr"
Jul 12 22:11:06.022311 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.022168    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx9tp\" (UniqueName: \"kubernetes.io/projected/f91f26b5-1055-4423-a442-da2443b2d44e-kube-api-access-mx9tp\") pod \"kube-storage-version-migrator-operator-799cb7f958-dqc2v\" (UID: \"f91f26b5-1055-4423-a442-da2443b2d44e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dqc2v"
Jul 12 22:11:06.022311 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.022236    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xt77\" (UniqueName: \"kubernetes.io/projected/aa72db1a-a0ab-45e1-b5e4-c4ebb5de7640-kube-api-access-7xt77\") pod \"network-check-source-869dcfcdcd-c9nrk\" (UID: \"aa72db1a-a0ab-45e1-b5e4-c4ebb5de7640\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-c9nrk"
Jul 12 22:11:06.022311 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.022255    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc370348-fbc2-4e67-96ce-a1129336bf21-service-ca-bundle\") pod \"router-default-7cb4cc88b6-fscqr\" (UID: \"bc370348-fbc2-4e67-96ce-a1129336bf21\") " pod="openshift-ingress/router-default-7cb4cc88b6-fscqr"
Jul 12 22:11:06.022738 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.022716    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f91f26b5-1055-4423-a442-da2443b2d44e-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-dqc2v\" (UID: \"f91f26b5-1055-4423-a442-da2443b2d44e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dqc2v"
Jul 12 22:11:06.025642 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.025616    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f91f26b5-1055-4423-a442-da2443b2d44e-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-dqc2v\" (UID: \"f91f26b5-1055-4423-a442-da2443b2d44e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dqc2v"
Jul 12 22:11:06.033732 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.033704    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xt77\" (UniqueName: \"kubernetes.io/projected/aa72db1a-a0ab-45e1-b5e4-c4ebb5de7640-kube-api-access-7xt77\") pod \"network-check-source-869dcfcdcd-c9nrk\" (UID: \"aa72db1a-a0ab-45e1-b5e4-c4ebb5de7640\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-c9nrk"
Jul 12 22:11:06.034172 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.034152    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx9tp\" (UniqueName: \"kubernetes.io/projected/f91f26b5-1055-4423-a442-da2443b2d44e-kube-api-access-mx9tp\") pod \"kube-storage-version-migrator-operator-799cb7f958-dqc2v\" (UID: \"f91f26b5-1055-4423-a442-da2443b2d44e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dqc2v"
Jul 12 22:11:06.034277 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.034187    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjx4d\" (UniqueName: \"kubernetes.io/projected/772a2a08-9010-4063-ba53-040e1e5881d1-kube-api-access-vjx4d\") pod \"volume-data-source-validator-78495597f6-gf2x8\" (UID: \"772a2a08-9010-4063-ba53-040e1e5881d1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gf2x8"
Jul 12 22:11:06.090319 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.090285    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gf2x8"
Jul 12 22:11:06.122626 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.122580    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc370348-fbc2-4e67-96ce-a1129336bf21-service-ca-bundle\") pod \"router-default-7cb4cc88b6-fscqr\" (UID: \"bc370348-fbc2-4e67-96ce-a1129336bf21\") " pod="openshift-ingress/router-default-7cb4cc88b6-fscqr"
Jul 12 22:11:06.122626 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.122632    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/43823729-01b0-480f-83bd-faf0b86abcdc-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-6zwcc\" (UID: \"43823729-01b0-480f-83bd-faf0b86abcdc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-6zwcc"
Jul 12 22:11:06.122842 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.122696    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc370348-fbc2-4e67-96ce-a1129336bf21-metrics-certs\") pod \"router-default-7cb4cc88b6-fscqr\" (UID: \"bc370348-fbc2-4e67-96ce-a1129336bf21\") " pod="openshift-ingress/router-default-7cb4cc88b6-fscqr"
Jul 12 22:11:06.122842 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:06.122743    2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 12 22:11:06.122842 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:06.122760    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc370348-fbc2-4e67-96ce-a1129336bf21-service-ca-bundle podName:bc370348-fbc2-4e67-96ce-a1129336bf21 nodeName:}" failed. No retries permitted until 2026-07-12 22:11:06.622741534 +0000 UTC m=+38.936244865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bc370348-fbc2-4e67-96ce-a1129336bf21-service-ca-bundle") pod "router-default-7cb4cc88b6-fscqr" (UID: "bc370348-fbc2-4e67-96ce-a1129336bf21") : configmap references non-existent config key: service-ca.crt
Jul 12 22:11:06.122842 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:06.122799    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43823729-01b0-480f-83bd-faf0b86abcdc-samples-operator-tls podName:43823729-01b0-480f-83bd-faf0b86abcdc nodeName:}" failed. No retries permitted until 2026-07-12 22:11:06.62278357 +0000 UTC m=+38.936286915 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/43823729-01b0-480f-83bd-faf0b86abcdc-samples-operator-tls") pod "cluster-samples-operator-65db858894-6zwcc" (UID: "43823729-01b0-480f-83bd-faf0b86abcdc") : secret "samples-operator-tls" not found
Jul 12 22:11:06.122842 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.122744    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-248f2\" (UID: \"25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-248f2"
Jul 12 22:11:06.122842 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.122829    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bc370348-fbc2-4e67-96ce-a1129336bf21-stats-auth\") pod \"router-default-7cb4cc88b6-fscqr\" (UID: \"bc370348-fbc2-4e67-96ce-a1129336bf21\") " pod="openshift-ingress/router-default-7cb4cc88b6-fscqr"
Jul 12 22:11:06.123103 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.122851    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmzwm\" (UniqueName: \"kubernetes.io/projected/43823729-01b0-480f-83bd-faf0b86abcdc-kube-api-access-tmzwm\") pod \"cluster-samples-operator-65db858894-6zwcc\" (UID: \"43823729-01b0-480f-83bd-faf0b86abcdc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-6zwcc"
Jul 12 22:11:06.123103 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.122871    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-248f2\" (UID: \"25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-248f2"
Jul 12 22:11:06.123103 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.122896    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkwc7\" (UniqueName: \"kubernetes.io/projected/25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa-kube-api-access-wkwc7\") pod \"cluster-monitoring-operator-5f7f494dd4-248f2\" (UID: \"25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-248f2"
Jul 12 22:11:06.123103 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.122930    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs8zt\" (UniqueName: \"kubernetes.io/projected/bc370348-fbc2-4e67-96ce-a1129336bf21-kube-api-access-rs8zt\") pod \"router-default-7cb4cc88b6-fscqr\" (UID: \"bc370348-fbc2-4e67-96ce-a1129336bf21\") " pod="openshift-ingress/router-default-7cb4cc88b6-fscqr"
Jul 12 22:11:06.123103 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:06.122801    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 12 22:11:06.123103 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.122987    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bc370348-fbc2-4e67-96ce-a1129336bf21-default-certificate\") pod \"router-default-7cb4cc88b6-fscqr\" (UID: \"bc370348-fbc2-4e67-96ce-a1129336bf21\") " pod="openshift-ingress/router-default-7cb4cc88b6-fscqr"
Jul 12 22:11:06.123103 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:06.123027    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa-cluster-monitoring-operator-tls podName:25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa nodeName:}" failed. No retries permitted until 2026-07-12 22:11:06.623016985 +0000 UTC m=+38.936520318 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-248f2" (UID: "25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa") : secret "cluster-monitoring-operator-tls" not found
Jul 12 22:11:06.123103 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:06.122810    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 12 22:11:06.123103 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:06.123085    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc370348-fbc2-4e67-96ce-a1129336bf21-metrics-certs podName:bc370348-fbc2-4e67-96ce-a1129336bf21 nodeName:}" failed. No retries permitted until 2026-07-12 22:11:06.623070772 +0000 UTC m=+38.936574104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bc370348-fbc2-4e67-96ce-a1129336bf21-metrics-certs") pod "router-default-7cb4cc88b6-fscqr" (UID: "bc370348-fbc2-4e67-96ce-a1129336bf21") : secret "router-metrics-certs-default" not found
Jul 12 22:11:06.123684 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.123663    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-248f2\" (UID: \"25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-248f2"
Jul 12 22:11:06.125870 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.125852    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bc370348-fbc2-4e67-96ce-a1129336bf21-default-certificate\") pod \"router-default-7cb4cc88b6-fscqr\" (UID: \"bc370348-fbc2-4e67-96ce-a1129336bf21\") " pod="openshift-ingress/router-default-7cb4cc88b6-fscqr"
Jul 12 22:11:06.125968 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.125935    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bc370348-fbc2-4e67-96ce-a1129336bf21-stats-auth\") pod \"router-default-7cb4cc88b6-fscqr\" (UID: \"bc370348-fbc2-4e67-96ce-a1129336bf21\") " pod="openshift-ingress/router-default-7cb4cc88b6-fscqr"
Jul 12 22:11:06.130207 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.130183    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-c9nrk"
Jul 12 22:11:06.134066 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.134043    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs8zt\" (UniqueName: \"kubernetes.io/projected/bc370348-fbc2-4e67-96ce-a1129336bf21-kube-api-access-rs8zt\") pod \"router-default-7cb4cc88b6-fscqr\" (UID: \"bc370348-fbc2-4e67-96ce-a1129336bf21\") " pod="openshift-ingress/router-default-7cb4cc88b6-fscqr"
Jul 12 22:11:06.134223 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.134205    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmzwm\" (UniqueName: \"kubernetes.io/projected/43823729-01b0-480f-83bd-faf0b86abcdc-kube-api-access-tmzwm\") pod \"cluster-samples-operator-65db858894-6zwcc\" (UID: \"43823729-01b0-480f-83bd-faf0b86abcdc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-6zwcc"
Jul 12 22:11:06.134323 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.134302    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkwc7\" (UniqueName: \"kubernetes.io/projected/25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa-kube-api-access-wkwc7\") pod \"cluster-monitoring-operator-5f7f494dd4-248f2\" (UID: \"25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-248f2"
Jul 12 22:11:06.154264 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.154233    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dqc2v"
Jul 12 22:11:06.225041 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.224388    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3e0d19f-0837-473f-a4eb-89df81150888-metrics-tls\") pod \"dns-default-jhqcn\" (UID: \"e3e0d19f-0837-473f-a4eb-89df81150888\") " pod="openshift-dns/dns-default-jhqcn"
Jul 12 22:11:06.225041 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:06.224596    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 12 22:11:06.225041 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:06.224663    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3e0d19f-0837-473f-a4eb-89df81150888-metrics-tls podName:e3e0d19f-0837-473f-a4eb-89df81150888 nodeName:}" failed. No retries permitted until 2026-07-12 22:11:08.224643041 +0000 UTC m=+40.538146375 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e3e0d19f-0837-473f-a4eb-89df81150888-metrics-tls") pod "dns-default-jhqcn" (UID: "e3e0d19f-0837-473f-a4eb-89df81150888") : secret "dns-default-metrics-tls" not found
Jul 12 22:11:06.252958 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.242814    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7t5r"
Jul 12 22:11:06.252958 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.251072    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-st67z"
Jul 12 22:11:06.253563 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.253333    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 12 22:11:06.255845 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.254655    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-npkdh\""
Jul 12 22:11:06.255845 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.254853    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 12 22:11:06.257004 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.256986    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gf2x8"]
Jul 12 22:11:06.305712 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.305689    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-c9nrk"]
Jul 12 22:11:06.308083 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:11:06.308054    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa72db1a_a0ab_45e1_b5e4_c4ebb5de7640.slice/crio-d75fb2227458614f2b58913615078659ce28f9820a50090e5edbd870f41908c4 WatchSource:0}: Error finding container d75fb2227458614f2b58913615078659ce28f9820a50090e5edbd870f41908c4: Status 404 returned error can't find the container with id d75fb2227458614f2b58913615078659ce28f9820a50090e5edbd870f41908c4
Jul 12 22:11:06.326948 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.326918    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dqc2v"]
Jul 12 22:11:06.331566 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:11:06.331517    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf91f26b5_1055_4423_a442_da2443b2d44e.slice/crio-a05238adb21bb10b96494390f88c874100e75191710ede63d9d8c5daad086be7 WatchSource:0}: Error finding container a05238adb21bb10b96494390f88c874100e75191710ede63d9d8c5daad086be7: Status 404 returned error can't find the container with id a05238adb21bb10b96494390f88c874100e75191710ede63d9d8c5daad086be7
Jul 12 22:11:06.561248 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.561166    2560 generic.go:356] "Generic (PLEG): container finished" podID="912f085a-8942-4512-98f9-c8ba189509c2" containerID="d4fd6c1df53376524ee0f31d2bbcf475bef01fd2d489d22c8de1437bbf06ff71" exitCode=0
Jul 12 22:11:06.561384 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.561251    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kvvgp" event={"ID":"912f085a-8942-4512-98f9-c8ba189509c2","Type":"ContainerDied","Data":"d4fd6c1df53376524ee0f31d2bbcf475bef01fd2d489d22c8de1437bbf06ff71"}
Jul 12 22:11:06.562932 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.562904    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dqc2v" event={"ID":"f91f26b5-1055-4423-a442-da2443b2d44e","Type":"ContainerStarted","Data":"a05238adb21bb10b96494390f88c874100e75191710ede63d9d8c5daad086be7"}
Jul 12 22:11:06.564237 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.564207    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-c9nrk" event={"ID":"aa72db1a-a0ab-45e1-b5e4-c4ebb5de7640","Type":"ContainerStarted","Data":"d75fb2227458614f2b58913615078659ce28f9820a50090e5edbd870f41908c4"}
Jul 12 22:11:06.565671 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.565618    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gf2x8" event={"ID":"772a2a08-9010-4063-ba53-040e1e5881d1","Type":"ContainerStarted","Data":"9154250c9c3a7e73af7d8367bd29d6999dcf13a1730aaaf325c6270d47a49e55"}
Jul 12 22:11:06.567041 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.566996    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8965j" event={"ID":"fd201c7d-6b11-430a-b079-9a16e001c434","Type":"ContainerStarted","Data":"0b667fa8c904a56d6f135137810cf88a23194eb2bd3b4b31e3c93fbee97d2dee"}
Jul 12 22:11:06.628195 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.628162    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc370348-fbc2-4e67-96ce-a1129336bf21-service-ca-bundle\") pod \"router-default-7cb4cc88b6-fscqr\" (UID: \"bc370348-fbc2-4e67-96ce-a1129336bf21\") " pod="openshift-ingress/router-default-7cb4cc88b6-fscqr"
Jul 12 22:11:06.628320 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.628215    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/43823729-01b0-480f-83bd-faf0b86abcdc-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-6zwcc\" (UID: \"43823729-01b0-480f-83bd-faf0b86abcdc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-6zwcc"
Jul 12 22:11:06.628320 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.628266    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc370348-fbc2-4e67-96ce-a1129336bf21-metrics-certs\") pod \"router-default-7cb4cc88b6-fscqr\" (UID: \"bc370348-fbc2-4e67-96ce-a1129336bf21\") " pod="openshift-ingress/router-default-7cb4cc88b6-fscqr"
Jul 12 22:11:06.628320 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:06.628302    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-248f2\" (UID: \"25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-248f2"
Jul 12 22:11:06.628425 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:06.628371    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc370348-fbc2-4e67-96ce-a1129336bf21-service-ca-bundle podName:bc370348-fbc2-4e67-96ce-a1129336bf21 nodeName:}" failed. No retries permitted until 2026-07-12 22:11:07.628345926 +0000 UTC m=+39.941849273 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bc370348-fbc2-4e67-96ce-a1129336bf21-service-ca-bundle") pod "router-default-7cb4cc88b6-fscqr" (UID: "bc370348-fbc2-4e67-96ce-a1129336bf21") : configmap references non-existent config key: service-ca.crt
Jul 12 22:11:06.628425 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:06.628413    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 12 22:11:06.628490 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:06.628455    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa-cluster-monitoring-operator-tls podName:25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa nodeName:}" failed. No retries permitted until 2026-07-12 22:11:07.628440987 +0000 UTC m=+39.941944351 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-248f2" (UID: "25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa") : secret "cluster-monitoring-operator-tls" not found
Jul 12 22:11:06.628526 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:06.628505    2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 12 22:11:06.628526 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:06.628527    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43823729-01b0-480f-83bd-faf0b86abcdc-samples-operator-tls podName:43823729-01b0-480f-83bd-faf0b86abcdc nodeName:}" failed. No retries permitted until 2026-07-12 22:11:07.628519106 +0000 UTC m=+39.942022437 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/43823729-01b0-480f-83bd-faf0b86abcdc-samples-operator-tls") pod "cluster-samples-operator-65db858894-6zwcc" (UID: "43823729-01b0-480f-83bd-faf0b86abcdc") : secret "samples-operator-tls" not found
Jul 12 22:11:06.628526 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:06.628583    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 12 22:11:06.628712 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:06.628622    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc370348-fbc2-4e67-96ce-a1129336bf21-metrics-certs podName:bc370348-fbc2-4e67-96ce-a1129336bf21 nodeName:}" failed. No retries permitted until 2026-07-12 22:11:07.628597411 +0000 UTC m=+39.942100742 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bc370348-fbc2-4e67-96ce-a1129336bf21-metrics-certs") pod "router-default-7cb4cc88b6-fscqr" (UID: "bc370348-fbc2-4e67-96ce-a1129336bf21") : secret "router-metrics-certs-default" not found
Jul 12 22:11:07.574926 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:07.574885    2560 generic.go:356] "Generic (PLEG): container finished" podID="912f085a-8942-4512-98f9-c8ba189509c2" containerID="2ea3afdf21666308196bddf8710b044559cb2575869626cff7eead57b209068c" exitCode=0
Jul 12 22:11:07.575831 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:07.574967    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kvvgp" event={"ID":"912f085a-8942-4512-98f9-c8ba189509c2","Type":"ContainerDied","Data":"2ea3afdf21666308196bddf8710b044559cb2575869626cff7eead57b209068c"}
Jul 12 22:11:07.637665 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:07.637621    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc370348-fbc2-4e67-96ce-a1129336bf21-service-ca-bundle\") pod \"router-default-7cb4cc88b6-fscqr\" (UID: \"bc370348-fbc2-4e67-96ce-a1129336bf21\") " pod="openshift-ingress/router-default-7cb4cc88b6-fscqr"
Jul 12 22:11:07.637839 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:07.637688    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/43823729-01b0-480f-83bd-faf0b86abcdc-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-6zwcc\" (UID: \"43823729-01b0-480f-83bd-faf0b86abcdc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-6zwcc"
Jul 12 22:11:07.637839 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:07.637750    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc370348-fbc2-4e67-96ce-a1129336bf21-metrics-certs\") pod \"router-default-7cb4cc88b6-fscqr\" (UID: \"bc370348-fbc2-4e67-96ce-a1129336bf21\") " pod="openshift-ingress/router-default-7cb4cc88b6-fscqr"
Jul 12 22:11:07.637839 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:07.637806    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-248f2\" (UID: \"25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-248f2"
Jul 12 22:11:07.637996 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:07.637968    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 12 22:11:07.638113 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:07.638039    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa-cluster-monitoring-operator-tls podName:25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa nodeName:}" failed. No retries permitted until 2026-07-12 22:11:09.638019909 +0000 UTC m=+41.951523264 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-248f2" (UID: "25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa") : secret "cluster-monitoring-operator-tls" not found
Jul 12 22:11:07.638193 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:07.638117    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc370348-fbc2-4e67-96ce-a1129336bf21-service-ca-bundle podName:bc370348-fbc2-4e67-96ce-a1129336bf21 nodeName:}" failed. No retries permitted until 2026-07-12 22:11:09.63810711 +0000 UTC m=+41.951610457 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bc370348-fbc2-4e67-96ce-a1129336bf21-service-ca-bundle") pod "router-default-7cb4cc88b6-fscqr" (UID: "bc370348-fbc2-4e67-96ce-a1129336bf21") : configmap references non-existent config key: service-ca.crt
Jul 12 22:11:07.638244 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:07.638196    2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 12 22:11:07.638244 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:07.638231    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43823729-01b0-480f-83bd-faf0b86abcdc-samples-operator-tls podName:43823729-01b0-480f-83bd-faf0b86abcdc nodeName:}" failed. No retries permitted until 2026-07-12 22:11:09.638219942 +0000 UTC m=+41.951723290 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/43823729-01b0-480f-83bd-faf0b86abcdc-samples-operator-tls") pod "cluster-samples-operator-65db858894-6zwcc" (UID: "43823729-01b0-480f-83bd-faf0b86abcdc") : secret "samples-operator-tls" not found
Jul 12 22:11:07.638320 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:07.638290    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 12 22:11:07.638320 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:07.638318    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc370348-fbc2-4e67-96ce-a1129336bf21-metrics-certs podName:bc370348-fbc2-4e67-96ce-a1129336bf21 nodeName:}" failed. No retries permitted until 2026-07-12 22:11:09.63830871 +0000 UTC m=+41.951812056 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bc370348-fbc2-4e67-96ce-a1129336bf21-metrics-certs") pod "router-default-7cb4cc88b6-fscqr" (UID: "bc370348-fbc2-4e67-96ce-a1129336bf21") : secret "router-metrics-certs-default" not found
Jul 12 22:11:08.243774 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:08.243508    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3e0d19f-0837-473f-a4eb-89df81150888-metrics-tls\") pod \"dns-default-jhqcn\" (UID: \"e3e0d19f-0837-473f-a4eb-89df81150888\") " pod="openshift-dns/dns-default-jhqcn"
Jul 12 22:11:08.243943 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:08.243672    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 12 22:11:08.243943 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:08.243856    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3e0d19f-0837-473f-a4eb-89df81150888-metrics-tls podName:e3e0d19f-0837-473f-a4eb-89df81150888 nodeName:}" failed. No retries permitted until 2026-07-12 22:11:12.243840526 +0000 UTC m=+44.557343861 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e3e0d19f-0837-473f-a4eb-89df81150888-metrics-tls") pod "dns-default-jhqcn" (UID: "e3e0d19f-0837-473f-a4eb-89df81150888") : secret "dns-default-metrics-tls" not found
Jul 12 22:11:08.583942 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:08.583806    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kvvgp" event={"ID":"912f085a-8942-4512-98f9-c8ba189509c2","Type":"ContainerStarted","Data":"b1d340b840f16b6c25e84719ae5ae38f61e9415d654ebe1692eb4288b8dbd974"}
Jul 12 22:11:08.622119 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:08.621133    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kvvgp" podStartSLOduration=10.031452384 podStartE2EDuration="40.621115702s" totalImagesPullingTime="30.589663318s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 22:10:28 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:11:08.611000781 +0000 UTC m=+40.924504139" watchObservedRunningTime="2026-07-12 22:11:08.621115702 +0000 UTC m=+40.934619055"
Jul 12 22:11:09.657450 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:09.657050    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/43823729-01b0-480f-83bd-faf0b86abcdc-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-6zwcc\" (UID: \"43823729-01b0-480f-83bd-faf0b86abcdc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-6zwcc"
Jul 12 22:11:09.657450 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:09.657153    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc370348-fbc2-4e67-96ce-a1129336bf21-metrics-certs\") pod \"router-default-7cb4cc88b6-fscqr\" (UID: \"bc370348-fbc2-4e67-96ce-a1129336bf21\") " pod="openshift-ingress/router-default-7cb4cc88b6-fscqr"
Jul 12 22:11:09.657450 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:09.657210    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-248f2\" (UID: \"25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-248f2"
Jul 12 22:11:09.657450 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:09.657227    2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 12 22:11:09.657450 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:09.657305    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43823729-01b0-480f-83bd-faf0b86abcdc-samples-operator-tls podName:43823729-01b0-480f-83bd-faf0b86abcdc nodeName:}" failed. No retries permitted until 2026-07-12 22:11:13.657283486 +0000 UTC m=+45.970786842 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/43823729-01b0-480f-83bd-faf0b86abcdc-samples-operator-tls") pod "cluster-samples-operator-65db858894-6zwcc" (UID: "43823729-01b0-480f-83bd-faf0b86abcdc") : secret "samples-operator-tls" not found
Jul 12 22:11:09.657450 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:09.657315    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 12 22:11:09.657450 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:09.657329    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc370348-fbc2-4e67-96ce-a1129336bf21-service-ca-bundle\") pod \"router-default-7cb4cc88b6-fscqr\" (UID: \"bc370348-fbc2-4e67-96ce-a1129336bf21\") " pod="openshift-ingress/router-default-7cb4cc88b6-fscqr"
Jul 12 22:11:09.657450 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:09.657356    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 12 22:11:09.657450 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:09.657375    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc370348-fbc2-4e67-96ce-a1129336bf21-metrics-certs podName:bc370348-fbc2-4e67-96ce-a1129336bf21 nodeName:}" failed. No retries permitted until 2026-07-12 22:11:13.657359384 +0000 UTC m=+45.970862733 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bc370348-fbc2-4e67-96ce-a1129336bf21-metrics-certs") pod "router-default-7cb4cc88b6-fscqr" (UID: "bc370348-fbc2-4e67-96ce-a1129336bf21") : secret "router-metrics-certs-default" not found
Jul 12 22:11:09.657450 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:09.657405    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa-cluster-monitoring-operator-tls podName:25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa nodeName:}" failed. No retries permitted until 2026-07-12 22:11:13.65739038 +0000 UTC m=+45.970893730 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-248f2" (UID: "25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa") : secret "cluster-monitoring-operator-tls" not found
Jul 12 22:11:09.657450 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:09.657428    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc370348-fbc2-4e67-96ce-a1129336bf21-service-ca-bundle podName:bc370348-fbc2-4e67-96ce-a1129336bf21 nodeName:}" failed. No retries permitted until 2026-07-12 22:11:13.657415788 +0000 UTC m=+45.970919123 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bc370348-fbc2-4e67-96ce-a1129336bf21-service-ca-bundle") pod "router-default-7cb4cc88b6-fscqr" (UID: "bc370348-fbc2-4e67-96ce-a1129336bf21") : configmap references non-existent config key: service-ca.crt
Jul 12 22:11:10.528509 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:10.528474    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778d449776-slpdg"]
Jul 12 22:11:10.566556 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:10.566511    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778d449776-slpdg"]
Jul 12 22:11:10.566733 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:10.566642    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778d449776-slpdg"
Jul 12 22:11:10.573322 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:10.573287    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 12 22:11:10.576873 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:10.576849    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 12 22:11:10.577031 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:10.576933    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 12 22:11:10.577031 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:10.576949    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 12 22:11:10.577031 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:10.576933    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 12 22:11:10.577031 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:10.576960    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 12 22:11:10.577254 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:10.577133    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 12 22:11:10.666031 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:10.665993    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-778d449776-slpdg\" (UID: \"ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778d449776-slpdg"
Jul 12 22:11:10.666411 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:10.666073    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-778d449776-slpdg\" (UID: \"ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778d449776-slpdg"
Jul 12 22:11:10.666411 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:10.666096    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67-ca\") pod \"cluster-proxy-proxy-agent-778d449776-slpdg\" (UID: \"ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778d449776-slpdg"
Jul 12 22:11:10.666411 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:10.666131    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ffsc\" (UniqueName: \"kubernetes.io/projected/ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67-kube-api-access-6ffsc\") pod \"cluster-proxy-proxy-agent-778d449776-slpdg\" (UID: \"ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778d449776-slpdg"
Jul 12 22:11:10.666411 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:10.666209    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67-additional-service-ca\") pod \"cluster-proxy-proxy-agent-778d449776-slpdg\" (UID: \"ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778d449776-slpdg"
Jul 12 22:11:10.666411 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:10.666268    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67-hub\") pod \"cluster-proxy-proxy-agent-778d449776-slpdg\" (UID: \"ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778d449776-slpdg"
Jul 12 22:11:10.767374 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:10.767334    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67-additional-service-ca\") pod \"cluster-proxy-proxy-agent-778d449776-slpdg\" (UID: \"ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778d449776-slpdg"
Jul 12 22:11:10.767567 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:10.767405    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67-hub\") pod \"cluster-proxy-proxy-agent-778d449776-slpdg\" (UID: \"ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778d449776-slpdg"
Jul 12 22:11:10.767567 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:10.767454    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-778d449776-slpdg\" (UID: \"ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778d449776-slpdg"
Jul 12 22:11:10.767567 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:10.767527    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-778d449776-slpdg\" (UID: \"ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778d449776-slpdg"
Jul 12 22:11:10.767718 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:10.767583    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67-ca\") pod \"cluster-proxy-proxy-agent-778d449776-slpdg\" (UID: \"ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778d449776-slpdg"
Jul 12 22:11:10.767718 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:10.767626    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ffsc\" (UniqueName: \"kubernetes.io/projected/ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67-kube-api-access-6ffsc\") pod \"cluster-proxy-proxy-agent-778d449776-slpdg\" (UID: \"ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778d449776-slpdg"
Jul 12 22:11:10.768409 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:10.768173    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67-additional-service-ca\") pod \"cluster-proxy-proxy-agent-778d449776-slpdg\" (UID: \"ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778d449776-slpdg"
Jul 12 22:11:10.770855 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:10.770833    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67-hub\") pod \"cluster-proxy-proxy-agent-778d449776-slpdg\" (UID: \"ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778d449776-slpdg"
Jul 12 22:11:10.770855 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:10.770845    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-778d449776-slpdg\" (UID: \"ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778d449776-slpdg"
Jul 12 22:11:10.770985 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:10.770888    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67-ca\") pod \"cluster-proxy-proxy-agent-778d449776-slpdg\" (UID: \"ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778d449776-slpdg"
Jul 12 22:11:10.771065 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:10.771049    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-778d449776-slpdg\" (UID: \"ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778d449776-slpdg"
Jul 12 22:11:10.778956 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:10.778901    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ffsc\" (UniqueName: \"kubernetes.io/projected/ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67-kube-api-access-6ffsc\") pod \"cluster-proxy-proxy-agent-778d449776-slpdg\" (UID: \"ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778d449776-slpdg"
Jul 12 22:11:10.892980 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:10.892950    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778d449776-slpdg"
Jul 12 22:11:11.027885 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:11.027854    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mjg7l"]
Jul 12 22:11:11.043301 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:11.043237    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mjg7l"
Jul 12 22:11:11.047929 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:11.047906    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 12 22:11:11.048154 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:11.048134    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 12 22:11:11.048207 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:11.048164    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5sgr4\""
Jul 12 22:11:11.048426 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:11.048406    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 12 22:11:11.048514 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:11.048440    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 12 22:11:11.052570 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:11.052551    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mjg7l"]
Jul 12 22:11:11.071191 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:11.071160    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/05f8f52f-6343-4d4c-af3b-028de417ace6-data-volume\") pod \"insights-runtime-extractor-mjg7l\" (UID: \"05f8f52f-6343-4d4c-af3b-028de417ace6\") " pod="openshift-insights/insights-runtime-extractor-mjg7l"
Jul 12 22:11:11.071336 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:11.071276    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/05f8f52f-6343-4d4c-af3b-028de417ace6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mjg7l\" (UID: \"05f8f52f-6343-4d4c-af3b-028de417ace6\") " pod="openshift-insights/insights-runtime-extractor-mjg7l"
Jul 12 22:11:11.071336 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:11.071303    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7csr\" (UniqueName: \"kubernetes.io/projected/05f8f52f-6343-4d4c-af3b-028de417ace6-kube-api-access-q7csr\") pod \"insights-runtime-extractor-mjg7l\" (UID: \"05f8f52f-6343-4d4c-af3b-028de417ace6\") " pod="openshift-insights/insights-runtime-extractor-mjg7l"
Jul 12 22:11:11.071413 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:11.071357    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/05f8f52f-6343-4d4c-af3b-028de417ace6-crio-socket\") pod \"insights-runtime-extractor-mjg7l\" (UID: \"05f8f52f-6343-4d4c-af3b-028de417ace6\") " pod="openshift-insights/insights-runtime-extractor-mjg7l"
Jul 12 22:11:11.071413 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:11.071374    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/05f8f52f-6343-4d4c-af3b-028de417ace6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mjg7l\" (UID: \"05f8f52f-6343-4d4c-af3b-028de417ace6\") " pod="openshift-insights/insights-runtime-extractor-mjg7l"
Jul 12 22:11:11.173029 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:11.172990    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/05f8f52f-6343-4d4c-af3b-028de417ace6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mjg7l\" (UID: \"05f8f52f-6343-4d4c-af3b-028de417ace6\") " pod="openshift-insights/insights-runtime-extractor-mjg7l"
Jul 12 22:11:11.173174 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:11.173041    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7csr\" (UniqueName: \"kubernetes.io/projected/05f8f52f-6343-4d4c-af3b-028de417ace6-kube-api-access-q7csr\") pod \"insights-runtime-extractor-mjg7l\" (UID: \"05f8f52f-6343-4d4c-af3b-028de417ace6\") " pod="openshift-insights/insights-runtime-extractor-mjg7l"
Jul 12 22:11:11.173174 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:11.173140    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 12 22:11:11.173268 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:11.173232    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/05f8f52f-6343-4d4c-af3b-028de417ace6-kube-rbac-proxy-cm podName:05f8f52f-6343-4d4c-af3b-028de417ace6 nodeName:}" failed. No retries permitted until 2026-07-12 22:11:11.673210959 +0000 UTC m=+43.986714296 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/05f8f52f-6343-4d4c-af3b-028de417ace6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mjg7l" (UID: "05f8f52f-6343-4d4c-af3b-028de417ace6") : configmap "kube-rbac-proxy" not found
Jul 12 22:11:11.173345 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:11.173311    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/05f8f52f-6343-4d4c-af3b-028de417ace6-crio-socket\") pod \"insights-runtime-extractor-mjg7l\" (UID: \"05f8f52f-6343-4d4c-af3b-028de417ace6\") " pod="openshift-insights/insights-runtime-extractor-mjg7l"
Jul 12 22:11:11.173345 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:11.173340    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/05f8f52f-6343-4d4c-af3b-028de417ace6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mjg7l\" (UID: \"05f8f52f-6343-4d4c-af3b-028de417ace6\") " pod="openshift-insights/insights-runtime-extractor-mjg7l"
Jul 12 22:11:11.173598 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:11.173377    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/05f8f52f-6343-4d4c-af3b-028de417ace6-data-volume\") pod \"insights-runtime-extractor-mjg7l\" (UID: \"05f8f52f-6343-4d4c-af3b-028de417ace6\") " pod="openshift-insights/insights-runtime-extractor-mjg7l"
Jul 12 22:11:11.173598 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:11.173442    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 12 22:11:11.173598 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:11.173506    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05f8f52f-6343-4d4c-af3b-028de417ace6-insights-runtime-extractor-tls podName:05f8f52f-6343-4d4c-af3b-028de417ace6 nodeName:}" failed. No retries permitted until 2026-07-12 22:11:11.673490226 +0000 UTC m=+43.986993563 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/05f8f52f-6343-4d4c-af3b-028de417ace6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mjg7l" (UID: "05f8f52f-6343-4d4c-af3b-028de417ace6") : secret "insights-runtime-extractor-tls" not found
Jul 12 22:11:11.173598 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:11.173563    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/05f8f52f-6343-4d4c-af3b-028de417ace6-crio-socket\") pod \"insights-runtime-extractor-mjg7l\" (UID: \"05f8f52f-6343-4d4c-af3b-028de417ace6\") " pod="openshift-insights/insights-runtime-extractor-mjg7l"
Jul 12 22:11:11.173787 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:11.173735    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/05f8f52f-6343-4d4c-af3b-028de417ace6-data-volume\") pod \"insights-runtime-extractor-mjg7l\" (UID: \"05f8f52f-6343-4d4c-af3b-028de417ace6\") " pod="openshift-insights/insights-runtime-extractor-mjg7l"
Jul 12 22:11:11.182236 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:11.182211    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7csr\" (UniqueName: \"kubernetes.io/projected/05f8f52f-6343-4d4c-af3b-028de417ace6-kube-api-access-q7csr\") pod \"insights-runtime-extractor-mjg7l\" (UID: \"05f8f52f-6343-4d4c-af3b-028de417ace6\") " pod="openshift-insights/insights-runtime-extractor-mjg7l"
Jul 12 22:11:11.275172 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:11.275139    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778d449776-slpdg"]
Jul 12 22:11:11.278182 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:11:11.278158    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff3d768b_0f04_4ae3_b7fd_b500d5fa3e67.slice/crio-af3b5f51390ae3a693e50499d0facfffc1da26df9d33c8c63343c8f4dc8919ea WatchSource:0}: Error finding container af3b5f51390ae3a693e50499d0facfffc1da26df9d33c8c63343c8f4dc8919ea: Status 404 returned error can't find the container with id af3b5f51390ae3a693e50499d0facfffc1da26df9d33c8c63343c8f4dc8919ea
Jul 12 22:11:11.592718 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:11.592678    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gf2x8" event={"ID":"772a2a08-9010-4063-ba53-040e1e5881d1","Type":"ContainerStarted","Data":"283ce296055fad949ce2cceb0b35396093c1286f047537b947492ac431b17e3b"}
Jul 12 22:11:11.594282 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:11.594252    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8965j" event={"ID":"fd201c7d-6b11-430a-b079-9a16e001c434","Type":"ContainerStarted","Data":"bdc02b7c178b78a06266c5792504339c3a6a7f43eafdc871b5c916445c40a86e"}
Jul 12 22:11:11.595546 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:11.595500    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778d449776-slpdg" event={"ID":"ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67","Type":"ContainerStarted","Data":"af3b5f51390ae3a693e50499d0facfffc1da26df9d33c8c63343c8f4dc8919ea"}
Jul 12 22:11:11.597033 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:11.597005    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dqc2v" event={"ID":"f91f26b5-1055-4423-a442-da2443b2d44e","Type":"ContainerStarted","Data":"7c8e3967e332984da4469d0e778b50ddd9c935a9f38624578ac18f483b439015"}
Jul 12 22:11:11.598852 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:11.598831    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-c9nrk" event={"ID":"aa72db1a-a0ab-45e1-b5e4-c4ebb5de7640","Type":"ContainerStarted","Data":"788c8ceeb910bdcb9f013004f3371b29540a00ff1f0756cd97154e3433e65968"}
Jul 12 22:11:11.608928 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:11.608891    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gf2x8" podStartSLOduration=1.7665172359999999 podStartE2EDuration="6.608878751s" totalImagesPullingTime="4.842361515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:11:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:11:11.60808308 +0000 UTC m=+43.921586435" watchObservedRunningTime="2026-07-12 22:11:11.608878751 +0000 UTC m=+43.922382083"
Jul 12 22:11:11.627465 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:11.627415    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-c9nrk" podStartSLOduration=1.786133307 podStartE2EDuration="6.627397796s" totalImagesPullingTime="4.841264489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:11:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:11:11.626863709 +0000 UTC m=+43.940367063" watchObservedRunningTime="2026-07-12 22:11:11.627397796 +0000 UTC m=+43.940901147"
Jul 12 22:11:11.653029 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:11.652973    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dqc2v" podStartSLOduration=1.8482795589999998 podStartE2EDuration="6.652954952s" totalImagesPullingTime="4.804675393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:11:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:11:11.652362349 +0000 UTC m=+43.965865706" watchObservedRunningTime="2026-07-12 22:11:11.652954952 +0000 UTC m=+43.966458306"
Jul 12 22:11:11.674297 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:11.674240    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8965j" podStartSLOduration=2.4765348019999998 podStartE2EDuration="7.674224878s" totalImagesPullingTime="5.197690076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:11:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:11:11.673411979 +0000 UTC m=+43.986915332" watchObservedRunningTime="2026-07-12 22:11:11.674224878 +0000 UTC m=+43.987728231"
Jul 12 22:11:11.678920 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:11.677896    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/05f8f52f-6343-4d4c-af3b-028de417ace6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mjg7l\" (UID: \"05f8f52f-6343-4d4c-af3b-028de417ace6\") " pod="openshift-insights/insights-runtime-extractor-mjg7l"
Jul 12 22:11:11.678920 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:11.677997    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/05f8f52f-6343-4d4c-af3b-028de417ace6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mjg7l\" (UID: \"05f8f52f-6343-4d4c-af3b-028de417ace6\") " pod="openshift-insights/insights-runtime-extractor-mjg7l"
Jul 12 22:11:11.678920 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:11.678489    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 12 22:11:11.678920 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:11.678579    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/05f8f52f-6343-4d4c-af3b-028de417ace6-kube-rbac-proxy-cm podName:05f8f52f-6343-4d4c-af3b-028de417ace6 nodeName:}" failed. No retries permitted until 2026-07-12 22:11:12.678560564 +0000 UTC m=+44.992063898 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/05f8f52f-6343-4d4c-af3b-028de417ace6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mjg7l" (UID: "05f8f52f-6343-4d4c-af3b-028de417ace6") : configmap "kube-rbac-proxy" not found
Jul 12 22:11:11.678920 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:11.678600    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 12 22:11:11.678920 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:11.678652    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05f8f52f-6343-4d4c-af3b-028de417ace6-insights-runtime-extractor-tls podName:05f8f52f-6343-4d4c-af3b-028de417ace6 nodeName:}" failed. No retries permitted until 2026-07-12 22:11:12.67863629 +0000 UTC m=+44.992139623 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/05f8f52f-6343-4d4c-af3b-028de417ace6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mjg7l" (UID: "05f8f52f-6343-4d4c-af3b-028de417ace6") : secret "insights-runtime-extractor-tls" not found
Jul 12 22:11:12.283408 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:12.283368    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3e0d19f-0837-473f-a4eb-89df81150888-metrics-tls\") pod \"dns-default-jhqcn\" (UID: \"e3e0d19f-0837-473f-a4eb-89df81150888\") " pod="openshift-dns/dns-default-jhqcn"
Jul 12 22:11:12.283610 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:12.283552    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 12 22:11:12.283670 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:12.283623    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3e0d19f-0837-473f-a4eb-89df81150888-metrics-tls podName:e3e0d19f-0837-473f-a4eb-89df81150888 nodeName:}" failed. No retries permitted until 2026-07-12 22:11:20.283600826 +0000 UTC m=+52.597104179 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e3e0d19f-0837-473f-a4eb-89df81150888-metrics-tls") pod "dns-default-jhqcn" (UID: "e3e0d19f-0837-473f-a4eb-89df81150888") : secret "dns-default-metrics-tls" not found
Jul 12 22:11:12.427357 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:12.427321    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g4qw4_05ebf682-f7b0-4267-8a22-46d0c41ecba9/dns-node-resolver/0.log"
Jul 12 22:11:12.687278 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:12.687241    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/05f8f52f-6343-4d4c-af3b-028de417ace6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mjg7l\" (UID: \"05f8f52f-6343-4d4c-af3b-028de417ace6\") " pod="openshift-insights/insights-runtime-extractor-mjg7l"
Jul 12 22:11:12.687821 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:12.687333    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/05f8f52f-6343-4d4c-af3b-028de417ace6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mjg7l\" (UID: \"05f8f52f-6343-4d4c-af3b-028de417ace6\") " pod="openshift-insights/insights-runtime-extractor-mjg7l"
Jul 12 22:11:12.687821 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:12.687742    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 12 22:11:12.687821 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:12.687749    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 12 22:11:12.687990 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:12.687857    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05f8f52f-6343-4d4c-af3b-028de417ace6-insights-runtime-extractor-tls podName:05f8f52f-6343-4d4c-af3b-028de417ace6 nodeName:}" failed. No retries permitted until 2026-07-12 22:11:14.687783987 +0000 UTC m=+47.001287319 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/05f8f52f-6343-4d4c-af3b-028de417ace6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mjg7l" (UID: "05f8f52f-6343-4d4c-af3b-028de417ace6") : secret "insights-runtime-extractor-tls" not found
Jul 12 22:11:12.687990 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:12.687876    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/05f8f52f-6343-4d4c-af3b-028de417ace6-kube-rbac-proxy-cm podName:05f8f52f-6343-4d4c-af3b-028de417ace6 nodeName:}" failed. No retries permitted until 2026-07-12 22:11:14.687865678 +0000 UTC m=+47.001369010 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/05f8f52f-6343-4d4c-af3b-028de417ace6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mjg7l" (UID: "05f8f52f-6343-4d4c-af3b-028de417ace6") : configmap "kube-rbac-proxy" not found
Jul 12 22:11:13.627409 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:13.627377    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zkk8b_c946a409-971a-4760-b311-5278608e31f9/node-ca/0.log"
Jul 12 22:11:13.696398 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:13.696351    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-248f2\" (UID: \"25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-248f2"
Jul 12 22:11:13.696863 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:13.696420    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc370348-fbc2-4e67-96ce-a1129336bf21-service-ca-bundle\") pod \"router-default-7cb4cc88b6-fscqr\" (UID: \"bc370348-fbc2-4e67-96ce-a1129336bf21\") " pod="openshift-ingress/router-default-7cb4cc88b6-fscqr"
Jul 12 22:11:13.696863 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:13.696459    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/43823729-01b0-480f-83bd-faf0b86abcdc-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-6zwcc\" (UID: \"43823729-01b0-480f-83bd-faf0b86abcdc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-6zwcc"
Jul 12 22:11:13.696863 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:13.696491    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc370348-fbc2-4e67-96ce-a1129336bf21-metrics-certs\") pod \"router-default-7cb4cc88b6-fscqr\" (UID: \"bc370348-fbc2-4e67-96ce-a1129336bf21\") " pod="openshift-ingress/router-default-7cb4cc88b6-fscqr"
Jul 12 22:11:13.696863 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:13.696521    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 12 22:11:13.696863 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:13.696605    2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 12 22:11:13.696863 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:13.696605    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc370348-fbc2-4e67-96ce-a1129336bf21-service-ca-bundle podName:bc370348-fbc2-4e67-96ce-a1129336bf21 nodeName:}" failed. No retries permitted until 2026-07-12 22:11:21.696579903 +0000 UTC m=+54.010083257 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bc370348-fbc2-4e67-96ce-a1129336bf21-service-ca-bundle") pod "router-default-7cb4cc88b6-fscqr" (UID: "bc370348-fbc2-4e67-96ce-a1129336bf21") : configmap references non-existent config key: service-ca.crt
Jul 12 22:11:13.696863 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:13.696645    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 12 22:11:13.696863 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:13.696672    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa-cluster-monitoring-operator-tls podName:25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa nodeName:}" failed. No retries permitted until 2026-07-12 22:11:21.696652766 +0000 UTC m=+54.010156111 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-248f2" (UID: "25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa") : secret "cluster-monitoring-operator-tls" not found
Jul 12 22:11:13.696863 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:13.696693    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43823729-01b0-480f-83bd-faf0b86abcdc-samples-operator-tls podName:43823729-01b0-480f-83bd-faf0b86abcdc nodeName:}" failed. No retries permitted until 2026-07-12 22:11:21.696680597 +0000 UTC m=+54.010183935 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/43823729-01b0-480f-83bd-faf0b86abcdc-samples-operator-tls") pod "cluster-samples-operator-65db858894-6zwcc" (UID: "43823729-01b0-480f-83bd-faf0b86abcdc") : secret "samples-operator-tls" not found
Jul 12 22:11:13.696863 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:13.696709    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc370348-fbc2-4e67-96ce-a1129336bf21-metrics-certs podName:bc370348-fbc2-4e67-96ce-a1129336bf21 nodeName:}" failed. No retries permitted until 2026-07-12 22:11:21.696700058 +0000 UTC m=+54.010203401 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bc370348-fbc2-4e67-96ce-a1129336bf21-metrics-certs") pod "router-default-7cb4cc88b6-fscqr" (UID: "bc370348-fbc2-4e67-96ce-a1129336bf21") : secret "router-metrics-certs-default" not found
Jul 12 22:11:14.230780 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:14.230746    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8965j_fd201c7d-6b11-430a-b079-9a16e001c434/serve-healthcheck-canary/0.log"
Jul 12 22:11:14.704460 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:14.704417    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/05f8f52f-6343-4d4c-af3b-028de417ace6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mjg7l\" (UID: \"05f8f52f-6343-4d4c-af3b-028de417ace6\") " pod="openshift-insights/insights-runtime-extractor-mjg7l"
Jul 12 22:11:14.704984 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:14.704591    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 12 22:11:14.704984 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:14.704590    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/05f8f52f-6343-4d4c-af3b-028de417ace6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mjg7l\" (UID: \"05f8f52f-6343-4d4c-af3b-028de417ace6\") " pod="openshift-insights/insights-runtime-extractor-mjg7l"
Jul 12 22:11:14.704984 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:14.704658    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/05f8f52f-6343-4d4c-af3b-028de417ace6-kube-rbac-proxy-cm podName:05f8f52f-6343-4d4c-af3b-028de417ace6 nodeName:}" failed. No retries permitted until 2026-07-12 22:11:18.704638104 +0000 UTC m=+51.018141436 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/05f8f52f-6343-4d4c-af3b-028de417ace6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mjg7l" (UID: "05f8f52f-6343-4d4c-af3b-028de417ace6") : configmap "kube-rbac-proxy" not found
Jul 12 22:11:14.704984 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:14.704688    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 12 22:11:14.704984 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:14.704756    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05f8f52f-6343-4d4c-af3b-028de417ace6-insights-runtime-extractor-tls podName:05f8f52f-6343-4d4c-af3b-028de417ace6 nodeName:}" failed. No retries permitted until 2026-07-12 22:11:18.704737962 +0000 UTC m=+51.018241307 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/05f8f52f-6343-4d4c-af3b-028de417ace6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mjg7l" (UID: "05f8f52f-6343-4d4c-af3b-028de417ace6") : secret "insights-runtime-extractor-tls" not found
Jul 12 22:11:16.616825 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:16.616710    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778d449776-slpdg" event={"ID":"ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67","Type":"ContainerStarted","Data":"18efd2db3c06b8e6147b94ed0acb3fbcf9431f00c9f5262ece0676323968ee0b"}
Jul 12 22:11:16.616825 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:16.616752    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778d449776-slpdg" event={"ID":"ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67","Type":"ContainerStarted","Data":"f268a5e235ba77107fc72ed66dd02d6a42797108396fc2415173ff0f8cb57f42"}
Jul 12 22:11:16.616825 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:16.616765    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778d449776-slpdg" event={"ID":"ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67","Type":"ContainerStarted","Data":"8934324f290fa947750394cb0d0d1e6312bcaa0de97fc784717c1e19f372a4ba"}
Jul 12 22:11:16.641293 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:16.641238    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778d449776-slpdg" podStartSLOduration=1.832760723 podStartE2EDuration="6.641222709s" totalImagesPullingTime="4.808461986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:11:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:11:16.637395061 +0000 UTC m=+48.950898415" watchObservedRunningTime="2026-07-12 22:11:16.641222709 +0000 UTC m=+48.954726063"
Jul 12 22:11:18.739379 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:18.739340    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/05f8f52f-6343-4d4c-af3b-028de417ace6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mjg7l\" (UID: \"05f8f52f-6343-4d4c-af3b-028de417ace6\") " pod="openshift-insights/insights-runtime-extractor-mjg7l"
Jul 12 22:11:18.739778 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:18.739408    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/05f8f52f-6343-4d4c-af3b-028de417ace6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mjg7l\" (UID: \"05f8f52f-6343-4d4c-af3b-028de417ace6\") " pod="openshift-insights/insights-runtime-extractor-mjg7l"
Jul 12 22:11:18.739778 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:18.739459    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 12 22:11:18.739778 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:18.739548    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/05f8f52f-6343-4d4c-af3b-028de417ace6-kube-rbac-proxy-cm podName:05f8f52f-6343-4d4c-af3b-028de417ace6 nodeName:}" failed. No retries permitted until 2026-07-12 22:11:26.739514055 +0000 UTC m=+59.053017407 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/05f8f52f-6343-4d4c-af3b-028de417ace6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mjg7l" (UID: "05f8f52f-6343-4d4c-af3b-028de417ace6") : configmap "kube-rbac-proxy" not found
Jul 12 22:11:18.739778 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:18.739570    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 12 22:11:18.739778 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:18.739635    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05f8f52f-6343-4d4c-af3b-028de417ace6-insights-runtime-extractor-tls podName:05f8f52f-6343-4d4c-af3b-028de417ace6 nodeName:}" failed. No retries permitted until 2026-07-12 22:11:26.739622747 +0000 UTC m=+59.053126079 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/05f8f52f-6343-4d4c-af3b-028de417ace6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mjg7l" (UID: "05f8f52f-6343-4d4c-af3b-028de417ace6") : secret "insights-runtime-extractor-tls" not found
Jul 12 22:11:20.353574 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:20.353515    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3e0d19f-0837-473f-a4eb-89df81150888-metrics-tls\") pod \"dns-default-jhqcn\" (UID: \"e3e0d19f-0837-473f-a4eb-89df81150888\") " pod="openshift-dns/dns-default-jhqcn"
Jul 12 22:11:20.356055 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:20.356031    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3e0d19f-0837-473f-a4eb-89df81150888-metrics-tls\") pod \"dns-default-jhqcn\" (UID: \"e3e0d19f-0837-473f-a4eb-89df81150888\") " pod="openshift-dns/dns-default-jhqcn"
Jul 12 22:11:20.409193 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:20.409155    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jhqcn"
Jul 12 22:11:20.555096 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:20.555063    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jhqcn"]
Jul 12 22:11:20.557909 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:11:20.557879    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3e0d19f_0837_473f_a4eb_89df81150888.slice/crio-acb40d8f062e2a2ba420807ac5de2b1d76a50874d7bbd42d05c06849e22657c0 WatchSource:0}: Error finding container acb40d8f062e2a2ba420807ac5de2b1d76a50874d7bbd42d05c06849e22657c0: Status 404 returned error can't find the container with id acb40d8f062e2a2ba420807ac5de2b1d76a50874d7bbd42d05c06849e22657c0
Jul 12 22:11:20.629595 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:20.629497    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jhqcn" event={"ID":"e3e0d19f-0837-473f-a4eb-89df81150888","Type":"ContainerStarted","Data":"acb40d8f062e2a2ba420807ac5de2b1d76a50874d7bbd42d05c06849e22657c0"}
Jul 12 22:11:20.895666 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:20.895573    2560 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778d449776-slpdg" podUID="ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 12 22:11:21.766387 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:21.766342    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-248f2\" (UID: \"25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-248f2"
Jul 12 22:11:21.766888 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:21.766415    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc370348-fbc2-4e67-96ce-a1129336bf21-service-ca-bundle\") pod \"router-default-7cb4cc88b6-fscqr\" (UID: \"bc370348-fbc2-4e67-96ce-a1129336bf21\") " pod="openshift-ingress/router-default-7cb4cc88b6-fscqr"
Jul 12 22:11:21.766888 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:21.766437    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/43823729-01b0-480f-83bd-faf0b86abcdc-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-6zwcc\" (UID: \"43823729-01b0-480f-83bd-faf0b86abcdc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-6zwcc"
Jul 12 22:11:21.766888 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:21.766457    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc370348-fbc2-4e67-96ce-a1129336bf21-metrics-certs\") pod \"router-default-7cb4cc88b6-fscqr\" (UID: \"bc370348-fbc2-4e67-96ce-a1129336bf21\") " pod="openshift-ingress/router-default-7cb4cc88b6-fscqr"
Jul 12 22:11:21.766888 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:21.766560    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 12 22:11:21.766888 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:21.766633    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bc370348-fbc2-4e67-96ce-a1129336bf21-service-ca-bundle podName:bc370348-fbc2-4e67-96ce-a1129336bf21 nodeName:}" failed. No retries permitted until 2026-07-12 22:11:37.766610471 +0000 UTC m=+70.080113821 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bc370348-fbc2-4e67-96ce-a1129336bf21-service-ca-bundle") pod "router-default-7cb4cc88b6-fscqr" (UID: "bc370348-fbc2-4e67-96ce-a1129336bf21") : configmap references non-existent config key: service-ca.crt
Jul 12 22:11:21.766888 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:21.766661    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa-cluster-monitoring-operator-tls podName:25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa nodeName:}" failed. No retries permitted until 2026-07-12 22:11:37.76665113 +0000 UTC m=+70.080154468 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-248f2" (UID: "25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa") : secret "cluster-monitoring-operator-tls" not found
Jul 12 22:11:21.769325 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:21.769299    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/43823729-01b0-480f-83bd-faf0b86abcdc-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-6zwcc\" (UID: \"43823729-01b0-480f-83bd-faf0b86abcdc\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-6zwcc"
Jul 12 22:11:21.769450 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:21.769350    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc370348-fbc2-4e67-96ce-a1129336bf21-metrics-certs\") pod \"router-default-7cb4cc88b6-fscqr\" (UID: \"bc370348-fbc2-4e67-96ce-a1129336bf21\") " pod="openshift-ingress/router-default-7cb4cc88b6-fscqr"
Jul 12 22:11:21.854168 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:21.854125    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-6zwcc"
Jul 12 22:11:22.011993 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:22.011956    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-6zwcc"]
Jul 12 22:11:22.637294 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:22.637247    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-6zwcc" event={"ID":"43823729-01b0-480f-83bd-faf0b86abcdc","Type":"ContainerStarted","Data":"859a5a9e6ad9fa72a5670cf926fa8ffb689341c2ffd03e434c531c3e6822ed76"}
Jul 12 22:11:23.643686 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:23.643644    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jhqcn" event={"ID":"e3e0d19f-0837-473f-a4eb-89df81150888","Type":"ContainerStarted","Data":"b45f85f7d52eda7b33c51be8048e1c8b86b6c188001b759d4898dbe68da315ad"}
Jul 12 22:11:23.644167 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:23.643707    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jhqcn" event={"ID":"e3e0d19f-0837-473f-a4eb-89df81150888","Type":"ContainerStarted","Data":"ae606638bc7c0fe4bcdb3674d57e5e9ba4970e9aa234d1f9321a5bd4f6fec8c0"}
Jul 12 22:11:23.644167 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:23.643769    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jhqcn"
Jul 12 22:11:23.663613 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:23.663552    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jhqcn" podStartSLOduration=17.215838799 podStartE2EDuration="19.663515837s" totalImagesPullingTime="2.447677038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:11:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:11:23.662893754 +0000 UTC m=+55.976397109" watchObservedRunningTime="2026-07-12 22:11:23.663515837 +0000 UTC m=+55.977019192"
Jul 12 22:11:24.649410 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:24.649325    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-6zwcc" event={"ID":"43823729-01b0-480f-83bd-faf0b86abcdc","Type":"ContainerStarted","Data":"04e61c66258e343709e3787f4fd203fbe1595cd8efea43c61993eba38247e7a7"}
Jul 12 22:11:24.649410 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:24.649390    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-6zwcc" event={"ID":"43823729-01b0-480f-83bd-faf0b86abcdc","Type":"ContainerStarted","Data":"708149658afef7dbdb6f0922db6f7366b70d24a15ba6d8d9f7474cd4c4f238e0"}
Jul 12 22:11:24.667099 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:24.667047    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-6zwcc" podStartSLOduration=17.35955464 podStartE2EDuration="19.667030558s" totalImagesPullingTime="2.307475918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:11:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:11:24.666423607 +0000 UTC m=+56.979926964" watchObservedRunningTime="2026-07-12 22:11:24.667030558 +0000 UTC m=+56.980533912"
Jul 12 22:11:26.810668 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:26.810633    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/05f8f52f-6343-4d4c-af3b-028de417ace6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mjg7l\" (UID: \"05f8f52f-6343-4d4c-af3b-028de417ace6\") " pod="openshift-insights/insights-runtime-extractor-mjg7l"
Jul 12 22:11:26.811042 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:26.810710    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/05f8f52f-6343-4d4c-af3b-028de417ace6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mjg7l\" (UID: \"05f8f52f-6343-4d4c-af3b-028de417ace6\") " pod="openshift-insights/insights-runtime-extractor-mjg7l"
Jul 12 22:11:26.811171 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:26.811150    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/05f8f52f-6343-4d4c-af3b-028de417ace6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mjg7l\" (UID: \"05f8f52f-6343-4d4c-af3b-028de417ace6\") " pod="openshift-insights/insights-runtime-extractor-mjg7l"
Jul 12 22:11:26.813093 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:26.813075    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/05f8f52f-6343-4d4c-af3b-028de417ace6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mjg7l\" (UID: \"05f8f52f-6343-4d4c-af3b-028de417ace6\") " pod="openshift-insights/insights-runtime-extractor-mjg7l"
Jul 12 22:11:26.952747 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:26.952715    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mjg7l"
Jul 12 22:11:27.093476 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:27.093442    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mjg7l"]
Jul 12 22:11:27.096765 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:11:27.096739    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05f8f52f_6343_4d4c_af3b_028de417ace6.slice/crio-d2b6e1dcb8d078aabc79b127e05b05d12b2ed5f1de17e4d76b84df8353d269c8 WatchSource:0}: Error finding container d2b6e1dcb8d078aabc79b127e05b05d12b2ed5f1de17e4d76b84df8353d269c8: Status 404 returned error can't find the container with id d2b6e1dcb8d078aabc79b127e05b05d12b2ed5f1de17e4d76b84df8353d269c8
Jul 12 22:11:27.661217 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:27.661179    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mjg7l" event={"ID":"05f8f52f-6343-4d4c-af3b-028de417ace6","Type":"ContainerStarted","Data":"d2b6e1dcb8d078aabc79b127e05b05d12b2ed5f1de17e4d76b84df8353d269c8"}
Jul 12 22:11:30.561959 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:30.561926    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-52pbf"
Jul 12 22:11:30.895030 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:30.894989    2560 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778d449776-slpdg" podUID="ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 12 22:11:31.677429 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:31.677393    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mjg7l" event={"ID":"05f8f52f-6343-4d4c-af3b-028de417ace6","Type":"ContainerStarted","Data":"92dbe768ab9141896e0139decbc2c54133e8752e4ea4b8f156344d2f5671329f"}
Jul 12 22:11:32.683222 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:32.683184    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mjg7l" event={"ID":"05f8f52f-6343-4d4c-af3b-028de417ace6","Type":"ContainerStarted","Data":"81785564a8e15a2302137109ee06c1929d978b986cb059c6c7efce2839c58a79"}
Jul 12 22:11:33.488617 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.488578    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-tlbnn"]
Jul 12 22:11:33.491740 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.491712    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-tlbnn"
Jul 12 22:11:33.499778 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.499750    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5f8fccb5d7-fzjx6"]
Jul 12 22:11:33.502999 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.502983    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f8fccb5d7-fzjx6"
Jul 12 22:11:33.506564 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:33.504987    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-76psl\" is forbidden: User \"system:node:ip-10-0-135-47.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-network-console\": no relationship found between node 'ip-10-0-135-47.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-network-console\"/\"default-dockercfg-76psl\"" type="*v1.Secret"
Jul 12 22:11:33.506564 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:33.505162    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"networking-console-plugin\" is forbidden: User \"system:node:ip-10-0-135-47.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-network-console\": no relationship found between node 'ip-10-0-135-47.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" type="*v1.ConfigMap"
Jul 12 22:11:33.506564 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.505221    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 12 22:11:33.506564 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.506067    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 12 22:11:33.506564 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.506199    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 12 22:11:33.506564 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.506556    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hjmtz\""
Jul 12 22:11:33.507296 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.507273    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 12 22:11:33.512565 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.512525    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 12 22:11:33.523487 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.523457    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f8fccb5d7-fzjx6"]
Jul 12 22:11:33.525450 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.525423    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-tlbnn"]
Jul 12 22:11:33.565884 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.565845    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-bound-sa-token\") pod \"image-registry-5f8fccb5d7-fzjx6\" (UID: \"2c761ba8-bf2b-44fa-a5b0-af2ac82da425\") " pod="openshift-image-registry/image-registry-5f8fccb5d7-fzjx6"
Jul 12 22:11:33.566068 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.565913    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-registry-certificates\") pod \"image-registry-5f8fccb5d7-fzjx6\" (UID: \"2c761ba8-bf2b-44fa-a5b0-af2ac82da425\") " pod="openshift-image-registry/image-registry-5f8fccb5d7-fzjx6"
Jul 12 22:11:33.566068 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.565940    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-registry-tls\") pod \"image-registry-5f8fccb5d7-fzjx6\" (UID: \"2c761ba8-bf2b-44fa-a5b0-af2ac82da425\") " pod="openshift-image-registry/image-registry-5f8fccb5d7-fzjx6"
Jul 12 22:11:33.566068 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.565969    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7af611d3-24c8-4266-88d2-23b4523117e2-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-tlbnn\" (UID: \"7af611d3-24c8-4266-88d2-23b4523117e2\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-tlbnn"
Jul 12 22:11:33.566068 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.566040    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-installation-pull-secrets\") pod \"image-registry-5f8fccb5d7-fzjx6\" (UID: \"2c761ba8-bf2b-44fa-a5b0-af2ac82da425\") " pod="openshift-image-registry/image-registry-5f8fccb5d7-fzjx6"
Jul 12 22:11:33.566260 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.566086    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-image-registry-private-configuration\") pod \"image-registry-5f8fccb5d7-fzjx6\" (UID: \"2c761ba8-bf2b-44fa-a5b0-af2ac82da425\") " pod="openshift-image-registry/image-registry-5f8fccb5d7-fzjx6"
Jul 12 22:11:33.566260 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.566117    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-ca-trust-extracted\") pod \"image-registry-5f8fccb5d7-fzjx6\" (UID: \"2c761ba8-bf2b-44fa-a5b0-af2ac82da425\") " pod="openshift-image-registry/image-registry-5f8fccb5d7-fzjx6"
Jul 12 22:11:33.566260 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.566248    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxn8n\" (UniqueName: \"kubernetes.io/projected/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-kube-api-access-sxn8n\") pod \"image-registry-5f8fccb5d7-fzjx6\" (UID: \"2c761ba8-bf2b-44fa-a5b0-af2ac82da425\") " pod="openshift-image-registry/image-registry-5f8fccb5d7-fzjx6"
Jul 12 22:11:33.566374 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.566299    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7af611d3-24c8-4266-88d2-23b4523117e2-nginx-conf\") pod \"networking-console-plugin-6cf856758c-tlbnn\" (UID: \"7af611d3-24c8-4266-88d2-23b4523117e2\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-tlbnn"
Jul 12 22:11:33.566374 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.566361    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-trusted-ca\") pod \"image-registry-5f8fccb5d7-fzjx6\" (UID: \"2c761ba8-bf2b-44fa-a5b0-af2ac82da425\") " pod="openshift-image-registry/image-registry-5f8fccb5d7-fzjx6"
Jul 12 22:11:33.614144 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.614110    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5f8fccb5d7-fzjx6"]
Jul 12 22:11:33.614387 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:11:33.614363    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-sxn8n registry-certificates registry-tls trusted-ca], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-5f8fccb5d7-fzjx6" podUID="2c761ba8-bf2b-44fa-a5b0-af2ac82da425"
Jul 12 22:11:33.652477 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.651784    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jhqcn"
Jul 12 22:11:33.665517 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.665479    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6b5b4b6c75-vtlnj"]
Jul 12 22:11:33.667351 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.667318    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-bound-sa-token\") pod \"image-registry-5f8fccb5d7-fzjx6\" (UID: \"2c761ba8-bf2b-44fa-a5b0-af2ac82da425\") " pod="openshift-image-registry/image-registry-5f8fccb5d7-fzjx6"
Jul 12 22:11:33.667493 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.667368    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-registry-certificates\") pod \"image-registry-5f8fccb5d7-fzjx6\" (UID: \"2c761ba8-bf2b-44fa-a5b0-af2ac82da425\") " pod="openshift-image-registry/image-registry-5f8fccb5d7-fzjx6"
Jul 12 22:11:33.667493 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.667392    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-registry-tls\") pod \"image-registry-5f8fccb5d7-fzjx6\" (UID: \"2c761ba8-bf2b-44fa-a5b0-af2ac82da425\") " pod="openshift-image-registry/image-registry-5f8fccb5d7-fzjx6"
Jul 12 22:11:33.667493 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.667425    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7af611d3-24c8-4266-88d2-23b4523117e2-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-tlbnn\" (UID: \"7af611d3-24c8-4266-88d2-23b4523117e2\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-tlbnn"
Jul 12 22:11:33.667493 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.667463    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-installation-pull-secrets\") pod \"image-registry-5f8fccb5d7-fzjx6\" (UID: \"2c761ba8-bf2b-44fa-a5b0-af2ac82da425\") " pod="openshift-image-registry/image-registry-5f8fccb5d7-fzjx6"
Jul 12 22:11:33.667493 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.667488    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-image-registry-private-configuration\") pod \"image-registry-5f8fccb5d7-fzjx6\" (UID: \"2c761ba8-bf2b-44fa-a5b0-af2ac82da425\") " pod="openshift-image-registry/image-registry-5f8fccb5d7-fzjx6"
Jul 12 22:11:33.667723 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.667557    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-ca-trust-extracted\") pod \"image-registry-5f8fccb5d7-fzjx6\" (UID: \"2c761ba8-bf2b-44fa-a5b0-af2ac82da425\") " pod="openshift-image-registry/image-registry-5f8fccb5d7-fzjx6"
Jul 12 22:11:33.667723 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.667694    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxn8n\" (UniqueName: \"kubernetes.io/projected/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-kube-api-access-sxn8n\") pod \"image-registry-5f8fccb5d7-fzjx6\" (UID: \"2c761ba8-bf2b-44fa-a5b0-af2ac82da425\") " pod="openshift-image-registry/image-registry-5f8fccb5d7-fzjx6"
Jul 12 22:11:33.667817 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.667739    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7af611d3-24c8-4266-88d2-23b4523117e2-nginx-conf\") pod \"networking-console-plugin-6cf856758c-tlbnn\" (UID: \"7af611d3-24c8-4266-88d2-23b4523117e2\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-tlbnn"
Jul 12 22:11:33.667817 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.667777    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-trusted-ca\") pod \"image-registry-5f8fccb5d7-fzjx6\" (UID: \"2c761ba8-bf2b-44fa-a5b0-af2ac82da425\") " pod="openshift-image-registry/image-registry-5f8fccb5d7-fzjx6"
Jul 12 22:11:33.668192 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.668167    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-ca-trust-extracted\") pod \"image-registry-5f8fccb5d7-fzjx6\" (UID: \"2c761ba8-bf2b-44fa-a5b0-af2ac82da425\") " pod="openshift-image-registry/image-registry-5f8fccb5d7-fzjx6"
Jul 12 22:11:33.668655 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.668598    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-registry-certificates\") pod \"image-registry-5f8fccb5d7-fzjx6\" (UID: \"2c761ba8-bf2b-44fa-a5b0-af2ac82da425\") " pod="openshift-image-registry/image-registry-5f8fccb5d7-fzjx6"
Jul 12 22:11:33.669223 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.669198    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-trusted-ca\") pod \"image-registry-5f8fccb5d7-fzjx6\" (UID: \"2c761ba8-bf2b-44fa-a5b0-af2ac82da425\") " pod="openshift-image-registry/image-registry-5f8fccb5d7-fzjx6"
Jul 12 22:11:33.670917 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.670894    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-installation-pull-secrets\") pod \"image-registry-5f8fccb5d7-fzjx6\" (UID: \"2c761ba8-bf2b-44fa-a5b0-af2ac82da425\") " pod="openshift-image-registry/image-registry-5f8fccb5d7-fzjx6"
Jul 12 22:11:33.671102 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.671079    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7af611d3-24c8-4266-88d2-23b4523117e2-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-tlbnn\" (UID: \"7af611d3-24c8-4266-88d2-23b4523117e2\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-tlbnn"
Jul 12 22:11:33.671466 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.671444    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-registry-tls\") pod \"image-registry-5f8fccb5d7-fzjx6\" (UID: \"2c761ba8-bf2b-44fa-a5b0-af2ac82da425\") " pod="openshift-image-registry/image-registry-5f8fccb5d7-fzjx6"
Jul 12 22:11:33.671616 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.671597    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-image-registry-private-configuration\") pod \"image-registry-5f8fccb5d7-fzjx6\" (UID: \"2c761ba8-bf2b-44fa-a5b0-af2ac82da425\") " pod="openshift-image-registry/image-registry-5f8fccb5d7-fzjx6"
Jul 12 22:11:33.671688 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.671608    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b5b4b6c75-vtlnj"
Jul 12 22:11:33.686675 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.686607    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b5b4b6c75-vtlnj"]
Jul 12 22:11:33.688411 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.688380    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f8fccb5d7-fzjx6"
Jul 12 22:11:33.688615 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.688596    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxn8n\" (UniqueName: \"kubernetes.io/projected/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-kube-api-access-sxn8n\") pod \"image-registry-5f8fccb5d7-fzjx6\" (UID: \"2c761ba8-bf2b-44fa-a5b0-af2ac82da425\") " pod="openshift-image-registry/image-registry-5f8fccb5d7-fzjx6"
Jul 12 22:11:33.692281 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.692252    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-bound-sa-token\") pod \"image-registry-5f8fccb5d7-fzjx6\" (UID: \"2c761ba8-bf2b-44fa-a5b0-af2ac82da425\") " pod="openshift-image-registry/image-registry-5f8fccb5d7-fzjx6"
Jul 12 22:11:33.714313 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.714284    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f8fccb5d7-fzjx6"
Jul 12 22:11:33.768865 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.768780    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-bound-sa-token\") pod \"2c761ba8-bf2b-44fa-a5b0-af2ac82da425\" (UID: \"2c761ba8-bf2b-44fa-a5b0-af2ac82da425\") "
Jul 12 22:11:33.768865 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.768830    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-installation-pull-secrets\") pod \"2c761ba8-bf2b-44fa-a5b0-af2ac82da425\" (UID: \"2c761ba8-bf2b-44fa-a5b0-af2ac82da425\") "
Jul 12 22:11:33.769089 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.768879    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-registry-certificates\") pod \"2c761ba8-bf2b-44fa-a5b0-af2ac82da425\" (UID: \"2c761ba8-bf2b-44fa-a5b0-af2ac82da425\") "
Jul 12 22:11:33.769089 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.768942    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxn8n\" (UniqueName: \"kubernetes.io/projected/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-kube-api-access-sxn8n\") pod \"2c761ba8-bf2b-44fa-a5b0-af2ac82da425\" (UID: \"2c761ba8-bf2b-44fa-a5b0-af2ac82da425\") "
Jul 12 22:11:33.769089 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.768989    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-image-registry-private-configuration\") pod \"2c761ba8-bf2b-44fa-a5b0-af2ac82da425\" (UID: \"2c761ba8-bf2b-44fa-a5b0-af2ac82da425\") "
Jul 12 22:11:33.769089 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.769009    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-registry-tls\") pod \"2c761ba8-bf2b-44fa-a5b0-af2ac82da425\" (UID: \"2c761ba8-bf2b-44fa-a5b0-af2ac82da425\") "
Jul 12 22:11:33.769089 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.769044    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-ca-trust-extracted\") pod \"2c761ba8-bf2b-44fa-a5b0-af2ac82da425\" (UID: \"2c761ba8-bf2b-44fa-a5b0-af2ac82da425\") "
Jul 12 22:11:33.769089 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.769075    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-trusted-ca\") pod \"2c761ba8-bf2b-44fa-a5b0-af2ac82da425\" (UID: \"2c761ba8-bf2b-44fa-a5b0-af2ac82da425\") "
Jul 12 22:11:33.769333 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.769272    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ede4fc85-270f-40d3-8e4e-8f0d47d673bd-registry-tls\") pod \"image-registry-6b5b4b6c75-vtlnj\" (UID: \"ede4fc85-270f-40d3-8e4e-8f0d47d673bd\") " pod="openshift-image-registry/image-registry-6b5b4b6c75-vtlnj"
Jul 12 22:11:33.769371 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.769339    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ede4fc85-270f-40d3-8e4e-8f0d47d673bd-installation-pull-secrets\") pod \"image-registry-6b5b4b6c75-vtlnj\" (UID: \"ede4fc85-270f-40d3-8e4e-8f0d47d673bd\") " pod="openshift-image-registry/image-registry-6b5b4b6c75-vtlnj"
Jul 12 22:11:33.769426 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.769378    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ede4fc85-270f-40d3-8e4e-8f0d47d673bd-trusted-ca\") pod \"image-registry-6b5b4b6c75-vtlnj\" (UID: \"ede4fc85-270f-40d3-8e4e-8f0d47d673bd\") " pod="openshift-image-registry/image-registry-6b5b4b6c75-vtlnj"
Jul 12 22:11:33.769426 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.769403    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ede4fc85-270f-40d3-8e4e-8f0d47d673bd-ca-trust-extracted\") pod \"image-registry-6b5b4b6c75-vtlnj\" (UID: \"ede4fc85-270f-40d3-8e4e-8f0d47d673bd\") " pod="openshift-image-registry/image-registry-6b5b4b6c75-vtlnj"
Jul 12 22:11:33.769426 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.769421    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ede4fc85-270f-40d3-8e4e-8f0d47d673bd-registry-certificates\") pod \"image-registry-6b5b4b6c75-vtlnj\" (UID: \"ede4fc85-270f-40d3-8e4e-8f0d47d673bd\") " pod="openshift-image-registry/image-registry-6b5b4b6c75-vtlnj"
Jul 12 22:11:33.769600 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.769489    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ede4fc85-270f-40d3-8e4e-8f0d47d673bd-image-registry-private-configuration\") pod \"image-registry-6b5b4b6c75-vtlnj\" (UID: \"ede4fc85-270f-40d3-8e4e-8f0d47d673bd\") " pod="openshift-image-registry/image-registry-6b5b4b6c75-vtlnj"
Jul 12 22:11:33.769600 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.769513    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ede4fc85-270f-40d3-8e4e-8f0d47d673bd-bound-sa-token\") pod \"image-registry-6b5b4b6c75-vtlnj\" (UID: \"ede4fc85-270f-40d3-8e4e-8f0d47d673bd\") " pod="openshift-image-registry/image-registry-6b5b4b6c75-vtlnj"
Jul 12 22:11:33.769600 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.769554    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf92v\" (UniqueName: \"kubernetes.io/projected/ede4fc85-270f-40d3-8e4e-8f0d47d673bd-kube-api-access-wf92v\") pod \"image-registry-6b5b4b6c75-vtlnj\" (UID: \"ede4fc85-270f-40d3-8e4e-8f0d47d673bd\") " pod="openshift-image-registry/image-registry-6b5b4b6c75-vtlnj"
Jul 12 22:11:33.769914 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.769678    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "2c761ba8-bf2b-44fa-a5b0-af2ac82da425" (UID: "2c761ba8-bf2b-44fa-a5b0-af2ac82da425"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:11:33.769914 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.769772    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "2c761ba8-bf2b-44fa-a5b0-af2ac82da425" (UID: "2c761ba8-bf2b-44fa-a5b0-af2ac82da425"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 12 22:11:33.770312 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.770283    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "2c761ba8-bf2b-44fa-a5b0-af2ac82da425" (UID: "2c761ba8-bf2b-44fa-a5b0-af2ac82da425"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 12 22:11:33.771576 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.771524    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "2c761ba8-bf2b-44fa-a5b0-af2ac82da425" (UID: "2c761ba8-bf2b-44fa-a5b0-af2ac82da425"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 12 22:11:33.771700 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.771675    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-kube-api-access-sxn8n" (OuterVolumeSpecName: "kube-api-access-sxn8n") pod "2c761ba8-bf2b-44fa-a5b0-af2ac82da425" (UID: "2c761ba8-bf2b-44fa-a5b0-af2ac82da425"). InnerVolumeSpecName "kube-api-access-sxn8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 22:11:33.771757 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.771722    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "2c761ba8-bf2b-44fa-a5b0-af2ac82da425" (UID: "2c761ba8-bf2b-44fa-a5b0-af2ac82da425"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 22:11:33.772257 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.772179    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2c761ba8-bf2b-44fa-a5b0-af2ac82da425" (UID: "2c761ba8-bf2b-44fa-a5b0-af2ac82da425"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 22:11:33.773577 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.773551    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "2c761ba8-bf2b-44fa-a5b0-af2ac82da425" (UID: "2c761ba8-bf2b-44fa-a5b0-af2ac82da425"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 12 22:11:33.870156 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.870119    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ede4fc85-270f-40d3-8e4e-8f0d47d673bd-registry-tls\") pod \"image-registry-6b5b4b6c75-vtlnj\" (UID: \"ede4fc85-270f-40d3-8e4e-8f0d47d673bd\") " pod="openshift-image-registry/image-registry-6b5b4b6c75-vtlnj"
Jul 12 22:11:33.870344 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.870175    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ede4fc85-270f-40d3-8e4e-8f0d47d673bd-installation-pull-secrets\") pod \"image-registry-6b5b4b6c75-vtlnj\" (UID: \"ede4fc85-270f-40d3-8e4e-8f0d47d673bd\") " pod="openshift-image-registry/image-registry-6b5b4b6c75-vtlnj"
Jul 12 22:11:33.870344 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.870206    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ede4fc85-270f-40d3-8e4e-8f0d47d673bd-trusted-ca\") pod \"image-registry-6b5b4b6c75-vtlnj\" (UID: \"ede4fc85-270f-40d3-8e4e-8f0d47d673bd\") " pod="openshift-image-registry/image-registry-6b5b4b6c75-vtlnj"
Jul 12 22:11:33.870344 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.870234    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ede4fc85-270f-40d3-8e4e-8f0d47d673bd-ca-trust-extracted\") pod \"image-registry-6b5b4b6c75-vtlnj\" (UID: \"ede4fc85-270f-40d3-8e4e-8f0d47d673bd\") " pod="openshift-image-registry/image-registry-6b5b4b6c75-vtlnj"
Jul 12 22:11:33.870344 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.870260    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ede4fc85-270f-40d3-8e4e-8f0d47d673bd-registry-certificates\") pod \"image-registry-6b5b4b6c75-vtlnj\" (UID: \"ede4fc85-270f-40d3-8e4e-8f0d47d673bd\") " pod="openshift-image-registry/image-registry-6b5b4b6c75-vtlnj"
Jul 12 22:11:33.870344 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.870313    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ede4fc85-270f-40d3-8e4e-8f0d47d673bd-image-registry-private-configuration\") pod \"image-registry-6b5b4b6c75-vtlnj\" (UID: \"ede4fc85-270f-40d3-8e4e-8f0d47d673bd\") " pod="openshift-image-registry/image-registry-6b5b4b6c75-vtlnj"
Jul 12 22:11:33.870344 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.870338    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ede4fc85-270f-40d3-8e4e-8f0d47d673bd-bound-sa-token\") pod \"image-registry-6b5b4b6c75-vtlnj\" (UID: \"ede4fc85-270f-40d3-8e4e-8f0d47d673bd\") " pod="openshift-image-registry/image-registry-6b5b4b6c75-vtlnj"
Jul 12 22:11:33.870637 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.870362    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf92v\" (UniqueName: \"kubernetes.io/projected/ede4fc85-270f-40d3-8e4e-8f0d47d673bd-kube-api-access-wf92v\") pod \"image-registry-6b5b4b6c75-vtlnj\" (UID: \"ede4fc85-270f-40d3-8e4e-8f0d47d673bd\") " pod="openshift-image-registry/image-registry-6b5b4b6c75-vtlnj"
Jul 12 22:11:33.870637 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.870464    2560 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-registry-certificates\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:11:33.870637 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.870482    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxn8n\" (UniqueName: \"kubernetes.io/projected/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-kube-api-access-sxn8n\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:11:33.870637 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.870496    2560 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-image-registry-private-configuration\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:11:33.870637 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.870509    2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-registry-tls\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:11:33.870637 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.870521    2560 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-ca-trust-extracted\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:11:33.870637 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.870553    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-trusted-ca\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:11:33.870637 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.870566    2560 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-bound-sa-token\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:11:33.870637 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.870579    2560 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2c761ba8-bf2b-44fa-a5b0-af2ac82da425-installation-pull-secrets\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:11:33.871636 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.871505    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ede4fc85-270f-40d3-8e4e-8f0d47d673bd-trusted-ca\") pod \"image-registry-6b5b4b6c75-vtlnj\" (UID: \"ede4fc85-270f-40d3-8e4e-8f0d47d673bd\") " pod="openshift-image-registry/image-registry-6b5b4b6c75-vtlnj"
Jul 12 22:11:33.871636 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.871567    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ede4fc85-270f-40d3-8e4e-8f0d47d673bd-ca-trust-extracted\") pod \"image-registry-6b5b4b6c75-vtlnj\" (UID: \"ede4fc85-270f-40d3-8e4e-8f0d47d673bd\") " pod="openshift-image-registry/image-registry-6b5b4b6c75-vtlnj"
Jul 12 22:11:33.871914 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.871695    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ede4fc85-270f-40d3-8e4e-8f0d47d673bd-registry-certificates\") pod \"image-registry-6b5b4b6c75-vtlnj\" (UID: \"ede4fc85-270f-40d3-8e4e-8f0d47d673bd\") " pod="openshift-image-registry/image-registry-6b5b4b6c75-vtlnj"
Jul 12 22:11:33.873227 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.873198    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ede4fc85-270f-40d3-8e4e-8f0d47d673bd-registry-tls\") pod \"image-registry-6b5b4b6c75-vtlnj\" (UID: \"ede4fc85-270f-40d3-8e4e-8f0d47d673bd\") " pod="openshift-image-registry/image-registry-6b5b4b6c75-vtlnj"
Jul 12 22:11:33.873597 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.873576    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ede4fc85-270f-40d3-8e4e-8f0d47d673bd-image-registry-private-configuration\") pod \"image-registry-6b5b4b6c75-vtlnj\" (UID: \"ede4fc85-270f-40d3-8e4e-8f0d47d673bd\") " pod="openshift-image-registry/image-registry-6b5b4b6c75-vtlnj"
Jul 12 22:11:33.873767 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.873748    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ede4fc85-270f-40d3-8e4e-8f0d47d673bd-installation-pull-secrets\") pod \"image-registry-6b5b4b6c75-vtlnj\" (UID: \"ede4fc85-270f-40d3-8e4e-8f0d47d673bd\") " pod="openshift-image-registry/image-registry-6b5b4b6c75-vtlnj"
Jul 12 22:11:33.879043 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.879013    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ede4fc85-270f-40d3-8e4e-8f0d47d673bd-bound-sa-token\") pod \"image-registry-6b5b4b6c75-vtlnj\" (UID: \"ede4fc85-270f-40d3-8e4e-8f0d47d673bd\") " pod="openshift-image-registry/image-registry-6b5b4b6c75-vtlnj"
Jul 12 22:11:33.879226 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.879209    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf92v\" (UniqueName: \"kubernetes.io/projected/ede4fc85-270f-40d3-8e4e-8f0d47d673bd-kube-api-access-wf92v\") pod \"image-registry-6b5b4b6c75-vtlnj\" (UID: \"ede4fc85-270f-40d3-8e4e-8f0d47d673bd\") " pod="openshift-image-registry/image-registry-6b5b4b6c75-vtlnj"
Jul 12 22:11:33.971119 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.971078    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/551bf86c-bd1a-40be-b977-41205da830c2-metrics-certs\") pod \"network-metrics-daemon-st67z\" (UID: \"551bf86c-bd1a-40be-b977-41205da830c2\") " pod="openshift-multus/network-metrics-daemon-st67z"
Jul 12 22:11:33.974082 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.974048    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 12 22:11:33.984205 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:33.984178    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/551bf86c-bd1a-40be-b977-41205da830c2-metrics-certs\") pod \"network-metrics-daemon-st67z\" (UID: \"551bf86c-bd1a-40be-b977-41205da830c2\") " pod="openshift-multus/network-metrics-daemon-st67z"
Jul 12 22:11:34.010732 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:34.010700    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b5b4b6c75-vtlnj"
Jul 12 22:11:34.172818 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:34.172753    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpv7p\" (UniqueName: \"kubernetes.io/projected/bcf4ee57-809d-4e8b-be7f-593b88689c16-kube-api-access-bpv7p\") pod \"network-check-target-cn4d2\" (UID: \"bcf4ee57-809d-4e8b-be7f-593b88689c16\") " pod="openshift-network-diagnostics/network-check-target-cn4d2"
Jul 12 22:11:34.172989 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:34.172826    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/700bcb2a-4c93-4abe-b403-9b404b3731e3-original-pull-secret\") pod \"global-pull-secret-syncer-b7t5r\" (UID: \"700bcb2a-4c93-4abe-b403-9b404b3731e3\") " pod="kube-system/global-pull-secret-syncer-b7t5r"
Jul 12 22:11:34.175812 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:34.175785    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 12 22:11:34.175956 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:34.175891    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpv7p\" (UniqueName: \"kubernetes.io/projected/bcf4ee57-809d-4e8b-be7f-593b88689c16-kube-api-access-bpv7p\") pod \"network-check-target-cn4d2\" (UID: \"bcf4ee57-809d-4e8b-be7f-593b88689c16\") " pod="openshift-network-diagnostics/network-check-target-cn4d2"
Jul 12 22:11:34.177889 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:34.177868    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-npkdh\""
Jul 12 22:11:34.185591 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:34.185571    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-st67z"
Jul 12 22:11:34.185850 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:34.185826    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/700bcb2a-4c93-4abe-b403-9b404b3731e3-original-pull-secret\") pod \"global-pull-secret-syncer-b7t5r\" (UID: \"700bcb2a-4c93-4abe-b403-9b404b3731e3\") " pod="kube-system/global-pull-secret-syncer-b7t5r"
Jul 12 22:11:34.353504 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:34.353477    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bgjrf\""
Jul 12 22:11:34.361362 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:34.361332    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cn4d2"
Jul 12 22:11:34.468809 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:34.468729    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b7t5r"
Jul 12 22:11:34.593058 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:34.593027    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 12 22:11:34.599854 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:34.599826    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7af611d3-24c8-4266-88d2-23b4523117e2-nginx-conf\") pod \"networking-console-plugin-6cf856758c-tlbnn\" (UID: \"7af611d3-24c8-4266-88d2-23b4523117e2\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-tlbnn"
Jul 12 22:11:34.606379 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:34.606359    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-76psl\""
Jul 12 22:11:34.700086 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:34.697474    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f8fccb5d7-fzjx6"
Jul 12 22:11:34.702216 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:34.702190    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-tlbnn"
Jul 12 22:11:34.774179 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:34.774142    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5f8fccb5d7-fzjx6"]
Jul 12 22:11:34.779276 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:34.779245    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5f8fccb5d7-fzjx6"]
Jul 12 22:11:34.873824 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:34.873796    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cn4d2"]
Jul 12 22:11:34.878085 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:11:34.878058    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcf4ee57_809d_4e8b_be7f_593b88689c16.slice/crio-1721dcfaccf253daedb76d9c3bf7d3aa604ed38c5c10b2894653a825784d4c1d WatchSource:0}: Error finding container 1721dcfaccf253daedb76d9c3bf7d3aa604ed38c5c10b2894653a825784d4c1d: Status 404 returned error can't find the container with id 1721dcfaccf253daedb76d9c3bf7d3aa604ed38c5c10b2894653a825784d4c1d
Jul 12 22:11:34.911643 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:34.911602    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-tlbnn"]
Jul 12 22:11:34.918626 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:11:34.918599    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7af611d3_24c8_4266_88d2_23b4523117e2.slice/crio-c6016c7b9908d2c0d6469052bfeb9fe6fd828fe1445e581fde1e67f26d50a16b WatchSource:0}: Error finding container c6016c7b9908d2c0d6469052bfeb9fe6fd828fe1445e581fde1e67f26d50a16b: Status 404 returned error can't find the container with id c6016c7b9908d2c0d6469052bfeb9fe6fd828fe1445e581fde1e67f26d50a16b
Jul 12 22:11:35.008571 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:35.008522    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b7t5r"]
Jul 12 22:11:35.010405 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:11:35.010376    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod700bcb2a_4c93_4abe_b403_9b404b3731e3.slice/crio-cce13697e3ad2359bd19cc6e5c571d6a64dd67d8a6879a63640e20f170fa2cbd WatchSource:0}: Error finding container cce13697e3ad2359bd19cc6e5c571d6a64dd67d8a6879a63640e20f170fa2cbd: Status 404 returned error can't find the container with id cce13697e3ad2359bd19cc6e5c571d6a64dd67d8a6879a63640e20f170fa2cbd
Jul 12 22:11:35.023806 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:35.023772    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-st67z"]
Jul 12 22:11:35.027686 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:11:35.027660    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod551bf86c_bd1a_40be_b977_41205da830c2.slice/crio-dd35325d782e581177dd389780c446f4f56a5f90b8016a1ec3673417822b014d WatchSource:0}: Error finding container dd35325d782e581177dd389780c446f4f56a5f90b8016a1ec3673417822b014d: Status 404 returned error can't find the container with id dd35325d782e581177dd389780c446f4f56a5f90b8016a1ec3673417822b014d
Jul 12 22:11:35.028129 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:35.028106    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b5b4b6c75-vtlnj"]
Jul 12 22:11:35.030483 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:11:35.030458    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podede4fc85_270f_40d3_8e4e_8f0d47d673bd.slice/crio-bde7b98bc657f839c034394ec9c431523ea09852a78ea5aa9abc816852bb3ec2 WatchSource:0}: Error finding container bde7b98bc657f839c034394ec9c431523ea09852a78ea5aa9abc816852bb3ec2: Status 404 returned error can't find the container with id bde7b98bc657f839c034394ec9c431523ea09852a78ea5aa9abc816852bb3ec2
Jul 12 22:11:35.705292 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:35.705252    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mjg7l" event={"ID":"05f8f52f-6343-4d4c-af3b-028de417ace6","Type":"ContainerStarted","Data":"9eab46a5f0c4d24101d85fa2f45bfb2214416e4707607d4b6be93779f87a98e7"}
Jul 12 22:11:35.708087 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:35.708053    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-st67z" event={"ID":"551bf86c-bd1a-40be-b977-41205da830c2","Type":"ContainerStarted","Data":"dd35325d782e581177dd389780c446f4f56a5f90b8016a1ec3673417822b014d"}
Jul 12 22:11:35.709803 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:35.709777    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-tlbnn" event={"ID":"7af611d3-24c8-4266-88d2-23b4523117e2","Type":"ContainerStarted","Data":"c6016c7b9908d2c0d6469052bfeb9fe6fd828fe1445e581fde1e67f26d50a16b"}
Jul 12 22:11:35.712467 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:35.712440    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b5b4b6c75-vtlnj" event={"ID":"ede4fc85-270f-40d3-8e4e-8f0d47d673bd","Type":"ContainerStarted","Data":"021504f2111baaae2ea2778a1c5411d7c9650aa78b866b50247b341e618b465d"}
Jul 12 22:11:35.712591 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:35.712488    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b5b4b6c75-vtlnj" event={"ID":"ede4fc85-270f-40d3-8e4e-8f0d47d673bd","Type":"ContainerStarted","Data":"bde7b98bc657f839c034394ec9c431523ea09852a78ea5aa9abc816852bb3ec2"}
Jul 12 22:11:35.713097 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:35.713073    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6b5b4b6c75-vtlnj"
Jul 12 22:11:35.714522 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:35.714476    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b7t5r" event={"ID":"700bcb2a-4c93-4abe-b403-9b404b3731e3","Type":"ContainerStarted","Data":"cce13697e3ad2359bd19cc6e5c571d6a64dd67d8a6879a63640e20f170fa2cbd"}
Jul 12 22:11:35.717208 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:35.716672    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cn4d2" event={"ID":"bcf4ee57-809d-4e8b-be7f-593b88689c16","Type":"ContainerStarted","Data":"96f5a56a3fd8fa2832c9fba61b3f3a80d8206c872c7d6c0698ec4ae1fba30c8d"}
Jul 12 22:11:35.717208 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:35.716713    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cn4d2" event={"ID":"bcf4ee57-809d-4e8b-be7f-593b88689c16","Type":"ContainerStarted","Data":"1721dcfaccf253daedb76d9c3bf7d3aa604ed38c5c10b2894653a825784d4c1d"}
Jul 12 22:11:35.717208 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:35.717170    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cn4d2"
Jul 12 22:11:35.727023 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:35.726959    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mjg7l" podStartSLOduration=18.277405329 podStartE2EDuration="25.726939893s" totalImagesPullingTime="7.449534564s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 22:11:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:11:35.726761704 +0000 UTC m=+68.040265058" watchObservedRunningTime="2026-07-12 22:11:35.726939893 +0000 UTC m=+68.040443248"
Jul 12 22:11:35.753434 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:35.753372    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6b5b4b6c75-vtlnj" podStartSLOduration=2.753351625 podStartE2EDuration="2.753351625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:11:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:11:35.752591624 +0000 UTC m=+68.066094976" watchObservedRunningTime="2026-07-12 22:11:35.753351625 +0000 UTC m=+68.066854977"
Jul 12 22:11:35.770947 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:35.770578    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-cn4d2" podStartSLOduration=67.770561927 podStartE2EDuration="1m7.770561927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:10:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:11:35.769794691 +0000 UTC m=+68.083298046" watchObservedRunningTime="2026-07-12 22:11:35.770561927 +0000 UTC m=+68.084065283"
Jul 12 22:11:36.246602 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:36.246563    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c761ba8-bf2b-44fa-a5b0-af2ac82da425" path="/var/lib/kubelet/pods/2c761ba8-bf2b-44fa-a5b0-af2ac82da425/volumes"
Jul 12 22:11:36.721918 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:36.721881    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-tlbnn" event={"ID":"7af611d3-24c8-4266-88d2-23b4523117e2","Type":"ContainerStarted","Data":"8eeca5eae33972f1b18f05fe957eaa9cb832112f7d7bbed33fce1386f33b0311"}
Jul 12 22:11:36.724225 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:36.724089    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-st67z" event={"ID":"551bf86c-bd1a-40be-b977-41205da830c2","Type":"ContainerStarted","Data":"def9b3c4accb14806feccc88e76dc09234cc3ed714a7aac903b1aadcee8fd182"}
Jul 12 22:11:36.724225 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:36.724149    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-st67z" event={"ID":"551bf86c-bd1a-40be-b977-41205da830c2","Type":"ContainerStarted","Data":"98eb7557e0fa6f9a6ef137366e131b8f38efb4919f6ded48cf3b9106cfa9e0b3"}
Jul 12 22:11:36.739639 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:36.739589    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-tlbnn" podStartSLOduration=2.161119353 podStartE2EDuration="3.73956967s" totalImagesPullingTime="1.578450317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:11:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:11:36.738242952 +0000 UTC m=+69.051746306" watchObservedRunningTime="2026-07-12 22:11:36.73956967 +0000 UTC m=+69.053073023"
Jul 12 22:11:36.762609 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:36.759084    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-st67z" podStartSLOduration=67.292181778 podStartE2EDuration="1m8.7590613s" totalImagesPullingTime="1.466879522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:10:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:11:36.756835502 +0000 UTC m=+69.070338858" watchObservedRunningTime="2026-07-12 22:11:36.7590613 +0000 UTC m=+69.072564654"
Jul 12 22:11:37.806121 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:37.804974    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-248f2\" (UID: \"25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-248f2"
Jul 12 22:11:37.806121 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:37.805084    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc370348-fbc2-4e67-96ce-a1129336bf21-service-ca-bundle\") pod \"router-default-7cb4cc88b6-fscqr\" (UID: \"bc370348-fbc2-4e67-96ce-a1129336bf21\") " pod="openshift-ingress/router-default-7cb4cc88b6-fscqr"
Jul 12 22:11:37.806121 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:37.805891    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc370348-fbc2-4e67-96ce-a1129336bf21-service-ca-bundle\") pod \"router-default-7cb4cc88b6-fscqr\" (UID: \"bc370348-fbc2-4e67-96ce-a1129336bf21\") " pod="openshift-ingress/router-default-7cb4cc88b6-fscqr"
Jul 12 22:11:37.808490 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:37.808463    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-248f2\" (UID: \"25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-248f2"
Jul 12 22:11:37.970189 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:37.970160    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-wdpmv\""
Jul 12 22:11:37.975189 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:37.975166    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-c7pws\""
Jul 12 22:11:37.977234 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:37.977212    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-248f2"
Jul 12 22:11:37.983507 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:37.983483    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7cb4cc88b6-fscqr"
Jul 12 22:11:39.122375 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:39.122175    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7cb4cc88b6-fscqr"]
Jul 12 22:11:39.124871 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:11:39.124844    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc370348_fbc2_4e67_96ce_a1129336bf21.slice/crio-f36fd724d0541eb19c55465b535bd59aadca68086405dd137fc886f43e12667f WatchSource:0}: Error finding container f36fd724d0541eb19c55465b535bd59aadca68086405dd137fc886f43e12667f: Status 404 returned error can't find the container with id f36fd724d0541eb19c55465b535bd59aadca68086405dd137fc886f43e12667f
Jul 12 22:11:39.135860 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:39.135833    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-248f2"]
Jul 12 22:11:39.139816 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:11:39.139791    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25cdd4ee_5dbf_4ce5_b5dd_c4bf9a1f6afa.slice/crio-92fa5dc0d35a599aef8321087d90a9d99b75269dc76231f434b6c12a565af939 WatchSource:0}: Error finding container 92fa5dc0d35a599aef8321087d90a9d99b75269dc76231f434b6c12a565af939: Status 404 returned error can't find the container with id 92fa5dc0d35a599aef8321087d90a9d99b75269dc76231f434b6c12a565af939
Jul 12 22:11:39.741176 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:39.741079    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7cb4cc88b6-fscqr" event={"ID":"bc370348-fbc2-4e67-96ce-a1129336bf21","Type":"ContainerStarted","Data":"3dc06823b1774c4eb0907fe0de052b0dd8df9cae8f6e857e8a89a7bab1c42bd5"}
Jul 12 22:11:39.741176 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:39.741161    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7cb4cc88b6-fscqr" event={"ID":"bc370348-fbc2-4e67-96ce-a1129336bf21","Type":"ContainerStarted","Data":"f36fd724d0541eb19c55465b535bd59aadca68086405dd137fc886f43e12667f"}
Jul 12 22:11:39.742343 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:39.742317    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-248f2" event={"ID":"25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa","Type":"ContainerStarted","Data":"92fa5dc0d35a599aef8321087d90a9d99b75269dc76231f434b6c12a565af939"}
Jul 12 22:11:39.743733 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:39.743715    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b7t5r" event={"ID":"700bcb2a-4c93-4abe-b403-9b404b3731e3","Type":"ContainerStarted","Data":"4bcd59b77da829de1bc7f2c32abb5b5ac578e5a30bf2611d761f0cdfae1c037f"}
Jul 12 22:11:39.762292 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:39.762243    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7cb4cc88b6-fscqr" podStartSLOduration=34.762225068 podStartE2EDuration="34.762225068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:11:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:11:39.761171275 +0000 UTC m=+72.074674641" watchObservedRunningTime="2026-07-12 22:11:39.762225068 +0000 UTC m=+72.075728424"
Jul 12 22:11:39.778169 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:39.778109    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-b7t5r" podStartSLOduration=67.437054209 podStartE2EDuration="1m11.778091765s" totalImagesPullingTime="4.341037556s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 22:10:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:11:39.777136639 +0000 UTC m=+72.090639994" watchObservedRunningTime="2026-07-12 22:11:39.778091765 +0000 UTC m=+72.091595118"
Jul 12 22:11:39.984493 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:39.984455    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7cb4cc88b6-fscqr"
Jul 12 22:11:39.987748 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:39.987721    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7cb4cc88b6-fscqr"
Jul 12 22:11:40.749608 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:40.749553    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7cb4cc88b6-fscqr"
Jul 12 22:11:40.751169 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:40.751146    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7cb4cc88b6-fscqr"
Jul 12 22:11:40.894817 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:40.894776    2560 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778d449776-slpdg" podUID="ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 12 22:11:40.894978 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:40.894875    2560 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778d449776-slpdg"
Jul 12 22:11:40.895456 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:40.895414    2560 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"18efd2db3c06b8e6147b94ed0acb3fbcf9431f00c9f5262ece0676323968ee0b"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778d449776-slpdg" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 12 22:11:40.895550 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:40.895480    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778d449776-slpdg" podUID="ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67" containerName="service-proxy" containerID="cri-o://18efd2db3c06b8e6147b94ed0acb3fbcf9431f00c9f5262ece0676323968ee0b" gracePeriod=30
Jul 12 22:11:41.754888 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:41.754791    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-248f2" event={"ID":"25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa","Type":"ContainerStarted","Data":"c0c494effe122bf07f29a51f8181bae6d56a7c7cdc526e3b1a7de49b85808909"}
Jul 12 22:11:41.757302 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:41.757260    2560 generic.go:356] "Generic (PLEG): container finished" podID="ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67" containerID="18efd2db3c06b8e6147b94ed0acb3fbcf9431f00c9f5262ece0676323968ee0b" exitCode=2
Jul 12 22:11:41.757430 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:41.757333    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778d449776-slpdg" event={"ID":"ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67","Type":"ContainerDied","Data":"18efd2db3c06b8e6147b94ed0acb3fbcf9431f00c9f5262ece0676323968ee0b"}
Jul 12 22:11:41.757430 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:41.757369    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-778d449776-slpdg" event={"ID":"ff3d768b-0f04-4ae3-b7fd-b500d5fa3e67","Type":"ContainerStarted","Data":"0a7b697bb2b79530488eebb83d817206b6d575dc5ffa0873d6de70a78c38d834"}
Jul 12 22:11:41.775364 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:41.775307    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-248f2" podStartSLOduration=34.516953222 podStartE2EDuration="36.775292531s" totalImagesPullingTime="2.258339309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:11:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:11:41.774391736 +0000 UTC m=+74.087895089" watchObservedRunningTime="2026-07-12 22:11:41.775292531 +0000 UTC m=+74.088795881"
Jul 12 22:11:57.707122 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:57.707016    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-28frw"]
Jul 12 22:11:57.713038 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:57.713015    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-28frw"
Jul 12 22:11:57.716080 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:57.716056    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 12 22:11:57.717646 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:57.717624    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 12 22:11:57.717755 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:57.717631    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 12 22:11:57.717755 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:57.717662    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 12 22:11:57.718099 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:57.718085    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-k7pht\""
Jul 12 22:11:57.733458 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:57.733437    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6b5b4b6c75-vtlnj"
Jul 12 22:11:57.857495 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:57.857458    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b2356589-c594-4a1b-8b54-c45101644a44-node-exporter-textfile\") pod \"node-exporter-28frw\" (UID: \"b2356589-c594-4a1b-8b54-c45101644a44\") " pod="openshift-monitoring/node-exporter-28frw"
Jul 12 22:11:57.857495 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:57.857501    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s6zc\" (UniqueName: \"kubernetes.io/projected/b2356589-c594-4a1b-8b54-c45101644a44-kube-api-access-8s6zc\") pod \"node-exporter-28frw\" (UID: \"b2356589-c594-4a1b-8b54-c45101644a44\") " pod="openshift-monitoring/node-exporter-28frw"
Jul 12 22:11:57.857713 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:57.857559    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b2356589-c594-4a1b-8b54-c45101644a44-node-exporter-accelerators-collector-config\") pod \"node-exporter-28frw\" (UID: \"b2356589-c594-4a1b-8b54-c45101644a44\") " pod="openshift-monitoring/node-exporter-28frw"
Jul 12 22:11:57.857713 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:57.857603    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2356589-c594-4a1b-8b54-c45101644a44-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-28frw\" (UID: \"b2356589-c594-4a1b-8b54-c45101644a44\") " pod="openshift-monitoring/node-exporter-28frw"
Jul 12 22:11:57.857713 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:57.857670    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2356589-c594-4a1b-8b54-c45101644a44-sys\") pod \"node-exporter-28frw\" (UID: \"b2356589-c594-4a1b-8b54-c45101644a44\") " pod="openshift-monitoring/node-exporter-28frw"
Jul 12 22:11:57.857865 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:57.857842    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b2356589-c594-4a1b-8b54-c45101644a44-node-exporter-tls\") pod \"node-exporter-28frw\" (UID: \"b2356589-c594-4a1b-8b54-c45101644a44\") " pod="openshift-monitoring/node-exporter-28frw"
Jul 12 22:11:57.857923 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:57.857899    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2356589-c594-4a1b-8b54-c45101644a44-metrics-client-ca\") pod \"node-exporter-28frw\" (UID: \"b2356589-c594-4a1b-8b54-c45101644a44\") " pod="openshift-monitoring/node-exporter-28frw"
Jul 12 22:11:57.857978 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:57.857947    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b2356589-c594-4a1b-8b54-c45101644a44-root\") pod \"node-exporter-28frw\" (UID: \"b2356589-c594-4a1b-8b54-c45101644a44\") " pod="openshift-monitoring/node-exporter-28frw"
Jul 12 22:11:57.858035 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:57.857976    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b2356589-c594-4a1b-8b54-c45101644a44-node-exporter-wtmp\") pod \"node-exporter-28frw\" (UID: \"b2356589-c594-4a1b-8b54-c45101644a44\") " pod="openshift-monitoring/node-exporter-28frw"
Jul 12 22:11:57.959336 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:57.959253    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b2356589-c594-4a1b-8b54-c45101644a44-node-exporter-textfile\") pod \"node-exporter-28frw\" (UID: \"b2356589-c594-4a1b-8b54-c45101644a44\") " pod="openshift-monitoring/node-exporter-28frw"
Jul 12 22:11:57.959336 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:57.959291    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s6zc\" (UniqueName: \"kubernetes.io/projected/b2356589-c594-4a1b-8b54-c45101644a44-kube-api-access-8s6zc\") pod \"node-exporter-28frw\" (UID: \"b2356589-c594-4a1b-8b54-c45101644a44\") " pod="openshift-monitoring/node-exporter-28frw"
Jul 12 22:11:57.959336 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:57.959320    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b2356589-c594-4a1b-8b54-c45101644a44-node-exporter-accelerators-collector-config\") pod \"node-exporter-28frw\" (UID: \"b2356589-c594-4a1b-8b54-c45101644a44\") " pod="openshift-monitoring/node-exporter-28frw"
Jul 12 22:11:57.959641 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:57.959340    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2356589-c594-4a1b-8b54-c45101644a44-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-28frw\" (UID: \"b2356589-c594-4a1b-8b54-c45101644a44\") " pod="openshift-monitoring/node-exporter-28frw"
Jul 12 22:11:57.959641 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:57.959368    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2356589-c594-4a1b-8b54-c45101644a44-sys\") pod \"node-exporter-28frw\" (UID: \"b2356589-c594-4a1b-8b54-c45101644a44\") " pod="openshift-monitoring/node-exporter-28frw"
Jul 12 22:11:57.959641 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:57.959388    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b2356589-c594-4a1b-8b54-c45101644a44-node-exporter-tls\") pod \"node-exporter-28frw\" (UID: \"b2356589-c594-4a1b-8b54-c45101644a44\") " pod="openshift-monitoring/node-exporter-28frw"
Jul 12 22:11:57.959641 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:57.959419    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2356589-c594-4a1b-8b54-c45101644a44-metrics-client-ca\") pod \"node-exporter-28frw\" (UID: \"b2356589-c594-4a1b-8b54-c45101644a44\") " pod="openshift-monitoring/node-exporter-28frw"
Jul 12 22:11:57.959641 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:57.959436    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b2356589-c594-4a1b-8b54-c45101644a44-root\") pod \"node-exporter-28frw\" (UID: \"b2356589-c594-4a1b-8b54-c45101644a44\") " pod="openshift-monitoring/node-exporter-28frw"
Jul 12 22:11:57.959641 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:57.959437    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2356589-c594-4a1b-8b54-c45101644a44-sys\") pod \"node-exporter-28frw\" (UID: \"b2356589-c594-4a1b-8b54-c45101644a44\") " pod="openshift-monitoring/node-exporter-28frw"
Jul 12 22:11:57.959641 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:57.959451    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b2356589-c594-4a1b-8b54-c45101644a44-node-exporter-wtmp\") pod \"node-exporter-28frw\" (UID: \"b2356589-c594-4a1b-8b54-c45101644a44\") " pod="openshift-monitoring/node-exporter-28frw"
Jul 12 22:11:57.959641 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:57.959563    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b2356589-c594-4a1b-8b54-c45101644a44-root\") pod \"node-exporter-28frw\" (UID: \"b2356589-c594-4a1b-8b54-c45101644a44\") " pod="openshift-monitoring/node-exporter-28frw"
Jul 12 22:11:57.959944 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:57.959696    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b2356589-c594-4a1b-8b54-c45101644a44-node-exporter-textfile\") pod \"node-exporter-28frw\" (UID: \"b2356589-c594-4a1b-8b54-c45101644a44\") " pod="openshift-monitoring/node-exporter-28frw"
Jul 12 22:11:57.959944 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:57.959912    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b2356589-c594-4a1b-8b54-c45101644a44-node-exporter-wtmp\") pod \"node-exporter-28frw\" (UID: \"b2356589-c594-4a1b-8b54-c45101644a44\") " pod="openshift-monitoring/node-exporter-28frw"
Jul 12 22:11:57.960046 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:57.959940    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b2356589-c594-4a1b-8b54-c45101644a44-node-exporter-accelerators-collector-config\") pod \"node-exporter-28frw\" (UID: \"b2356589-c594-4a1b-8b54-c45101644a44\") " pod="openshift-monitoring/node-exporter-28frw"
Jul 12 22:11:57.960105 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:57.960050    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2356589-c594-4a1b-8b54-c45101644a44-metrics-client-ca\") pod \"node-exporter-28frw\" (UID: \"b2356589-c594-4a1b-8b54-c45101644a44\") " pod="openshift-monitoring/node-exporter-28frw"
Jul 12 22:11:57.961999 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:57.961969    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b2356589-c594-4a1b-8b54-c45101644a44-node-exporter-tls\") pod \"node-exporter-28frw\" (UID: \"b2356589-c594-4a1b-8b54-c45101644a44\") " pod="openshift-monitoring/node-exporter-28frw"
Jul 12 22:11:57.961999 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:57.961970    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2356589-c594-4a1b-8b54-c45101644a44-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-28frw\" (UID: \"b2356589-c594-4a1b-8b54-c45101644a44\") " pod="openshift-monitoring/node-exporter-28frw"
Jul 12 22:11:57.969355 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:57.969326    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s6zc\" (UniqueName: \"kubernetes.io/projected/b2356589-c594-4a1b-8b54-c45101644a44-kube-api-access-8s6zc\") pod \"node-exporter-28frw\" (UID: \"b2356589-c594-4a1b-8b54-c45101644a44\") " pod="openshift-monitoring/node-exporter-28frw"
Jul 12 22:11:58.022780 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:58.022748    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-28frw"
Jul 12 22:11:58.033771 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:11:58.033729    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2356589_c594_4a1b_8b54_c45101644a44.slice/crio-2cd0c5e7cf9c38b87453c05c48a70e3a48fb2ef16c10f707e419ada8e5752950 WatchSource:0}: Error finding container 2cd0c5e7cf9c38b87453c05c48a70e3a48fb2ef16c10f707e419ada8e5752950: Status 404 returned error can't find the container with id 2cd0c5e7cf9c38b87453c05c48a70e3a48fb2ef16c10f707e419ada8e5752950
Jul 12 22:11:58.823481 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:58.823435    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-28frw" event={"ID":"b2356589-c594-4a1b-8b54-c45101644a44","Type":"ContainerStarted","Data":"2cd0c5e7cf9c38b87453c05c48a70e3a48fb2ef16c10f707e419ada8e5752950"}
Jul 12 22:11:59.828087 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:59.828048    2560 generic.go:356] "Generic (PLEG): container finished" podID="b2356589-c594-4a1b-8b54-c45101644a44" containerID="18db5e06b61298147937b60e1ba3b58bd4e7aaed69b7773c4d1c50fd83bdc3b6" exitCode=0
Jul 12 22:11:59.828459 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:11:59.828145    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-28frw" event={"ID":"b2356589-c594-4a1b-8b54-c45101644a44","Type":"ContainerDied","Data":"18db5e06b61298147937b60e1ba3b58bd4e7aaed69b7773c4d1c50fd83bdc3b6"}
Jul 12 22:12:00.834504 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:12:00.834472    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-28frw" event={"ID":"b2356589-c594-4a1b-8b54-c45101644a44","Type":"ContainerStarted","Data":"5dff315959b21237d174d0a1675956cf2f230b205d43f6f8d6ead7199b3879bc"}
Jul 12 22:12:00.834915 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:12:00.834544    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-28frw" event={"ID":"b2356589-c594-4a1b-8b54-c45101644a44","Type":"ContainerStarted","Data":"9e7911ce06e69a4114c0af24d45c5ff89122513c63190dfa087cd404c4aef3ec"}
Jul 12 22:12:00.857071 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:12:00.857015    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-28frw" podStartSLOduration=2.783138636 podStartE2EDuration="3.856997336s" totalImagesPullingTime="1.001361912s" totalInitContainerRuntime="72.496788ms" isStatefulPod=true podCreationTimestamp="2026-07-12 22:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:12:00.856588067 +0000 UTC m=+93.170091421" watchObservedRunningTime="2026-07-12 22:12:00.856997336 +0000 UTC m=+93.170500689"
Jul 12 22:12:07.731416 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:12:07.731376    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-cn4d2"
Jul 12 22:12:31.949559 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:12:31.949438    2560 generic.go:356] "Generic (PLEG): container finished" podID="f91f26b5-1055-4423-a442-da2443b2d44e" containerID="7c8e3967e332984da4469d0e778b50ddd9c935a9f38624578ac18f483b439015" exitCode=0
Jul 12 22:12:31.949559 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:12:31.949487    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dqc2v" event={"ID":"f91f26b5-1055-4423-a442-da2443b2d44e","Type":"ContainerDied","Data":"7c8e3967e332984da4469d0e778b50ddd9c935a9f38624578ac18f483b439015"}
Jul 12 22:12:31.950094 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:12:31.949939    2560 scope.go:117] "RemoveContainer" containerID="7c8e3967e332984da4469d0e778b50ddd9c935a9f38624578ac18f483b439015"
Jul 12 22:12:32.955671 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:12:32.955632    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-dqc2v" event={"ID":"f91f26b5-1055-4423-a442-da2443b2d44e","Type":"ContainerStarted","Data":"01338f6a4a127cf85a2a0657449d68d13d72ae40539e97ec34b6fdeb1ceb0753"}
Jul 12 22:15:09.753353 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:09.753318    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56d7986bbd-8wmxx"]
Jul 12 22:15:09.756513 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:09.756495    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-56d7986bbd-8wmxx"
Jul 12 22:15:09.763058 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:09.762862    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 12 22:15:09.763058 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:09.762948    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 12 22:15:09.763058 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:09.762970    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 12 22:15:09.763204 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:09.763146    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 12 22:15:09.763338 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:09.763321    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-smkw7\""
Jul 12 22:15:09.781314 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:09.781283    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56d7986bbd-8wmxx"]
Jul 12 22:15:09.886365 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:09.886324    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/af400fc7-0820-496b-83d4-6ffea9eedac1-webhook-cert\") pod \"opendatahub-operator-controller-manager-56d7986bbd-8wmxx\" (UID: \"af400fc7-0820-496b-83d4-6ffea9eedac1\") " pod="opendatahub/opendatahub-operator-controller-manager-56d7986bbd-8wmxx"
Jul 12 22:15:09.886586 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:09.886374    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh6zq\" (UniqueName: \"kubernetes.io/projected/af400fc7-0820-496b-83d4-6ffea9eedac1-kube-api-access-wh6zq\") pod \"opendatahub-operator-controller-manager-56d7986bbd-8wmxx\" (UID: \"af400fc7-0820-496b-83d4-6ffea9eedac1\") " pod="opendatahub/opendatahub-operator-controller-manager-56d7986bbd-8wmxx"
Jul 12 22:15:09.886586 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:09.886453    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af400fc7-0820-496b-83d4-6ffea9eedac1-tmp\") pod \"opendatahub-operator-controller-manager-56d7986bbd-8wmxx\" (UID: \"af400fc7-0820-496b-83d4-6ffea9eedac1\") " pod="opendatahub/opendatahub-operator-controller-manager-56d7986bbd-8wmxx"
Jul 12 22:15:09.886586 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:09.886475    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/af400fc7-0820-496b-83d4-6ffea9eedac1-manifests\") pod \"opendatahub-operator-controller-manager-56d7986bbd-8wmxx\" (UID: \"af400fc7-0820-496b-83d4-6ffea9eedac1\") " pod="opendatahub/opendatahub-operator-controller-manager-56d7986bbd-8wmxx"
Jul 12 22:15:09.886586 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:09.886497    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/af400fc7-0820-496b-83d4-6ffea9eedac1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56d7986bbd-8wmxx\" (UID: \"af400fc7-0820-496b-83d4-6ffea9eedac1\") " pod="opendatahub/opendatahub-operator-controller-manager-56d7986bbd-8wmxx"
Jul 12 22:15:09.987252 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:09.987216    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af400fc7-0820-496b-83d4-6ffea9eedac1-tmp\") pod \"opendatahub-operator-controller-manager-56d7986bbd-8wmxx\" (UID: \"af400fc7-0820-496b-83d4-6ffea9eedac1\") " pod="opendatahub/opendatahub-operator-controller-manager-56d7986bbd-8wmxx"
Jul 12 22:15:09.987252 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:09.987257    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/af400fc7-0820-496b-83d4-6ffea9eedac1-manifests\") pod \"opendatahub-operator-controller-manager-56d7986bbd-8wmxx\" (UID: \"af400fc7-0820-496b-83d4-6ffea9eedac1\") " pod="opendatahub/opendatahub-operator-controller-manager-56d7986bbd-8wmxx"
Jul 12 22:15:09.987487 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:09.987278    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/af400fc7-0820-496b-83d4-6ffea9eedac1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56d7986bbd-8wmxx\" (UID: \"af400fc7-0820-496b-83d4-6ffea9eedac1\") " pod="opendatahub/opendatahub-operator-controller-manager-56d7986bbd-8wmxx"
Jul 12 22:15:09.987487 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:09.987313    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/af400fc7-0820-496b-83d4-6ffea9eedac1-webhook-cert\") pod \"opendatahub-operator-controller-manager-56d7986bbd-8wmxx\" (UID: \"af400fc7-0820-496b-83d4-6ffea9eedac1\") " pod="opendatahub/opendatahub-operator-controller-manager-56d7986bbd-8wmxx"
Jul 12 22:15:09.987487 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:09.987340    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh6zq\" (UniqueName: \"kubernetes.io/projected/af400fc7-0820-496b-83d4-6ffea9eedac1-kube-api-access-wh6zq\") pod \"opendatahub-operator-controller-manager-56d7986bbd-8wmxx\" (UID: \"af400fc7-0820-496b-83d4-6ffea9eedac1\") " pod="opendatahub/opendatahub-operator-controller-manager-56d7986bbd-8wmxx"
Jul 12 22:15:09.987753 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:09.987723    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/af400fc7-0820-496b-83d4-6ffea9eedac1-manifests\") pod \"opendatahub-operator-controller-manager-56d7986bbd-8wmxx\" (UID: \"af400fc7-0820-496b-83d4-6ffea9eedac1\") " pod="opendatahub/opendatahub-operator-controller-manager-56d7986bbd-8wmxx"
Jul 12 22:15:09.989622 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:09.989603    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af400fc7-0820-496b-83d4-6ffea9eedac1-tmp\") pod \"opendatahub-operator-controller-manager-56d7986bbd-8wmxx\" (UID: \"af400fc7-0820-496b-83d4-6ffea9eedac1\") " pod="opendatahub/opendatahub-operator-controller-manager-56d7986bbd-8wmxx"
Jul 12 22:15:09.989940 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:09.989917    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/af400fc7-0820-496b-83d4-6ffea9eedac1-webhook-cert\") pod \"opendatahub-operator-controller-manager-56d7986bbd-8wmxx\" (UID: \"af400fc7-0820-496b-83d4-6ffea9eedac1\") " pod="opendatahub/opendatahub-operator-controller-manager-56d7986bbd-8wmxx"
Jul 12 22:15:09.989981 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:09.989968    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/af400fc7-0820-496b-83d4-6ffea9eedac1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56d7986bbd-8wmxx\" (UID: \"af400fc7-0820-496b-83d4-6ffea9eedac1\") " pod="opendatahub/opendatahub-operator-controller-manager-56d7986bbd-8wmxx"
Jul 12 22:15:10.008834 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:10.008780    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh6zq\" (UniqueName: \"kubernetes.io/projected/af400fc7-0820-496b-83d4-6ffea9eedac1-kube-api-access-wh6zq\") pod \"opendatahub-operator-controller-manager-56d7986bbd-8wmxx\" (UID: \"af400fc7-0820-496b-83d4-6ffea9eedac1\") " pod="opendatahub/opendatahub-operator-controller-manager-56d7986bbd-8wmxx"
Jul 12 22:15:10.067015 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:10.066979    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-56d7986bbd-8wmxx"
Jul 12 22:15:10.227358 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:10.227327    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56d7986bbd-8wmxx"]
Jul 12 22:15:10.230986 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:15:10.230960    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf400fc7_0820_496b_83d4_6ffea9eedac1.slice/crio-18d1bb1f03276eae68e52a38c7801fbd102aff781e09fd186441a4f95ab06a98 WatchSource:0}: Error finding container 18d1bb1f03276eae68e52a38c7801fbd102aff781e09fd186441a4f95ab06a98: Status 404 returned error can't find the container with id 18d1bb1f03276eae68e52a38c7801fbd102aff781e09fd186441a4f95ab06a98
Jul 12 22:15:10.503542 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:10.503482    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56d7986bbd-8wmxx" event={"ID":"af400fc7-0820-496b-83d4-6ffea9eedac1","Type":"ContainerStarted","Data":"18d1bb1f03276eae68e52a38c7801fbd102aff781e09fd186441a4f95ab06a98"}
Jul 12 22:15:13.518751 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:13.518657    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56d7986bbd-8wmxx" event={"ID":"af400fc7-0820-496b-83d4-6ffea9eedac1","Type":"ContainerStarted","Data":"680298303e935e3e174218f781ea1a24ec008b52c65a48f1b04ba42e07881698"}
Jul 12 22:15:14.523630 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:14.523592    2560 generic.go:356] "Generic (PLEG): container finished" podID="af400fc7-0820-496b-83d4-6ffea9eedac1" containerID="680298303e935e3e174218f781ea1a24ec008b52c65a48f1b04ba42e07881698" exitCode=0
Jul 12 22:15:14.524138 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:14.523635    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56d7986bbd-8wmxx" event={"ID":"af400fc7-0820-496b-83d4-6ffea9eedac1","Type":"ContainerDied","Data":"680298303e935e3e174218f781ea1a24ec008b52c65a48f1b04ba42e07881698"}
Jul 12 22:15:15.530316 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:15.530270    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56d7986bbd-8wmxx" event={"ID":"af400fc7-0820-496b-83d4-6ffea9eedac1","Type":"ContainerStarted","Data":"d72303740f121e4470d807386479a06fdfd43935eb896f4d755847233ae6eb11"}
Jul 12 22:15:15.530719 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:15.530346    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-56d7986bbd-8wmxx"
Jul 12 22:15:15.580938 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:15.580882    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-56d7986bbd-8wmxx" podStartSLOduration=2.237171255 podStartE2EDuration="6.580864047s" totalImagesPullingTime="3.329745769s" totalInitContainerRuntime="1.013947023s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:15:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:15:15.580048578 +0000 UTC m=+287.893551932" watchObservedRunningTime="2026-07-12 22:15:15.580864047 +0000 UTC m=+287.894367401"
Jul 12 22:15:26.536562 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:26.536507    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-56d7986bbd-8wmxx"
Jul 12 22:15:28.124141 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:28.124104    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52pbf_1b1f17bc-a992-4c72-8fdc-a33a3c3649c4/ovn-acl-logging/0.log"
Jul 12 22:15:28.130458 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:28.130429    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52pbf_1b1f17bc-a992-4c72-8fdc-a33a3c3649c4/ovn-acl-logging/0.log"
Jul 12 22:15:28.131763 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:28.131741    2560 kubelet.go:1648] "Image garbage collection succeeded"
Jul 12 22:15:29.402556 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:29.401581    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5747db99d8-nm6x2"]
Jul 12 22:15:29.406281 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:29.406254    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5747db99d8-nm6x2"
Jul 12 22:15:29.410293 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:29.410266    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 12 22:15:29.410293 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:29.410281    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-26nhb\""
Jul 12 22:15:29.410459 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:29.410276    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 12 22:15:29.432471 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:29.432434    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5747db99d8-nm6x2"]
Jul 12 22:15:29.443182 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:29.443154    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0b68ad9-67bc-4274-9319-6a9e5979d35e-tls-certs\") pod \"kube-auth-proxy-5747db99d8-nm6x2\" (UID: \"b0b68ad9-67bc-4274-9319-6a9e5979d35e\") " pod="openshift-ingress/kube-auth-proxy-5747db99d8-nm6x2"
Jul 12 22:15:29.443336 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:29.443208    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq2jh\" (UniqueName: \"kubernetes.io/projected/b0b68ad9-67bc-4274-9319-6a9e5979d35e-kube-api-access-xq2jh\") pod \"kube-auth-proxy-5747db99d8-nm6x2\" (UID: \"b0b68ad9-67bc-4274-9319-6a9e5979d35e\") " pod="openshift-ingress/kube-auth-proxy-5747db99d8-nm6x2"
Jul 12 22:15:29.443336 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:29.443243    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0b68ad9-67bc-4274-9319-6a9e5979d35e-tmp\") pod \"kube-auth-proxy-5747db99d8-nm6x2\" (UID: \"b0b68ad9-67bc-4274-9319-6a9e5979d35e\") " pod="openshift-ingress/kube-auth-proxy-5747db99d8-nm6x2"
Jul 12 22:15:29.544507 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:29.544468    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0b68ad9-67bc-4274-9319-6a9e5979d35e-tls-certs\") pod \"kube-auth-proxy-5747db99d8-nm6x2\" (UID: \"b0b68ad9-67bc-4274-9319-6a9e5979d35e\") " pod="openshift-ingress/kube-auth-proxy-5747db99d8-nm6x2"
Jul 12 22:15:29.544718 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:29.544576    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq2jh\" (UniqueName: \"kubernetes.io/projected/b0b68ad9-67bc-4274-9319-6a9e5979d35e-kube-api-access-xq2jh\") pod \"kube-auth-proxy-5747db99d8-nm6x2\" (UID: \"b0b68ad9-67bc-4274-9319-6a9e5979d35e\") " pod="openshift-ingress/kube-auth-proxy-5747db99d8-nm6x2"
Jul 12 22:15:29.544718 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:29.544613    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0b68ad9-67bc-4274-9319-6a9e5979d35e-tmp\") pod \"kube-auth-proxy-5747db99d8-nm6x2\" (UID: \"b0b68ad9-67bc-4274-9319-6a9e5979d35e\") " pod="openshift-ingress/kube-auth-proxy-5747db99d8-nm6x2"
Jul 12 22:15:29.544718 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:15:29.544663    2560 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 12 22:15:29.544894 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:15:29.544751    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0b68ad9-67bc-4274-9319-6a9e5979d35e-tls-certs podName:b0b68ad9-67bc-4274-9319-6a9e5979d35e nodeName:}" failed. No retries permitted until 2026-07-12 22:15:30.044732231 +0000 UTC m=+302.358235573 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b0b68ad9-67bc-4274-9319-6a9e5979d35e-tls-certs") pod "kube-auth-proxy-5747db99d8-nm6x2" (UID: "b0b68ad9-67bc-4274-9319-6a9e5979d35e") : secret "kube-auth-proxy-tls" not found
Jul 12 22:15:29.547099 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:29.547065    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0b68ad9-67bc-4274-9319-6a9e5979d35e-tmp\") pod \"kube-auth-proxy-5747db99d8-nm6x2\" (UID: \"b0b68ad9-67bc-4274-9319-6a9e5979d35e\") " pod="openshift-ingress/kube-auth-proxy-5747db99d8-nm6x2"
Jul 12 22:15:29.567224 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:29.567195    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq2jh\" (UniqueName: \"kubernetes.io/projected/b0b68ad9-67bc-4274-9319-6a9e5979d35e-kube-api-access-xq2jh\") pod \"kube-auth-proxy-5747db99d8-nm6x2\" (UID: \"b0b68ad9-67bc-4274-9319-6a9e5979d35e\") " pod="openshift-ingress/kube-auth-proxy-5747db99d8-nm6x2"
Jul 12 22:15:29.787180 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:29.787106    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-bv65k"]
Jul 12 22:15:29.790769 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:29.790745    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-bv65k"
Jul 12 22:15:29.793998 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:29.793976    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-wjwqt\""
Jul 12 22:15:29.794614 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:29.794599    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 12 22:15:29.819140 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:29.819114    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-bv65k"]
Jul 12 22:15:29.847516 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:29.847485    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjsk9\" (UniqueName: \"kubernetes.io/projected/e56c0e2b-43b5-47df-9181-62cd21c51201-kube-api-access-tjsk9\") pod \"odh-model-controller-7998c67dc4-bv65k\" (UID: \"e56c0e2b-43b5-47df-9181-62cd21c51201\") " pod="opendatahub/odh-model-controller-7998c67dc4-bv65k"
Jul 12 22:15:29.847704 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:29.847607    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e56c0e2b-43b5-47df-9181-62cd21c51201-cert\") pod \"odh-model-controller-7998c67dc4-bv65k\" (UID: \"e56c0e2b-43b5-47df-9181-62cd21c51201\") " pod="opendatahub/odh-model-controller-7998c67dc4-bv65k"
Jul 12 22:15:29.948678 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:29.948634    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e56c0e2b-43b5-47df-9181-62cd21c51201-cert\") pod \"odh-model-controller-7998c67dc4-bv65k\" (UID: \"e56c0e2b-43b5-47df-9181-62cd21c51201\") " pod="opendatahub/odh-model-controller-7998c67dc4-bv65k"
Jul 12 22:15:29.948962 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:29.948722    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjsk9\" (UniqueName: \"kubernetes.io/projected/e56c0e2b-43b5-47df-9181-62cd21c51201-kube-api-access-tjsk9\") pod \"odh-model-controller-7998c67dc4-bv65k\" (UID: \"e56c0e2b-43b5-47df-9181-62cd21c51201\") " pod="opendatahub/odh-model-controller-7998c67dc4-bv65k"
Jul 12 22:15:29.948962 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:15:29.948800    2560 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 12 22:15:29.948962 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:15:29.948873    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e56c0e2b-43b5-47df-9181-62cd21c51201-cert podName:e56c0e2b-43b5-47df-9181-62cd21c51201 nodeName:}" failed. No retries permitted until 2026-07-12 22:15:30.448853769 +0000 UTC m=+302.762357105 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e56c0e2b-43b5-47df-9181-62cd21c51201-cert") pod "odh-model-controller-7998c67dc4-bv65k" (UID: "e56c0e2b-43b5-47df-9181-62cd21c51201") : secret "odh-model-controller-webhook-cert" not found
Jul 12 22:15:29.964169 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:29.964133    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjsk9\" (UniqueName: \"kubernetes.io/projected/e56c0e2b-43b5-47df-9181-62cd21c51201-kube-api-access-tjsk9\") pod \"odh-model-controller-7998c67dc4-bv65k\" (UID: \"e56c0e2b-43b5-47df-9181-62cd21c51201\") " pod="opendatahub/odh-model-controller-7998c67dc4-bv65k"
Jul 12 22:15:30.049323 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:30.049242    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0b68ad9-67bc-4274-9319-6a9e5979d35e-tls-certs\") pod \"kube-auth-proxy-5747db99d8-nm6x2\" (UID: \"b0b68ad9-67bc-4274-9319-6a9e5979d35e\") " pod="openshift-ingress/kube-auth-proxy-5747db99d8-nm6x2"
Jul 12 22:15:30.051775 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:30.051752    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0b68ad9-67bc-4274-9319-6a9e5979d35e-tls-certs\") pod \"kube-auth-proxy-5747db99d8-nm6x2\" (UID: \"b0b68ad9-67bc-4274-9319-6a9e5979d35e\") " pod="openshift-ingress/kube-auth-proxy-5747db99d8-nm6x2"
Jul 12 22:15:30.315749 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:30.315717    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5747db99d8-nm6x2"
Jul 12 22:15:30.452410 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:30.452375    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e56c0e2b-43b5-47df-9181-62cd21c51201-cert\") pod \"odh-model-controller-7998c67dc4-bv65k\" (UID: \"e56c0e2b-43b5-47df-9181-62cd21c51201\") " pod="opendatahub/odh-model-controller-7998c67dc4-bv65k"
Jul 12 22:15:30.455772 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:30.455741    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e56c0e2b-43b5-47df-9181-62cd21c51201-cert\") pod \"odh-model-controller-7998c67dc4-bv65k\" (UID: \"e56c0e2b-43b5-47df-9181-62cd21c51201\") " pod="opendatahub/odh-model-controller-7998c67dc4-bv65k"
Jul 12 22:15:30.477007 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:30.476976    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5747db99d8-nm6x2"]
Jul 12 22:15:30.483178 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:15:30.483147    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0b68ad9_67bc_4274_9319_6a9e5979d35e.slice/crio-3fa3acbb2ad023d923abbec88fb486f87adb3feb17c8ae2aa99d882f2d6e3aa0 WatchSource:0}: Error finding container 3fa3acbb2ad023d923abbec88fb486f87adb3feb17c8ae2aa99d882f2d6e3aa0: Status 404 returned error can't find the container with id 3fa3acbb2ad023d923abbec88fb486f87adb3feb17c8ae2aa99d882f2d6e3aa0
Jul 12 22:15:30.485288 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:30.485272    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 12 22:15:30.589841 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:30.589761    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5747db99d8-nm6x2" event={"ID":"b0b68ad9-67bc-4274-9319-6a9e5979d35e","Type":"ContainerStarted","Data":"3fa3acbb2ad023d923abbec88fb486f87adb3feb17c8ae2aa99d882f2d6e3aa0"}
Jul 12 22:15:30.701074 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:30.701040    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-bv65k"
Jul 12 22:15:30.875303 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:30.875278    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-bv65k"]
Jul 12 22:15:30.877345 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:15:30.877316    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode56c0e2b_43b5_47df_9181_62cd21c51201.slice/crio-b7e77c39f62624cd12ad36306705cc54203e0b46a89729fe76c1329d4a7f703c WatchSource:0}: Error finding container b7e77c39f62624cd12ad36306705cc54203e0b46a89729fe76c1329d4a7f703c: Status 404 returned error can't find the container with id b7e77c39f62624cd12ad36306705cc54203e0b46a89729fe76c1329d4a7f703c
Jul 12 22:15:31.159505 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:31.159420    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-c4vs6"]
Jul 12 22:15:31.163091 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:31.163067    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-c4vs6"
Jul 12 22:15:31.167188 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:31.167161    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 12 22:15:31.179042 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:31.179015    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-c4vs6"]
Jul 12 22:15:31.259683 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:31.259645    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83198b60-f427-4933-949e-78c94964ccb3-tls-certs\") pod \"model-serving-api-54d89bd5fc-c4vs6\" (UID: \"83198b60-f427-4933-949e-78c94964ccb3\") " pod="opendatahub/model-serving-api-54d89bd5fc-c4vs6"
Jul 12 22:15:31.259896 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:31.259756    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jnl5\" (UniqueName: \"kubernetes.io/projected/83198b60-f427-4933-949e-78c94964ccb3-kube-api-access-6jnl5\") pod \"model-serving-api-54d89bd5fc-c4vs6\" (UID: \"83198b60-f427-4933-949e-78c94964ccb3\") " pod="opendatahub/model-serving-api-54d89bd5fc-c4vs6"
Jul 12 22:15:31.360675 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:31.360633    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jnl5\" (UniqueName: \"kubernetes.io/projected/83198b60-f427-4933-949e-78c94964ccb3-kube-api-access-6jnl5\") pod \"model-serving-api-54d89bd5fc-c4vs6\" (UID: \"83198b60-f427-4933-949e-78c94964ccb3\") " pod="opendatahub/model-serving-api-54d89bd5fc-c4vs6"
Jul 12 22:15:31.360877 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:31.360746    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83198b60-f427-4933-949e-78c94964ccb3-tls-certs\") pod \"model-serving-api-54d89bd5fc-c4vs6\" (UID: \"83198b60-f427-4933-949e-78c94964ccb3\") " pod="opendatahub/model-serving-api-54d89bd5fc-c4vs6"
Jul 12 22:15:31.360949 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:15:31.360870    2560 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 12 22:15:31.360949 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:15:31.360945    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83198b60-f427-4933-949e-78c94964ccb3-tls-certs podName:83198b60-f427-4933-949e-78c94964ccb3 nodeName:}" failed. No retries permitted until 2026-07-12 22:15:31.860924258 +0000 UTC m=+304.174427594 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/83198b60-f427-4933-949e-78c94964ccb3-tls-certs") pod "model-serving-api-54d89bd5fc-c4vs6" (UID: "83198b60-f427-4933-949e-78c94964ccb3") : secret "model-serving-api-tls" not found
Jul 12 22:15:31.397022 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:31.396989    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jnl5\" (UniqueName: \"kubernetes.io/projected/83198b60-f427-4933-949e-78c94964ccb3-kube-api-access-6jnl5\") pod \"model-serving-api-54d89bd5fc-c4vs6\" (UID: \"83198b60-f427-4933-949e-78c94964ccb3\") " pod="opendatahub/model-serving-api-54d89bd5fc-c4vs6"
Jul 12 22:15:31.595858 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:31.595822    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-bv65k" event={"ID":"e56c0e2b-43b5-47df-9181-62cd21c51201","Type":"ContainerStarted","Data":"b7e77c39f62624cd12ad36306705cc54203e0b46a89729fe76c1329d4a7f703c"}
Jul 12 22:15:31.775418 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:31.775386    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-bv7ct"]
Jul 12 22:15:31.779036 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:31.779014    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-bv7ct"
Jul 12 22:15:31.782177 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:31.782139    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-sqtxz\""
Jul 12 22:15:31.782984 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:31.782498    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 12 22:15:31.782984 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:31.782726    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 12 22:15:31.782984 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:31.782793    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 12 22:15:31.783200 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:31.783000    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 12 22:15:31.806752 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:31.806727    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-bv7ct"]
Jul 12 22:15:31.865605 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:31.865510    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9022283-b163-40ff-8a81-7df1a7dc6d5e-local-certs\") pod \"istiod-openshift-gateway-7574459656-bv7ct\" (UID: \"e9022283-b163-40ff-8a81-7df1a7dc6d5e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bv7ct"
Jul 12 22:15:31.865605 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:31.865589    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9022283-b163-40ff-8a81-7df1a7dc6d5e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-bv7ct\" (UID: \"e9022283-b163-40ff-8a81-7df1a7dc6d5e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bv7ct"
Jul 12 22:15:31.865825 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:31.865622    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9022283-b163-40ff-8a81-7df1a7dc6d5e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-bv7ct\" (UID: \"e9022283-b163-40ff-8a81-7df1a7dc6d5e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bv7ct"
Jul 12 22:15:31.865825 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:31.865656    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9022283-b163-40ff-8a81-7df1a7dc6d5e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-bv7ct\" (UID: \"e9022283-b163-40ff-8a81-7df1a7dc6d5e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bv7ct"
Jul 12 22:15:31.865825 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:31.865693    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n29tj\" (UniqueName: \"kubernetes.io/projected/e9022283-b163-40ff-8a81-7df1a7dc6d5e-kube-api-access-n29tj\") pod \"istiod-openshift-gateway-7574459656-bv7ct\" (UID: \"e9022283-b163-40ff-8a81-7df1a7dc6d5e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bv7ct"
Jul 12 22:15:31.865825 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:31.865733    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9022283-b163-40ff-8a81-7df1a7dc6d5e-istio-token\") pod \"istiod-openshift-gateway-7574459656-bv7ct\" (UID: \"e9022283-b163-40ff-8a81-7df1a7dc6d5e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bv7ct"
Jul 12 22:15:31.865825 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:31.865757    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e9022283-b163-40ff-8a81-7df1a7dc6d5e-cacerts\") pod \"istiod-openshift-gateway-7574459656-bv7ct\" (UID: \"e9022283-b163-40ff-8a81-7df1a7dc6d5e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bv7ct"
Jul 12 22:15:31.865825 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:31.865798    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83198b60-f427-4933-949e-78c94964ccb3-tls-certs\") pod \"model-serving-api-54d89bd5fc-c4vs6\" (UID: \"83198b60-f427-4933-949e-78c94964ccb3\") " pod="opendatahub/model-serving-api-54d89bd5fc-c4vs6"
Jul 12 22:15:31.869259 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:31.869226    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83198b60-f427-4933-949e-78c94964ccb3-tls-certs\") pod \"model-serving-api-54d89bd5fc-c4vs6\" (UID: \"83198b60-f427-4933-949e-78c94964ccb3\") " pod="opendatahub/model-serving-api-54d89bd5fc-c4vs6"
Jul 12 22:15:31.968409 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:31.967328    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n29tj\" (UniqueName: \"kubernetes.io/projected/e9022283-b163-40ff-8a81-7df1a7dc6d5e-kube-api-access-n29tj\") pod \"istiod-openshift-gateway-7574459656-bv7ct\" (UID: \"e9022283-b163-40ff-8a81-7df1a7dc6d5e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bv7ct"
Jul 12 22:15:31.968409 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:31.967389    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9022283-b163-40ff-8a81-7df1a7dc6d5e-istio-token\") pod \"istiod-openshift-gateway-7574459656-bv7ct\" (UID: \"e9022283-b163-40ff-8a81-7df1a7dc6d5e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bv7ct"
Jul 12 22:15:31.968409 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:31.967414    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e9022283-b163-40ff-8a81-7df1a7dc6d5e-cacerts\") pod \"istiod-openshift-gateway-7574459656-bv7ct\" (UID: \"e9022283-b163-40ff-8a81-7df1a7dc6d5e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bv7ct"
Jul 12 22:15:31.968409 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:31.967484    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9022283-b163-40ff-8a81-7df1a7dc6d5e-local-certs\") pod \"istiod-openshift-gateway-7574459656-bv7ct\" (UID: \"e9022283-b163-40ff-8a81-7df1a7dc6d5e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bv7ct"
Jul 12 22:15:31.968409 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:31.967521    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9022283-b163-40ff-8a81-7df1a7dc6d5e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-bv7ct\" (UID: \"e9022283-b163-40ff-8a81-7df1a7dc6d5e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bv7ct"
Jul 12 22:15:31.968409 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:31.967569    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9022283-b163-40ff-8a81-7df1a7dc6d5e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-bv7ct\" (UID: \"e9022283-b163-40ff-8a81-7df1a7dc6d5e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bv7ct"
Jul 12 22:15:31.968409 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:31.967606    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9022283-b163-40ff-8a81-7df1a7dc6d5e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-bv7ct\" (UID: \"e9022283-b163-40ff-8a81-7df1a7dc6d5e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bv7ct"
Jul 12 22:15:31.969416 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:31.969386    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9022283-b163-40ff-8a81-7df1a7dc6d5e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-bv7ct\" (UID: \"e9022283-b163-40ff-8a81-7df1a7dc6d5e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bv7ct"
Jul 12 22:15:31.977054 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:31.976992    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9022283-b163-40ff-8a81-7df1a7dc6d5e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-bv7ct\" (UID: \"e9022283-b163-40ff-8a81-7df1a7dc6d5e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bv7ct"
Jul 12 22:15:31.978246 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:31.977865    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9022283-b163-40ff-8a81-7df1a7dc6d5e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-bv7ct\" (UID: \"e9022283-b163-40ff-8a81-7df1a7dc6d5e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bv7ct"
Jul 12 22:15:31.978246 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:31.978168    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e9022283-b163-40ff-8a81-7df1a7dc6d5e-cacerts\") pod \"istiod-openshift-gateway-7574459656-bv7ct\" (UID: \"e9022283-b163-40ff-8a81-7df1a7dc6d5e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bv7ct"
Jul 12 22:15:31.979044 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:31.979002    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9022283-b163-40ff-8a81-7df1a7dc6d5e-local-certs\") pod \"istiod-openshift-gateway-7574459656-bv7ct\" (UID: \"e9022283-b163-40ff-8a81-7df1a7dc6d5e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bv7ct"
Jul 12 22:15:31.983098 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:31.983054    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9022283-b163-40ff-8a81-7df1a7dc6d5e-istio-token\") pod \"istiod-openshift-gateway-7574459656-bv7ct\" (UID: \"e9022283-b163-40ff-8a81-7df1a7dc6d5e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bv7ct"
Jul 12 22:15:31.984505 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:31.984460    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n29tj\" (UniqueName: \"kubernetes.io/projected/e9022283-b163-40ff-8a81-7df1a7dc6d5e-kube-api-access-n29tj\") pod \"istiod-openshift-gateway-7574459656-bv7ct\" (UID: \"e9022283-b163-40ff-8a81-7df1a7dc6d5e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-bv7ct"
Jul 12 22:15:32.076383 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:32.076350    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-c4vs6"
Jul 12 22:15:32.094713 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:32.094685    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-bv7ct"
Jul 12 22:15:34.261740 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:34.261663    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-c4vs6"]
Jul 12 22:15:34.266296 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:15:34.266254    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83198b60_f427_4933_949e_78c94964ccb3.slice/crio-4cb0af2e3a79ec28e3f7d157a33f6c0f41343d88369d843ed894ff3367c9afd4 WatchSource:0}: Error finding container 4cb0af2e3a79ec28e3f7d157a33f6c0f41343d88369d843ed894ff3367c9afd4: Status 404 returned error can't find the container with id 4cb0af2e3a79ec28e3f7d157a33f6c0f41343d88369d843ed894ff3367c9afd4
Jul 12 22:15:34.305919 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:34.305896    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-bv7ct"]
Jul 12 22:15:34.308357 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:15:34.308329    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9022283_b163_40ff_8a81_7df1a7dc6d5e.slice/crio-8371f69e58500bf157a6120767e3f36b8c4548c163640b65064437d6cf1af82e WatchSource:0}: Error finding container 8371f69e58500bf157a6120767e3f36b8c4548c163640b65064437d6cf1af82e: Status 404 returned error can't find the container with id 8371f69e58500bf157a6120767e3f36b8c4548c163640b65064437d6cf1af82e
Jul 12 22:15:34.610827 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:34.610753    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-bv65k" event={"ID":"e56c0e2b-43b5-47df-9181-62cd21c51201","Type":"ContainerStarted","Data":"e34505261a1125b1b10e142359465fc5f027c00e791793955fc2abf3e70744ff"}
Jul 12 22:15:34.610984 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:34.610873    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-bv65k"
Jul 12 22:15:34.611985 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:34.611960    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-c4vs6" event={"ID":"83198b60-f427-4933-949e-78c94964ccb3","Type":"ContainerStarted","Data":"4cb0af2e3a79ec28e3f7d157a33f6c0f41343d88369d843ed894ff3367c9afd4"}
Jul 12 22:15:34.613001 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:34.612983    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-bv7ct" event={"ID":"e9022283-b163-40ff-8a81-7df1a7dc6d5e","Type":"ContainerStarted","Data":"8371f69e58500bf157a6120767e3f36b8c4548c163640b65064437d6cf1af82e"}
Jul 12 22:15:34.614320 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:34.614298    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5747db99d8-nm6x2" event={"ID":"b0b68ad9-67bc-4274-9319-6a9e5979d35e","Type":"ContainerStarted","Data":"e5693619e8824d275b905792e22061e2681b4e59fe0228a49ed375e8a41e14a3"}
Jul 12 22:15:34.630824 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:34.630771    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-bv65k" podStartSLOduration=2.399739658 podStartE2EDuration="5.630756316s" totalImagesPullingTime="3.231016658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:15:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:15:34.628989076 +0000 UTC m=+306.942492434" watchObservedRunningTime="2026-07-12 22:15:34.630756316 +0000 UTC m=+306.944259671"
Jul 12 22:15:34.648322 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:34.648254    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5747db99d8-nm6x2" podStartSLOduration=2.066992279 podStartE2EDuration="5.648230493s" totalImagesPullingTime="3.581238214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:15:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:15:34.645854759 +0000 UTC m=+306.959358114" watchObservedRunningTime="2026-07-12 22:15:34.648230493 +0000 UTC m=+306.961733848"
Jul 12 22:15:36.626362 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:36.626319    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-c4vs6" event={"ID":"83198b60-f427-4933-949e-78c94964ccb3","Type":"ContainerStarted","Data":"2a60241a655a0eab2f46ea083461f8cb3def3aa5b321d01d63e36301f90fab34"}
Jul 12 22:15:36.626857 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:36.626517    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-c4vs6"
Jul 12 22:15:36.649238 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:36.649171    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-c4vs6" podStartSLOduration=3.819469159 podStartE2EDuration="5.649147333s" totalImagesPullingTime="1.829678174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:15:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:15:36.647875398 +0000 UTC m=+308.961378757" watchObservedRunningTime="2026-07-12 22:15:36.649147333 +0000 UTC m=+308.962650689"
Jul 12 22:15:37.119290 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:37.119245    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:15:37.119389 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:37.119332    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:15:37.634312 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:37.634274    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-bv7ct" event={"ID":"e9022283-b163-40ff-8a81-7df1a7dc6d5e","Type":"ContainerStarted","Data":"6bfe1404b52246edf90f13efc3e556e02a74b4f4d29fda5d34ff82a9a4b8ab47"}
Jul 12 22:15:37.634716 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:37.634343    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-bv7ct"
Jul 12 22:15:37.662907 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:37.662854    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-bv7ct" podStartSLOduration=3.8544111130000003 podStartE2EDuration="6.662836282s" totalImagesPullingTime="2.808425169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:15:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:15:37.660228672 +0000 UTC m=+309.973732027" watchObservedRunningTime="2026-07-12 22:15:37.662836282 +0000 UTC m=+309.976339635"
Jul 12 22:15:38.640417 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:38.640382    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-bv7ct"
Jul 12 22:15:41.275522 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:41.275492    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-r26w4"]
Jul 12 22:15:41.279156 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:41.279133    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r26w4"
Jul 12 22:15:41.281928 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:41.281908    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 12 22:15:41.282017 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:41.281918    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-wj7hv\""
Jul 12 22:15:41.284800 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:41.284779    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-z8f69"]
Jul 12 22:15:41.287869 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:41.287851    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z8f69"
Jul 12 22:15:41.291211 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:41.291188    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-c4q7g\""
Jul 12 22:15:41.291341 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:41.291316    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 12 22:15:41.293301 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:41.293279    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-r26w4"]
Jul 12 22:15:41.303682 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:41.303647    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-z8f69"]
Jul 12 22:15:41.351046 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:41.351008    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aad212a3-b149-4734-b08e-0d2d90e21379-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-z8f69\" (UID: \"aad212a3-b149-4734-b08e-0d2d90e21379\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z8f69"
Jul 12 22:15:41.351046 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:41.351045    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dt6f\" (UniqueName: \"kubernetes.io/projected/aad212a3-b149-4734-b08e-0d2d90e21379-kube-api-access-8dt6f\") pod \"llmisvc-controller-manager-777c6bd7d5-z8f69\" (UID: \"aad212a3-b149-4734-b08e-0d2d90e21379\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z8f69"
Jul 12 22:15:41.351257 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:41.351071    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph65k\" (UniqueName: \"kubernetes.io/projected/1cd6224b-33bc-41ce-bb56-c25600cc46f1-kube-api-access-ph65k\") pod \"kserve-controller-manager-d6f4b65b8-r26w4\" (UID: \"1cd6224b-33bc-41ce-bb56-c25600cc46f1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-r26w4"
Jul 12 22:15:41.351257 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:41.351113    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1cd6224b-33bc-41ce-bb56-c25600cc46f1-cert\") pod \"kserve-controller-manager-d6f4b65b8-r26w4\" (UID: \"1cd6224b-33bc-41ce-bb56-c25600cc46f1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-r26w4"
Jul 12 22:15:41.452402 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:41.452367    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1cd6224b-33bc-41ce-bb56-c25600cc46f1-cert\") pod \"kserve-controller-manager-d6f4b65b8-r26w4\" (UID: \"1cd6224b-33bc-41ce-bb56-c25600cc46f1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-r26w4"
Jul 12 22:15:41.452605 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:41.452434    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aad212a3-b149-4734-b08e-0d2d90e21379-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-z8f69\" (UID: \"aad212a3-b149-4734-b08e-0d2d90e21379\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z8f69"
Jul 12 22:15:41.452605 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:41.452454    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dt6f\" (UniqueName: \"kubernetes.io/projected/aad212a3-b149-4734-b08e-0d2d90e21379-kube-api-access-8dt6f\") pod \"llmisvc-controller-manager-777c6bd7d5-z8f69\" (UID: \"aad212a3-b149-4734-b08e-0d2d90e21379\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z8f69"
Jul 12 22:15:41.452605 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:41.452474    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph65k\" (UniqueName: \"kubernetes.io/projected/1cd6224b-33bc-41ce-bb56-c25600cc46f1-kube-api-access-ph65k\") pod \"kserve-controller-manager-d6f4b65b8-r26w4\" (UID: \"1cd6224b-33bc-41ce-bb56-c25600cc46f1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-r26w4"
Jul 12 22:15:41.452605 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:15:41.452520    2560 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 12 22:15:41.452605 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:15:41.452593    2560 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 12 22:15:41.452824 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:15:41.452619    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cd6224b-33bc-41ce-bb56-c25600cc46f1-cert podName:1cd6224b-33bc-41ce-bb56-c25600cc46f1 nodeName:}" failed. No retries permitted until 2026-07-12 22:15:41.952601969 +0000 UTC m=+314.266105304 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1cd6224b-33bc-41ce-bb56-c25600cc46f1-cert") pod "kserve-controller-manager-d6f4b65b8-r26w4" (UID: "1cd6224b-33bc-41ce-bb56-c25600cc46f1") : secret "kserve-webhook-server-cert" not found
Jul 12 22:15:41.452824 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:15:41.452643    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aad212a3-b149-4734-b08e-0d2d90e21379-cert podName:aad212a3-b149-4734-b08e-0d2d90e21379 nodeName:}" failed. No retries permitted until 2026-07-12 22:15:41.952625004 +0000 UTC m=+314.266128340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/aad212a3-b149-4734-b08e-0d2d90e21379-cert") pod "llmisvc-controller-manager-777c6bd7d5-z8f69" (UID: "aad212a3-b149-4734-b08e-0d2d90e21379") : secret "llmisvc-webhook-server-cert" not found
Jul 12 22:15:41.462246 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:41.462218    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph65k\" (UniqueName: \"kubernetes.io/projected/1cd6224b-33bc-41ce-bb56-c25600cc46f1-kube-api-access-ph65k\") pod \"kserve-controller-manager-d6f4b65b8-r26w4\" (UID: \"1cd6224b-33bc-41ce-bb56-c25600cc46f1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-r26w4"
Jul 12 22:15:41.462353 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:41.462251    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dt6f\" (UniqueName: \"kubernetes.io/projected/aad212a3-b149-4734-b08e-0d2d90e21379-kube-api-access-8dt6f\") pod \"llmisvc-controller-manager-777c6bd7d5-z8f69\" (UID: \"aad212a3-b149-4734-b08e-0d2d90e21379\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z8f69"
Jul 12 22:15:41.956903 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:41.956859    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aad212a3-b149-4734-b08e-0d2d90e21379-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-z8f69\" (UID: \"aad212a3-b149-4734-b08e-0d2d90e21379\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z8f69"
Jul 12 22:15:41.956903 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:41.956909    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1cd6224b-33bc-41ce-bb56-c25600cc46f1-cert\") pod \"kserve-controller-manager-d6f4b65b8-r26w4\" (UID: \"1cd6224b-33bc-41ce-bb56-c25600cc46f1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-r26w4"
Jul 12 22:15:41.959566 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:41.959522    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1cd6224b-33bc-41ce-bb56-c25600cc46f1-cert\") pod \"kserve-controller-manager-d6f4b65b8-r26w4\" (UID: \"1cd6224b-33bc-41ce-bb56-c25600cc46f1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-r26w4"
Jul 12 22:15:41.959650 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:41.959548    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aad212a3-b149-4734-b08e-0d2d90e21379-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-z8f69\" (UID: \"aad212a3-b149-4734-b08e-0d2d90e21379\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z8f69"
Jul 12 22:15:42.190740 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:42.190712    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r26w4"
Jul 12 22:15:42.199860 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:42.199828    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z8f69"
Jul 12 22:15:42.347982 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:42.347952    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-r26w4"]
Jul 12 22:15:42.350525 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:15:42.350493    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd6224b_33bc_41ce_bb56_c25600cc46f1.slice/crio-6780661aa8997d4a686ed8df983c61ca6d545a8c9ec6252abf1c86f92bced4fd WatchSource:0}: Error finding container 6780661aa8997d4a686ed8df983c61ca6d545a8c9ec6252abf1c86f92bced4fd: Status 404 returned error can't find the container with id 6780661aa8997d4a686ed8df983c61ca6d545a8c9ec6252abf1c86f92bced4fd
Jul 12 22:15:42.382208 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:42.382179    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-z8f69"]
Jul 12 22:15:42.385053 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:15:42.385022    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podaad212a3_b149_4734_b08e_0d2d90e21379.slice/crio-cee6ca03c26a2c0adb189cda1727dae904d8917e0739a778173539e1b40b037d WatchSource:0}: Error finding container cee6ca03c26a2c0adb189cda1727dae904d8917e0739a778173539e1b40b037d: Status 404 returned error can't find the container with id cee6ca03c26a2c0adb189cda1727dae904d8917e0739a778173539e1b40b037d
Jul 12 22:15:42.657161 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:42.657123    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r26w4" event={"ID":"1cd6224b-33bc-41ce-bb56-c25600cc46f1","Type":"ContainerStarted","Data":"6780661aa8997d4a686ed8df983c61ca6d545a8c9ec6252abf1c86f92bced4fd"}
Jul 12 22:15:42.658107 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:42.658086    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z8f69" event={"ID":"aad212a3-b149-4734-b08e-0d2d90e21379","Type":"ContainerStarted","Data":"cee6ca03c26a2c0adb189cda1727dae904d8917e0739a778173539e1b40b037d"}
Jul 12 22:15:43.305088 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:43.305054    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-bv7ct"]
Jul 12 22:15:43.305319 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:43.305268    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-bv7ct" podUID="e9022283-b163-40ff-8a81-7df1a7dc6d5e" containerName="discovery" containerID="cri-o://6bfe1404b52246edf90f13efc3e556e02a74b4f4d29fda5d34ff82a9a4b8ab47" gracePeriod=30
Jul 12 22:15:43.652125 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:43.651963    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-bv7ct"
Jul 12 22:15:43.665626 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:43.665469    2560 generic.go:356] "Generic (PLEG): container finished" podID="e9022283-b163-40ff-8a81-7df1a7dc6d5e" containerID="6bfe1404b52246edf90f13efc3e556e02a74b4f4d29fda5d34ff82a9a4b8ab47" exitCode=0
Jul 12 22:15:43.665626 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:43.665608    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-bv7ct"
Jul 12 22:15:43.665900 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:43.665601    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-bv7ct" event={"ID":"e9022283-b163-40ff-8a81-7df1a7dc6d5e","Type":"ContainerDied","Data":"6bfe1404b52246edf90f13efc3e556e02a74b4f4d29fda5d34ff82a9a4b8ab47"}
Jul 12 22:15:43.665900 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:43.665752    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-bv7ct" event={"ID":"e9022283-b163-40ff-8a81-7df1a7dc6d5e","Type":"ContainerDied","Data":"8371f69e58500bf157a6120767e3f36b8c4548c163640b65064437d6cf1af82e"}
Jul 12 22:15:43.665900 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:43.665785    2560 scope.go:117] "RemoveContainer" containerID="6bfe1404b52246edf90f13efc3e556e02a74b4f4d29fda5d34ff82a9a4b8ab47"
Jul 12 22:15:43.776951 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:43.776918    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9022283-b163-40ff-8a81-7df1a7dc6d5e-local-certs\") pod \"e9022283-b163-40ff-8a81-7df1a7dc6d5e\" (UID: \"e9022283-b163-40ff-8a81-7df1a7dc6d5e\") "
Jul 12 22:15:43.777131 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:43.776968    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9022283-b163-40ff-8a81-7df1a7dc6d5e-istio-token\") pod \"e9022283-b163-40ff-8a81-7df1a7dc6d5e\" (UID: \"e9022283-b163-40ff-8a81-7df1a7dc6d5e\") "
Jul 12 22:15:43.777131 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:43.777001    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n29tj\" (UniqueName: \"kubernetes.io/projected/e9022283-b163-40ff-8a81-7df1a7dc6d5e-kube-api-access-n29tj\") pod \"e9022283-b163-40ff-8a81-7df1a7dc6d5e\" (UID: \"e9022283-b163-40ff-8a81-7df1a7dc6d5e\") "
Jul 12 22:15:43.777131 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:43.777043    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9022283-b163-40ff-8a81-7df1a7dc6d5e-istio-csr-dns-cert\") pod \"e9022283-b163-40ff-8a81-7df1a7dc6d5e\" (UID: \"e9022283-b163-40ff-8a81-7df1a7dc6d5e\") "
Jul 12 22:15:43.777131 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:43.777094    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e9022283-b163-40ff-8a81-7df1a7dc6d5e-cacerts\") pod \"e9022283-b163-40ff-8a81-7df1a7dc6d5e\" (UID: \"e9022283-b163-40ff-8a81-7df1a7dc6d5e\") "
Jul 12 22:15:43.777333 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:43.777138    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9022283-b163-40ff-8a81-7df1a7dc6d5e-istio-csr-ca-configmap\") pod \"e9022283-b163-40ff-8a81-7df1a7dc6d5e\" (UID: \"e9022283-b163-40ff-8a81-7df1a7dc6d5e\") "
Jul 12 22:15:43.777333 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:43.777193    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9022283-b163-40ff-8a81-7df1a7dc6d5e-istio-kubeconfig\") pod \"e9022283-b163-40ff-8a81-7df1a7dc6d5e\" (UID: \"e9022283-b163-40ff-8a81-7df1a7dc6d5e\") "
Jul 12 22:15:43.777797 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:43.777762    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9022283-b163-40ff-8a81-7df1a7dc6d5e-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "e9022283-b163-40ff-8a81-7df1a7dc6d5e" (UID: "e9022283-b163-40ff-8a81-7df1a7dc6d5e"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 12 22:15:43.779871 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:43.779652    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9022283-b163-40ff-8a81-7df1a7dc6d5e-istio-token" (OuterVolumeSpecName: "istio-token") pod "e9022283-b163-40ff-8a81-7df1a7dc6d5e" (UID: "e9022283-b163-40ff-8a81-7df1a7dc6d5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 22:15:43.779871 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:43.779852    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9022283-b163-40ff-8a81-7df1a7dc6d5e-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "e9022283-b163-40ff-8a81-7df1a7dc6d5e" (UID: "e9022283-b163-40ff-8a81-7df1a7dc6d5e"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 12 22:15:43.780051 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:43.779906    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9022283-b163-40ff-8a81-7df1a7dc6d5e-local-certs" (OuterVolumeSpecName: "local-certs") pod "e9022283-b163-40ff-8a81-7df1a7dc6d5e" (UID: "e9022283-b163-40ff-8a81-7df1a7dc6d5e"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:15:43.780051 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:43.780012    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9022283-b163-40ff-8a81-7df1a7dc6d5e-cacerts" (OuterVolumeSpecName: "cacerts") pod "e9022283-b163-40ff-8a81-7df1a7dc6d5e" (UID: "e9022283-b163-40ff-8a81-7df1a7dc6d5e"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 12 22:15:43.780465 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:43.780439    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9022283-b163-40ff-8a81-7df1a7dc6d5e-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "e9022283-b163-40ff-8a81-7df1a7dc6d5e" (UID: "e9022283-b163-40ff-8a81-7df1a7dc6d5e"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 12 22:15:43.780598 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:43.780501    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9022283-b163-40ff-8a81-7df1a7dc6d5e-kube-api-access-n29tj" (OuterVolumeSpecName: "kube-api-access-n29tj") pod "e9022283-b163-40ff-8a81-7df1a7dc6d5e" (UID: "e9022283-b163-40ff-8a81-7df1a7dc6d5e"). InnerVolumeSpecName "kube-api-access-n29tj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 22:15:43.786854 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:43.786746    2560 scope.go:117] "RemoveContainer" containerID="6bfe1404b52246edf90f13efc3e556e02a74b4f4d29fda5d34ff82a9a4b8ab47"
Jul 12 22:15:43.787174 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:15:43.787149    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bfe1404b52246edf90f13efc3e556e02a74b4f4d29fda5d34ff82a9a4b8ab47\": container with ID starting with 6bfe1404b52246edf90f13efc3e556e02a74b4f4d29fda5d34ff82a9a4b8ab47 not found: ID does not exist" containerID="6bfe1404b52246edf90f13efc3e556e02a74b4f4d29fda5d34ff82a9a4b8ab47"
Jul 12 22:15:43.787264 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:43.787188    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bfe1404b52246edf90f13efc3e556e02a74b4f4d29fda5d34ff82a9a4b8ab47"} err="failed to get container status \"6bfe1404b52246edf90f13efc3e556e02a74b4f4d29fda5d34ff82a9a4b8ab47\": rpc error: code = NotFound desc = could not find container \"6bfe1404b52246edf90f13efc3e556e02a74b4f4d29fda5d34ff82a9a4b8ab47\": container with ID starting with 6bfe1404b52246edf90f13efc3e556e02a74b4f4d29fda5d34ff82a9a4b8ab47 not found: ID does not exist"
Jul 12 22:15:43.878353 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:43.878263    2560 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e9022283-b163-40ff-8a81-7df1a7dc6d5e-istio-csr-dns-cert\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:15:43.878353 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:43.878297    2560 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e9022283-b163-40ff-8a81-7df1a7dc6d5e-cacerts\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:15:43.878353 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:43.878311    2560 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e9022283-b163-40ff-8a81-7df1a7dc6d5e-istio-csr-ca-configmap\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:15:43.878353 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:43.878324    2560 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e9022283-b163-40ff-8a81-7df1a7dc6d5e-istio-kubeconfig\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:15:43.878353 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:43.878335    2560 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9022283-b163-40ff-8a81-7df1a7dc6d5e-local-certs\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:15:43.878353 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:43.878342    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9022283-b163-40ff-8a81-7df1a7dc6d5e-istio-token\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:15:43.878353 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:43.878350    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n29tj\" (UniqueName: \"kubernetes.io/projected/e9022283-b163-40ff-8a81-7df1a7dc6d5e-kube-api-access-n29tj\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:15:43.998219 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:43.998181    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-bv7ct"]
Jul 12 22:15:44.015942 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:44.015901    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-bv7ct"]
Jul 12 22:15:44.246239 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:44.246162    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9022283-b163-40ff-8a81-7df1a7dc6d5e" path="/var/lib/kubelet/pods/e9022283-b163-40ff-8a81-7df1a7dc6d5e/volumes"
Jul 12 22:15:45.621593 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:45.621573    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-bv65k"
Jul 12 22:15:45.679355 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:45.678873    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r26w4" event={"ID":"1cd6224b-33bc-41ce-bb56-c25600cc46f1","Type":"ContainerStarted","Data":"1438a328b88e89293e96ee1966093ed6597ed47527375514ec5ad404942163bd"}
Jul 12 22:15:45.679355 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:45.679315    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r26w4"
Jul 12 22:15:45.715389 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:45.715337    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r26w4" podStartSLOduration=1.478402093 podStartE2EDuration="4.715320079s" totalImagesPullingTime="3.236917986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:15:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:15:45.714939792 +0000 UTC m=+318.028443147" watchObservedRunningTime="2026-07-12 22:15:45.715320079 +0000 UTC m=+318.028823432"
Jul 12 22:15:46.684175 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:46.684140    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z8f69" event={"ID":"aad212a3-b149-4734-b08e-0d2d90e21379","Type":"ContainerStarted","Data":"e753a9677ddcc3617be04cf61f6d332e9f955f198dd189ed54aa8a6e988acdb1"}
Jul 12 22:15:46.684591 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:46.684299    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z8f69"
Jul 12 22:15:46.703118 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:46.703068    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z8f69" podStartSLOduration=2.497244426 podStartE2EDuration="5.703051497s" totalImagesPullingTime="3.205807071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:15:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:15:46.701844706 +0000 UTC m=+319.015348061" watchObservedRunningTime="2026-07-12 22:15:46.703051497 +0000 UTC m=+319.016554850"
Jul 12 22:15:47.638346 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:15:47.638312    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-c4vs6"
Jul 12 22:16:17.690384 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:17.690356    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z8f69"
Jul 12 22:16:17.693349 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:17.693323    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-r26w4"
Jul 12 22:16:30.414830 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:30.414801    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-n7ghq"]
Jul 12 22:16:30.415218 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:30.415111    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9022283-b163-40ff-8a81-7df1a7dc6d5e" containerName="discovery"
Jul 12 22:16:30.415218 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:30.415121    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9022283-b163-40ff-8a81-7df1a7dc6d5e" containerName="discovery"
Jul 12 22:16:30.415218 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:30.415171    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9022283-b163-40ff-8a81-7df1a7dc6d5e" containerName="discovery"
Jul 12 22:16:30.419195 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:30.419172    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-n7ghq"
Jul 12 22:16:30.421984 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:30.421960    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-qjxnp\""
Jul 12 22:16:30.422207 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:30.422193    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 12 22:16:30.423206 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:30.423188    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 12 22:16:30.432641 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:30.432614    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-n7ghq"]
Jul 12 22:16:30.457725 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:30.457696    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crftd\" (UniqueName: \"kubernetes.io/projected/132039e2-abdb-45d0-9c87-15963e67cd91-kube-api-access-crftd\") pod \"kuadrant-operator-catalog-n7ghq\" (UID: \"132039e2-abdb-45d0-9c87-15963e67cd91\") " pod="kuadrant-system/kuadrant-operator-catalog-n7ghq"
Jul 12 22:16:30.558797 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:30.558764    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crftd\" (UniqueName: \"kubernetes.io/projected/132039e2-abdb-45d0-9c87-15963e67cd91-kube-api-access-crftd\") pod \"kuadrant-operator-catalog-n7ghq\" (UID: \"132039e2-abdb-45d0-9c87-15963e67cd91\") " pod="kuadrant-system/kuadrant-operator-catalog-n7ghq"
Jul 12 22:16:30.566903 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:30.566873    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crftd\" (UniqueName: \"kubernetes.io/projected/132039e2-abdb-45d0-9c87-15963e67cd91-kube-api-access-crftd\") pod \"kuadrant-operator-catalog-n7ghq\" (UID: \"132039e2-abdb-45d0-9c87-15963e67cd91\") " pod="kuadrant-system/kuadrant-operator-catalog-n7ghq"
Jul 12 22:16:30.730852 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:30.730753    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-n7ghq"
Jul 12 22:16:30.775577 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:30.775545    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-n7ghq"]
Jul 12 22:16:30.865706 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:30.865683    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-n7ghq"]
Jul 12 22:16:30.867989 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:16:30.867958    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod132039e2_abdb_45d0_9c87_15963e67cd91.slice/crio-ebc0e17bd6c9745da6a6fd79f7b0422fb865d701c2d30990027b72b9c2485f77 WatchSource:0}: Error finding container ebc0e17bd6c9745da6a6fd79f7b0422fb865d701c2d30990027b72b9c2485f77: Status 404 returned error can't find the container with id ebc0e17bd6c9745da6a6fd79f7b0422fb865d701c2d30990027b72b9c2485f77
Jul 12 22:16:30.987364 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:30.987289    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jslgr"]
Jul 12 22:16:30.991486 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:30.991468    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-jslgr"
Jul 12 22:16:30.998737 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:30.998692    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jslgr"]
Jul 12 22:16:31.062347 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:31.062309    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw55q\" (UniqueName: \"kubernetes.io/projected/64dcd566-6879-4260-8bd0-17767ee9fb29-kube-api-access-mw55q\") pod \"kuadrant-operator-catalog-jslgr\" (UID: \"64dcd566-6879-4260-8bd0-17767ee9fb29\") " pod="kuadrant-system/kuadrant-operator-catalog-jslgr"
Jul 12 22:16:31.163569 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:31.163516    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw55q\" (UniqueName: \"kubernetes.io/projected/64dcd566-6879-4260-8bd0-17767ee9fb29-kube-api-access-mw55q\") pod \"kuadrant-operator-catalog-jslgr\" (UID: \"64dcd566-6879-4260-8bd0-17767ee9fb29\") " pod="kuadrant-system/kuadrant-operator-catalog-jslgr"
Jul 12 22:16:31.171574 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:31.171548    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw55q\" (UniqueName: \"kubernetes.io/projected/64dcd566-6879-4260-8bd0-17767ee9fb29-kube-api-access-mw55q\") pod \"kuadrant-operator-catalog-jslgr\" (UID: \"64dcd566-6879-4260-8bd0-17767ee9fb29\") " pod="kuadrant-system/kuadrant-operator-catalog-jslgr"
Jul 12 22:16:31.302732 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:31.302647    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-jslgr"
Jul 12 22:16:31.439098 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:31.439070    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jslgr"]
Jul 12 22:16:31.442873 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:16:31.442838    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64dcd566_6879_4260_8bd0_17767ee9fb29.slice/crio-15702bfcf233af0dabc4a7bee47cbb722cde2006ea5ba10ecb21e798e8993083 WatchSource:0}: Error finding container 15702bfcf233af0dabc4a7bee47cbb722cde2006ea5ba10ecb21e798e8993083: Status 404 returned error can't find the container with id 15702bfcf233af0dabc4a7bee47cbb722cde2006ea5ba10ecb21e798e8993083
Jul 12 22:16:31.879376 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:31.879325    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-jslgr" event={"ID":"64dcd566-6879-4260-8bd0-17767ee9fb29","Type":"ContainerStarted","Data":"15702bfcf233af0dabc4a7bee47cbb722cde2006ea5ba10ecb21e798e8993083"}
Jul 12 22:16:31.880808 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:31.880779    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-n7ghq" event={"ID":"132039e2-abdb-45d0-9c87-15963e67cd91","Type":"ContainerStarted","Data":"ebc0e17bd6c9745da6a6fd79f7b0422fb865d701c2d30990027b72b9c2485f77"}
Jul 12 22:16:33.891861 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:33.891819    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-jslgr" event={"ID":"64dcd566-6879-4260-8bd0-17767ee9fb29","Type":"ContainerStarted","Data":"1a1c265048369e519de02bb3a53123fa6753d8115204ba26841d8c1a53b0ba13"}
Jul 12 22:16:33.893145 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:33.893121    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-n7ghq" event={"ID":"132039e2-abdb-45d0-9c87-15963e67cd91","Type":"ContainerStarted","Data":"f0cfc9ea63562d1442001d2a6ca41837347e28608252393ef3e5d5ac71837455"}
Jul 12 22:16:33.893248 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:33.893178    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-n7ghq" podUID="132039e2-abdb-45d0-9c87-15963e67cd91" containerName="registry-server" containerID="cri-o://f0cfc9ea63562d1442001d2a6ca41837347e28608252393ef3e5d5ac71837455" gracePeriod=2
Jul 12 22:16:33.908816 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:33.908775    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-jslgr" podStartSLOduration=2.309094431 podStartE2EDuration="3.908758936s" totalImagesPullingTime="1.599664505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:16:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:16:33.90756285 +0000 UTC m=+366.221066206" watchObservedRunningTime="2026-07-12 22:16:33.908758936 +0000 UTC m=+366.222262290"
Jul 12 22:16:33.923259 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:33.923199    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-n7ghq" podStartSLOduration=1.7516014100000001 podStartE2EDuration="3.923183264s" totalImagesPullingTime="2.171581854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:16:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:16:33.922408108 +0000 UTC m=+366.235911462" watchObservedRunningTime="2026-07-12 22:16:33.923183264 +0000 UTC m=+366.236686617"
Jul 12 22:16:34.147028 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:34.146967    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-n7ghq"
Jul 12 22:16:34.187861 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:34.187828    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crftd\" (UniqueName: \"kubernetes.io/projected/132039e2-abdb-45d0-9c87-15963e67cd91-kube-api-access-crftd\") pod \"132039e2-abdb-45d0-9c87-15963e67cd91\" (UID: \"132039e2-abdb-45d0-9c87-15963e67cd91\") "
Jul 12 22:16:34.190137 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:34.190104    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/132039e2-abdb-45d0-9c87-15963e67cd91-kube-api-access-crftd" (OuterVolumeSpecName: "kube-api-access-crftd") pod "132039e2-abdb-45d0-9c87-15963e67cd91" (UID: "132039e2-abdb-45d0-9c87-15963e67cd91"). InnerVolumeSpecName "kube-api-access-crftd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 22:16:34.288316 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:34.288280    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crftd\" (UniqueName: \"kubernetes.io/projected/132039e2-abdb-45d0-9c87-15963e67cd91-kube-api-access-crftd\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:16:34.898289 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:34.898257    2560 generic.go:356] "Generic (PLEG): container finished" podID="132039e2-abdb-45d0-9c87-15963e67cd91" containerID="f0cfc9ea63562d1442001d2a6ca41837347e28608252393ef3e5d5ac71837455" exitCode=0
Jul 12 22:16:34.898719 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:34.898299    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-n7ghq"
Jul 12 22:16:34.898719 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:34.898341    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-n7ghq" event={"ID":"132039e2-abdb-45d0-9c87-15963e67cd91","Type":"ContainerDied","Data":"f0cfc9ea63562d1442001d2a6ca41837347e28608252393ef3e5d5ac71837455"}
Jul 12 22:16:34.898719 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:34.898388    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-n7ghq" event={"ID":"132039e2-abdb-45d0-9c87-15963e67cd91","Type":"ContainerDied","Data":"ebc0e17bd6c9745da6a6fd79f7b0422fb865d701c2d30990027b72b9c2485f77"}
Jul 12 22:16:34.898719 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:34.898407    2560 scope.go:117] "RemoveContainer" containerID="f0cfc9ea63562d1442001d2a6ca41837347e28608252393ef3e5d5ac71837455"
Jul 12 22:16:34.907512 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:34.907491    2560 scope.go:117] "RemoveContainer" containerID="f0cfc9ea63562d1442001d2a6ca41837347e28608252393ef3e5d5ac71837455"
Jul 12 22:16:34.907807 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:16:34.907785    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0cfc9ea63562d1442001d2a6ca41837347e28608252393ef3e5d5ac71837455\": container with ID starting with f0cfc9ea63562d1442001d2a6ca41837347e28608252393ef3e5d5ac71837455 not found: ID does not exist" containerID="f0cfc9ea63562d1442001d2a6ca41837347e28608252393ef3e5d5ac71837455"
Jul 12 22:16:34.907884 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:34.907817    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0cfc9ea63562d1442001d2a6ca41837347e28608252393ef3e5d5ac71837455"} err="failed to get container status \"f0cfc9ea63562d1442001d2a6ca41837347e28608252393ef3e5d5ac71837455\": rpc error: code = NotFound desc = could not find container \"f0cfc9ea63562d1442001d2a6ca41837347e28608252393ef3e5d5ac71837455\": container with ID starting with f0cfc9ea63562d1442001d2a6ca41837347e28608252393ef3e5d5ac71837455 not found: ID does not exist"
Jul 12 22:16:34.914263 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:34.914236    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-n7ghq"]
Jul 12 22:16:34.921414 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:34.921391    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-n7ghq"]
Jul 12 22:16:36.245414 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:36.245379    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="132039e2-abdb-45d0-9c87-15963e67cd91" path="/var/lib/kubelet/pods/132039e2-abdb-45d0-9c87-15963e67cd91/volumes"
Jul 12 22:16:41.303786 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:41.303751    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-jslgr"
Jul 12 22:16:41.304409 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:41.303842    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-jslgr"
Jul 12 22:16:41.327526 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:41.327489    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-jslgr"
Jul 12 22:16:41.952892 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:16:41.952867    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-jslgr"
Jul 12 22:17:03.413238 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:03.413202    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-82qc5"]
Jul 12 22:17:03.413624 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:03.413562    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="132039e2-abdb-45d0-9c87-15963e67cd91" containerName="registry-server"
Jul 12 22:17:03.413624 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:03.413575    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="132039e2-abdb-45d0-9c87-15963e67cd91" containerName="registry-server"
Jul 12 22:17:03.413697 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:03.413639    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="132039e2-abdb-45d0-9c87-15963e67cd91" containerName="registry-server"
Jul 12 22:17:03.421006 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:03.420985    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-82qc5"
Jul 12 22:17:03.424239 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:03.424220    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-6g2mg\""
Jul 12 22:17:03.429768 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:03.429746    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-82qc5"]
Jul 12 22:17:03.551595 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:03.551561    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kfdp\" (UniqueName: \"kubernetes.io/projected/de5d7d02-92fe-4232-bb9f-a1c710f5fab4-kube-api-access-2kfdp\") pod \"authorino-operator-75cc8d5cb7-82qc5\" (UID: \"de5d7d02-92fe-4232-bb9f-a1c710f5fab4\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-82qc5"
Jul 12 22:17:03.652246 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:03.652209    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kfdp\" (UniqueName: \"kubernetes.io/projected/de5d7d02-92fe-4232-bb9f-a1c710f5fab4-kube-api-access-2kfdp\") pod \"authorino-operator-75cc8d5cb7-82qc5\" (UID: \"de5d7d02-92fe-4232-bb9f-a1c710f5fab4\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-82qc5"
Jul 12 22:17:03.668286 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:03.668211    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kfdp\" (UniqueName: \"kubernetes.io/projected/de5d7d02-92fe-4232-bb9f-a1c710f5fab4-kube-api-access-2kfdp\") pod \"authorino-operator-75cc8d5cb7-82qc5\" (UID: \"de5d7d02-92fe-4232-bb9f-a1c710f5fab4\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-82qc5"
Jul 12 22:17:03.732218 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:03.732181    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-82qc5"
Jul 12 22:17:03.879031 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:03.878940    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-82qc5"]
Jul 12 22:17:03.881423 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:17:03.881397    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde5d7d02_92fe_4232_bb9f_a1c710f5fab4.slice/crio-8048701a317480931c48313faa0943d0098954d226d7c8e95c73dca5d890d5af WatchSource:0}: Error finding container 8048701a317480931c48313faa0943d0098954d226d7c8e95c73dca5d890d5af: Status 404 returned error can't find the container with id 8048701a317480931c48313faa0943d0098954d226d7c8e95c73dca5d890d5af
Jul 12 22:17:04.028074 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:04.027988    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-82qc5" event={"ID":"de5d7d02-92fe-4232-bb9f-a1c710f5fab4","Type":"ContainerStarted","Data":"8048701a317480931c48313faa0943d0098954d226d7c8e95c73dca5d890d5af"}
Jul 12 22:17:06.668154 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:06.668114    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fxmp"]
Jul 12 22:17:06.671441 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:06.671419    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fxmp"
Jul 12 22:17:06.674272 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:06.674250    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-jlmlb\""
Jul 12 22:17:06.684153 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:06.684129    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fxmp"]
Jul 12 22:17:06.781633 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:06.781592    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmsj8\" (UniqueName: \"kubernetes.io/projected/54711632-e08c-4400-9fa5-77a792634eb3-kube-api-access-fmsj8\") pod \"limitador-operator-controller-manager-859d7b78bd-9fxmp\" (UID: \"54711632-e08c-4400-9fa5-77a792634eb3\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fxmp"
Jul 12 22:17:06.882505 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:06.882470    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmsj8\" (UniqueName: \"kubernetes.io/projected/54711632-e08c-4400-9fa5-77a792634eb3-kube-api-access-fmsj8\") pod \"limitador-operator-controller-manager-859d7b78bd-9fxmp\" (UID: \"54711632-e08c-4400-9fa5-77a792634eb3\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fxmp"
Jul 12 22:17:06.909652 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:06.909623    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmsj8\" (UniqueName: \"kubernetes.io/projected/54711632-e08c-4400-9fa5-77a792634eb3-kube-api-access-fmsj8\") pod \"limitador-operator-controller-manager-859d7b78bd-9fxmp\" (UID: \"54711632-e08c-4400-9fa5-77a792634eb3\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fxmp"
Jul 12 22:17:06.981771 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:06.981696    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fxmp"
Jul 12 22:17:07.046349 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:07.046294    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-82qc5" event={"ID":"de5d7d02-92fe-4232-bb9f-a1c710f5fab4","Type":"ContainerStarted","Data":"133f20164b6d9ea6841fedc81898350042d8c2651d8043470db01519ec433308"}
Jul 12 22:17:07.046680 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:07.046658    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-82qc5"
Jul 12 22:17:07.073645 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:07.073599    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-82qc5" podStartSLOduration=1.891366176 podStartE2EDuration="4.073581434s" totalImagesPullingTime="2.182215258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:17:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:17:07.071483519 +0000 UTC m=+399.384986885" watchObservedRunningTime="2026-07-12 22:17:07.073581434 +0000 UTC m=+399.387084788"
Jul 12 22:17:07.153108 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:07.153074    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fxmp"]
Jul 12 22:17:07.156348 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:17:07.156321    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54711632_e08c_4400_9fa5_77a792634eb3.slice/crio-7f48f8595825591b84f0dc459c8d7386dbd5174e1245ce08fb4f504d14a09fd7 WatchSource:0}: Error finding container 7f48f8595825591b84f0dc459c8d7386dbd5174e1245ce08fb4f504d14a09fd7: Status 404 returned error can't find the container with id 7f48f8595825591b84f0dc459c8d7386dbd5174e1245ce08fb4f504d14a09fd7
Jul 12 22:17:08.052103 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:08.052071    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fxmp" event={"ID":"54711632-e08c-4400-9fa5-77a792634eb3","Type":"ContainerStarted","Data":"7f48f8595825591b84f0dc459c8d7386dbd5174e1245ce08fb4f504d14a09fd7"}
Jul 12 22:17:09.058234 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:09.058124    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fxmp" event={"ID":"54711632-e08c-4400-9fa5-77a792634eb3","Type":"ContainerStarted","Data":"430dd8103fd8ef11dca145a96d0ba08848c4e8e902850711e4817d67ebba25b2"}
Jul 12 22:17:09.058234 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:09.058185    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fxmp"
Jul 12 22:17:09.075390 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:09.075339    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fxmp" podStartSLOduration=1.6404665029999999 podStartE2EDuration="3.075323812s" totalImagesPullingTime="1.434857309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:17:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:17:09.073374248 +0000 UTC m=+401.386877601" watchObservedRunningTime="2026-07-12 22:17:09.075323812 +0000 UTC m=+401.388827167"
Jul 12 22:17:11.714440 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:11.714405    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gn87p"]
Jul 12 22:17:11.717824 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:11.717806    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gn87p"
Jul 12 22:17:11.721234 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:11.721214    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-8jsv5\""
Jul 12 22:17:11.731844 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:11.731820    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gn87p"]
Jul 12 22:17:11.821190 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:11.821151    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6mcm\" (UniqueName: \"kubernetes.io/projected/392eafff-89ce-4eed-bc96-507683c5b012-kube-api-access-q6mcm\") pod \"kuadrant-operator-controller-manager-8684c9ff49-gn87p\" (UID: \"392eafff-89ce-4eed-bc96-507683c5b012\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gn87p"
Jul 12 22:17:11.821190 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:11.821198    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/392eafff-89ce-4eed-bc96-507683c5b012-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-gn87p\" (UID: \"392eafff-89ce-4eed-bc96-507683c5b012\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gn87p"
Jul 12 22:17:11.922281 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:11.922241    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6mcm\" (UniqueName: \"kubernetes.io/projected/392eafff-89ce-4eed-bc96-507683c5b012-kube-api-access-q6mcm\") pod \"kuadrant-operator-controller-manager-8684c9ff49-gn87p\" (UID: \"392eafff-89ce-4eed-bc96-507683c5b012\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gn87p"
Jul 12 22:17:11.922467 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:11.922291    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/392eafff-89ce-4eed-bc96-507683c5b012-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-gn87p\" (UID: \"392eafff-89ce-4eed-bc96-507683c5b012\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gn87p"
Jul 12 22:17:11.922720 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:11.922702    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/392eafff-89ce-4eed-bc96-507683c5b012-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-gn87p\" (UID: \"392eafff-89ce-4eed-bc96-507683c5b012\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gn87p"
Jul 12 22:17:11.933938 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:11.933907    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6mcm\" (UniqueName: \"kubernetes.io/projected/392eafff-89ce-4eed-bc96-507683c5b012-kube-api-access-q6mcm\") pod \"kuadrant-operator-controller-manager-8684c9ff49-gn87p\" (UID: \"392eafff-89ce-4eed-bc96-507683c5b012\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gn87p"
Jul 12 22:17:12.028702 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:12.028601    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gn87p"
Jul 12 22:17:12.169006 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:12.168859    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gn87p"]
Jul 12 22:17:12.171404 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:17:12.171377    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod392eafff_89ce_4eed_bc96_507683c5b012.slice/crio-d109deb5ef11e20499c57abe6f313079d93f92b6fa32f077425c0c4fbd310122 WatchSource:0}: Error finding container d109deb5ef11e20499c57abe6f313079d93f92b6fa32f077425c0c4fbd310122: Status 404 returned error can't find the container with id d109deb5ef11e20499c57abe6f313079d93f92b6fa32f077425c0c4fbd310122
Jul 12 22:17:13.084222 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:13.084187    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gn87p" event={"ID":"392eafff-89ce-4eed-bc96-507683c5b012","Type":"ContainerStarted","Data":"d109deb5ef11e20499c57abe6f313079d93f92b6fa32f077425c0c4fbd310122"}
Jul 12 22:17:14.223780 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:14.223743    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-2nhpx"]
Jul 12 22:17:14.227043 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:14.227028    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-2nhpx"
Jul 12 22:17:14.229749 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:14.229723    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 12 22:17:14.229872 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:14.229727    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-twqk8\""
Jul 12 22:17:14.239206 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:14.239185    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-2nhpx"]
Jul 12 22:17:14.343891 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:14.343852    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28vkh\" (UniqueName: \"kubernetes.io/projected/59f50f9c-506a-458e-b2cf-138d8094da03-kube-api-access-28vkh\") pod \"dns-operator-controller-manager-864dd78c4c-2nhpx\" (UID: \"59f50f9c-506a-458e-b2cf-138d8094da03\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-2nhpx"
Jul 12 22:17:14.444706 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:14.444658    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28vkh\" (UniqueName: \"kubernetes.io/projected/59f50f9c-506a-458e-b2cf-138d8094da03-kube-api-access-28vkh\") pod \"dns-operator-controller-manager-864dd78c4c-2nhpx\" (UID: \"59f50f9c-506a-458e-b2cf-138d8094da03\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-2nhpx"
Jul 12 22:17:14.462752 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:14.462711    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28vkh\" (UniqueName: \"kubernetes.io/projected/59f50f9c-506a-458e-b2cf-138d8094da03-kube-api-access-28vkh\") pod \"dns-operator-controller-manager-864dd78c4c-2nhpx\" (UID: \"59f50f9c-506a-458e-b2cf-138d8094da03\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-2nhpx"
Jul 12 22:17:14.538071 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:14.537969    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-2nhpx"
Jul 12 22:17:14.680242 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:14.680219    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-2nhpx"]
Jul 12 22:17:14.682481 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:17:14.682453    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59f50f9c_506a_458e_b2cf_138d8094da03.slice/crio-19e688e6752f9693e090a41e013dceaf6cb7e2612884a38c79c4f85f605959f4 WatchSource:0}: Error finding container 19e688e6752f9693e090a41e013dceaf6cb7e2612884a38c79c4f85f605959f4: Status 404 returned error can't find the container with id 19e688e6752f9693e090a41e013dceaf6cb7e2612884a38c79c4f85f605959f4
Jul 12 22:17:15.094726 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:15.094682    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-2nhpx" event={"ID":"59f50f9c-506a-458e-b2cf-138d8094da03","Type":"ContainerStarted","Data":"19e688e6752f9693e090a41e013dceaf6cb7e2612884a38c79c4f85f605959f4"}
Jul 12 22:17:18.055265 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:18.055189    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-82qc5"
Jul 12 22:17:18.114008 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:18.113970    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-2nhpx" event={"ID":"59f50f9c-506a-458e-b2cf-138d8094da03","Type":"ContainerStarted","Data":"c308c0bcf99d38249b810b8b0928ea2a4ce88ffcd369b1eb94a7f2433e55fd35"}
Jul 12 22:17:18.114212 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:18.114074    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-2nhpx"
Jul 12 22:17:18.115760 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:18.115736    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gn87p" event={"ID":"392eafff-89ce-4eed-bc96-507683c5b012","Type":"ContainerStarted","Data":"a0553a9e424749b838b21fb8bf8a3859fc9442d01441c4c2adb9ad68019a463a"}
Jul 12 22:17:18.115879 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:18.115863    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gn87p"
Jul 12 22:17:18.133762 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:18.133711    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-2nhpx" podStartSLOduration=1.175363729 podStartE2EDuration="4.133695034s" totalImagesPullingTime="2.958331305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:17:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:17:18.132462938 +0000 UTC m=+410.445966292" watchObservedRunningTime="2026-07-12 22:17:18.133695034 +0000 UTC m=+410.447198380"
Jul 12 22:17:18.152489 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:18.152427    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gn87p" podStartSLOduration=1.67071692 podStartE2EDuration="7.152410836s" totalImagesPullingTime="5.481693916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:17:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:17:18.150701135 +0000 UTC m=+410.464204491" watchObservedRunningTime="2026-07-12 22:17:18.152410836 +0000 UTC m=+410.465914190"
Jul 12 22:17:20.065841 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:20.065815    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fxmp"
Jul 12 22:17:20.487477 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:20.487396    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-8mm8s"]
Jul 12 22:17:20.490589 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:20.490568    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8mm8s"
Jul 12 22:17:20.493396 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:20.493374    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 12 22:17:20.493993 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:20.493978    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qs8cq\""
Jul 12 22:17:20.494155 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:20.494142    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 12 22:17:20.496015 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:20.495995    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/54bb394f-0e12-49ce-bb8b-d91088eddb45-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-8mm8s\" (UID: \"54bb394f-0e12-49ce-bb8b-d91088eddb45\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8mm8s"
Jul 12 22:17:20.496099 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:20.496035    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/54bb394f-0e12-49ce-bb8b-d91088eddb45-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-8mm8s\" (UID: \"54bb394f-0e12-49ce-bb8b-d91088eddb45\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8mm8s"
Jul 12 22:17:20.496163 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:20.496137    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2gxq\" (UniqueName: \"kubernetes.io/projected/54bb394f-0e12-49ce-bb8b-d91088eddb45-kube-api-access-v2gxq\") pod \"kuadrant-console-plugin-66b5774985-8mm8s\" (UID: \"54bb394f-0e12-49ce-bb8b-d91088eddb45\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8mm8s"
Jul 12 22:17:20.508403 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:20.508381    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-8mm8s"]
Jul 12 22:17:20.596652 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:20.596621    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/54bb394f-0e12-49ce-bb8b-d91088eddb45-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-8mm8s\" (UID: \"54bb394f-0e12-49ce-bb8b-d91088eddb45\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8mm8s"
Jul 12 22:17:20.596822 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:20.596704    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2gxq\" (UniqueName: \"kubernetes.io/projected/54bb394f-0e12-49ce-bb8b-d91088eddb45-kube-api-access-v2gxq\") pod \"kuadrant-console-plugin-66b5774985-8mm8s\" (UID: \"54bb394f-0e12-49ce-bb8b-d91088eddb45\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8mm8s"
Jul 12 22:17:20.596822 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:20.596757    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/54bb394f-0e12-49ce-bb8b-d91088eddb45-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-8mm8s\" (UID: \"54bb394f-0e12-49ce-bb8b-d91088eddb45\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8mm8s"
Jul 12 22:17:20.596908 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:17:20.596844    2560 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 12 22:17:20.596908 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:17:20.596897    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54bb394f-0e12-49ce-bb8b-d91088eddb45-plugin-serving-cert podName:54bb394f-0e12-49ce-bb8b-d91088eddb45 nodeName:}" failed. No retries permitted until 2026-07-12 22:17:21.096880449 +0000 UTC m=+413.410383783 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/54bb394f-0e12-49ce-bb8b-d91088eddb45-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-8mm8s" (UID: "54bb394f-0e12-49ce-bb8b-d91088eddb45") : secret "plugin-serving-cert" not found
Jul 12 22:17:20.597157 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:20.597138    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/54bb394f-0e12-49ce-bb8b-d91088eddb45-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-8mm8s\" (UID: \"54bb394f-0e12-49ce-bb8b-d91088eddb45\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8mm8s"
Jul 12 22:17:20.624881 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:20.624850    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2gxq\" (UniqueName: \"kubernetes.io/projected/54bb394f-0e12-49ce-bb8b-d91088eddb45-kube-api-access-v2gxq\") pod \"kuadrant-console-plugin-66b5774985-8mm8s\" (UID: \"54bb394f-0e12-49ce-bb8b-d91088eddb45\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8mm8s"
Jul 12 22:17:21.101368 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:21.101331    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/54bb394f-0e12-49ce-bb8b-d91088eddb45-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-8mm8s\" (UID: \"54bb394f-0e12-49ce-bb8b-d91088eddb45\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8mm8s"
Jul 12 22:17:21.103947 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:21.103921    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/54bb394f-0e12-49ce-bb8b-d91088eddb45-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-8mm8s\" (UID: \"54bb394f-0e12-49ce-bb8b-d91088eddb45\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8mm8s"
Jul 12 22:17:21.399732 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:21.399636    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8mm8s"
Jul 12 22:17:21.536237 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:21.536201    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-8mm8s"]
Jul 12 22:17:21.540111 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:17:21.540071    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54bb394f_0e12_49ce_bb8b_d91088eddb45.slice/crio-0286a33dccfba0876d5b325a2b1713d09137c40d56070f838b4371f64f834613 WatchSource:0}: Error finding container 0286a33dccfba0876d5b325a2b1713d09137c40d56070f838b4371f64f834613: Status 404 returned error can't find the container with id 0286a33dccfba0876d5b325a2b1713d09137c40d56070f838b4371f64f834613
Jul 12 22:17:22.134739 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:22.134706    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8mm8s" event={"ID":"54bb394f-0e12-49ce-bb8b-d91088eddb45","Type":"ContainerStarted","Data":"0286a33dccfba0876d5b325a2b1713d09137c40d56070f838b4371f64f834613"}
Jul 12 22:17:29.122496 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:29.122435    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-2nhpx"
Jul 12 22:17:29.123034 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:29.122521    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gn87p"
Jul 12 22:17:30.886033 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:30.885999    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gn87p"]
Jul 12 22:17:30.886510 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:30.886267    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gn87p" podUID="392eafff-89ce-4eed-bc96-507683c5b012" containerName="manager" containerID="cri-o://a0553a9e424749b838b21fb8bf8a3859fc9442d01441c4c2adb9ad68019a463a" gracePeriod=2
Jul 12 22:17:30.895478 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:17:30.895422    2560 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"392eafff-89ce-4eed-bc96-507683c5b012\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-gn87p\": pods \"kuadrant-operator-controller-manager-8684c9ff49-gn87p\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gn87p"
Jul 12 22:17:30.895678 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:30.895554    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gn87p"]
Jul 12 22:17:30.908137 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:30.907917    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fxmp"]
Jul 12 22:17:30.908774 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:30.908581    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fxmp" podUID="54711632-e08c-4400-9fa5-77a792634eb3" containerName="manager" containerID="cri-o://430dd8103fd8ef11dca145a96d0ba08848c4e8e902850711e4817d67ebba25b2" gracePeriod=2
Jul 12 22:17:30.910377 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:30.910345    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jd4kb"]
Jul 12 22:17:30.910785 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:30.910763    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="392eafff-89ce-4eed-bc96-507683c5b012" containerName="manager"
Jul 12 22:17:30.910785 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:30.910784    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="392eafff-89ce-4eed-bc96-507683c5b012" containerName="manager"
Jul 12 22:17:30.910956 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:30.910885    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="392eafff-89ce-4eed-bc96-507683c5b012" containerName="manager"
Jul 12 22:17:30.916743 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:30.916675    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fxmp"]
Jul 12 22:17:30.916849 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:30.916776    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jd4kb"
Jul 12 22:17:30.926394 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:17:30.926345    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-gn87p\" is forbidden: User \"system:node:ip-10-0-135-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-47.ec2.internal' and this object" podUID="392eafff-89ce-4eed-bc96-507683c5b012" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gn87p"
Jul 12 22:17:30.928600 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:17:30.928568    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9fxmp\" is forbidden: User \"system:node:ip-10-0-135-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-47.ec2.internal' and this object" podUID="54711632-e08c-4400-9fa5-77a792634eb3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fxmp"
Jul 12 22:17:30.930609 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:17:30.930579    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-gn87p\" is forbidden: User \"system:node:ip-10-0-135-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-47.ec2.internal' and this object" podUID="392eafff-89ce-4eed-bc96-507683c5b012" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gn87p"
Jul 12 22:17:30.937472 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:30.937340    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xzccs"]
Jul 12 22:17:30.937899 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:30.937877    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54711632-e08c-4400-9fa5-77a792634eb3" containerName="manager"
Jul 12 22:17:30.938035 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:30.937901    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="54711632-e08c-4400-9fa5-77a792634eb3" containerName="manager"
Jul 12 22:17:30.939406 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:30.939230    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="54711632-e08c-4400-9fa5-77a792634eb3" containerName="manager"
Jul 12 22:17:30.942512 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:30.942489    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jd4kb"]
Jul 12 22:17:30.942628 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:30.942610    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xzccs"
Jul 12 22:17:30.954449 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:17:30.954412    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9fxmp\" is forbidden: User \"system:node:ip-10-0-135-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-47.ec2.internal' and this object" podUID="54711632-e08c-4400-9fa5-77a792634eb3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fxmp"
Jul 12 22:17:30.954609 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:30.954580    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xzccs"]
Jul 12 22:17:30.956695 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:17:30.956470    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-gn87p\" is forbidden: User \"system:node:ip-10-0-135-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-47.ec2.internal' and this object" podUID="392eafff-89ce-4eed-bc96-507683c5b012" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gn87p"
Jul 12 22:17:30.987877 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:30.987844    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxx9m\" (UniqueName: \"kubernetes.io/projected/f013fece-befe-44ff-89f1-1b71c5888c7e-kube-api-access-pxx9m\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jd4kb\" (UID: \"f013fece-befe-44ff-89f1-1b71c5888c7e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jd4kb"
Jul 12 22:17:30.988041 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:30.987954    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdtj9\" (UniqueName: \"kubernetes.io/projected/e78a3ec0-5fae-4c7f-a62e-43571c60c7da-kube-api-access-mdtj9\") pod \"limitador-operator-controller-manager-859d7b78bd-xzccs\" (UID: \"e78a3ec0-5fae-4c7f-a62e-43571c60c7da\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xzccs"
Jul 12 22:17:30.988041 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:30.988023    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f013fece-befe-44ff-89f1-1b71c5888c7e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jd4kb\" (UID: \"f013fece-befe-44ff-89f1-1b71c5888c7e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jd4kb"
Jul 12 22:17:31.089566 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:31.089504    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f013fece-befe-44ff-89f1-1b71c5888c7e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jd4kb\" (UID: \"f013fece-befe-44ff-89f1-1b71c5888c7e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jd4kb"
Jul 12 22:17:31.089926 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:31.089631    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxx9m\" (UniqueName: \"kubernetes.io/projected/f013fece-befe-44ff-89f1-1b71c5888c7e-kube-api-access-pxx9m\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jd4kb\" (UID: \"f013fece-befe-44ff-89f1-1b71c5888c7e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jd4kb"
Jul 12 22:17:31.089926 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:31.089671    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdtj9\" (UniqueName: \"kubernetes.io/projected/e78a3ec0-5fae-4c7f-a62e-43571c60c7da-kube-api-access-mdtj9\") pod \"limitador-operator-controller-manager-859d7b78bd-xzccs\" (UID: \"e78a3ec0-5fae-4c7f-a62e-43571c60c7da\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xzccs"
Jul 12 22:17:31.090062 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:31.089988    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f013fece-befe-44ff-89f1-1b71c5888c7e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jd4kb\" (UID: \"f013fece-befe-44ff-89f1-1b71c5888c7e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jd4kb"
Jul 12 22:17:31.098953 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:31.098926    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxx9m\" (UniqueName: \"kubernetes.io/projected/f013fece-befe-44ff-89f1-1b71c5888c7e-kube-api-access-pxx9m\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jd4kb\" (UID: \"f013fece-befe-44ff-89f1-1b71c5888c7e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jd4kb"
Jul 12 22:17:31.099382 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:31.099360    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdtj9\" (UniqueName: \"kubernetes.io/projected/e78a3ec0-5fae-4c7f-a62e-43571c60c7da-kube-api-access-mdtj9\") pod \"limitador-operator-controller-manager-859d7b78bd-xzccs\" (UID: \"e78a3ec0-5fae-4c7f-a62e-43571c60c7da\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xzccs"
Jul 12 22:17:31.322884 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:31.322848    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jd4kb"
Jul 12 22:17:31.329870 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:31.329838    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xzccs"
Jul 12 22:17:43.454417 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:43.454344    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xzccs"]
Jul 12 22:17:43.458921 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:17:43.458894    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode78a3ec0_5fae_4c7f_a62e_43571c60c7da.slice/crio-c2581e8595e7df5c0791e0db31f507347f534074f007b8b97690643bfa9d9756 WatchSource:0}: Error finding container c2581e8595e7df5c0791e0db31f507347f534074f007b8b97690643bfa9d9756: Status 404 returned error can't find the container with id c2581e8595e7df5c0791e0db31f507347f534074f007b8b97690643bfa9d9756
Jul 12 22:17:43.468279 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:43.468133    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gn87p"
Jul 12 22:17:43.470490 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:17:43.470464    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-gn87p\" is forbidden: User \"system:node:ip-10-0-135-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-47.ec2.internal' and this object" podUID="392eafff-89ce-4eed-bc96-507683c5b012" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gn87p"
Jul 12 22:17:43.470602 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:43.470493    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fxmp"
Jul 12 22:17:43.472513 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:17:43.472493    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9fxmp\" is forbidden: User \"system:node:ip-10-0-135-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-47.ec2.internal' and this object" podUID="54711632-e08c-4400-9fa5-77a792634eb3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fxmp"
Jul 12 22:17:43.474573 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:17:43.474544    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-gn87p\" is forbidden: User \"system:node:ip-10-0-135-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-47.ec2.internal' and this object" podUID="392eafff-89ce-4eed-bc96-507683c5b012" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gn87p"
Jul 12 22:17:43.481831 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:43.481813    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jd4kb"]
Jul 12 22:17:43.484669 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:17:43.484630    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf013fece_befe_44ff_89f1_1b71c5888c7e.slice/crio-dad38b21e0eb0a22d3219150d93b7f6c1199af14d652ec3b3abd4fc62edbca97 WatchSource:0}: Error finding container dad38b21e0eb0a22d3219150d93b7f6c1199af14d652ec3b3abd4fc62edbca97: Status 404 returned error can't find the container with id dad38b21e0eb0a22d3219150d93b7f6c1199af14d652ec3b3abd4fc62edbca97
Jul 12 22:17:43.604128 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:43.604093    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6mcm\" (UniqueName: \"kubernetes.io/projected/392eafff-89ce-4eed-bc96-507683c5b012-kube-api-access-q6mcm\") pod \"392eafff-89ce-4eed-bc96-507683c5b012\" (UID: \"392eafff-89ce-4eed-bc96-507683c5b012\") "
Jul 12 22:17:43.604128 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:43.604135    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmsj8\" (UniqueName: \"kubernetes.io/projected/54711632-e08c-4400-9fa5-77a792634eb3-kube-api-access-fmsj8\") pod \"54711632-e08c-4400-9fa5-77a792634eb3\" (UID: \"54711632-e08c-4400-9fa5-77a792634eb3\") "
Jul 12 22:17:43.604341 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:43.604205    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/392eafff-89ce-4eed-bc96-507683c5b012-extensions-socket-volume\") pod \"392eafff-89ce-4eed-bc96-507683c5b012\" (UID: \"392eafff-89ce-4eed-bc96-507683c5b012\") "
Jul 12 22:17:43.604661 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:43.604627    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/392eafff-89ce-4eed-bc96-507683c5b012-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "392eafff-89ce-4eed-bc96-507683c5b012" (UID: "392eafff-89ce-4eed-bc96-507683c5b012"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:17:43.606370 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:43.606344    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/392eafff-89ce-4eed-bc96-507683c5b012-kube-api-access-q6mcm" (OuterVolumeSpecName: "kube-api-access-q6mcm") pod "392eafff-89ce-4eed-bc96-507683c5b012" (UID: "392eafff-89ce-4eed-bc96-507683c5b012"). InnerVolumeSpecName "kube-api-access-q6mcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 22:17:43.606448 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:43.606350    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54711632-e08c-4400-9fa5-77a792634eb3-kube-api-access-fmsj8" (OuterVolumeSpecName: "kube-api-access-fmsj8") pod "54711632-e08c-4400-9fa5-77a792634eb3" (UID: "54711632-e08c-4400-9fa5-77a792634eb3"). InnerVolumeSpecName "kube-api-access-fmsj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 22:17:43.704930 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:43.704895    2560 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/392eafff-89ce-4eed-bc96-507683c5b012-extensions-socket-volume\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:17:43.704930 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:43.704924    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6mcm\" (UniqueName: \"kubernetes.io/projected/392eafff-89ce-4eed-bc96-507683c5b012-kube-api-access-q6mcm\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:17:43.704930 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:43.704934    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmsj8\" (UniqueName: \"kubernetes.io/projected/54711632-e08c-4400-9fa5-77a792634eb3-kube-api-access-fmsj8\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:17:44.245188 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:44.245154    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="392eafff-89ce-4eed-bc96-507683c5b012" path="/var/lib/kubelet/pods/392eafff-89ce-4eed-bc96-507683c5b012/volumes"
Jul 12 22:17:44.245489 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:44.245477    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54711632-e08c-4400-9fa5-77a792634eb3" path="/var/lib/kubelet/pods/54711632-e08c-4400-9fa5-77a792634eb3/volumes"
Jul 12 22:17:44.259377 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:44.259353    2560 generic.go:356] "Generic (PLEG): container finished" podID="392eafff-89ce-4eed-bc96-507683c5b012" containerID="a0553a9e424749b838b21fb8bf8a3859fc9442d01441c4c2adb9ad68019a463a" exitCode=0
Jul 12 22:17:44.259491 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:44.259392    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gn87p"
Jul 12 22:17:44.259491 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:44.259459    2560 scope.go:117] "RemoveContainer" containerID="a0553a9e424749b838b21fb8bf8a3859fc9442d01441c4c2adb9ad68019a463a"
Jul 12 22:17:44.261431 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:44.261382    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jd4kb" event={"ID":"f013fece-befe-44ff-89f1-1b71c5888c7e","Type":"ContainerStarted","Data":"93c0ecafa02bec7239a7982815ff0f12d679ffb036510b83c524ce94daa5a48f"}
Jul 12 22:17:44.261593 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:44.261447    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jd4kb"
Jul 12 22:17:44.261593 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:44.261472    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jd4kb" event={"ID":"f013fece-befe-44ff-89f1-1b71c5888c7e","Type":"ContainerStarted","Data":"dad38b21e0eb0a22d3219150d93b7f6c1199af14d652ec3b3abd4fc62edbca97"}
Jul 12 22:17:44.261909 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:17:44.261884    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-gn87p\" is forbidden: User \"system:node:ip-10-0-135-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-47.ec2.internal' and this object" podUID="392eafff-89ce-4eed-bc96-507683c5b012" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gn87p"
Jul 12 22:17:44.263110 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:44.263084    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xzccs" event={"ID":"e78a3ec0-5fae-4c7f-a62e-43571c60c7da","Type":"ContainerStarted","Data":"a45611455f9fde9fcf6b8336e9ae41e6207e940b1ef14c2a532cb5a7e346457e"}
Jul 12 22:17:44.263217 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:44.263134    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xzccs" event={"ID":"e78a3ec0-5fae-4c7f-a62e-43571c60c7da","Type":"ContainerStarted","Data":"c2581e8595e7df5c0791e0db31f507347f534074f007b8b97690643bfa9d9756"}
Jul 12 22:17:44.263217 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:44.263164    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xzccs"
Jul 12 22:17:44.263990 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:17:44.263966    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-gn87p\" is forbidden: User \"system:node:ip-10-0-135-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-47.ec2.internal' and this object" podUID="392eafff-89ce-4eed-bc96-507683c5b012" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gn87p"
Jul 12 22:17:44.264439 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:44.264422    2560 generic.go:356] "Generic (PLEG): container finished" podID="54711632-e08c-4400-9fa5-77a792634eb3" containerID="430dd8103fd8ef11dca145a96d0ba08848c4e8e902850711e4817d67ebba25b2" exitCode=0
Jul 12 22:17:44.264519 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:44.264456    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fxmp"
Jul 12 22:17:44.266121 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:44.266064    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8mm8s" event={"ID":"54bb394f-0e12-49ce-bb8b-d91088eddb45","Type":"ContainerStarted","Data":"b152e9c3fc7927f76c5d84ba3dd9964a84e09fbff75a41095acc076ae61ac628"}
Jul 12 22:17:44.268982 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:44.268967    2560 scope.go:117] "RemoveContainer" containerID="a0553a9e424749b838b21fb8bf8a3859fc9442d01441c4c2adb9ad68019a463a"
Jul 12 22:17:44.269232 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:17:44.269215    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0553a9e424749b838b21fb8bf8a3859fc9442d01441c4c2adb9ad68019a463a\": container with ID starting with a0553a9e424749b838b21fb8bf8a3859fc9442d01441c4c2adb9ad68019a463a not found: ID does not exist" containerID="a0553a9e424749b838b21fb8bf8a3859fc9442d01441c4c2adb9ad68019a463a"
Jul 12 22:17:44.269296 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:44.269239    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0553a9e424749b838b21fb8bf8a3859fc9442d01441c4c2adb9ad68019a463a"} err="failed to get container status \"a0553a9e424749b838b21fb8bf8a3859fc9442d01441c4c2adb9ad68019a463a\": rpc error: code = NotFound desc = could not find container \"a0553a9e424749b838b21fb8bf8a3859fc9442d01441c4c2adb9ad68019a463a\": container with ID starting with a0553a9e424749b838b21fb8bf8a3859fc9442d01441c4c2adb9ad68019a463a not found: ID does not exist"
Jul 12 22:17:44.269296 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:44.269254    2560 scope.go:117] "RemoveContainer" containerID="430dd8103fd8ef11dca145a96d0ba08848c4e8e902850711e4817d67ebba25b2"
Jul 12 22:17:44.284824 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:44.284776    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jd4kb" podStartSLOduration=14.284759833 podStartE2EDuration="14.284759833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:17:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:17:44.283151311 +0000 UTC m=+436.596654666" watchObservedRunningTime="2026-07-12 22:17:44.284759833 +0000 UTC m=+436.598263191"
Jul 12 22:17:44.291282 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:44.291168    2560 scope.go:117] "RemoveContainer" containerID="430dd8103fd8ef11dca145a96d0ba08848c4e8e902850711e4817d67ebba25b2"
Jul 12 22:17:44.291475 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:17:44.291454    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"430dd8103fd8ef11dca145a96d0ba08848c4e8e902850711e4817d67ebba25b2\": container with ID starting with 430dd8103fd8ef11dca145a96d0ba08848c4e8e902850711e4817d67ebba25b2 not found: ID does not exist" containerID="430dd8103fd8ef11dca145a96d0ba08848c4e8e902850711e4817d67ebba25b2"
Jul 12 22:17:44.291520 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:44.291483    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"430dd8103fd8ef11dca145a96d0ba08848c4e8e902850711e4817d67ebba25b2"} err="failed to get container status \"430dd8103fd8ef11dca145a96d0ba08848c4e8e902850711e4817d67ebba25b2\": rpc error: code = NotFound desc = could not find container \"430dd8103fd8ef11dca145a96d0ba08848c4e8e902850711e4817d67ebba25b2\": container with ID starting with 430dd8103fd8ef11dca145a96d0ba08848c4e8e902850711e4817d67ebba25b2 not found: ID does not exist"
Jul 12 22:17:44.301215 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:44.301172    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xzccs" podStartSLOduration=14.301158444 podStartE2EDuration="14.301158444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:17:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:17:44.300126055 +0000 UTC m=+436.613629410" watchObservedRunningTime="2026-07-12 22:17:44.301158444 +0000 UTC m=+436.614661798"
Jul 12 22:17:44.302256 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:17:44.302235    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9fxmp\" is forbidden: User \"system:node:ip-10-0-135-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-47.ec2.internal' and this object" podUID="54711632-e08c-4400-9fa5-77a792634eb3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9fxmp"
Jul 12 22:17:44.319256 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:44.319206    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8mm8s" podStartSLOduration=2.4852853010000002 podStartE2EDuration="24.319191337s" totalImagesPullingTime="21.833906036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:17:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:17:44.317885601 +0000 UTC m=+436.631388957" watchObservedRunningTime="2026-07-12 22:17:44.319191337 +0000 UTC m=+436.632694691"
Jul 12 22:17:44.319991 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:17:44.319956    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-gn87p\" is forbidden: User \"system:node:ip-10-0-135-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-47.ec2.internal' and this object" podUID="392eafff-89ce-4eed-bc96-507683c5b012" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gn87p"
Jul 12 22:17:46.665036 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:46.664994    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jd4kb"]
Jul 12 22:17:46.665454 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:46.665223    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jd4kb" podUID="f013fece-befe-44ff-89f1-1b71c5888c7e" containerName="manager" containerID="cri-o://93c0ecafa02bec7239a7982815ff0f12d679ffb036510b83c524ce94daa5a48f" gracePeriod=10
Jul 12 22:17:46.898756 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:46.898723    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55m8v"]
Jul 12 22:17:46.902066 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:46.902050    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55m8v"
Jul 12 22:17:46.917142 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:46.917073    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55m8v"]
Jul 12 22:17:46.929580 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:46.929555    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jd4kb"
Jul 12 22:17:47.032792 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:47.032759    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f013fece-befe-44ff-89f1-1b71c5888c7e-extensions-socket-volume\") pod \"f013fece-befe-44ff-89f1-1b71c5888c7e\" (UID: \"f013fece-befe-44ff-89f1-1b71c5888c7e\") "
Jul 12 22:17:47.032967 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:47.032854    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxx9m\" (UniqueName: \"kubernetes.io/projected/f013fece-befe-44ff-89f1-1b71c5888c7e-kube-api-access-pxx9m\") pod \"f013fece-befe-44ff-89f1-1b71c5888c7e\" (UID: \"f013fece-befe-44ff-89f1-1b71c5888c7e\") "
Jul 12 22:17:47.032967 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:47.032932    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcsj4\" (UniqueName: \"kubernetes.io/projected/0f1d1e7b-bad9-43ff-be2f-0df8ea968968-kube-api-access-qcsj4\") pod \"kuadrant-operator-controller-manager-86878c769f-55m8v\" (UID: \"0f1d1e7b-bad9-43ff-be2f-0df8ea968968\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55m8v"
Jul 12 22:17:47.033042 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:47.032977    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f1d1e7b-bad9-43ff-be2f-0df8ea968968-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-55m8v\" (UID: \"0f1d1e7b-bad9-43ff-be2f-0df8ea968968\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55m8v"
Jul 12 22:17:47.033256 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:47.033222    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f013fece-befe-44ff-89f1-1b71c5888c7e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f013fece-befe-44ff-89f1-1b71c5888c7e" (UID: "f013fece-befe-44ff-89f1-1b71c5888c7e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:17:47.035106 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:47.035080    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f013fece-befe-44ff-89f1-1b71c5888c7e-kube-api-access-pxx9m" (OuterVolumeSpecName: "kube-api-access-pxx9m") pod "f013fece-befe-44ff-89f1-1b71c5888c7e" (UID: "f013fece-befe-44ff-89f1-1b71c5888c7e"). InnerVolumeSpecName "kube-api-access-pxx9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 22:17:47.133640 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:47.133600    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcsj4\" (UniqueName: \"kubernetes.io/projected/0f1d1e7b-bad9-43ff-be2f-0df8ea968968-kube-api-access-qcsj4\") pod \"kuadrant-operator-controller-manager-86878c769f-55m8v\" (UID: \"0f1d1e7b-bad9-43ff-be2f-0df8ea968968\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55m8v"
Jul 12 22:17:47.133640 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:47.133645    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f1d1e7b-bad9-43ff-be2f-0df8ea968968-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-55m8v\" (UID: \"0f1d1e7b-bad9-43ff-be2f-0df8ea968968\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55m8v"
Jul 12 22:17:47.133877 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:47.133683    2560 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f013fece-befe-44ff-89f1-1b71c5888c7e-extensions-socket-volume\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:17:47.133877 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:47.133692    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxx9m\" (UniqueName: \"kubernetes.io/projected/f013fece-befe-44ff-89f1-1b71c5888c7e-kube-api-access-pxx9m\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:17:47.134005 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:47.133989    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f1d1e7b-bad9-43ff-be2f-0df8ea968968-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-55m8v\" (UID: \"0f1d1e7b-bad9-43ff-be2f-0df8ea968968\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55m8v"
Jul 12 22:17:47.144659 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:47.144624    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcsj4\" (UniqueName: \"kubernetes.io/projected/0f1d1e7b-bad9-43ff-be2f-0df8ea968968-kube-api-access-qcsj4\") pod \"kuadrant-operator-controller-manager-86878c769f-55m8v\" (UID: \"0f1d1e7b-bad9-43ff-be2f-0df8ea968968\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55m8v"
Jul 12 22:17:47.212588 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:47.212460    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55m8v"
Jul 12 22:17:47.288622 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:47.288584    2560 generic.go:356] "Generic (PLEG): container finished" podID="f013fece-befe-44ff-89f1-1b71c5888c7e" containerID="93c0ecafa02bec7239a7982815ff0f12d679ffb036510b83c524ce94daa5a48f" exitCode=0
Jul 12 22:17:47.288789 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:47.288630    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jd4kb"
Jul 12 22:17:47.288789 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:47.288656    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jd4kb" event={"ID":"f013fece-befe-44ff-89f1-1b71c5888c7e","Type":"ContainerDied","Data":"93c0ecafa02bec7239a7982815ff0f12d679ffb036510b83c524ce94daa5a48f"}
Jul 12 22:17:47.288789 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:47.288731    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jd4kb" event={"ID":"f013fece-befe-44ff-89f1-1b71c5888c7e","Type":"ContainerDied","Data":"dad38b21e0eb0a22d3219150d93b7f6c1199af14d652ec3b3abd4fc62edbca97"}
Jul 12 22:17:47.288789 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:47.288752    2560 scope.go:117] "RemoveContainer" containerID="93c0ecafa02bec7239a7982815ff0f12d679ffb036510b83c524ce94daa5a48f"
Jul 12 22:17:47.299801 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:47.299781    2560 scope.go:117] "RemoveContainer" containerID="93c0ecafa02bec7239a7982815ff0f12d679ffb036510b83c524ce94daa5a48f"
Jul 12 22:17:47.300109 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:17:47.300088    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93c0ecafa02bec7239a7982815ff0f12d679ffb036510b83c524ce94daa5a48f\": container with ID starting with 93c0ecafa02bec7239a7982815ff0f12d679ffb036510b83c524ce94daa5a48f not found: ID does not exist" containerID="93c0ecafa02bec7239a7982815ff0f12d679ffb036510b83c524ce94daa5a48f"
Jul 12 22:17:47.300153 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:47.300117    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93c0ecafa02bec7239a7982815ff0f12d679ffb036510b83c524ce94daa5a48f"} err="failed to get container status \"93c0ecafa02bec7239a7982815ff0f12d679ffb036510b83c524ce94daa5a48f\": rpc error: code = NotFound desc = could not find container \"93c0ecafa02bec7239a7982815ff0f12d679ffb036510b83c524ce94daa5a48f\": container with ID starting with 93c0ecafa02bec7239a7982815ff0f12d679ffb036510b83c524ce94daa5a48f not found: ID does not exist"
Jul 12 22:17:47.316342 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:47.316306    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jd4kb"]
Jul 12 22:17:47.328723 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:47.328661    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jd4kb"]
Jul 12 22:17:47.387573 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:17:47.387523    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f1d1e7b_bad9_43ff_be2f_0df8ea968968.slice/crio-b03d3077d20927f2a99fe05cd2c5b45d17ec64e2c45ad1c04889217bbd669a93 WatchSource:0}: Error finding container b03d3077d20927f2a99fe05cd2c5b45d17ec64e2c45ad1c04889217bbd669a93: Status 404 returned error can't find the container with id b03d3077d20927f2a99fe05cd2c5b45d17ec64e2c45ad1c04889217bbd669a93
Jul 12 22:17:47.397423 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:47.397396    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55m8v"]
Jul 12 22:17:48.246158 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:48.246129    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f013fece-befe-44ff-89f1-1b71c5888c7e" path="/var/lib/kubelet/pods/f013fece-befe-44ff-89f1-1b71c5888c7e/volumes"
Jul 12 22:17:48.296516 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:48.296476    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55m8v" event={"ID":"0f1d1e7b-bad9-43ff-be2f-0df8ea968968","Type":"ContainerStarted","Data":"94290a02c5d231714cba78ec7665e3fef260b620cfe9144c71bca7442b108b89"}
Jul 12 22:17:48.296724 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:48.296551    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55m8v" event={"ID":"0f1d1e7b-bad9-43ff-be2f-0df8ea968968","Type":"ContainerStarted","Data":"b03d3077d20927f2a99fe05cd2c5b45d17ec64e2c45ad1c04889217bbd669a93"}
Jul 12 22:17:48.296724 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:48.296589    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55m8v"
Jul 12 22:17:48.317010 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:48.316958    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55m8v" podStartSLOduration=2.316941356 podStartE2EDuration="2.316941356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:17:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:17:48.314726788 +0000 UTC m=+440.628230142" watchObservedRunningTime="2026-07-12 22:17:48.316941356 +0000 UTC m=+440.630444709"
Jul 12 22:17:55.275443 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:55.275364    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xzccs"
Jul 12 22:17:59.303016 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:17:59.302980    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55m8v"
Jul 12 22:18:20.088232 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:20.088199    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-fxttl"]
Jul 12 22:18:20.088772 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:20.088526    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f013fece-befe-44ff-89f1-1b71c5888c7e" containerName="manager"
Jul 12 22:18:20.088772 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:20.088552    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f013fece-befe-44ff-89f1-1b71c5888c7e" containerName="manager"
Jul 12 22:18:20.088772 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:20.088612    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f013fece-befe-44ff-89f1-1b71c5888c7e" containerName="manager"
Jul 12 22:18:20.091731 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:20.091714    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-fxttl"
Jul 12 22:18:20.094522 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:20.094504    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-sswrt\""
Jul 12 22:18:20.097254 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:20.097234    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-fxttl"]
Jul 12 22:18:20.220326 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:20.220290    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h84vn\" (UniqueName: \"kubernetes.io/projected/1eeca096-ed5c-4a79-aafa-a0ab1c135aee-kube-api-access-h84vn\") pod \"authorino-75d556bd77-fxttl\" (UID: \"1eeca096-ed5c-4a79-aafa-a0ab1c135aee\") " pod="kuadrant-system/authorino-75d556bd77-fxttl"
Jul 12 22:18:20.320959 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:20.320923    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h84vn\" (UniqueName: \"kubernetes.io/projected/1eeca096-ed5c-4a79-aafa-a0ab1c135aee-kube-api-access-h84vn\") pod \"authorino-75d556bd77-fxttl\" (UID: \"1eeca096-ed5c-4a79-aafa-a0ab1c135aee\") " pod="kuadrant-system/authorino-75d556bd77-fxttl"
Jul 12 22:18:20.329955 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:20.329926    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h84vn\" (UniqueName: \"kubernetes.io/projected/1eeca096-ed5c-4a79-aafa-a0ab1c135aee-kube-api-access-h84vn\") pod \"authorino-75d556bd77-fxttl\" (UID: \"1eeca096-ed5c-4a79-aafa-a0ab1c135aee\") " pod="kuadrant-system/authorino-75d556bd77-fxttl"
Jul 12 22:18:20.402141 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:20.402054    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-fxttl"
Jul 12 22:18:20.541676 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:20.541648    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-fxttl"]
Jul 12 22:18:20.544212 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:18:20.544184    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1eeca096_ed5c_4a79_aafa_a0ab1c135aee.slice/crio-207fb78a6fb28514298a4c23870181f9064d398fbdc9184470d1235992714f11 WatchSource:0}: Error finding container 207fb78a6fb28514298a4c23870181f9064d398fbdc9184470d1235992714f11: Status 404 returned error can't find the container with id 207fb78a6fb28514298a4c23870181f9064d398fbdc9184470d1235992714f11
Jul 12 22:18:21.458276 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:21.458232    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-fxttl" event={"ID":"1eeca096-ed5c-4a79-aafa-a0ab1c135aee","Type":"ContainerStarted","Data":"207fb78a6fb28514298a4c23870181f9064d398fbdc9184470d1235992714f11"}
Jul 12 22:18:24.478360 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:24.478309    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-fxttl" event={"ID":"1eeca096-ed5c-4a79-aafa-a0ab1c135aee","Type":"ContainerStarted","Data":"f33da4794ce589d0d8996199b7162052fa51b693a416d1c769936b3bb8efc53a"}
Jul 12 22:18:24.493722 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:24.493681    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-fxttl" podStartSLOduration=1.502650244 podStartE2EDuration="4.493666979s" totalImagesPullingTime="2.991016735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:18:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:18:24.492808265 +0000 UTC m=+476.806311621" watchObservedRunningTime="2026-07-12 22:18:24.493666979 +0000 UTC m=+476.807170333"
Jul 12 22:18:49.520091 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:49.520054    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-fvjpc"]
Jul 12 22:18:49.525061 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:49.525033    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-fvjpc"
Jul 12 22:18:49.530004 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:49.529977    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-fvjpc"]
Jul 12 22:18:49.678043 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:49.678013    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfqff\" (UniqueName: \"kubernetes.io/projected/f2f941f6-9481-4717-87b0-93d6b107fff6-kube-api-access-dfqff\") pod \"authorino-5d4b67c446-fvjpc\" (UID: \"f2f941f6-9481-4717-87b0-93d6b107fff6\") " pod="kuadrant-system/authorino-5d4b67c446-fvjpc"
Jul 12 22:18:49.716163 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:49.716110    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-fvjpc"]
Jul 12 22:18:49.716428 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:18:49.716400    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-dfqff], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-fvjpc" podUID="f2f941f6-9481-4717-87b0-93d6b107fff6"
Jul 12 22:18:49.739875 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:49.739835    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-mvm8z"]
Jul 12 22:18:49.744278 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:49.744245    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-mvm8z"
Jul 12 22:18:49.747226 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:49.747207    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 12 22:18:49.752584 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:49.752560    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-mvm8z"]
Jul 12 22:18:49.770642 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:49.770561    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-mvm8z"]
Jul 12 22:18:49.770877 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:18:49.770852    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-lpjlb tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-lpjlb tls-cert]: context canceled" pod="kuadrant-system/authorino-76877994-mvm8z" podUID="2b1654bd-7a38-4e06-94ed-183b321511ac"
Jul 12 22:18:49.779019 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:49.778995    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfqff\" (UniqueName: \"kubernetes.io/projected/f2f941f6-9481-4717-87b0-93d6b107fff6-kube-api-access-dfqff\") pod \"authorino-5d4b67c446-fvjpc\" (UID: \"f2f941f6-9481-4717-87b0-93d6b107fff6\") " pod="kuadrant-system/authorino-5d4b67c446-fvjpc"
Jul 12 22:18:49.790934 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:49.790899    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfqff\" (UniqueName: \"kubernetes.io/projected/f2f941f6-9481-4717-87b0-93d6b107fff6-kube-api-access-dfqff\") pod \"authorino-5d4b67c446-fvjpc\" (UID: \"f2f941f6-9481-4717-87b0-93d6b107fff6\") " pod="kuadrant-system/authorino-5d4b67c446-fvjpc"
Jul 12 22:18:49.794756 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:49.794728    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-88c59b6b9-tpdzr"]
Jul 12 22:18:49.797844 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:49.797810    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-88c59b6b9-tpdzr"
Jul 12 22:18:49.806478 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:49.806451    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-88c59b6b9-tpdzr"]
Jul 12 22:18:49.880067 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:49.880026    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2b1654bd-7a38-4e06-94ed-183b321511ac-tls-cert\") pod \"authorino-76877994-mvm8z\" (UID: \"2b1654bd-7a38-4e06-94ed-183b321511ac\") " pod="kuadrant-system/authorino-76877994-mvm8z"
Jul 12 22:18:49.880253 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:49.880140    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpjlb\" (UniqueName: \"kubernetes.io/projected/2b1654bd-7a38-4e06-94ed-183b321511ac-kube-api-access-lpjlb\") pod \"authorino-76877994-mvm8z\" (UID: \"2b1654bd-7a38-4e06-94ed-183b321511ac\") " pod="kuadrant-system/authorino-76877994-mvm8z"
Jul 12 22:18:49.980696 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:49.980661    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpnkg\" (UniqueName: \"kubernetes.io/projected/91f5df03-b49d-45af-9f03-3ccc85703d3a-kube-api-access-rpnkg\") pod \"authorino-88c59b6b9-tpdzr\" (UID: \"91f5df03-b49d-45af-9f03-3ccc85703d3a\") " pod="kuadrant-system/authorino-88c59b6b9-tpdzr"
Jul 12 22:18:49.980858 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:49.980726    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/91f5df03-b49d-45af-9f03-3ccc85703d3a-tls-cert\") pod \"authorino-88c59b6b9-tpdzr\" (UID: \"91f5df03-b49d-45af-9f03-3ccc85703d3a\") " pod="kuadrant-system/authorino-88c59b6b9-tpdzr"
Jul 12 22:18:49.980858 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:49.980811    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpjlb\" (UniqueName: \"kubernetes.io/projected/2b1654bd-7a38-4e06-94ed-183b321511ac-kube-api-access-lpjlb\") pod \"authorino-76877994-mvm8z\" (UID: \"2b1654bd-7a38-4e06-94ed-183b321511ac\") " pod="kuadrant-system/authorino-76877994-mvm8z"
Jul 12 22:18:49.980934 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:49.980859    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2b1654bd-7a38-4e06-94ed-183b321511ac-tls-cert\") pod \"authorino-76877994-mvm8z\" (UID: \"2b1654bd-7a38-4e06-94ed-183b321511ac\") " pod="kuadrant-system/authorino-76877994-mvm8z"
Jul 12 22:18:49.983640 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:49.983616    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2b1654bd-7a38-4e06-94ed-183b321511ac-tls-cert\") pod \"authorino-76877994-mvm8z\" (UID: \"2b1654bd-7a38-4e06-94ed-183b321511ac\") " pod="kuadrant-system/authorino-76877994-mvm8z"
Jul 12 22:18:49.991294 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:49.991266    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpjlb\" (UniqueName: \"kubernetes.io/projected/2b1654bd-7a38-4e06-94ed-183b321511ac-kube-api-access-lpjlb\") pod \"authorino-76877994-mvm8z\" (UID: \"2b1654bd-7a38-4e06-94ed-183b321511ac\") " pod="kuadrant-system/authorino-76877994-mvm8z"
Jul 12 22:18:50.082050 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:50.082018    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpnkg\" (UniqueName: \"kubernetes.io/projected/91f5df03-b49d-45af-9f03-3ccc85703d3a-kube-api-access-rpnkg\") pod \"authorino-88c59b6b9-tpdzr\" (UID: \"91f5df03-b49d-45af-9f03-3ccc85703d3a\") " pod="kuadrant-system/authorino-88c59b6b9-tpdzr"
Jul 12 22:18:50.082250 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:50.082112    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/91f5df03-b49d-45af-9f03-3ccc85703d3a-tls-cert\") pod \"authorino-88c59b6b9-tpdzr\" (UID: \"91f5df03-b49d-45af-9f03-3ccc85703d3a\") " pod="kuadrant-system/authorino-88c59b6b9-tpdzr"
Jul 12 22:18:50.084817 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:50.084794    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/91f5df03-b49d-45af-9f03-3ccc85703d3a-tls-cert\") pod \"authorino-88c59b6b9-tpdzr\" (UID: \"91f5df03-b49d-45af-9f03-3ccc85703d3a\") " pod="kuadrant-system/authorino-88c59b6b9-tpdzr"
Jul 12 22:18:50.093680 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:50.093655    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpnkg\" (UniqueName: \"kubernetes.io/projected/91f5df03-b49d-45af-9f03-3ccc85703d3a-kube-api-access-rpnkg\") pod \"authorino-88c59b6b9-tpdzr\" (UID: \"91f5df03-b49d-45af-9f03-3ccc85703d3a\") " pod="kuadrant-system/authorino-88c59b6b9-tpdzr"
Jul 12 22:18:50.108348 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:50.108323    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-88c59b6b9-tpdzr"
Jul 12 22:18:50.245493 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:18:50.245449    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91f5df03_b49d_45af_9f03_3ccc85703d3a.slice/crio-742ba20f5e603be7fc3abebe40a339af36aaa9836cc9f5c420cdf9e1d8bcd470 WatchSource:0}: Error finding container 742ba20f5e603be7fc3abebe40a339af36aaa9836cc9f5c420cdf9e1d8bcd470: Status 404 returned error can't find the container with id 742ba20f5e603be7fc3abebe40a339af36aaa9836cc9f5c420cdf9e1d8bcd470
Jul 12 22:18:50.247998 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:50.247979    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-88c59b6b9-tpdzr"]
Jul 12 22:18:50.601714 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:50.601682    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-mvm8z"
Jul 12 22:18:50.601714 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:50.601703    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-88c59b6b9-tpdzr" event={"ID":"91f5df03-b49d-45af-9f03-3ccc85703d3a","Type":"ContainerStarted","Data":"742ba20f5e603be7fc3abebe40a339af36aaa9836cc9f5c420cdf9e1d8bcd470"}
Jul 12 22:18:50.602179 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:50.601827    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-fvjpc"
Jul 12 22:18:50.609921 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:50.609902    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-mvm8z"
Jul 12 22:18:50.612040 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:50.612021    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-fvjpc"
Jul 12 22:18:50.789504 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:50.789475    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpjlb\" (UniqueName: \"kubernetes.io/projected/2b1654bd-7a38-4e06-94ed-183b321511ac-kube-api-access-lpjlb\") pod \"2b1654bd-7a38-4e06-94ed-183b321511ac\" (UID: \"2b1654bd-7a38-4e06-94ed-183b321511ac\") "
Jul 12 22:18:50.789805 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:50.789555    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2b1654bd-7a38-4e06-94ed-183b321511ac-tls-cert\") pod \"2b1654bd-7a38-4e06-94ed-183b321511ac\" (UID: \"2b1654bd-7a38-4e06-94ed-183b321511ac\") "
Jul 12 22:18:50.789805 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:50.789662    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfqff\" (UniqueName: \"kubernetes.io/projected/f2f941f6-9481-4717-87b0-93d6b107fff6-kube-api-access-dfqff\") pod \"f2f941f6-9481-4717-87b0-93d6b107fff6\" (UID: \"f2f941f6-9481-4717-87b0-93d6b107fff6\") "
Jul 12 22:18:50.791815 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:50.791787    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b1654bd-7a38-4e06-94ed-183b321511ac-kube-api-access-lpjlb" (OuterVolumeSpecName: "kube-api-access-lpjlb") pod "2b1654bd-7a38-4e06-94ed-183b321511ac" (UID: "2b1654bd-7a38-4e06-94ed-183b321511ac"). InnerVolumeSpecName "kube-api-access-lpjlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 22:18:50.792404 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:50.792384    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b1654bd-7a38-4e06-94ed-183b321511ac-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "2b1654bd-7a38-4e06-94ed-183b321511ac" (UID: "2b1654bd-7a38-4e06-94ed-183b321511ac"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 12 22:18:50.792404 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:50.792398    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2f941f6-9481-4717-87b0-93d6b107fff6-kube-api-access-dfqff" (OuterVolumeSpecName: "kube-api-access-dfqff") pod "f2f941f6-9481-4717-87b0-93d6b107fff6" (UID: "f2f941f6-9481-4717-87b0-93d6b107fff6"). InnerVolumeSpecName "kube-api-access-dfqff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 22:18:50.890882 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:50.890795    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfqff\" (UniqueName: \"kubernetes.io/projected/f2f941f6-9481-4717-87b0-93d6b107fff6-kube-api-access-dfqff\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:18:50.890882 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:50.890824    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpjlb\" (UniqueName: \"kubernetes.io/projected/2b1654bd-7a38-4e06-94ed-183b321511ac-kube-api-access-lpjlb\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:18:50.890882 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:50.890834    2560 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2b1654bd-7a38-4e06-94ed-183b321511ac-tls-cert\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:18:51.607397 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:51.607366    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-fvjpc"
Jul 12 22:18:51.607928 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:51.607418    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-mvm8z"
Jul 12 22:18:51.607928 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:51.607409    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-88c59b6b9-tpdzr" event={"ID":"91f5df03-b49d-45af-9f03-3ccc85703d3a","Type":"ContainerStarted","Data":"0bdd9009a3091d6e8f2e70e628a95e83a37b2e7b013d783a34ffbea3bb030b8e"}
Jul 12 22:18:51.622668 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:51.622619    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-88c59b6b9-tpdzr" podStartSLOduration=2.19147285 podStartE2EDuration="2.62260171s" totalImagesPullingTime="431.12886ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:18:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:18:51.621971628 +0000 UTC m=+503.935474981" watchObservedRunningTime="2026-07-12 22:18:51.62260171 +0000 UTC m=+503.936105063"
Jul 12 22:18:51.652996 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:51.652968    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-fxttl"]
Jul 12 22:18:51.653207 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:51.653179    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-fxttl" podUID="1eeca096-ed5c-4a79-aafa-a0ab1c135aee" containerName="authorino" containerID="cri-o://f33da4794ce589d0d8996199b7162052fa51b693a416d1c769936b3bb8efc53a" gracePeriod=30
Jul 12 22:18:51.656434 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:51.656401    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-fvjpc"]
Jul 12 22:18:51.666961 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:51.666913    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-fvjpc"]
Jul 12 22:18:51.689499 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:51.689472    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-mvm8z"]
Jul 12 22:18:51.696433 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:51.696403    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-mvm8z"]
Jul 12 22:18:51.914281 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:51.914256    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-fxttl"
Jul 12 22:18:51.997264 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:51.997218    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h84vn\" (UniqueName: \"kubernetes.io/projected/1eeca096-ed5c-4a79-aafa-a0ab1c135aee-kube-api-access-h84vn\") pod \"1eeca096-ed5c-4a79-aafa-a0ab1c135aee\" (UID: \"1eeca096-ed5c-4a79-aafa-a0ab1c135aee\") "
Jul 12 22:18:51.999435 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:51.999405    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eeca096-ed5c-4a79-aafa-a0ab1c135aee-kube-api-access-h84vn" (OuterVolumeSpecName: "kube-api-access-h84vn") pod "1eeca096-ed5c-4a79-aafa-a0ab1c135aee" (UID: "1eeca096-ed5c-4a79-aafa-a0ab1c135aee"). InnerVolumeSpecName "kube-api-access-h84vn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 22:18:52.098120 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:52.098091    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h84vn\" (UniqueName: \"kubernetes.io/projected/1eeca096-ed5c-4a79-aafa-a0ab1c135aee-kube-api-access-h84vn\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:18:52.246265 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:52.246189    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b1654bd-7a38-4e06-94ed-183b321511ac" path="/var/lib/kubelet/pods/2b1654bd-7a38-4e06-94ed-183b321511ac/volumes"
Jul 12 22:18:52.246442 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:52.246429    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2f941f6-9481-4717-87b0-93d6b107fff6" path="/var/lib/kubelet/pods/f2f941f6-9481-4717-87b0-93d6b107fff6/volumes"
Jul 12 22:18:52.613405 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:52.613371    2560 generic.go:356] "Generic (PLEG): container finished" podID="1eeca096-ed5c-4a79-aafa-a0ab1c135aee" containerID="f33da4794ce589d0d8996199b7162052fa51b693a416d1c769936b3bb8efc53a" exitCode=0
Jul 12 22:18:52.613913 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:52.613413    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-fxttl"
Jul 12 22:18:52.613913 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:52.613469    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-fxttl" event={"ID":"1eeca096-ed5c-4a79-aafa-a0ab1c135aee","Type":"ContainerDied","Data":"f33da4794ce589d0d8996199b7162052fa51b693a416d1c769936b3bb8efc53a"}
Jul 12 22:18:52.613913 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:52.613515    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-fxttl" event={"ID":"1eeca096-ed5c-4a79-aafa-a0ab1c135aee","Type":"ContainerDied","Data":"207fb78a6fb28514298a4c23870181f9064d398fbdc9184470d1235992714f11"}
Jul 12 22:18:52.613913 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:52.613568    2560 scope.go:117] "RemoveContainer" containerID="f33da4794ce589d0d8996199b7162052fa51b693a416d1c769936b3bb8efc53a"
Jul 12 22:18:52.623349 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:52.623331    2560 scope.go:117] "RemoveContainer" containerID="f33da4794ce589d0d8996199b7162052fa51b693a416d1c769936b3bb8efc53a"
Jul 12 22:18:52.623692 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:18:52.623669    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f33da4794ce589d0d8996199b7162052fa51b693a416d1c769936b3bb8efc53a\": container with ID starting with f33da4794ce589d0d8996199b7162052fa51b693a416d1c769936b3bb8efc53a not found: ID does not exist" containerID="f33da4794ce589d0d8996199b7162052fa51b693a416d1c769936b3bb8efc53a"
Jul 12 22:18:52.623764 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:52.623700    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f33da4794ce589d0d8996199b7162052fa51b693a416d1c769936b3bb8efc53a"} err="failed to get container status \"f33da4794ce589d0d8996199b7162052fa51b693a416d1c769936b3bb8efc53a\": rpc error: code = NotFound desc = could not find container \"f33da4794ce589d0d8996199b7162052fa51b693a416d1c769936b3bb8efc53a\": container with ID starting with f33da4794ce589d0d8996199b7162052fa51b693a416d1c769936b3bb8efc53a not found: ID does not exist"
Jul 12 22:18:52.629916 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:52.629892    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-fxttl"]
Jul 12 22:18:52.634252 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:52.634232    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-fxttl"]
Jul 12 22:18:54.245424 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:18:54.245393    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1eeca096-ed5c-4a79-aafa-a0ab1c135aee" path="/var/lib/kubelet/pods/1eeca096-ed5c-4a79-aafa-a0ab1c135aee/volumes"
Jul 12 22:19:32.342033 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:19:32.341989    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-d2264"]
Jul 12 22:19:32.342460 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:19:32.342357    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1eeca096-ed5c-4a79-aafa-a0ab1c135aee" containerName="authorino"
Jul 12 22:19:32.342460 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:19:32.342374    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1eeca096-ed5c-4a79-aafa-a0ab1c135aee" containerName="authorino"
Jul 12 22:19:32.342460 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:19:32.342445    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1eeca096-ed5c-4a79-aafa-a0ab1c135aee" containerName="authorino"
Jul 12 22:19:32.346887 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:19:32.346870    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-d2264"
Jul 12 22:19:32.353507 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:19:32.353480    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 12 22:19:32.353651 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:19:32.353510    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-p8frm\""
Jul 12 22:19:32.360965 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:19:32.360936    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-d2264"]
Jul 12 22:19:32.442798 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:19:32.442746    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnk8l\" (UniqueName: \"kubernetes.io/projected/8b5f6930-2044-4c4e-b26d-65a6a273e4e2-kube-api-access-fnk8l\") pod \"payload-pre-processing-74584bb9f8-d2264\" (UID: \"8b5f6930-2044-4c4e-b26d-65a6a273e4e2\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-d2264"
Jul 12 22:19:32.442798 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:19:32.442794    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b5f6930-2044-4c4e-b26d-65a6a273e4e2-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-d2264\" (UID: \"8b5f6930-2044-4c4e-b26d-65a6a273e4e2\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-d2264"
Jul 12 22:19:32.544151 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:19:32.544113    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnk8l\" (UniqueName: \"kubernetes.io/projected/8b5f6930-2044-4c4e-b26d-65a6a273e4e2-kube-api-access-fnk8l\") pod \"payload-pre-processing-74584bb9f8-d2264\" (UID: \"8b5f6930-2044-4c4e-b26d-65a6a273e4e2\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-d2264"
Jul 12 22:19:32.544151 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:19:32.544154    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b5f6930-2044-4c4e-b26d-65a6a273e4e2-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-d2264\" (UID: \"8b5f6930-2044-4c4e-b26d-65a6a273e4e2\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-d2264"
Jul 12 22:19:32.544925 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:19:32.544900    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b5f6930-2044-4c4e-b26d-65a6a273e4e2-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-d2264\" (UID: \"8b5f6930-2044-4c4e-b26d-65a6a273e4e2\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-d2264"
Jul 12 22:19:32.552520 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:19:32.552493    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnk8l\" (UniqueName: \"kubernetes.io/projected/8b5f6930-2044-4c4e-b26d-65a6a273e4e2-kube-api-access-fnk8l\") pod \"payload-pre-processing-74584bb9f8-d2264\" (UID: \"8b5f6930-2044-4c4e-b26d-65a6a273e4e2\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-d2264"
Jul 12 22:19:32.658432 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:19:32.658346    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-d2264"
Jul 12 22:19:32.803558 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:19:32.803516    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-d2264"]
Jul 12 22:19:32.806084 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:19:32.806056    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b5f6930_2044_4c4e_b26d_65a6a273e4e2.slice/crio-ad72f681da8920fefdf1b64db6a2373582d826eec7bca73d9f06bac4d77f2b70 WatchSource:0}: Error finding container ad72f681da8920fefdf1b64db6a2373582d826eec7bca73d9f06bac4d77f2b70: Status 404 returned error can't find the container with id ad72f681da8920fefdf1b64db6a2373582d826eec7bca73d9f06bac4d77f2b70
Jul 12 22:19:33.806461 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:19:33.806395    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-d2264" event={"ID":"8b5f6930-2044-4c4e-b26d-65a6a273e4e2","Type":"ContainerStarted","Data":"ad72f681da8920fefdf1b64db6a2373582d826eec7bca73d9f06bac4d77f2b70"}
Jul 12 22:19:34.813244 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:19:34.813209    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-d2264" event={"ID":"8b5f6930-2044-4c4e-b26d-65a6a273e4e2","Type":"ContainerStarted","Data":"4bfe32faf349ffbd71e8475e40db08b2d9f7bd381efb798cabbb660de03e3d7f"}
Jul 12 22:19:34.813637 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:19:34.813338    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-d2264"
Jul 12 22:19:34.829638 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:19:34.829417    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-74584bb9f8-d2264" podStartSLOduration=1.511560304 podStartE2EDuration="2.82939629s" totalImagesPullingTime="1.317835986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:19:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:19:34.82888165 +0000 UTC m=+547.142385003" watchObservedRunningTime="2026-07-12 22:19:34.82939629 +0000 UTC m=+547.142899645"
Jul 12 22:19:45.822521 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:19:45.822488    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-d2264"
Jul 12 22:20:02.384596 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:02.384552    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-bm656"]
Jul 12 22:20:02.388009 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:02.387989    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bm656"
Jul 12 22:20:02.390819 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:02.390798    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 12 22:20:02.390978 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:02.390960    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-7867w\""
Jul 12 22:20:02.392060 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:02.392039    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 12 22:20:02.392208 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:02.392190    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 12 22:20:02.399171 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:02.399147    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-bm656"]
Jul 12 22:20:02.486712 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:02.486678    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f737042d-b618-47e6-8815-6dbc10c4bf17-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bm656\" (UID: \"f737042d-b618-47e6-8815-6dbc10c4bf17\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bm656"
Jul 12 22:20:02.486712 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:02.486713    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f737042d-b618-47e6-8815-6dbc10c4bf17-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bm656\" (UID: \"f737042d-b618-47e6-8815-6dbc10c4bf17\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bm656"
Jul 12 22:20:02.486914 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:02.486737    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f737042d-b618-47e6-8815-6dbc10c4bf17-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bm656\" (UID: \"f737042d-b618-47e6-8815-6dbc10c4bf17\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bm656"
Jul 12 22:20:02.486951 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:02.486908    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f737042d-b618-47e6-8815-6dbc10c4bf17-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bm656\" (UID: \"f737042d-b618-47e6-8815-6dbc10c4bf17\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bm656"
Jul 12 22:20:02.487002 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:02.486951    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79vhl\" (UniqueName: \"kubernetes.io/projected/f737042d-b618-47e6-8815-6dbc10c4bf17-kube-api-access-79vhl\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bm656\" (UID: \"f737042d-b618-47e6-8815-6dbc10c4bf17\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bm656"
Jul 12 22:20:02.487039 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:02.487015    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f737042d-b618-47e6-8815-6dbc10c4bf17-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bm656\" (UID: \"f737042d-b618-47e6-8815-6dbc10c4bf17\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bm656"
Jul 12 22:20:02.587709 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:02.587667    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f737042d-b618-47e6-8815-6dbc10c4bf17-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bm656\" (UID: \"f737042d-b618-47e6-8815-6dbc10c4bf17\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bm656"
Jul 12 22:20:02.587709 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:02.587705    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79vhl\" (UniqueName: \"kubernetes.io/projected/f737042d-b618-47e6-8815-6dbc10c4bf17-kube-api-access-79vhl\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bm656\" (UID: \"f737042d-b618-47e6-8815-6dbc10c4bf17\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bm656"
Jul 12 22:20:02.587973 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:02.587726    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f737042d-b618-47e6-8815-6dbc10c4bf17-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bm656\" (UID: \"f737042d-b618-47e6-8815-6dbc10c4bf17\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bm656"
Jul 12 22:20:02.587973 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:02.587861    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f737042d-b618-47e6-8815-6dbc10c4bf17-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bm656\" (UID: \"f737042d-b618-47e6-8815-6dbc10c4bf17\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bm656"
Jul 12 22:20:02.587973 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:02.587909    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f737042d-b618-47e6-8815-6dbc10c4bf17-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bm656\" (UID: \"f737042d-b618-47e6-8815-6dbc10c4bf17\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bm656"
Jul 12 22:20:02.587973 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:02.587943    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f737042d-b618-47e6-8815-6dbc10c4bf17-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bm656\" (UID: \"f737042d-b618-47e6-8815-6dbc10c4bf17\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bm656"
Jul 12 22:20:02.588207 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:02.588186    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f737042d-b618-47e6-8815-6dbc10c4bf17-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bm656\" (UID: \"f737042d-b618-47e6-8815-6dbc10c4bf17\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bm656"
Jul 12 22:20:02.588265 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:02.588233    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f737042d-b618-47e6-8815-6dbc10c4bf17-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bm656\" (UID: \"f737042d-b618-47e6-8815-6dbc10c4bf17\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bm656"
Jul 12 22:20:02.588308 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:02.588289    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f737042d-b618-47e6-8815-6dbc10c4bf17-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bm656\" (UID: \"f737042d-b618-47e6-8815-6dbc10c4bf17\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bm656"
Jul 12 22:20:02.590161 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:02.590129    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f737042d-b618-47e6-8815-6dbc10c4bf17-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bm656\" (UID: \"f737042d-b618-47e6-8815-6dbc10c4bf17\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bm656"
Jul 12 22:20:02.590380 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:02.590363    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f737042d-b618-47e6-8815-6dbc10c4bf17-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bm656\" (UID: \"f737042d-b618-47e6-8815-6dbc10c4bf17\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bm656"
Jul 12 22:20:02.595687 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:02.595665    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79vhl\" (UniqueName: \"kubernetes.io/projected/f737042d-b618-47e6-8815-6dbc10c4bf17-kube-api-access-79vhl\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bm656\" (UID: \"f737042d-b618-47e6-8815-6dbc10c4bf17\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bm656"
Jul 12 22:20:02.699143 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:02.699055    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bm656"
Jul 12 22:20:02.838244 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:02.838200    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-bm656"]
Jul 12 22:20:02.840668 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:20:02.840639    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf737042d_b618_47e6_8815_6dbc10c4bf17.slice/crio-f8453cac72c25e7fab3ba6c3e09720ba32253289b7eeeccebd007e2bad4d41f0 WatchSource:0}: Error finding container f8453cac72c25e7fab3ba6c3e09720ba32253289b7eeeccebd007e2bad4d41f0: Status 404 returned error can't find the container with id f8453cac72c25e7fab3ba6c3e09720ba32253289b7eeeccebd007e2bad4d41f0
Jul 12 22:20:02.954944 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:02.954850    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bm656" event={"ID":"f737042d-b618-47e6-8815-6dbc10c4bf17","Type":"ContainerStarted","Data":"f8453cac72c25e7fab3ba6c3e09720ba32253289b7eeeccebd007e2bad4d41f0"}
Jul 12 22:20:04.968549 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:04.968494    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bm656" event={"ID":"f737042d-b618-47e6-8815-6dbc10c4bf17","Type":"ContainerStarted","Data":"6db9975e02e3c822be36eafd67225714ec990596f2340c3e805e98bd384ef0fc"}
Jul 12 22:20:04.988958 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:04.988906    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bm656" podStartSLOduration=1.7481451730000002 podStartE2EDuration="2.988889419s" totalImagesPullingTime="1.240744246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:20:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:20:04.987974309 +0000 UTC m=+577.301477653" watchObservedRunningTime="2026-07-12 22:20:04.988889419 +0000 UTC m=+577.302392772"
Jul 12 22:20:12.700164 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:12.700129    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bm656"
Jul 12 22:20:12.700620 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:12.700198    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bm656"
Jul 12 22:20:12.713204 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:12.713178    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bm656"
Jul 12 22:20:13.018021 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:13.017936    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bm656"
Jul 12 22:20:28.161103 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:28.161077    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52pbf_1b1f17bc-a992-4c72-8fdc-a33a3c3649c4/ovn-acl-logging/0.log"
Jul 12 22:20:28.162826 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:28.162801    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52pbf_1b1f17bc-a992-4c72-8fdc-a33a3c3649c4/ovn-acl-logging/0.log"
Jul 12 22:20:44.071662 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:44.071631    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv"]
Jul 12 22:20:44.078090 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:44.078069    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv"
Jul 12 22:20:44.081001 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:44.080977    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 12 22:20:44.087752 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:44.087723    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv"]
Jul 12 22:20:44.146116 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:44.146081    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb7f33eb-f41d-45f1-8b64-7fa4acc07633-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv\" (UID: \"cb7f33eb-f41d-45f1-8b64-7fa4acc07633\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv"
Jul 12 22:20:44.146305 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:44.146159    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8k77\" (UniqueName: \"kubernetes.io/projected/cb7f33eb-f41d-45f1-8b64-7fa4acc07633-kube-api-access-n8k77\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv\" (UID: \"cb7f33eb-f41d-45f1-8b64-7fa4acc07633\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv"
Jul 12 22:20:44.146305 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:44.146202    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb7f33eb-f41d-45f1-8b64-7fa4acc07633-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv\" (UID: \"cb7f33eb-f41d-45f1-8b64-7fa4acc07633\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv"
Jul 12 22:20:44.146305 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:44.146276    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb7f33eb-f41d-45f1-8b64-7fa4acc07633-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv\" (UID: \"cb7f33eb-f41d-45f1-8b64-7fa4acc07633\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv"
Jul 12 22:20:44.146464 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:44.146309    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb7f33eb-f41d-45f1-8b64-7fa4acc07633-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv\" (UID: \"cb7f33eb-f41d-45f1-8b64-7fa4acc07633\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv"
Jul 12 22:20:44.146464 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:44.146341    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb7f33eb-f41d-45f1-8b64-7fa4acc07633-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv\" (UID: \"cb7f33eb-f41d-45f1-8b64-7fa4acc07633\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv"
Jul 12 22:20:44.247489 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:44.247462    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb7f33eb-f41d-45f1-8b64-7fa4acc07633-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv\" (UID: \"cb7f33eb-f41d-45f1-8b64-7fa4acc07633\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv"
Jul 12 22:20:44.247697 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:44.247501    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb7f33eb-f41d-45f1-8b64-7fa4acc07633-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv\" (UID: \"cb7f33eb-f41d-45f1-8b64-7fa4acc07633\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv"
Jul 12 22:20:44.247697 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:44.247521    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb7f33eb-f41d-45f1-8b64-7fa4acc07633-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv\" (UID: \"cb7f33eb-f41d-45f1-8b64-7fa4acc07633\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv"
Jul 12 22:20:44.247697 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:44.247557    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb7f33eb-f41d-45f1-8b64-7fa4acc07633-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv\" (UID: \"cb7f33eb-f41d-45f1-8b64-7fa4acc07633\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv"
Jul 12 22:20:44.247697 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:44.247625    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb7f33eb-f41d-45f1-8b64-7fa4acc07633-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv\" (UID: \"cb7f33eb-f41d-45f1-8b64-7fa4acc07633\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv"
Jul 12 22:20:44.247697 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:44.247680    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8k77\" (UniqueName: \"kubernetes.io/projected/cb7f33eb-f41d-45f1-8b64-7fa4acc07633-kube-api-access-n8k77\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv\" (UID: \"cb7f33eb-f41d-45f1-8b64-7fa4acc07633\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv"
Jul 12 22:20:44.247963 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:44.247929    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb7f33eb-f41d-45f1-8b64-7fa4acc07633-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv\" (UID: \"cb7f33eb-f41d-45f1-8b64-7fa4acc07633\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv"
Jul 12 22:20:44.248020 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:44.247973    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb7f33eb-f41d-45f1-8b64-7fa4acc07633-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv\" (UID: \"cb7f33eb-f41d-45f1-8b64-7fa4acc07633\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv"
Jul 12 22:20:44.248071 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:44.248054    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb7f33eb-f41d-45f1-8b64-7fa4acc07633-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv\" (UID: \"cb7f33eb-f41d-45f1-8b64-7fa4acc07633\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv"
Jul 12 22:20:44.250113 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:44.250090    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb7f33eb-f41d-45f1-8b64-7fa4acc07633-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv\" (UID: \"cb7f33eb-f41d-45f1-8b64-7fa4acc07633\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv"
Jul 12 22:20:44.250248 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:44.250233    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb7f33eb-f41d-45f1-8b64-7fa4acc07633-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv\" (UID: \"cb7f33eb-f41d-45f1-8b64-7fa4acc07633\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv"
Jul 12 22:20:44.255647 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:44.255628    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8k77\" (UniqueName: \"kubernetes.io/projected/cb7f33eb-f41d-45f1-8b64-7fa4acc07633-kube-api-access-n8k77\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv\" (UID: \"cb7f33eb-f41d-45f1-8b64-7fa4acc07633\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv"
Jul 12 22:20:44.389928 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:44.389833    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv"
Jul 12 22:20:44.530049 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:44.530022    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv"]
Jul 12 22:20:44.531640 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:20:44.531616    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb7f33eb_f41d_45f1_8b64_7fa4acc07633.slice/crio-119ff92ded541cb7d370b88397f68b72462440c71620534d6ab7239fda357165 WatchSource:0}: Error finding container 119ff92ded541cb7d370b88397f68b72462440c71620534d6ab7239fda357165: Status 404 returned error can't find the container with id 119ff92ded541cb7d370b88397f68b72462440c71620534d6ab7239fda357165
Jul 12 22:20:44.533071 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:44.533049    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 12 22:20:45.156079 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:45.156043    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv" event={"ID":"cb7f33eb-f41d-45f1-8b64-7fa4acc07633","Type":"ContainerStarted","Data":"85ee3a08376109e30c60e9e594ca062520f897a445160673b737919e67fa68a3"}
Jul 12 22:20:45.156454 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:45.156097    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv" event={"ID":"cb7f33eb-f41d-45f1-8b64-7fa4acc07633","Type":"ContainerStarted","Data":"119ff92ded541cb7d370b88397f68b72462440c71620534d6ab7239fda357165"}
Jul 12 22:20:45.173514 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:45.173453    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv" podStartSLOduration=0.994912923 podStartE2EDuration="1.173432737s" totalImagesPullingTime="178.519814ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:20:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:20:45.172908072 +0000 UTC m=+617.486411429" watchObservedRunningTime="2026-07-12 22:20:45.173432737 +0000 UTC m=+617.486936093"
Jul 12 22:20:54.390931 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:54.390833    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv"
Jul 12 22:20:54.390931 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:54.390902    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv"
Jul 12 22:20:54.404281 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:54.404251    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv"
Jul 12 22:20:55.218619 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:20:55.218581    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv"
Jul 12 22:22:14.987976 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:22:14.987944    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75b966d6fb-284gh"]
Jul 12 22:22:14.991339 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:22:14.991320    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75b966d6fb-284gh"
Jul 12 22:22:14.998880 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:22:14.998841    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75b966d6fb-284gh"]
Jul 12 22:22:15.110956 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:22:15.110917    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thwrm\" (UniqueName: \"kubernetes.io/projected/a4fb8815-1f53-43ad-b617-0333ef625b60-kube-api-access-thwrm\") pod \"authorino-75b966d6fb-284gh\" (UID: \"a4fb8815-1f53-43ad-b617-0333ef625b60\") " pod="kuadrant-system/authorino-75b966d6fb-284gh"
Jul 12 22:22:15.111123 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:22:15.110963    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a4fb8815-1f53-43ad-b617-0333ef625b60-tls-cert\") pod \"authorino-75b966d6fb-284gh\" (UID: \"a4fb8815-1f53-43ad-b617-0333ef625b60\") " pod="kuadrant-system/authorino-75b966d6fb-284gh"
Jul 12 22:22:15.211930 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:22:15.211871    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thwrm\" (UniqueName: \"kubernetes.io/projected/a4fb8815-1f53-43ad-b617-0333ef625b60-kube-api-access-thwrm\") pod \"authorino-75b966d6fb-284gh\" (UID: \"a4fb8815-1f53-43ad-b617-0333ef625b60\") " pod="kuadrant-system/authorino-75b966d6fb-284gh"
Jul 12 22:22:15.212095 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:22:15.211948    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a4fb8815-1f53-43ad-b617-0333ef625b60-tls-cert\") pod \"authorino-75b966d6fb-284gh\" (UID: \"a4fb8815-1f53-43ad-b617-0333ef625b60\") " pod="kuadrant-system/authorino-75b966d6fb-284gh"
Jul 12 22:22:15.214523 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:22:15.214498    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a4fb8815-1f53-43ad-b617-0333ef625b60-tls-cert\") pod \"authorino-75b966d6fb-284gh\" (UID: \"a4fb8815-1f53-43ad-b617-0333ef625b60\") " pod="kuadrant-system/authorino-75b966d6fb-284gh"
Jul 12 22:22:15.220792 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:22:15.220762    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thwrm\" (UniqueName: \"kubernetes.io/projected/a4fb8815-1f53-43ad-b617-0333ef625b60-kube-api-access-thwrm\") pod \"authorino-75b966d6fb-284gh\" (UID: \"a4fb8815-1f53-43ad-b617-0333ef625b60\") " pod="kuadrant-system/authorino-75b966d6fb-284gh"
Jul 12 22:22:15.302073 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:22:15.301986    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75b966d6fb-284gh"
Jul 12 22:22:15.444601 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:22:15.444577    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75b966d6fb-284gh"]
Jul 12 22:22:15.446770 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:22:15.446740    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4fb8815_1f53_43ad_b617_0333ef625b60.slice/crio-4a921034aebf684acfd253cff75235190a59dd5624f415669e00ac5d7558aa44 WatchSource:0}: Error finding container 4a921034aebf684acfd253cff75235190a59dd5624f415669e00ac5d7558aa44: Status 404 returned error can't find the container with id 4a921034aebf684acfd253cff75235190a59dd5624f415669e00ac5d7558aa44
Jul 12 22:22:15.580523 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:22:15.580484    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75b966d6fb-284gh" event={"ID":"a4fb8815-1f53-43ad-b617-0333ef625b60","Type":"ContainerStarted","Data":"4a921034aebf684acfd253cff75235190a59dd5624f415669e00ac5d7558aa44"}
Jul 12 22:22:16.590571 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:22:16.589811    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75b966d6fb-284gh" event={"ID":"a4fb8815-1f53-43ad-b617-0333ef625b60","Type":"ContainerStarted","Data":"acfb5dc4d60d7dbef077000812bef633db75fc3ed39abf367cf185b612622e51"}
Jul 12 22:22:16.608973 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:22:16.608909    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75b966d6fb-284gh" podStartSLOduration=2.184914916 podStartE2EDuration="2.608884053s" totalImagesPullingTime="423.969137ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:22:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:22:16.607016084 +0000 UTC m=+708.920519441" watchObservedRunningTime="2026-07-12 22:22:16.608884053 +0000 UTC m=+708.922387409"
Jul 12 22:22:16.643667 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:22:16.643558    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-88c59b6b9-tpdzr"]
Jul 12 22:22:16.643851 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:22:16.643759    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-88c59b6b9-tpdzr" podUID="91f5df03-b49d-45af-9f03-3ccc85703d3a" containerName="authorino" containerID="cri-o://0bdd9009a3091d6e8f2e70e628a95e83a37b2e7b013d783a34ffbea3bb030b8e" gracePeriod=30
Jul 12 22:22:16.922346 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:22:16.922313    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-88c59b6b9-tpdzr"
Jul 12 22:22:17.028020 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:22:17.027989    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rpnkg\" (UniqueName: \"kubernetes.io/projected/91f5df03-b49d-45af-9f03-3ccc85703d3a-kube-api-access-rpnkg\") pod \"91f5df03-b49d-45af-9f03-3ccc85703d3a\" (UID: \"91f5df03-b49d-45af-9f03-3ccc85703d3a\") "
Jul 12 22:22:17.028215 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:22:17.028108    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/91f5df03-b49d-45af-9f03-3ccc85703d3a-tls-cert\") pod \"91f5df03-b49d-45af-9f03-3ccc85703d3a\" (UID: \"91f5df03-b49d-45af-9f03-3ccc85703d3a\") "
Jul 12 22:22:17.030567 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:22:17.030504    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91f5df03-b49d-45af-9f03-3ccc85703d3a-kube-api-access-rpnkg" (OuterVolumeSpecName: "kube-api-access-rpnkg") pod "91f5df03-b49d-45af-9f03-3ccc85703d3a" (UID: "91f5df03-b49d-45af-9f03-3ccc85703d3a"). InnerVolumeSpecName "kube-api-access-rpnkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 22:22:17.040016 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:22:17.039981    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91f5df03-b49d-45af-9f03-3ccc85703d3a-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "91f5df03-b49d-45af-9f03-3ccc85703d3a" (UID: "91f5df03-b49d-45af-9f03-3ccc85703d3a"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 12 22:22:17.128903 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:22:17.128867    2560 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/91f5df03-b49d-45af-9f03-3ccc85703d3a-tls-cert\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:22:17.128903 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:22:17.128899    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpnkg\" (UniqueName: \"kubernetes.io/projected/91f5df03-b49d-45af-9f03-3ccc85703d3a-kube-api-access-rpnkg\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:22:17.597191 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:22:17.597153    2560 generic.go:356] "Generic (PLEG): container finished" podID="91f5df03-b49d-45af-9f03-3ccc85703d3a" containerID="0bdd9009a3091d6e8f2e70e628a95e83a37b2e7b013d783a34ffbea3bb030b8e" exitCode=0
Jul 12 22:22:17.597668 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:22:17.597419    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-88c59b6b9-tpdzr"
Jul 12 22:22:17.597769 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:22:17.597740    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-88c59b6b9-tpdzr" event={"ID":"91f5df03-b49d-45af-9f03-3ccc85703d3a","Type":"ContainerDied","Data":"0bdd9009a3091d6e8f2e70e628a95e83a37b2e7b013d783a34ffbea3bb030b8e"}
Jul 12 22:22:17.597897 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:22:17.597788    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-88c59b6b9-tpdzr" event={"ID":"91f5df03-b49d-45af-9f03-3ccc85703d3a","Type":"ContainerDied","Data":"742ba20f5e603be7fc3abebe40a339af36aaa9836cc9f5c420cdf9e1d8bcd470"}
Jul 12 22:22:17.597897 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:22:17.597814    2560 scope.go:117] "RemoveContainer" containerID="0bdd9009a3091d6e8f2e70e628a95e83a37b2e7b013d783a34ffbea3bb030b8e"
Jul 12 22:22:17.608901 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:22:17.608883    2560 scope.go:117] "RemoveContainer" containerID="0bdd9009a3091d6e8f2e70e628a95e83a37b2e7b013d783a34ffbea3bb030b8e"
Jul 12 22:22:17.609234 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:22:17.609215    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bdd9009a3091d6e8f2e70e628a95e83a37b2e7b013d783a34ffbea3bb030b8e\": container with ID starting with 0bdd9009a3091d6e8f2e70e628a95e83a37b2e7b013d783a34ffbea3bb030b8e not found: ID does not exist" containerID="0bdd9009a3091d6e8f2e70e628a95e83a37b2e7b013d783a34ffbea3bb030b8e"
Jul 12 22:22:17.609285 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:22:17.609242    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bdd9009a3091d6e8f2e70e628a95e83a37b2e7b013d783a34ffbea3bb030b8e"} err="failed to get container status \"0bdd9009a3091d6e8f2e70e628a95e83a37b2e7b013d783a34ffbea3bb030b8e\": rpc error: code = NotFound desc = could not find container \"0bdd9009a3091d6e8f2e70e628a95e83a37b2e7b013d783a34ffbea3bb030b8e\": container with ID starting with 0bdd9009a3091d6e8f2e70e628a95e83a37b2e7b013d783a34ffbea3bb030b8e not found: ID does not exist"
Jul 12 22:22:17.617742 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:22:17.617713    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-88c59b6b9-tpdzr"]
Jul 12 22:22:17.623940 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:22:17.623905    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-88c59b6b9-tpdzr"]
Jul 12 22:22:18.248975 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:22:18.248941    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91f5df03-b49d-45af-9f03-3ccc85703d3a" path="/var/lib/kubelet/pods/91f5df03-b49d-45af-9f03-3ccc85703d3a/volumes"
Jul 12 22:25:28.190088 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:25:28.190016    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52pbf_1b1f17bc-a992-4c72-8fdc-a33a3c3649c4/ovn-acl-logging/0.log"
Jul 12 22:25:28.190592 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:25:28.190216    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52pbf_1b1f17bc-a992-4c72-8fdc-a33a3c3649c4/ovn-acl-logging/0.log"
Jul 12 22:28:20.088312 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:28:20.088235    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55m8v"]
Jul 12 22:28:20.090892 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:28:20.088479    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55m8v" podUID="0f1d1e7b-bad9-43ff-be2f-0df8ea968968" containerName="manager" containerID="cri-o://94290a02c5d231714cba78ec7665e3fef260b620cfe9144c71bca7442b108b89" gracePeriod=10
Jul 12 22:28:22.252195 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:28:22.252162    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55m8v"
Jul 12 22:28:22.341902 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:28:22.341865    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qcsj4\" (UniqueName: \"kubernetes.io/projected/0f1d1e7b-bad9-43ff-be2f-0df8ea968968-kube-api-access-qcsj4\") pod \"0f1d1e7b-bad9-43ff-be2f-0df8ea968968\" (UID: \"0f1d1e7b-bad9-43ff-be2f-0df8ea968968\") "
Jul 12 22:28:22.342067 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:28:22.341956    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f1d1e7b-bad9-43ff-be2f-0df8ea968968-extensions-socket-volume\") pod \"0f1d1e7b-bad9-43ff-be2f-0df8ea968968\" (UID: \"0f1d1e7b-bad9-43ff-be2f-0df8ea968968\") "
Jul 12 22:28:22.342431 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:28:22.342381    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f1d1e7b-bad9-43ff-be2f-0df8ea968968-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0f1d1e7b-bad9-43ff-be2f-0df8ea968968" (UID: "0f1d1e7b-bad9-43ff-be2f-0df8ea968968"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:28:22.344152 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:28:22.344128    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f1d1e7b-bad9-43ff-be2f-0df8ea968968-kube-api-access-qcsj4" (OuterVolumeSpecName: "kube-api-access-qcsj4") pod "0f1d1e7b-bad9-43ff-be2f-0df8ea968968" (UID: "0f1d1e7b-bad9-43ff-be2f-0df8ea968968"). InnerVolumeSpecName "kube-api-access-qcsj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 22:28:22.383369 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:28:22.383336    2560 generic.go:356] "Generic (PLEG): container finished" podID="0f1d1e7b-bad9-43ff-be2f-0df8ea968968" containerID="94290a02c5d231714cba78ec7665e3fef260b620cfe9144c71bca7442b108b89" exitCode=0
Jul 12 22:28:22.383561 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:28:22.383380    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55m8v"
Jul 12 22:28:22.383561 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:28:22.383432    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55m8v" event={"ID":"0f1d1e7b-bad9-43ff-be2f-0df8ea968968","Type":"ContainerDied","Data":"94290a02c5d231714cba78ec7665e3fef260b620cfe9144c71bca7442b108b89"}
Jul 12 22:28:22.383561 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:28:22.383497    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55m8v" event={"ID":"0f1d1e7b-bad9-43ff-be2f-0df8ea968968","Type":"ContainerDied","Data":"b03d3077d20927f2a99fe05cd2c5b45d17ec64e2c45ad1c04889217bbd669a93"}
Jul 12 22:28:22.383561 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:28:22.383527    2560 scope.go:117] "RemoveContainer" containerID="94290a02c5d231714cba78ec7665e3fef260b620cfe9144c71bca7442b108b89"
Jul 12 22:28:22.395544 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:28:22.395505    2560 scope.go:117] "RemoveContainer" containerID="94290a02c5d231714cba78ec7665e3fef260b620cfe9144c71bca7442b108b89"
Jul 12 22:28:22.395840 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:28:22.395819    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94290a02c5d231714cba78ec7665e3fef260b620cfe9144c71bca7442b108b89\": container with ID starting with 94290a02c5d231714cba78ec7665e3fef260b620cfe9144c71bca7442b108b89 not found: ID does not exist" containerID="94290a02c5d231714cba78ec7665e3fef260b620cfe9144c71bca7442b108b89"
Jul 12 22:28:22.395897 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:28:22.395848    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94290a02c5d231714cba78ec7665e3fef260b620cfe9144c71bca7442b108b89"} err="failed to get container status \"94290a02c5d231714cba78ec7665e3fef260b620cfe9144c71bca7442b108b89\": rpc error: code = NotFound desc = could not find container \"94290a02c5d231714cba78ec7665e3fef260b620cfe9144c71bca7442b108b89\": container with ID starting with 94290a02c5d231714cba78ec7665e3fef260b620cfe9144c71bca7442b108b89 not found: ID does not exist"
Jul 12 22:28:22.406660 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:28:22.406492    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55m8v"]
Jul 12 22:28:22.408363 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:28:22.408346    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-55m8v"]
Jul 12 22:28:22.443176 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:28:22.443144    2560 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f1d1e7b-bad9-43ff-be2f-0df8ea968968-extensions-socket-volume\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:28:22.443176 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:28:22.443170    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcsj4\" (UniqueName: \"kubernetes.io/projected/0f1d1e7b-bad9-43ff-be2f-0df8ea968968-kube-api-access-qcsj4\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:28:24.246552 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:28:24.246506    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f1d1e7b-bad9-43ff-be2f-0df8ea968968" path="/var/lib/kubelet/pods/0f1d1e7b-bad9-43ff-be2f-0df8ea968968/volumes"
Jul 12 22:29:21.238144 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:29:21.238110    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vkvst"]
Jul 12 22:29:21.238641 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:29:21.238459    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f1d1e7b-bad9-43ff-be2f-0df8ea968968" containerName="manager"
Jul 12 22:29:21.238641 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:29:21.238470    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f1d1e7b-bad9-43ff-be2f-0df8ea968968" containerName="manager"
Jul 12 22:29:21.238641 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:29:21.238483    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91f5df03-b49d-45af-9f03-3ccc85703d3a" containerName="authorino"
Jul 12 22:29:21.238641 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:29:21.238488    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="91f5df03-b49d-45af-9f03-3ccc85703d3a" containerName="authorino"
Jul 12 22:29:21.238641 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:29:21.238559    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="91f5df03-b49d-45af-9f03-3ccc85703d3a" containerName="authorino"
Jul 12 22:29:21.238641 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:29:21.238569    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f1d1e7b-bad9-43ff-be2f-0df8ea968968" containerName="manager"
Jul 12 22:29:21.241628 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:29:21.241607    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vkvst"
Jul 12 22:29:21.245060 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:29:21.245039    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-8jsv5\""
Jul 12 22:29:21.252740 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:29:21.252716    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vkvst"]
Jul 12 22:29:21.349400 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:29:21.349357    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl5r2\" (UniqueName: \"kubernetes.io/projected/03b1cf92-7bdc-4944-a52e-f6c0f8c5bd3c-kube-api-access-sl5r2\") pod \"kuadrant-operator-controller-manager-86878c769f-vkvst\" (UID: \"03b1cf92-7bdc-4944-a52e-f6c0f8c5bd3c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vkvst"
Jul 12 22:29:21.349622 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:29:21.349474    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/03b1cf92-7bdc-4944-a52e-f6c0f8c5bd3c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vkvst\" (UID: \"03b1cf92-7bdc-4944-a52e-f6c0f8c5bd3c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vkvst"
Jul 12 22:29:21.450297 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:29:21.450254    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/03b1cf92-7bdc-4944-a52e-f6c0f8c5bd3c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vkvst\" (UID: \"03b1cf92-7bdc-4944-a52e-f6c0f8c5bd3c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vkvst"
Jul 12 22:29:21.450452 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:29:21.450328    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl5r2\" (UniqueName: \"kubernetes.io/projected/03b1cf92-7bdc-4944-a52e-f6c0f8c5bd3c-kube-api-access-sl5r2\") pod \"kuadrant-operator-controller-manager-86878c769f-vkvst\" (UID: \"03b1cf92-7bdc-4944-a52e-f6c0f8c5bd3c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vkvst"
Jul 12 22:29:21.450700 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:29:21.450679    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/03b1cf92-7bdc-4944-a52e-f6c0f8c5bd3c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vkvst\" (UID: \"03b1cf92-7bdc-4944-a52e-f6c0f8c5bd3c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vkvst"
Jul 12 22:29:21.464682 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:29:21.464654    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl5r2\" (UniqueName: \"kubernetes.io/projected/03b1cf92-7bdc-4944-a52e-f6c0f8c5bd3c-kube-api-access-sl5r2\") pod \"kuadrant-operator-controller-manager-86878c769f-vkvst\" (UID: \"03b1cf92-7bdc-4944-a52e-f6c0f8c5bd3c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vkvst"
Jul 12 22:29:21.552199 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:29:21.552117    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vkvst"
Jul 12 22:29:21.698866 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:29:21.698838    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vkvst"]
Jul 12 22:29:21.701793 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:29:21.701759    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b1cf92_7bdc_4944_a52e_f6c0f8c5bd3c.slice/crio-27238bfffd4dcc39d491c71b2ab378366a1199859c67951258874703a9406dea WatchSource:0}: Error finding container 27238bfffd4dcc39d491c71b2ab378366a1199859c67951258874703a9406dea: Status 404 returned error can't find the container with id 27238bfffd4dcc39d491c71b2ab378366a1199859c67951258874703a9406dea
Jul 12 22:29:22.682224 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:29:22.682186    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vkvst" event={"ID":"03b1cf92-7bdc-4944-a52e-f6c0f8c5bd3c","Type":"ContainerStarted","Data":"6062034d648a81e49d7eff98e9671e32ef38d52b8303062ad2b56e8a66aae5ad"}
Jul 12 22:29:22.682632 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:29:22.682236    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vkvst" event={"ID":"03b1cf92-7bdc-4944-a52e-f6c0f8c5bd3c","Type":"ContainerStarted","Data":"27238bfffd4dcc39d491c71b2ab378366a1199859c67951258874703a9406dea"}
Jul 12 22:29:22.682632 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:29:22.682313    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vkvst"
Jul 12 22:29:22.705843 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:29:22.705790    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vkvst" podStartSLOduration=1.705768794 podStartE2EDuration="1.705768794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:29:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:29:22.703456654 +0000 UTC m=+1135.016960007" watchObservedRunningTime="2026-07-12 22:29:22.705768794 +0000 UTC m=+1135.019272148"
Jul 12 22:29:33.689596 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:29:33.689555    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vkvst"
Jul 12 22:30:00.136962 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:30:00.136889    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29731590-6j5t9"]
Jul 12 22:30:00.140381 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:30:00.140365    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731590-6j5t9"
Jul 12 22:30:00.142811 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:30:00.142789    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlrdf\""
Jul 12 22:30:00.149056 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:30:00.149034    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29731590-6j5t9"]
Jul 12 22:30:00.151091 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:30:00.151074    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731590-6j5t9"
Jul 12 22:30:00.296097 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:30:00.296075    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29731590-6j5t9"]
Jul 12 22:30:00.297739 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:30:00.297713    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73e2be03_630b_4a4e_a56f_63bf767afd29.slice/crio-b99986b7c8b5f51389d874c7c243280dd788c7201df2d80953a54530b91619c6 WatchSource:0}: Error finding container b99986b7c8b5f51389d874c7c243280dd788c7201df2d80953a54530b91619c6: Status 404 returned error can't find the container with id b99986b7c8b5f51389d874c7c243280dd788c7201df2d80953a54530b91619c6
Jul 12 22:30:00.299754 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:30:00.299736    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 12 22:30:00.870276 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:30:00.870233    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731590-6j5t9" event={"ID":"73e2be03-630b-4a4e-a56f-63bf767afd29","Type":"ContainerStarted","Data":"b99986b7c8b5f51389d874c7c243280dd788c7201df2d80953a54530b91619c6"}
Jul 12 22:30:02.883895 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:30:02.883853    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731590-6j5t9" event={"ID":"73e2be03-630b-4a4e-a56f-63bf767afd29","Type":"ContainerStarted","Data":"bd03372a7376d1f3ad28b59749307ac82ae4d4043fc917946966f5fba5d3a738"}
Jul 12 22:30:02.900702 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:30:02.900650    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731590-6j5t9" podStartSLOduration=0.791324954 podStartE2EDuration="2.900633982s" totalImagesPullingTime="2.109309028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:30:02.89959015 +0000 UTC m=+1175.213093504" watchObservedRunningTime="2026-07-12 22:30:02.900633982 +0000 UTC m=+1175.214137392"
Jul 12 22:30:03.900070 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:30:03.900034    2560 generic.go:356] "Generic (PLEG): container finished" podID="73e2be03-630b-4a4e-a56f-63bf767afd29" containerID="bd03372a7376d1f3ad28b59749307ac82ae4d4043fc917946966f5fba5d3a738" exitCode=0
Jul 12 22:30:03.900479 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:30:03.900121    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731590-6j5t9" event={"ID":"73e2be03-630b-4a4e-a56f-63bf767afd29","Type":"ContainerDied","Data":"bd03372a7376d1f3ad28b59749307ac82ae4d4043fc917946966f5fba5d3a738"}
Jul 12 22:30:05.065649 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:30:05.065618    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731590-6j5t9"
Jul 12 22:30:05.914035 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:30:05.914005    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731590-6j5t9"
Jul 12 22:30:05.914035 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:30:05.914015    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731590-6j5t9" event={"ID":"73e2be03-630b-4a4e-a56f-63bf767afd29","Type":"ContainerDied","Data":"b99986b7c8b5f51389d874c7c243280dd788c7201df2d80953a54530b91619c6"}
Jul 12 22:30:05.914311 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:30:05.914066    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b99986b7c8b5f51389d874c7c243280dd788c7201df2d80953a54530b91619c6"
Jul 12 22:30:28.222658 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:30:28.222626    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52pbf_1b1f17bc-a992-4c72-8fdc-a33a3c3649c4/ovn-acl-logging/0.log"
Jul 12 22:30:28.223137 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:30:28.223116    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52pbf_1b1f17bc-a992-4c72-8fdc-a33a3c3649c4/ovn-acl-logging/0.log"
Jul 12 22:35:28.254897 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:35:28.254864    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52pbf_1b1f17bc-a992-4c72-8fdc-a33a3c3649c4/ovn-acl-logging/0.log"
Jul 12 22:35:28.257334 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:35:28.255990    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52pbf_1b1f17bc-a992-4c72-8fdc-a33a3c3649c4/ovn-acl-logging/0.log"
Jul 12 22:36:01.746156 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:01.746115    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbe7f53b-swp24"]
Jul 12 22:36:01.746761 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:01.746481    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73e2be03-630b-4a4e-a56f-63bf767afd29" containerName="cleanup"
Jul 12 22:36:01.746761 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:01.746493    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="73e2be03-630b-4a4e-a56f-63bf767afd29" containerName="cleanup"
Jul 12 22:36:01.746761 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:01.746587    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="73e2be03-630b-4a4e-a56f-63bf767afd29" containerName="cleanup"
Jul 12 22:36:01.750006 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:01.749987    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbe7f53b-swp24"
Jul 12 22:36:01.752750 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:01.752728    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 12 22:36:01.752856 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:01.752727    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlrdf\""
Jul 12 22:36:01.756837 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:01.756811    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbe7f53b-swp24"]
Jul 12 22:36:01.867835 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:01.867797    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87445187-9e4e-4548-8676-005e2be4eeef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-cbe7f53b-swp24\" (UID: \"87445187-9e4e-4548-8676-005e2be4eeef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbe7f53b-swp24"
Jul 12 22:36:01.969146 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:01.969112    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87445187-9e4e-4548-8676-005e2be4eeef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-cbe7f53b-swp24\" (UID: \"87445187-9e4e-4548-8676-005e2be4eeef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbe7f53b-swp24"
Jul 12 22:36:01.969782 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:01.969761    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87445187-9e4e-4548-8676-005e2be4eeef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-cbe7f53b-swp24\" (UID: \"87445187-9e4e-4548-8676-005e2be4eeef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbe7f53b-swp24"
Jul 12 22:36:02.061652 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:02.061560    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbe7f53b-swp24"
Jul 12 22:36:02.201519 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:02.201492    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbe7f53b-swp24"]
Jul 12 22:36:02.203778 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:36:02.203746    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87445187_9e4e_4548_8676_005e2be4eeef.slice/crio-628d94b6a06b3b3417a82ea7eec33dd438df7c380a5c8de67f6b55fa17df258c WatchSource:0}: Error finding container 628d94b6a06b3b3417a82ea7eec33dd438df7c380a5c8de67f6b55fa17df258c: Status 404 returned error can't find the container with id 628d94b6a06b3b3417a82ea7eec33dd438df7c380a5c8de67f6b55fa17df258c
Jul 12 22:36:02.706389 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:02.706299    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbe7f53b-swp24" event={"ID":"87445187-9e4e-4548-8676-005e2be4eeef","Type":"ContainerStarted","Data":"2625ea5c0d8c5e0471522ce91d1bf31e7f3a25521824025e4dd1cb676eab8fa1"}
Jul 12 22:36:02.706583 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:02.706388    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbe7f53b-swp24" event={"ID":"87445187-9e4e-4548-8676-005e2be4eeef","Type":"ContainerStarted","Data":"628d94b6a06b3b3417a82ea7eec33dd438df7c380a5c8de67f6b55fa17df258c"}
Jul 12 22:36:02.721759 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:02.721712    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbe7f53b-swp24" podStartSLOduration=1.721694091 podStartE2EDuration="1.721694091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:36:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:36:02.720567564 +0000 UTC m=+1535.034070920" watchObservedRunningTime="2026-07-12 22:36:02.721694091 +0000 UTC m=+1535.035197444"
Jul 12 22:36:03.712423 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:03.712386    2560 generic.go:356] "Generic (PLEG): container finished" podID="87445187-9e4e-4548-8676-005e2be4eeef" containerID="2625ea5c0d8c5e0471522ce91d1bf31e7f3a25521824025e4dd1cb676eab8fa1" exitCode=7
Jul 12 22:36:03.712852 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:03.712478    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbe7f53b-swp24" event={"ID":"87445187-9e4e-4548-8676-005e2be4eeef","Type":"ContainerDied","Data":"2625ea5c0d8c5e0471522ce91d1bf31e7f3a25521824025e4dd1cb676eab8fa1"}
Jul 12 22:36:03.712852 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:03.712687    2560 scope.go:117] "RemoveContainer" containerID="2625ea5c0d8c5e0471522ce91d1bf31e7f3a25521824025e4dd1cb676eab8fa1"
Jul 12 22:36:04.719309 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:04.719222    2560 generic.go:356] "Generic (PLEG): container finished" podID="87445187-9e4e-4548-8676-005e2be4eeef" containerID="9e07854302b0e15dc32d414dacdbaa122accf3417915c7eddf5a3bbe940453e7" exitCode=7
Jul 12 22:36:04.719309 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:04.719298    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbe7f53b-swp24" event={"ID":"87445187-9e4e-4548-8676-005e2be4eeef","Type":"ContainerDied","Data":"9e07854302b0e15dc32d414dacdbaa122accf3417915c7eddf5a3bbe940453e7"}
Jul 12 22:36:04.719958 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:04.719350    2560 scope.go:117] "RemoveContainer" containerID="2625ea5c0d8c5e0471522ce91d1bf31e7f3a25521824025e4dd1cb676eab8fa1"
Jul 12 22:36:04.719958 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:04.719605    2560 scope.go:117] "RemoveContainer" containerID="9e07854302b0e15dc32d414dacdbaa122accf3417915c7eddf5a3bbe940453e7"
Jul 12 22:36:04.719958 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:36:04.719848    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-cbe7f53b-swp24_odh-ai-gateway-infra(87445187-9e4e-4548-8676-005e2be4eeef)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbe7f53b-swp24" podUID="87445187-9e4e-4548-8676-005e2be4eeef"
Jul 12 22:36:05.727208 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:05.727178    2560 scope.go:117] "RemoveContainer" containerID="9e07854302b0e15dc32d414dacdbaa122accf3417915c7eddf5a3bbe940453e7"
Jul 12 22:36:05.727650 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:36:05.727391    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-cbe7f53b-swp24_odh-ai-gateway-infra(87445187-9e4e-4548-8676-005e2be4eeef)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbe7f53b-swp24" podUID="87445187-9e4e-4548-8676-005e2be4eeef"
Jul 12 22:36:17.240812 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:17.240772    2560 scope.go:117] "RemoveContainer" containerID="9e07854302b0e15dc32d414dacdbaa122accf3417915c7eddf5a3bbe940453e7"
Jul 12 22:36:17.796286 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:17.796250    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbe7f53b-swp24" event={"ID":"87445187-9e4e-4548-8676-005e2be4eeef","Type":"ContainerStarted","Data":"4c72de6686e1888225c102cf2eef3b13c11c5adb12868dc53491b7351bae6789"}
Jul 12 22:36:18.802379 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:18.802340    2560 generic.go:356] "Generic (PLEG): container finished" podID="87445187-9e4e-4548-8676-005e2be4eeef" containerID="4c72de6686e1888225c102cf2eef3b13c11c5adb12868dc53491b7351bae6789" exitCode=0
Jul 12 22:36:18.802846 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:18.802406    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbe7f53b-swp24" event={"ID":"87445187-9e4e-4548-8676-005e2be4eeef","Type":"ContainerDied","Data":"4c72de6686e1888225c102cf2eef3b13c11c5adb12868dc53491b7351bae6789"}
Jul 12 22:36:18.802846 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:18.802471    2560 scope.go:117] "RemoveContainer" containerID="9e07854302b0e15dc32d414dacdbaa122accf3417915c7eddf5a3bbe940453e7"
Jul 12 22:36:18.824805 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:18.824764    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbe7f53b-swp24"]
Jul 12 22:36:19.948787 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:19.948762    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbe7f53b-swp24"
Jul 12 22:36:20.026730 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:20.026697    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87445187-9e4e-4548-8676-005e2be4eeef-maas-api-service-ca\") pod \"87445187-9e4e-4548-8676-005e2be4eeef\" (UID: \"87445187-9e4e-4548-8676-005e2be4eeef\") "
Jul 12 22:36:20.027057 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:20.027035    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87445187-9e4e-4548-8676-005e2be4eeef-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "87445187-9e4e-4548-8676-005e2be4eeef" (UID: "87445187-9e4e-4548-8676-005e2be4eeef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 12 22:36:20.127885 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:20.127852    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87445187-9e4e-4548-8676-005e2be4eeef-maas-api-service-ca\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:36:20.815068 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:20.815035    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbe7f53b-swp24" event={"ID":"87445187-9e4e-4548-8676-005e2be4eeef","Type":"ContainerDied","Data":"628d94b6a06b3b3417a82ea7eec33dd438df7c380a5c8de67f6b55fa17df258c"}
Jul 12 22:36:20.815240 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:20.815074    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbe7f53b-swp24"
Jul 12 22:36:20.815240 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:20.815083    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="628d94b6a06b3b3417a82ea7eec33dd438df7c380a5c8de67f6b55fa17df258c"
Jul 12 22:36:20.847986 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:20.847952    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbe7f53b-swp24"]
Jul 12 22:36:20.850221 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:20.850199    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbe7f53b-swp24"]
Jul 12 22:36:22.246294 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:22.246255    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87445187-9e4e-4548-8676-005e2be4eeef" path="/var/lib/kubelet/pods/87445187-9e4e-4548-8676-005e2be4eeef/volumes"
Jul 12 22:36:51.788689 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:51.788637    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbe7f53b-nn7b7"]
Jul 12 22:36:51.789168 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:51.789051    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87445187-9e4e-4548-8676-005e2be4eeef" containerName="revoke-keys"
Jul 12 22:36:51.789168 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:51.789064    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="87445187-9e4e-4548-8676-005e2be4eeef" containerName="revoke-keys"
Jul 12 22:36:51.789168 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:51.789086    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87445187-9e4e-4548-8676-005e2be4eeef" containerName="revoke-keys"
Jul 12 22:36:51.789168 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:51.789092    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="87445187-9e4e-4548-8676-005e2be4eeef" containerName="revoke-keys"
Jul 12 22:36:51.789168 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:51.789148    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="87445187-9e4e-4548-8676-005e2be4eeef" containerName="revoke-keys"
Jul 12 22:36:51.789168 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:51.789158    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="87445187-9e4e-4548-8676-005e2be4eeef" containerName="revoke-keys"
Jul 12 22:36:51.792336 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:51.792319    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbe7f53b-nn7b7"
Jul 12 22:36:51.795000 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:51.794975    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 12 22:36:51.795000 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:51.794993    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlrdf\""
Jul 12 22:36:51.800089 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:51.799855    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbe7f53b-nn7b7"]
Jul 12 22:36:51.915681 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:51.915637    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b161689-298b-4034-a6e8-3c4a04a1046f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-cbe7f53b-nn7b7\" (UID: \"7b161689-298b-4034-a6e8-3c4a04a1046f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbe7f53b-nn7b7"
Jul 12 22:36:52.017098 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:52.017061    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b161689-298b-4034-a6e8-3c4a04a1046f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-cbe7f53b-nn7b7\" (UID: \"7b161689-298b-4034-a6e8-3c4a04a1046f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbe7f53b-nn7b7"
Jul 12 22:36:52.017758 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:52.017741    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b161689-298b-4034-a6e8-3c4a04a1046f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-cbe7f53b-nn7b7\" (UID: \"7b161689-298b-4034-a6e8-3c4a04a1046f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbe7f53b-nn7b7"
Jul 12 22:36:52.103484 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:52.103451    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbe7f53b-nn7b7"
Jul 12 22:36:52.243512 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:36:52.243484    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b161689_298b_4034_a6e8_3c4a04a1046f.slice/crio-4d915ffc491329886599dce8b2a6c3ca41a6fc8a3c3c01a2f97c07cd67a8faba WatchSource:0}: Error finding container 4d915ffc491329886599dce8b2a6c3ca41a6fc8a3c3c01a2f97c07cd67a8faba: Status 404 returned error can't find the container with id 4d915ffc491329886599dce8b2a6c3ca41a6fc8a3c3c01a2f97c07cd67a8faba
Jul 12 22:36:52.248105 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:52.248083    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbe7f53b-nn7b7"]
Jul 12 22:36:52.974185 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:52.974144    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbe7f53b-nn7b7" event={"ID":"7b161689-298b-4034-a6e8-3c4a04a1046f","Type":"ContainerStarted","Data":"5944a1b168c6a7653eb4c712caa58302db2ee2125e6234ca01199ec8c823d15a"}
Jul 12 22:36:52.974600 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:52.974203    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbe7f53b-nn7b7" event={"ID":"7b161689-298b-4034-a6e8-3c4a04a1046f","Type":"ContainerStarted","Data":"4d915ffc491329886599dce8b2a6c3ca41a6fc8a3c3c01a2f97c07cd67a8faba"}
Jul 12 22:36:52.991078 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:52.991026    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbe7f53b-nn7b7" podStartSLOduration=1.991009639 podStartE2EDuration="1.991009639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:36:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:36:52.990554496 +0000 UTC m=+1585.304057849" watchObservedRunningTime="2026-07-12 22:36:52.991009639 +0000 UTC m=+1585.304512992"
Jul 12 22:36:53.981335 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:53.981298    2560 generic.go:356] "Generic (PLEG): container finished" podID="7b161689-298b-4034-a6e8-3c4a04a1046f" containerID="5944a1b168c6a7653eb4c712caa58302db2ee2125e6234ca01199ec8c823d15a" exitCode=0
Jul 12 22:36:53.981776 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:53.981385    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbe7f53b-nn7b7" event={"ID":"7b161689-298b-4034-a6e8-3c4a04a1046f","Type":"ContainerDied","Data":"5944a1b168c6a7653eb4c712caa58302db2ee2125e6234ca01199ec8c823d15a"}
Jul 12 22:36:55.125511 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:55.125477    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbe7f53b-nn7b7"
Jul 12 22:36:55.246630 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:55.246525    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b161689-298b-4034-a6e8-3c4a04a1046f-maas-api-service-ca\") pod \"7b161689-298b-4034-a6e8-3c4a04a1046f\" (UID: \"7b161689-298b-4034-a6e8-3c4a04a1046f\") "
Jul 12 22:36:55.246853 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:55.246829    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b161689-298b-4034-a6e8-3c4a04a1046f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7b161689-298b-4034-a6e8-3c4a04a1046f" (UID: "7b161689-298b-4034-a6e8-3c4a04a1046f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 12 22:36:55.246916 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:55.246904    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b161689-298b-4034-a6e8-3c4a04a1046f-maas-api-service-ca\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:36:55.992415 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:55.992389    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbe7f53b-nn7b7"
Jul 12 22:36:55.992611 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:55.992386    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cbe7f53b-nn7b7" event={"ID":"7b161689-298b-4034-a6e8-3c4a04a1046f","Type":"ContainerDied","Data":"4d915ffc491329886599dce8b2a6c3ca41a6fc8a3c3c01a2f97c07cd67a8faba"}
Jul 12 22:36:55.992611 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:36:55.992505    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d915ffc491329886599dce8b2a6c3ca41a6fc8a3c3c01a2f97c07cd67a8faba"
Jul 12 22:37:17.620373 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:17.620330    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"]
Jul 12 22:37:17.620810 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:17.620786    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b161689-298b-4034-a6e8-3c4a04a1046f" containerName="revoke-keys"
Jul 12 22:37:17.620810 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:17.620810    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b161689-298b-4034-a6e8-3c4a04a1046f" containerName="revoke-keys"
Jul 12 22:37:17.621041 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:17.620849    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87445187-9e4e-4548-8676-005e2be4eeef" containerName="revoke-keys"
Jul 12 22:37:17.621041 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:17.620857    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="87445187-9e4e-4548-8676-005e2be4eeef" containerName="revoke-keys"
Jul 12 22:37:17.621041 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:17.620968    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b161689-298b-4034-a6e8-3c4a04a1046f" containerName="revoke-keys"
Jul 12 22:37:17.621041 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:17.620982    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="87445187-9e4e-4548-8676-005e2be4eeef" containerName="revoke-keys"
Jul 12 22:37:17.627832 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:17.627808    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"
Jul 12 22:37:17.630592 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:17.630567    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 12 22:37:17.630735 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:17.630567    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-e6750e7c-gw-openshift-default-dockercfg-68frx\""
Jul 12 22:37:17.637381 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:17.636746    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"]
Jul 12 22:37:17.744961 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:17.744931    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-credential-socket\") pod \"e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s\" (UID: \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\") " pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"
Jul 12 22:37:17.745122 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:17.744975    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-istio-token\") pod \"e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s\" (UID: \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\") " pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"
Jul 12 22:37:17.745122 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:17.744998    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-istio-envoy\") pod \"e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s\" (UID: \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\") " pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"
Jul 12 22:37:17.745122 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:17.745086    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-istiod-ca-cert\") pod \"e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s\" (UID: \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\") " pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"
Jul 12 22:37:17.745222 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:17.745135    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-istio-data\") pod \"e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s\" (UID: \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\") " pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"
Jul 12 22:37:17.745222 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:17.745163    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-istio-podinfo\") pod \"e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s\" (UID: \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\") " pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"
Jul 12 22:37:17.745222 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:17.745184    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-workload-certs\") pod \"e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s\" (UID: \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\") " pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"
Jul 12 22:37:17.745222 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:17.745216    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc9m8\" (UniqueName: \"kubernetes.io/projected/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-kube-api-access-vc9m8\") pod \"e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s\" (UID: \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\") " pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"
Jul 12 22:37:17.745364 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:17.745275    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-workload-socket\") pod \"e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s\" (UID: \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\") " pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"
Jul 12 22:37:17.846166 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:17.846128    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-credential-socket\") pod \"e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s\" (UID: \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\") " pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"
Jul 12 22:37:17.846323 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:17.846182    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-istio-token\") pod \"e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s\" (UID: \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\") " pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"
Jul 12 22:37:17.846323 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:17.846304    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-istio-envoy\") pod \"e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s\" (UID: \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\") " pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"
Jul 12 22:37:17.846455 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:17.846355    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-istiod-ca-cert\") pod \"e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s\" (UID: \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\") " pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"
Jul 12 22:37:17.846455 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:17.846408    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-istio-data\") pod \"e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s\" (UID: \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\") " pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"
Jul 12 22:37:17.846455 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:17.846436    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-istio-podinfo\") pod \"e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s\" (UID: \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\") " pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"
Jul 12 22:37:17.846638 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:17.846464    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-workload-certs\") pod \"e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s\" (UID: \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\") " pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"
Jul 12 22:37:17.846638 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:17.846510    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc9m8\" (UniqueName: \"kubernetes.io/projected/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-kube-api-access-vc9m8\") pod \"e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s\" (UID: \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\") " pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"
Jul 12 22:37:17.846638 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:17.846567    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-credential-socket\") pod \"e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s\" (UID: \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\") " pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"
Jul 12 22:37:17.846638 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:17.846590    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-workload-socket\") pod \"e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s\" (UID: \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\") " pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"
Jul 12 22:37:17.846836 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:17.846815    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-workload-certs\") pod \"e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s\" (UID: \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\") " pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"
Jul 12 22:37:17.846886 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:17.846863    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-workload-socket\") pod \"e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s\" (UID: \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\") " pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"
Jul 12 22:37:17.847043 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:17.847021    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-istio-data\") pod \"e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s\" (UID: \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\") " pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"
Jul 12 22:37:17.847115 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:17.847105    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-istiod-ca-cert\") pod \"e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s\" (UID: \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\") " pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"
Jul 12 22:37:17.848736 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:17.848717    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-istio-envoy\") pod \"e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s\" (UID: \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\") " pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"
Jul 12 22:37:17.848950 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:17.848934    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-istio-podinfo\") pod \"e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s\" (UID: \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\") " pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"
Jul 12 22:37:17.854356 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:17.854335    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-istio-token\") pod \"e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s\" (UID: \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\") " pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"
Jul 12 22:37:17.854729 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:17.854710    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc9m8\" (UniqueName: \"kubernetes.io/projected/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-kube-api-access-vc9m8\") pod \"e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s\" (UID: \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\") " pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"
Jul 12 22:37:17.941599 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:17.941478    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"
Jul 12 22:37:18.090230 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:18.090195    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"]
Jul 12 22:37:18.093492 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:37:18.093463    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab4fb1b_9591_4f48_ab50_0018a5e9dd79.slice/crio-e6b2ace0442f9be99766ef01ee4e4cda63eb809b08f034993e46799dbd1047da WatchSource:0}: Error finding container e6b2ace0442f9be99766ef01ee4e4cda63eb809b08f034993e46799dbd1047da: Status 404 returned error can't find the container with id e6b2ace0442f9be99766ef01ee4e4cda63eb809b08f034993e46799dbd1047da
Jul 12 22:37:18.095626 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:18.095608    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 12 22:37:18.102953 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:18.102925    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s" event={"ID":"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79","Type":"ContainerStarted","Data":"e6b2ace0442f9be99766ef01ee4e4cda63eb809b08f034993e46799dbd1047da"}
Jul 12 22:37:20.643236 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:20.643200    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:37:20.643558 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:20.643268    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:37:20.643558 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:20.643298    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:37:21.122314 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:21.122283    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s" event={"ID":"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79","Type":"ContainerStarted","Data":"abcbb4d5351632827c2c1c5203da2ddaea2330a1e4edf9d9b0f94e63c3a9a2c5"}
Jul 12 22:37:21.142298 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:21.142248    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s" podStartSLOduration=1.595055624 podStartE2EDuration="4.142229119s" totalImagesPullingTime="2.547173495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:37:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:37:21.14051691 +0000 UTC m=+1613.454020261" watchObservedRunningTime="2026-07-12 22:37:21.142229119 +0000 UTC m=+1613.455732472"
Jul 12 22:37:21.941957 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:21.941922    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"
Jul 12 22:37:21.947190 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:21.947162    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"
Jul 12 22:37:22.127686 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:22.127652    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"
Jul 12 22:37:22.128875 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:22.128854    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"
Jul 12 22:37:23.418592 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:23.418560    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6750e7c-2ktwf"]
Jul 12 22:37:23.422611 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:23.422586    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6750e7c-2ktwf"
Jul 12 22:37:23.425021 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:23.424994    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlrdf\""
Jul 12 22:37:23.425422 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:23.425402    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 12 22:37:23.429146 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:23.429118    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6750e7c-2ktwf"]
Jul 12 22:37:23.505328 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:23.505285    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78282258-7278-4906-9a4a-96581c6c5241-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e6750e7c-2ktwf\" (UID: \"78282258-7278-4906-9a4a-96581c6c5241\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6750e7c-2ktwf"
Jul 12 22:37:23.606570 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:23.606510    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78282258-7278-4906-9a4a-96581c6c5241-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e6750e7c-2ktwf\" (UID: \"78282258-7278-4906-9a4a-96581c6c5241\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6750e7c-2ktwf"
Jul 12 22:37:23.607124 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:23.607106    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78282258-7278-4906-9a4a-96581c6c5241-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e6750e7c-2ktwf\" (UID: \"78282258-7278-4906-9a4a-96581c6c5241\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6750e7c-2ktwf"
Jul 12 22:37:23.738616 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:23.738499    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6750e7c-2ktwf"
Jul 12 22:37:23.901722 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:23.901695    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6750e7c-2ktwf"]
Jul 12 22:37:23.903755 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:37:23.903728    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78282258_7278_4906_9a4a_96581c6c5241.slice/crio-8fa6423ed550dd27ec668ed1e4b16924f84d22f7a9c822fa90a5fe644896824c WatchSource:0}: Error finding container 8fa6423ed550dd27ec668ed1e4b16924f84d22f7a9c822fa90a5fe644896824c: Status 404 returned error can't find the container with id 8fa6423ed550dd27ec668ed1e4b16924f84d22f7a9c822fa90a5fe644896824c
Jul 12 22:37:24.139688 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:24.139659    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6750e7c-2ktwf" event={"ID":"78282258-7278-4906-9a4a-96581c6c5241","Type":"ContainerStarted","Data":"8fa6423ed550dd27ec668ed1e4b16924f84d22f7a9c822fa90a5fe644896824c"}
Jul 12 22:37:25.146292 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:25.146258    2560 generic.go:356] "Generic (PLEG): container finished" podID="78282258-7278-4906-9a4a-96581c6c5241" containerID="060243ad8540cb8194bba93c800c3d34dee31f62766c706d54a40e66b45f49ae" exitCode=7
Jul 12 22:37:25.146693 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:25.146353    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6750e7c-2ktwf" event={"ID":"78282258-7278-4906-9a4a-96581c6c5241","Type":"ContainerDied","Data":"060243ad8540cb8194bba93c800c3d34dee31f62766c706d54a40e66b45f49ae"}
Jul 12 22:37:25.146693 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:25.146505    2560 scope.go:117] "RemoveContainer" containerID="060243ad8540cb8194bba93c800c3d34dee31f62766c706d54a40e66b45f49ae"
Jul 12 22:37:26.153384 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:26.153296    2560 generic.go:356] "Generic (PLEG): container finished" podID="78282258-7278-4906-9a4a-96581c6c5241" containerID="1b81661c08386507f38df31ab9a5c4cf7d550d7ce981fea1f85087405296ccd4" exitCode=7
Jul 12 22:37:26.153384 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:26.153349    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6750e7c-2ktwf" event={"ID":"78282258-7278-4906-9a4a-96581c6c5241","Type":"ContainerDied","Data":"1b81661c08386507f38df31ab9a5c4cf7d550d7ce981fea1f85087405296ccd4"}
Jul 12 22:37:26.153891 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:26.153399    2560 scope.go:117] "RemoveContainer" containerID="060243ad8540cb8194bba93c800c3d34dee31f62766c706d54a40e66b45f49ae"
Jul 12 22:37:26.153891 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:26.153604    2560 scope.go:117] "RemoveContainer" containerID="1b81661c08386507f38df31ab9a5c4cf7d550d7ce981fea1f85087405296ccd4"
Jul 12 22:37:26.153891 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:37:26.153826    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-e6750e7c-2ktwf_odh-ai-gateway-infra(78282258-7278-4906-9a4a-96581c6c5241)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6750e7c-2ktwf" podUID="78282258-7278-4906-9a4a-96581c6c5241"
Jul 12 22:37:27.161015 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:27.160987    2560 scope.go:117] "RemoveContainer" containerID="1b81661c08386507f38df31ab9a5c4cf7d550d7ce981fea1f85087405296ccd4"
Jul 12 22:37:27.161424 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:37:27.161184    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-e6750e7c-2ktwf_odh-ai-gateway-infra(78282258-7278-4906-9a4a-96581c6c5241)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6750e7c-2ktwf" podUID="78282258-7278-4906-9a4a-96581c6c5241"
Jul 12 22:37:35.210604 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:35.210503    2560 generic.go:356] "Generic (PLEG): container finished" podID="0ab4fb1b-9591-4f48-ab50-0018a5e9dd79" containerID="abcbb4d5351632827c2c1c5203da2ddaea2330a1e4edf9d9b0f94e63c3a9a2c5" exitCode=0
Jul 12 22:37:35.210604 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:35.210583    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s" event={"ID":"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79","Type":"ContainerDied","Data":"abcbb4d5351632827c2c1c5203da2ddaea2330a1e4edf9d9b0f94e63c3a9a2c5"}
Jul 12 22:37:35.211169 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:35.211005    2560 scope.go:117] "RemoveContainer" containerID="abcbb4d5351632827c2c1c5203da2ddaea2330a1e4edf9d9b0f94e63c3a9a2c5"
Jul 12 22:37:35.212085 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:35.212054    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:37:35.212198 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:35.212139    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:37:35.212198 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:35.212190    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:37:36.217152 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:36.217117    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s" event={"ID":"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79","Type":"ContainerStarted","Data":"a110e0e0df9351a420e652b0da63d59768c07f9e435cda6f075eab997659ebd6"}
Jul 12 22:37:36.943002 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:36.942971    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"
Jul 12 22:37:36.943342 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:36.943317    2560 patch_prober.go:28] interesting pod/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Jul 12 22:37:36.943503 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:36.943479    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s" podUID="0ab4fb1b-9591-4f48-ab50-0018a5e9dd79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Jul 12 22:37:37.128074 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:37.128027    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"
Jul 12 22:37:37.943000 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:37.942953    2560 patch_prober.go:28] interesting pod/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Jul 12 22:37:37.943497 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:37.943041    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s" podUID="0ab4fb1b-9591-4f48-ab50-0018a5e9dd79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Jul 12 22:37:38.942602 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:38.942558    2560 patch_prober.go:28] interesting pod/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Jul 12 22:37:38.942924 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:38.942893    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s" podUID="0ab4fb1b-9591-4f48-ab50-0018a5e9dd79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Jul 12 22:37:39.943301 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:39.942837    2560 patch_prober.go:28] interesting pod/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Jul 12 22:37:39.943301 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:39.942904    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s" podUID="0ab4fb1b-9591-4f48-ab50-0018a5e9dd79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Jul 12 22:37:40.946251 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:40.946220    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"
Jul 12 22:37:40.947271 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:40.947249    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"
Jul 12 22:37:42.240019 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:42.239976    2560 scope.go:117] "RemoveContainer" containerID="1b81661c08386507f38df31ab9a5c4cf7d550d7ce981fea1f85087405296ccd4"
Jul 12 22:37:43.260008 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:43.259910    2560 generic.go:356] "Generic (PLEG): container finished" podID="78282258-7278-4906-9a4a-96581c6c5241" containerID="14eb107a04c782d8bf517d434707e222c5c9ba07f98dd4be80b84ba6093145a8" exitCode=0
Jul 12 22:37:43.260008 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:43.259993    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6750e7c-2ktwf" event={"ID":"78282258-7278-4906-9a4a-96581c6c5241","Type":"ContainerDied","Data":"14eb107a04c782d8bf517d434707e222c5c9ba07f98dd4be80b84ba6093145a8"}
Jul 12 22:37:43.260423 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:43.260079    2560 scope.go:117] "RemoveContainer" containerID="1b81661c08386507f38df31ab9a5c4cf7d550d7ce981fea1f85087405296ccd4"
Jul 12 22:37:44.286684 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:44.286653    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6750e7c-2ktwf"]
Jul 12 22:37:44.405518 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:44.405494    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6750e7c-2ktwf"
Jul 12 22:37:44.498875 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:44.498846    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78282258-7278-4906-9a4a-96581c6c5241-maas-api-service-ca\") pod \"78282258-7278-4906-9a4a-96581c6c5241\" (UID: \"78282258-7278-4906-9a4a-96581c6c5241\") "
Jul 12 22:37:44.499222 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:44.499199    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78282258-7278-4906-9a4a-96581c6c5241-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "78282258-7278-4906-9a4a-96581c6c5241" (UID: "78282258-7278-4906-9a4a-96581c6c5241"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 12 22:37:44.600381 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:44.600344    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78282258-7278-4906-9a4a-96581c6c5241-maas-api-service-ca\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:37:45.271550 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:45.271502    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6750e7c-2ktwf" event={"ID":"78282258-7278-4906-9a4a-96581c6c5241","Type":"ContainerDied","Data":"8fa6423ed550dd27ec668ed1e4b16924f84d22f7a9c822fa90a5fe644896824c"}
Jul 12 22:37:45.271550 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:45.271547    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6750e7c-2ktwf"
Jul 12 22:37:45.271813 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:45.271576    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fa6423ed550dd27ec668ed1e4b16924f84d22f7a9c822fa90a5fe644896824c"
Jul 12 22:37:45.299117 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:45.299088    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6750e7c-2ktwf"]
Jul 12 22:37:45.304591 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:45.304565    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6750e7c-2ktwf"]
Jul 12 22:37:46.245379 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:37:46.245342    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78282258-7278-4906-9a4a-96581c6c5241" path="/var/lib/kubelet/pods/78282258-7278-4906-9a4a-96581c6c5241/volumes"
Jul 12 22:38:23.468210 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:23.468174    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6750e7c-5ccnn"]
Jul 12 22:38:23.468686 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:23.468542    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78282258-7278-4906-9a4a-96581c6c5241" containerName="revoke-keys"
Jul 12 22:38:23.468686 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:23.468554    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="78282258-7278-4906-9a4a-96581c6c5241" containerName="revoke-keys"
Jul 12 22:38:23.468686 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:23.468568    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78282258-7278-4906-9a4a-96581c6c5241" containerName="revoke-keys"
Jul 12 22:38:23.468686 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:23.468574    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="78282258-7278-4906-9a4a-96581c6c5241" containerName="revoke-keys"
Jul 12 22:38:23.468686 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:23.468658    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="78282258-7278-4906-9a4a-96581c6c5241" containerName="revoke-keys"
Jul 12 22:38:23.468686 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:23.468668    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="78282258-7278-4906-9a4a-96581c6c5241" containerName="revoke-keys"
Jul 12 22:38:23.468686 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:23.468675    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="78282258-7278-4906-9a4a-96581c6c5241" containerName="revoke-keys"
Jul 12 22:38:23.471815 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:23.471795    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6750e7c-5ccnn"
Jul 12 22:38:23.474413 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:23.474383    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 12 22:38:23.474524 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:23.474391    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlrdf\""
Jul 12 22:38:23.478880 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:23.478855    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6750e7c-5ccnn"]
Jul 12 22:38:23.536637 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:23.536599    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6653b93-7999-41af-9812-a21428c64eb3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e6750e7c-5ccnn\" (UID: \"c6653b93-7999-41af-9812-a21428c64eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6750e7c-5ccnn"
Jul 12 22:38:23.538959 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:23.538937    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"]
Jul 12 22:38:23.539155 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:23.539137    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s" podUID="0ab4fb1b-9591-4f48-ab50-0018a5e9dd79" containerName="istio-proxy" containerID="cri-o://a110e0e0df9351a420e652b0da63d59768c07f9e435cda6f075eab997659ebd6" gracePeriod=30
Jul 12 22:38:23.638134 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:23.638094    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6653b93-7999-41af-9812-a21428c64eb3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e6750e7c-5ccnn\" (UID: \"c6653b93-7999-41af-9812-a21428c64eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6750e7c-5ccnn"
Jul 12 22:38:23.638771 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:23.638752    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6653b93-7999-41af-9812-a21428c64eb3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e6750e7c-5ccnn\" (UID: \"c6653b93-7999-41af-9812-a21428c64eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6750e7c-5ccnn"
Jul 12 22:38:23.783144 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:23.783054    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6750e7c-5ccnn"
Jul 12 22:38:23.922172 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:23.922137    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6750e7c-5ccnn"]
Jul 12 22:38:23.923828 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:38:23.923796    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6653b93_7999_41af_9812_a21428c64eb3.slice/crio-005c20cff088e8a2d0c70b47cd2e5eaccfd2d81f41ff1d6fd3d778f5be81dd62 WatchSource:0}: Error finding container 005c20cff088e8a2d0c70b47cd2e5eaccfd2d81f41ff1d6fd3d778f5be81dd62: Status 404 returned error can't find the container with id 005c20cff088e8a2d0c70b47cd2e5eaccfd2d81f41ff1d6fd3d778f5be81dd62
Jul 12 22:38:24.468602 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:24.468493    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6750e7c-5ccnn" event={"ID":"c6653b93-7999-41af-9812-a21428c64eb3","Type":"ContainerStarted","Data":"0cd8e31c25d33d89426918ea0ca3a7ee6141f83b3fe7971911faeabc544db600"}
Jul 12 22:38:24.468602 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:24.468580    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6750e7c-5ccnn" event={"ID":"c6653b93-7999-41af-9812-a21428c64eb3","Type":"ContainerStarted","Data":"005c20cff088e8a2d0c70b47cd2e5eaccfd2d81f41ff1d6fd3d778f5be81dd62"}
Jul 12 22:38:24.487853 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:24.487801    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6750e7c-5ccnn" podStartSLOduration=1.487784469 podStartE2EDuration="1.487784469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:38:24.486630031 +0000 UTC m=+1676.800133385" watchObservedRunningTime="2026-07-12 22:38:24.487784469 +0000 UTC m=+1676.801287822"
Jul 12 22:38:25.476106 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:25.476068    2560 generic.go:356] "Generic (PLEG): container finished" podID="c6653b93-7999-41af-9812-a21428c64eb3" containerID="0cd8e31c25d33d89426918ea0ca3a7ee6141f83b3fe7971911faeabc544db600" exitCode=0
Jul 12 22:38:25.476463 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:25.476160    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6750e7c-5ccnn" event={"ID":"c6653b93-7999-41af-9812-a21428c64eb3","Type":"ContainerDied","Data":"0cd8e31c25d33d89426918ea0ca3a7ee6141f83b3fe7971911faeabc544db600"}
Jul 12 22:38:25.949306 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:25.949273    2560 patch_prober.go:28] interesting pod/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 12 22:38:25.949472 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:25.949328    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s" podUID="0ab4fb1b-9591-4f48-ab50-0018a5e9dd79" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 12 22:38:26.617888 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:26.617858    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6750e7c-5ccnn"
Jul 12 22:38:26.767849 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:26.767762    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6653b93-7999-41af-9812-a21428c64eb3-maas-api-service-ca\") pod \"c6653b93-7999-41af-9812-a21428c64eb3\" (UID: \"c6653b93-7999-41af-9812-a21428c64eb3\") "
Jul 12 22:38:26.768147 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:26.768126    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6653b93-7999-41af-9812-a21428c64eb3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c6653b93-7999-41af-9812-a21428c64eb3" (UID: "c6653b93-7999-41af-9812-a21428c64eb3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 12 22:38:26.868958 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:26.868923    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6653b93-7999-41af-9812-a21428c64eb3-maas-api-service-ca\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:38:27.488256 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:27.488223    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6750e7c-5ccnn" event={"ID":"c6653b93-7999-41af-9812-a21428c64eb3","Type":"ContainerDied","Data":"005c20cff088e8a2d0c70b47cd2e5eaccfd2d81f41ff1d6fd3d778f5be81dd62"}
Jul 12 22:38:27.488256 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:27.488248    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e6750e7c-5ccnn"
Jul 12 22:38:27.488459 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:27.488270    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="005c20cff088e8a2d0c70b47cd2e5eaccfd2d81f41ff1d6fd3d778f5be81dd62"
Jul 12 22:38:28.802109 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:28.802045    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"
Jul 12 22:38:28.887132 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:28.887098    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-istio-envoy\") pod \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\" (UID: \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\") "
Jul 12 22:38:28.887280 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:28.887143    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-istio-token\") pod \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\" (UID: \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\") "
Jul 12 22:38:28.887280 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:28.887181    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-workload-certs\") pod \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\" (UID: \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\") "
Jul 12 22:38:28.887280 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:28.887227    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-workload-socket\") pod \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\" (UID: \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\") "
Jul 12 22:38:28.887280 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:28.887265    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-istiod-ca-cert\") pod \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\" (UID: \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\") "
Jul 12 22:38:28.887518 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:28.887312    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-credential-socket\") pod \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\" (UID: \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\") "
Jul 12 22:38:28.887518 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:28.887349    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-istio-podinfo\") pod \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\" (UID: \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\") "
Jul 12 22:38:28.887518 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:28.887381    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-istio-data\") pod \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\" (UID: \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\") "
Jul 12 22:38:28.887518 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:28.887446    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vc9m8\" (UniqueName: \"kubernetes.io/projected/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-kube-api-access-vc9m8\") pod \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\" (UID: \"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79\") "
Jul 12 22:38:28.887771 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:28.887521    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0ab4fb1b-9591-4f48-ab50-0018a5e9dd79" (UID: "0ab4fb1b-9591-4f48-ab50-0018a5e9dd79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:38:28.887771 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:28.887581    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0ab4fb1b-9591-4f48-ab50-0018a5e9dd79" (UID: "0ab4fb1b-9591-4f48-ab50-0018a5e9dd79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:38:28.887874 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:28.887795    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-workload-socket\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:38:28.887874 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:28.887816    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-credential-socket\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:38:28.887982 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:28.887965    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0ab4fb1b-9591-4f48-ab50-0018a5e9dd79" (UID: "0ab4fb1b-9591-4f48-ab50-0018a5e9dd79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:38:28.888085 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:28.888065    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0ab4fb1b-9591-4f48-ab50-0018a5e9dd79" (UID: "0ab4fb1b-9591-4f48-ab50-0018a5e9dd79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 12 22:38:28.888798 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:28.888724    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-istio-data" (OuterVolumeSpecName: "istio-data") pod "0ab4fb1b-9591-4f48-ab50-0018a5e9dd79" (UID: "0ab4fb1b-9591-4f48-ab50-0018a5e9dd79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:38:28.890860 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:28.890835    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-istio-token" (OuterVolumeSpecName: "istio-token") pod "0ab4fb1b-9591-4f48-ab50-0018a5e9dd79" (UID: "0ab4fb1b-9591-4f48-ab50-0018a5e9dd79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 22:38:28.891008 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:28.890983    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0ab4fb1b-9591-4f48-ab50-0018a5e9dd79" (UID: "0ab4fb1b-9591-4f48-ab50-0018a5e9dd79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 12 22:38:28.891096 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:28.891056    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-kube-api-access-vc9m8" (OuterVolumeSpecName: "kube-api-access-vc9m8") pod "0ab4fb1b-9591-4f48-ab50-0018a5e9dd79" (UID: "0ab4fb1b-9591-4f48-ab50-0018a5e9dd79"). InnerVolumeSpecName "kube-api-access-vc9m8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 22:38:28.891223 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:28.891202    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0ab4fb1b-9591-4f48-ab50-0018a5e9dd79" (UID: "0ab4fb1b-9591-4f48-ab50-0018a5e9dd79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:38:28.988723 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:28.988690    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-workload-certs\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:38:28.988723 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:28.988718    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-istiod-ca-cert\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:38:28.988723 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:28.988726    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-istio-podinfo\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:38:28.988723 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:28.988734    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-istio-data\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:38:28.988981 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:28.988744    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc9m8\" (UniqueName: \"kubernetes.io/projected/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-kube-api-access-vc9m8\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:38:28.988981 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:28.988752    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-istio-envoy\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:38:28.988981 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:28.988760    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79-istio-token\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:38:29.499798 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:29.499767    2560 generic.go:356] "Generic (PLEG): container finished" podID="0ab4fb1b-9591-4f48-ab50-0018a5e9dd79" containerID="a110e0e0df9351a420e652b0da63d59768c07f9e435cda6f075eab997659ebd6" exitCode=0
Jul 12 22:38:29.499963 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:29.499825    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"
Jul 12 22:38:29.499963 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:29.499858    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s" event={"ID":"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79","Type":"ContainerDied","Data":"a110e0e0df9351a420e652b0da63d59768c07f9e435cda6f075eab997659ebd6"}
Jul 12 22:38:29.499963 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:29.499918    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s" event={"ID":"0ab4fb1b-9591-4f48-ab50-0018a5e9dd79","Type":"ContainerDied","Data":"e6b2ace0442f9be99766ef01ee4e4cda63eb809b08f034993e46799dbd1047da"}
Jul 12 22:38:29.499963 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:29.499948    2560 scope.go:117] "RemoveContainer" containerID="a110e0e0df9351a420e652b0da63d59768c07f9e435cda6f075eab997659ebd6"
Jul 12 22:38:29.509864 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:29.509846    2560 scope.go:117] "RemoveContainer" containerID="abcbb4d5351632827c2c1c5203da2ddaea2330a1e4edf9d9b0f94e63c3a9a2c5"
Jul 12 22:38:29.517860 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:29.517841    2560 scope.go:117] "RemoveContainer" containerID="a110e0e0df9351a420e652b0da63d59768c07f9e435cda6f075eab997659ebd6"
Jul 12 22:38:29.518116 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:38:29.518091    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a110e0e0df9351a420e652b0da63d59768c07f9e435cda6f075eab997659ebd6\": container with ID starting with a110e0e0df9351a420e652b0da63d59768c07f9e435cda6f075eab997659ebd6 not found: ID does not exist" containerID="a110e0e0df9351a420e652b0da63d59768c07f9e435cda6f075eab997659ebd6"
Jul 12 22:38:29.518272 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:29.518123    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a110e0e0df9351a420e652b0da63d59768c07f9e435cda6f075eab997659ebd6"} err="failed to get container status \"a110e0e0df9351a420e652b0da63d59768c07f9e435cda6f075eab997659ebd6\": rpc error: code = NotFound desc = could not find container \"a110e0e0df9351a420e652b0da63d59768c07f9e435cda6f075eab997659ebd6\": container with ID starting with a110e0e0df9351a420e652b0da63d59768c07f9e435cda6f075eab997659ebd6 not found: ID does not exist"
Jul 12 22:38:29.518272 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:29.518140    2560 scope.go:117] "RemoveContainer" containerID="abcbb4d5351632827c2c1c5203da2ddaea2330a1e4edf9d9b0f94e63c3a9a2c5"
Jul 12 22:38:29.518492 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:38:29.518472    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abcbb4d5351632827c2c1c5203da2ddaea2330a1e4edf9d9b0f94e63c3a9a2c5\": container with ID starting with abcbb4d5351632827c2c1c5203da2ddaea2330a1e4edf9d9b0f94e63c3a9a2c5 not found: ID does not exist" containerID="abcbb4d5351632827c2c1c5203da2ddaea2330a1e4edf9d9b0f94e63c3a9a2c5"
Jul 12 22:38:29.518581 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:29.518501    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abcbb4d5351632827c2c1c5203da2ddaea2330a1e4edf9d9b0f94e63c3a9a2c5"} err="failed to get container status \"abcbb4d5351632827c2c1c5203da2ddaea2330a1e4edf9d9b0f94e63c3a9a2c5\": rpc error: code = NotFound desc = could not find container \"abcbb4d5351632827c2c1c5203da2ddaea2330a1e4edf9d9b0f94e63c3a9a2c5\": container with ID starting with abcbb4d5351632827c2c1c5203da2ddaea2330a1e4edf9d9b0f94e63c3a9a2c5 not found: ID does not exist"
Jul 12 22:38:29.520623 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:29.520601    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"]
Jul 12 22:38:29.526298 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:29.526276    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-e6750e7c-gw-openshift-default-5c549f9cf9-2cn8s"]
Jul 12 22:38:30.246405 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:30.246370    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ab4fb1b-9591-4f48-ab50-0018a5e9dd79" path="/var/lib/kubelet/pods/0ab4fb1b-9591-4f48-ab50-0018a5e9dd79/volumes"
Jul 12 22:38:35.160677 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.160645    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"]
Jul 12 22:38:35.161052 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.161039    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ab4fb1b-9591-4f48-ab50-0018a5e9dd79" containerName="istio-proxy"
Jul 12 22:38:35.161094 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.161054    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ab4fb1b-9591-4f48-ab50-0018a5e9dd79" containerName="istio-proxy"
Jul 12 22:38:35.161094 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.161069    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78282258-7278-4906-9a4a-96581c6c5241" containerName="revoke-keys"
Jul 12 22:38:35.161094 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.161075    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="78282258-7278-4906-9a4a-96581c6c5241" containerName="revoke-keys"
Jul 12 22:38:35.161094 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.161084    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6653b93-7999-41af-9812-a21428c64eb3" containerName="revoke-keys"
Jul 12 22:38:35.161094 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.161093    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6653b93-7999-41af-9812-a21428c64eb3" containerName="revoke-keys"
Jul 12 22:38:35.161243 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.161113    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ab4fb1b-9591-4f48-ab50-0018a5e9dd79" containerName="istio-proxy"
Jul 12 22:38:35.161243 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.161119    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ab4fb1b-9591-4f48-ab50-0018a5e9dd79" containerName="istio-proxy"
Jul 12 22:38:35.161243 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.161179    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ab4fb1b-9591-4f48-ab50-0018a5e9dd79" containerName="istio-proxy"
Jul 12 22:38:35.161243 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.161189    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ab4fb1b-9591-4f48-ab50-0018a5e9dd79" containerName="istio-proxy"
Jul 12 22:38:35.161243 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.161197    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6653b93-7999-41af-9812-a21428c64eb3" containerName="revoke-keys"
Jul 12 22:38:35.165898 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.165875    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"
Jul 12 22:38:35.168546 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.168510    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 12 22:38:35.168648 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.168581    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c725b4f0-openshift-default-dockercfg-x2rvp\""
Jul 12 22:38:35.176137 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.176113    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"]
Jul 12 22:38:35.345662 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.345623    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff016c4-cea3-460f-9d4b-d89659b9b257-workload-socket\") pod \"e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72\" (UID: \"0ff016c4-cea3-460f-9d4b-d89659b9b257\") " pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"
Jul 12 22:38:35.345662 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.345664    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ff016c4-cea3-460f-9d4b-d89659b9b257-istio-token\") pod \"e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72\" (UID: \"0ff016c4-cea3-460f-9d4b-d89659b9b257\") " pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"
Jul 12 22:38:35.345888 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.345688    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff016c4-cea3-460f-9d4b-d89659b9b257-istiod-ca-cert\") pod \"e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72\" (UID: \"0ff016c4-cea3-460f-9d4b-d89659b9b257\") " pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"
Jul 12 22:38:35.345888 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.345704    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hjbr\" (UniqueName: \"kubernetes.io/projected/0ff016c4-cea3-460f-9d4b-d89659b9b257-kube-api-access-6hjbr\") pod \"e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72\" (UID: \"0ff016c4-cea3-460f-9d4b-d89659b9b257\") " pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"
Jul 12 22:38:35.345888 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.345797    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff016c4-cea3-460f-9d4b-d89659b9b257-credential-socket\") pod \"e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72\" (UID: \"0ff016c4-cea3-460f-9d4b-d89659b9b257\") " pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"
Jul 12 22:38:35.345888 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.345825    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff016c4-cea3-460f-9d4b-d89659b9b257-istio-data\") pod \"e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72\" (UID: \"0ff016c4-cea3-460f-9d4b-d89659b9b257\") " pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"
Jul 12 22:38:35.345888 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.345862    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff016c4-cea3-460f-9d4b-d89659b9b257-workload-certs\") pod \"e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72\" (UID: \"0ff016c4-cea3-460f-9d4b-d89659b9b257\") " pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"
Jul 12 22:38:35.345888 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.345885    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff016c4-cea3-460f-9d4b-d89659b9b257-istio-envoy\") pod \"e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72\" (UID: \"0ff016c4-cea3-460f-9d4b-d89659b9b257\") " pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"
Jul 12 22:38:35.346096 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.345918    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff016c4-cea3-460f-9d4b-d89659b9b257-istio-podinfo\") pod \"e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72\" (UID: \"0ff016c4-cea3-460f-9d4b-d89659b9b257\") " pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"
Jul 12 22:38:35.446956 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.446860    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff016c4-cea3-460f-9d4b-d89659b9b257-credential-socket\") pod \"e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72\" (UID: \"0ff016c4-cea3-460f-9d4b-d89659b9b257\") " pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"
Jul 12 22:38:35.446956 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.446900    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff016c4-cea3-460f-9d4b-d89659b9b257-istio-data\") pod \"e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72\" (UID: \"0ff016c4-cea3-460f-9d4b-d89659b9b257\") " pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"
Jul 12 22:38:35.446956 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.446927    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff016c4-cea3-460f-9d4b-d89659b9b257-workload-certs\") pod \"e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72\" (UID: \"0ff016c4-cea3-460f-9d4b-d89659b9b257\") " pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"
Jul 12 22:38:35.446956 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.446944    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff016c4-cea3-460f-9d4b-d89659b9b257-istio-envoy\") pod \"e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72\" (UID: \"0ff016c4-cea3-460f-9d4b-d89659b9b257\") " pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"
Jul 12 22:38:35.447267 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.446971    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff016c4-cea3-460f-9d4b-d89659b9b257-istio-podinfo\") pod \"e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72\" (UID: \"0ff016c4-cea3-460f-9d4b-d89659b9b257\") " pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"
Jul 12 22:38:35.447267 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.447000    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff016c4-cea3-460f-9d4b-d89659b9b257-workload-socket\") pod \"e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72\" (UID: \"0ff016c4-cea3-460f-9d4b-d89659b9b257\") " pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"
Jul 12 22:38:35.447267 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.447015    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ff016c4-cea3-460f-9d4b-d89659b9b257-istio-token\") pod \"e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72\" (UID: \"0ff016c4-cea3-460f-9d4b-d89659b9b257\") " pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"
Jul 12 22:38:35.447267 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.447037    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff016c4-cea3-460f-9d4b-d89659b9b257-istiod-ca-cert\") pod \"e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72\" (UID: \"0ff016c4-cea3-460f-9d4b-d89659b9b257\") " pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"
Jul 12 22:38:35.447267 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.447052    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hjbr\" (UniqueName: \"kubernetes.io/projected/0ff016c4-cea3-460f-9d4b-d89659b9b257-kube-api-access-6hjbr\") pod \"e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72\" (UID: \"0ff016c4-cea3-460f-9d4b-d89659b9b257\") " pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"
Jul 12 22:38:35.447506 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.447327    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff016c4-cea3-460f-9d4b-d89659b9b257-workload-certs\") pod \"e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72\" (UID: \"0ff016c4-cea3-460f-9d4b-d89659b9b257\") " pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"
Jul 12 22:38:35.447506 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.447404    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff016c4-cea3-460f-9d4b-d89659b9b257-credential-socket\") pod \"e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72\" (UID: \"0ff016c4-cea3-460f-9d4b-d89659b9b257\") " pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"
Jul 12 22:38:35.447634 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.447557    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff016c4-cea3-460f-9d4b-d89659b9b257-istio-data\") pod \"e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72\" (UID: \"0ff016c4-cea3-460f-9d4b-d89659b9b257\") " pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"
Jul 12 22:38:35.447689 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.447631    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff016c4-cea3-460f-9d4b-d89659b9b257-workload-socket\") pod \"e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72\" (UID: \"0ff016c4-cea3-460f-9d4b-d89659b9b257\") " pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"
Jul 12 22:38:35.447928 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.447905    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff016c4-cea3-460f-9d4b-d89659b9b257-istiod-ca-cert\") pod \"e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72\" (UID: \"0ff016c4-cea3-460f-9d4b-d89659b9b257\") " pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"
Jul 12 22:38:35.449398 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.449382    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff016c4-cea3-460f-9d4b-d89659b9b257-istio-envoy\") pod \"e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72\" (UID: \"0ff016c4-cea3-460f-9d4b-d89659b9b257\") " pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"
Jul 12 22:38:35.449716 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.449699    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff016c4-cea3-460f-9d4b-d89659b9b257-istio-podinfo\") pod \"e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72\" (UID: \"0ff016c4-cea3-460f-9d4b-d89659b9b257\") " pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"
Jul 12 22:38:35.454993 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.454970    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ff016c4-cea3-460f-9d4b-d89659b9b257-istio-token\") pod \"e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72\" (UID: \"0ff016c4-cea3-460f-9d4b-d89659b9b257\") " pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"
Jul 12 22:38:35.455073 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.455043    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hjbr\" (UniqueName: \"kubernetes.io/projected/0ff016c4-cea3-460f-9d4b-d89659b9b257-kube-api-access-6hjbr\") pod \"e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72\" (UID: \"0ff016c4-cea3-460f-9d4b-d89659b9b257\") " pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"
Jul 12 22:38:35.478163 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.478131    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"
Jul 12 22:38:35.627775 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.627740    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"]
Jul 12 22:38:35.630262 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:38:35.630233    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ff016c4_cea3_460f_9d4b_d89659b9b257.slice/crio-b7466a625cb7dac7697f9e48f612fedf608af7bf48e444fbe8c4a216045eb2cf WatchSource:0}: Error finding container b7466a625cb7dac7697f9e48f612fedf608af7bf48e444fbe8c4a216045eb2cf: Status 404 returned error can't find the container with id b7466a625cb7dac7697f9e48f612fedf608af7bf48e444fbe8c4a216045eb2cf
Jul 12 22:38:35.632704 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.632657    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:38:35.632787 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.632760    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:38:35.632825 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:35.632806    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:38:36.546606 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:36.546565    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72" event={"ID":"0ff016c4-cea3-460f-9d4b-d89659b9b257","Type":"ContainerStarted","Data":"5cdb3171b218b501656efee932a3da919119ede01d9a0c6c4e444b49c58c2bc6"}
Jul 12 22:38:36.547247 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:36.546627    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72" event={"ID":"0ff016c4-cea3-460f-9d4b-d89659b9b257","Type":"ContainerStarted","Data":"b7466a625cb7dac7697f9e48f612fedf608af7bf48e444fbe8c4a216045eb2cf"}
Jul 12 22:38:36.573024 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:36.572967    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72" podStartSLOduration=1.572950334 podStartE2EDuration="1.572950334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:38:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:38:36.571238175 +0000 UTC m=+1688.884741528" watchObservedRunningTime="2026-07-12 22:38:36.572950334 +0000 UTC m=+1688.886453688"
Jul 12 22:38:37.478361 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:37.478317    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"
Jul 12 22:38:37.483417 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:37.483389    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"
Jul 12 22:38:37.552112 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:37.552081    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"
Jul 12 22:38:37.553039 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:37.553020    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"
Jul 12 22:38:41.450658 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:41.450626    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c725b4f0-kclpj"]
Jul 12 22:38:41.454316 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:41.454300    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c725b4f0-kclpj"
Jul 12 22:38:41.457043 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:41.456995    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 12 22:38:41.457043 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:41.457001    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlrdf\""
Jul 12 22:38:41.460834 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:41.460804    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c725b4f0-kclpj"]
Jul 12 22:38:41.599967 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:41.599921    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11cf1a6b-3e3a-4d82-8d69-04166eeef07f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c725b4f0-kclpj\" (UID: \"11cf1a6b-3e3a-4d82-8d69-04166eeef07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c725b4f0-kclpj"
Jul 12 22:38:41.701317 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:41.701219    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11cf1a6b-3e3a-4d82-8d69-04166eeef07f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c725b4f0-kclpj\" (UID: \"11cf1a6b-3e3a-4d82-8d69-04166eeef07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c725b4f0-kclpj"
Jul 12 22:38:41.701923 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:41.701901    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11cf1a6b-3e3a-4d82-8d69-04166eeef07f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c725b4f0-kclpj\" (UID: \"11cf1a6b-3e3a-4d82-8d69-04166eeef07f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c725b4f0-kclpj"
Jul 12 22:38:41.766672 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:41.766635    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c725b4f0-kclpj"
Jul 12 22:38:41.950436 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:41.950358    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c725b4f0-kclpj"]
Jul 12 22:38:42.578136 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:42.578103    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c725b4f0-kclpj" event={"ID":"11cf1a6b-3e3a-4d82-8d69-04166eeef07f","Type":"ContainerStarted","Data":"4593634a456d73d22b4dab61eb58ec938316b6fd03045d4de53680d7725b99f4"}
Jul 12 22:38:42.578517 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:42.578167    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c725b4f0-kclpj" event={"ID":"11cf1a6b-3e3a-4d82-8d69-04166eeef07f","Type":"ContainerStarted","Data":"c97f353131f70cbc88e4c3efcee9e98bdad12d5ad559186b00b48f7dcdcf1fae"}
Jul 12 22:38:42.593937 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:42.593881    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c725b4f0-kclpj" podStartSLOduration=1.5938644050000002 podStartE2EDuration="1.593864405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:38:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:38:42.593013845 +0000 UTC m=+1694.906517201" watchObservedRunningTime="2026-07-12 22:38:42.593864405 +0000 UTC m=+1694.907367758"
Jul 12 22:38:43.584366 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:43.584324    2560 generic.go:356] "Generic (PLEG): container finished" podID="11cf1a6b-3e3a-4d82-8d69-04166eeef07f" containerID="4593634a456d73d22b4dab61eb58ec938316b6fd03045d4de53680d7725b99f4" exitCode=7
Jul 12 22:38:43.584819 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:43.584373    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c725b4f0-kclpj" event={"ID":"11cf1a6b-3e3a-4d82-8d69-04166eeef07f","Type":"ContainerDied","Data":"4593634a456d73d22b4dab61eb58ec938316b6fd03045d4de53680d7725b99f4"}
Jul 12 22:38:43.584819 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:43.584594    2560 scope.go:117] "RemoveContainer" containerID="4593634a456d73d22b4dab61eb58ec938316b6fd03045d4de53680d7725b99f4"
Jul 12 22:38:44.591819 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:44.591786    2560 generic.go:356] "Generic (PLEG): container finished" podID="11cf1a6b-3e3a-4d82-8d69-04166eeef07f" containerID="0629f8e33f131dfc3cc3980f50b8f3e266f4c73b11701f011bfbcd3e2bab0aa1" exitCode=0
Jul 12 22:38:44.592288 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:44.591840    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c725b4f0-kclpj" event={"ID":"11cf1a6b-3e3a-4d82-8d69-04166eeef07f","Type":"ContainerDied","Data":"0629f8e33f131dfc3cc3980f50b8f3e266f4c73b11701f011bfbcd3e2bab0aa1"}
Jul 12 22:38:44.592288 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:44.591896    2560 scope.go:117] "RemoveContainer" containerID="4593634a456d73d22b4dab61eb58ec938316b6fd03045d4de53680d7725b99f4"
Jul 12 22:38:45.751491 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:45.751459    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c725b4f0-kclpj"
Jul 12 22:38:45.839998 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:45.839959    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11cf1a6b-3e3a-4d82-8d69-04166eeef07f-maas-api-service-ca\") pod \"11cf1a6b-3e3a-4d82-8d69-04166eeef07f\" (UID: \"11cf1a6b-3e3a-4d82-8d69-04166eeef07f\") "
Jul 12 22:38:45.840377 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:45.840349    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11cf1a6b-3e3a-4d82-8d69-04166eeef07f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "11cf1a6b-3e3a-4d82-8d69-04166eeef07f" (UID: "11cf1a6b-3e3a-4d82-8d69-04166eeef07f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 12 22:38:45.941118 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:45.941026    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11cf1a6b-3e3a-4d82-8d69-04166eeef07f-maas-api-service-ca\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:38:46.606352 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:46.606321    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c725b4f0-kclpj" event={"ID":"11cf1a6b-3e3a-4d82-8d69-04166eeef07f","Type":"ContainerDied","Data":"c97f353131f70cbc88e4c3efcee9e98bdad12d5ad559186b00b48f7dcdcf1fae"}
Jul 12 22:38:46.606524 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:46.606374    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c97f353131f70cbc88e4c3efcee9e98bdad12d5ad559186b00b48f7dcdcf1fae"
Jul 12 22:38:46.606524 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:46.606374    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c725b4f0-kclpj"
Jul 12 22:38:47.612776 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:47.612695    2560 generic.go:356] "Generic (PLEG): container finished" podID="0ff016c4-cea3-460f-9d4b-d89659b9b257" containerID="5cdb3171b218b501656efee932a3da919119ede01d9a0c6c4e444b49c58c2bc6" exitCode=0
Jul 12 22:38:47.613170 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:47.612770    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72" event={"ID":"0ff016c4-cea3-460f-9d4b-d89659b9b257","Type":"ContainerDied","Data":"5cdb3171b218b501656efee932a3da919119ede01d9a0c6c4e444b49c58c2bc6"}
Jul 12 22:38:47.613241 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:47.613222    2560 scope.go:117] "RemoveContainer" containerID="5cdb3171b218b501656efee932a3da919119ede01d9a0c6c4e444b49c58c2bc6"
Jul 12 22:38:47.614249 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:47.614221    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:38:47.614312 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:47.614300    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:38:47.614358 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:47.614345    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:38:48.640398 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:48.640171    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72" event={"ID":"0ff016c4-cea3-460f-9d4b-d89659b9b257","Type":"ContainerStarted","Data":"4e87e66fdfd4908ea95a1bdebbfcaff10f8138ad5fec1a274f0a33b08834bfa9"}
Jul 12 22:38:49.478726 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:49.478689    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"
Jul 12 22:38:49.479963 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:49.479940    2560 patch_prober.go:28] interesting pod/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Jul 12 22:38:49.480022 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:49.480001    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72" podUID="0ff016c4-cea3-460f-9d4b-d89659b9b257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Jul 12 22:38:50.478862 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:50.478824    2560 patch_prober.go:28] interesting pod/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Jul 12 22:38:50.479271 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:50.478880    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72" podUID="0ff016c4-cea3-460f-9d4b-d89659b9b257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Jul 12 22:38:51.479552 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:51.479494    2560 patch_prober.go:28] interesting pod/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Jul 12 22:38:51.479940 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:51.479583    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72" podUID="0ff016c4-cea3-460f-9d4b-d89659b9b257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Jul 12 22:38:52.482791 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:52.482755    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"
Jul 12 22:38:52.483259 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:52.483240    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"
Jul 12 22:38:52.484126 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:38:52.484104    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"
Jul 12 22:39:07.537666 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:07.537629    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"]
Jul 12 22:39:07.538151 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:07.537827    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72" podUID="0ff016c4-cea3-460f-9d4b-d89659b9b257" containerName="istio-proxy" containerID="cri-o://4e87e66fdfd4908ea95a1bdebbfcaff10f8138ad5fec1a274f0a33b08834bfa9" gracePeriod=30
Jul 12 22:39:12.765451 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:12.765414    2560 generic.go:356] "Generic (PLEG): container finished" podID="0ff016c4-cea3-460f-9d4b-d89659b9b257" containerID="4e87e66fdfd4908ea95a1bdebbfcaff10f8138ad5fec1a274f0a33b08834bfa9" exitCode=0
Jul 12 22:39:12.765906 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:12.765460    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72" event={"ID":"0ff016c4-cea3-460f-9d4b-d89659b9b257","Type":"ContainerDied","Data":"4e87e66fdfd4908ea95a1bdebbfcaff10f8138ad5fec1a274f0a33b08834bfa9"}
Jul 12 22:39:12.765906 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:12.765559    2560 scope.go:117] "RemoveContainer" containerID="5cdb3171b218b501656efee932a3da919119ede01d9a0c6c4e444b49c58c2bc6"
Jul 12 22:39:12.806773 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:12.806706    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"
Jul 12 22:39:12.888320 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:12.888288    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff016c4-cea3-460f-9d4b-d89659b9b257-workload-socket\") pod \"0ff016c4-cea3-460f-9d4b-d89659b9b257\" (UID: \"0ff016c4-cea3-460f-9d4b-d89659b9b257\") "
Jul 12 22:39:12.888487 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:12.888333    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff016c4-cea3-460f-9d4b-d89659b9b257-istio-envoy\") pod \"0ff016c4-cea3-460f-9d4b-d89659b9b257\" (UID: \"0ff016c4-cea3-460f-9d4b-d89659b9b257\") "
Jul 12 22:39:12.888487 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:12.888354    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ff016c4-cea3-460f-9d4b-d89659b9b257-istio-token\") pod \"0ff016c4-cea3-460f-9d4b-d89659b9b257\" (UID: \"0ff016c4-cea3-460f-9d4b-d89659b9b257\") "
Jul 12 22:39:12.888487 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:12.888378    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff016c4-cea3-460f-9d4b-d89659b9b257-credential-socket\") pod \"0ff016c4-cea3-460f-9d4b-d89659b9b257\" (UID: \"0ff016c4-cea3-460f-9d4b-d89659b9b257\") "
Jul 12 22:39:12.888487 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:12.888404    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hjbr\" (UniqueName: \"kubernetes.io/projected/0ff016c4-cea3-460f-9d4b-d89659b9b257-kube-api-access-6hjbr\") pod \"0ff016c4-cea3-460f-9d4b-d89659b9b257\" (UID: \"0ff016c4-cea3-460f-9d4b-d89659b9b257\") "
Jul 12 22:39:12.888487 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:12.888418    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff016c4-cea3-460f-9d4b-d89659b9b257-istiod-ca-cert\") pod \"0ff016c4-cea3-460f-9d4b-d89659b9b257\" (UID: \"0ff016c4-cea3-460f-9d4b-d89659b9b257\") "
Jul 12 22:39:12.888774 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:12.888502    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff016c4-cea3-460f-9d4b-d89659b9b257-istio-podinfo\") pod \"0ff016c4-cea3-460f-9d4b-d89659b9b257\" (UID: \"0ff016c4-cea3-460f-9d4b-d89659b9b257\") "
Jul 12 22:39:12.888774 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:12.888568    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff016c4-cea3-460f-9d4b-d89659b9b257-istio-data\") pod \"0ff016c4-cea3-460f-9d4b-d89659b9b257\" (UID: \"0ff016c4-cea3-460f-9d4b-d89659b9b257\") "
Jul 12 22:39:12.888774 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:12.888612    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff016c4-cea3-460f-9d4b-d89659b9b257-workload-certs\") pod \"0ff016c4-cea3-460f-9d4b-d89659b9b257\" (UID: \"0ff016c4-cea3-460f-9d4b-d89659b9b257\") "
Jul 12 22:39:12.888774 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:12.888604    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff016c4-cea3-460f-9d4b-d89659b9b257-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0ff016c4-cea3-460f-9d4b-d89659b9b257" (UID: "0ff016c4-cea3-460f-9d4b-d89659b9b257"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:39:12.888774 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:12.888684    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff016c4-cea3-460f-9d4b-d89659b9b257-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0ff016c4-cea3-460f-9d4b-d89659b9b257" (UID: "0ff016c4-cea3-460f-9d4b-d89659b9b257"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:39:12.889045 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:12.888885    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff016c4-cea3-460f-9d4b-d89659b9b257-workload-socket\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:39:12.889045 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:12.888903    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff016c4-cea3-460f-9d4b-d89659b9b257-credential-socket\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:39:12.889045 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:12.888993    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ff016c4-cea3-460f-9d4b-d89659b9b257-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0ff016c4-cea3-460f-9d4b-d89659b9b257" (UID: "0ff016c4-cea3-460f-9d4b-d89659b9b257"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 12 22:39:12.889324 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:12.889261    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff016c4-cea3-460f-9d4b-d89659b9b257-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0ff016c4-cea3-460f-9d4b-d89659b9b257" (UID: "0ff016c4-cea3-460f-9d4b-d89659b9b257"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:39:12.890374 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:12.890312    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff016c4-cea3-460f-9d4b-d89659b9b257-istio-data" (OuterVolumeSpecName: "istio-data") pod "0ff016c4-cea3-460f-9d4b-d89659b9b257" (UID: "0ff016c4-cea3-460f-9d4b-d89659b9b257"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:39:12.890828 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:12.890800    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff016c4-cea3-460f-9d4b-d89659b9b257-kube-api-access-6hjbr" (OuterVolumeSpecName: "kube-api-access-6hjbr") pod "0ff016c4-cea3-460f-9d4b-d89659b9b257" (UID: "0ff016c4-cea3-460f-9d4b-d89659b9b257"). InnerVolumeSpecName "kube-api-access-6hjbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 22:39:12.891054 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:12.891029    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff016c4-cea3-460f-9d4b-d89659b9b257-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0ff016c4-cea3-460f-9d4b-d89659b9b257" (UID: "0ff016c4-cea3-460f-9d4b-d89659b9b257"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:39:12.891054 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:12.891026    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff016c4-cea3-460f-9d4b-d89659b9b257-istio-token" (OuterVolumeSpecName: "istio-token") pod "0ff016c4-cea3-460f-9d4b-d89659b9b257" (UID: "0ff016c4-cea3-460f-9d4b-d89659b9b257"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 22:39:12.891803 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:12.891787    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ff016c4-cea3-460f-9d4b-d89659b9b257-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0ff016c4-cea3-460f-9d4b-d89659b9b257" (UID: "0ff016c4-cea3-460f-9d4b-d89659b9b257"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 12 22:39:12.989599 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:12.989560    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff016c4-cea3-460f-9d4b-d89659b9b257-workload-certs\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:39:12.989599 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:12.989591    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff016c4-cea3-460f-9d4b-d89659b9b257-istio-envoy\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:39:12.989599 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:12.989600    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ff016c4-cea3-460f-9d4b-d89659b9b257-istio-token\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:39:12.989599 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:12.989608    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hjbr\" (UniqueName: \"kubernetes.io/projected/0ff016c4-cea3-460f-9d4b-d89659b9b257-kube-api-access-6hjbr\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:39:12.990121 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:12.989618    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff016c4-cea3-460f-9d4b-d89659b9b257-istiod-ca-cert\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:39:12.990121 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:12.989626    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff016c4-cea3-460f-9d4b-d89659b9b257-istio-podinfo\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:39:12.990121 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:12.989634    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff016c4-cea3-460f-9d4b-d89659b9b257-istio-data\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:39:13.402187 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:13.402158    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55d0a565-n5khk"]
Jul 12 22:39:13.402522 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:13.402510    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ff016c4-cea3-460f-9d4b-d89659b9b257" containerName="istio-proxy"
Jul 12 22:39:13.402588 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:13.402524    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ff016c4-cea3-460f-9d4b-d89659b9b257" containerName="istio-proxy"
Jul 12 22:39:13.402588 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:13.402546    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11cf1a6b-3e3a-4d82-8d69-04166eeef07f" containerName="revoke-keys"
Jul 12 22:39:13.402588 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:13.402552    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="11cf1a6b-3e3a-4d82-8d69-04166eeef07f" containerName="revoke-keys"
Jul 12 22:39:13.402684 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:13.402628    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ff016c4-cea3-460f-9d4b-d89659b9b257" containerName="istio-proxy"
Jul 12 22:39:13.402684 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:13.402637    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ff016c4-cea3-460f-9d4b-d89659b9b257" containerName="istio-proxy"
Jul 12 22:39:13.402684 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:13.402645    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="11cf1a6b-3e3a-4d82-8d69-04166eeef07f" containerName="revoke-keys"
Jul 12 22:39:13.402684 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:13.402651    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="11cf1a6b-3e3a-4d82-8d69-04166eeef07f" containerName="revoke-keys"
Jul 12 22:39:13.405975 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:13.405955    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55d0a565-n5khk"
Jul 12 22:39:13.408379 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:13.408361    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlrdf\""
Jul 12 22:39:13.408623 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:13.408609    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 12 22:39:13.412159 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:13.412137    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55d0a565-n5khk"]
Jul 12 22:39:13.493384 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:13.493347    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3020733f-d891-46d3-848a-63b5be41e466-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-55d0a565-n5khk\" (UID: \"3020733f-d891-46d3-848a-63b5be41e466\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55d0a565-n5khk"
Jul 12 22:39:13.594234 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:13.594175    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3020733f-d891-46d3-848a-63b5be41e466-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-55d0a565-n5khk\" (UID: \"3020733f-d891-46d3-848a-63b5be41e466\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55d0a565-n5khk"
Jul 12 22:39:13.594932 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:13.594907    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3020733f-d891-46d3-848a-63b5be41e466-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-55d0a565-n5khk\" (UID: \"3020733f-d891-46d3-848a-63b5be41e466\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55d0a565-n5khk"
Jul 12 22:39:13.718477 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:13.718387    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55d0a565-n5khk"
Jul 12 22:39:13.775650 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:13.775386    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72" event={"ID":"0ff016c4-cea3-460f-9d4b-d89659b9b257","Type":"ContainerDied","Data":"b7466a625cb7dac7697f9e48f612fedf608af7bf48e444fbe8c4a216045eb2cf"}
Jul 12 22:39:13.775650 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:13.775462    2560 scope.go:117] "RemoveContainer" containerID="4e87e66fdfd4908ea95a1bdebbfcaff10f8138ad5fec1a274f0a33b08834bfa9"
Jul 12 22:39:13.775650 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:13.775397    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"
Jul 12 22:39:13.805192 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:13.805163    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"]
Jul 12 22:39:13.810348 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:13.810321    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c725b4f0-openshift-default-557568ff5c-z8n72"]
Jul 12 22:39:13.897548 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:13.897498    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55d0a565-n5khk"]
Jul 12 22:39:13.900889 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:39:13.900857    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3020733f_d891_46d3_848a_63b5be41e466.slice/crio-b50dd2d892887f7938921dcdd1ff49e4531fec5d37d1fab29f85e6bffb9b4e2e WatchSource:0}: Error finding container b50dd2d892887f7938921dcdd1ff49e4531fec5d37d1fab29f85e6bffb9b4e2e: Status 404 returned error can't find the container with id b50dd2d892887f7938921dcdd1ff49e4531fec5d37d1fab29f85e6bffb9b4e2e
Jul 12 22:39:14.245586 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:14.245526    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ff016c4-cea3-460f-9d4b-d89659b9b257" path="/var/lib/kubelet/pods/0ff016c4-cea3-460f-9d4b-d89659b9b257/volumes"
Jul 12 22:39:14.783477 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:14.783393    2560 generic.go:356] "Generic (PLEG): container finished" podID="3020733f-d891-46d3-848a-63b5be41e466" containerID="4671ee564a7dddf428f559f17da93a05b6fbfa8b6cbdcbdfddeae3ddc65f502f" exitCode=7
Jul 12 22:39:14.783865 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:14.783473    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55d0a565-n5khk" event={"ID":"3020733f-d891-46d3-848a-63b5be41e466","Type":"ContainerDied","Data":"4671ee564a7dddf428f559f17da93a05b6fbfa8b6cbdcbdfddeae3ddc65f502f"}
Jul 12 22:39:14.783865 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:14.783523    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55d0a565-n5khk" event={"ID":"3020733f-d891-46d3-848a-63b5be41e466","Type":"ContainerStarted","Data":"b50dd2d892887f7938921dcdd1ff49e4531fec5d37d1fab29f85e6bffb9b4e2e"}
Jul 12 22:39:14.783865 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:14.783641    2560 scope.go:117] "RemoveContainer" containerID="4671ee564a7dddf428f559f17da93a05b6fbfa8b6cbdcbdfddeae3ddc65f502f"
Jul 12 22:39:15.790845 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:15.790756    2560 generic.go:356] "Generic (PLEG): container finished" podID="3020733f-d891-46d3-848a-63b5be41e466" containerID="1e46c0d0944ccc302244405ef1cb4ed563a50be87192cd36ccfc5809a00aa0cd" exitCode=7
Jul 12 22:39:15.791268 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:15.790844    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55d0a565-n5khk" event={"ID":"3020733f-d891-46d3-848a-63b5be41e466","Type":"ContainerDied","Data":"1e46c0d0944ccc302244405ef1cb4ed563a50be87192cd36ccfc5809a00aa0cd"}
Jul 12 22:39:15.791268 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:15.790915    2560 scope.go:117] "RemoveContainer" containerID="4671ee564a7dddf428f559f17da93a05b6fbfa8b6cbdcbdfddeae3ddc65f502f"
Jul 12 22:39:15.791268 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:15.791012    2560 scope.go:117] "RemoveContainer" containerID="1e46c0d0944ccc302244405ef1cb4ed563a50be87192cd36ccfc5809a00aa0cd"
Jul 12 22:39:15.791268 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:39:15.791259    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-55d0a565-n5khk_odh-ai-gateway-infra(3020733f-d891-46d3-848a-63b5be41e466)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55d0a565-n5khk" podUID="3020733f-d891-46d3-848a-63b5be41e466"
Jul 12 22:39:16.797722 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:16.797697    2560 scope.go:117] "RemoveContainer" containerID="1e46c0d0944ccc302244405ef1cb4ed563a50be87192cd36ccfc5809a00aa0cd"
Jul 12 22:39:16.798088 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:39:16.797868    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-55d0a565-n5khk_odh-ai-gateway-infra(3020733f-d891-46d3-848a-63b5be41e466)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55d0a565-n5khk" podUID="3020733f-d891-46d3-848a-63b5be41e466"
Jul 12 22:39:29.239905 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:29.239866    2560 scope.go:117] "RemoveContainer" containerID="1e46c0d0944ccc302244405ef1cb4ed563a50be87192cd36ccfc5809a00aa0cd"
Jul 12 22:39:29.873276 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:29.873239    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55d0a565-n5khk" event={"ID":"3020733f-d891-46d3-848a-63b5be41e466","Type":"ContainerStarted","Data":"f89d769e3411b53456c9bf2c337848e3ec8589dd570b8e5cb12e4ec980a1d083"}
Jul 12 22:39:30.881077 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:30.881040    2560 generic.go:356] "Generic (PLEG): container finished" podID="3020733f-d891-46d3-848a-63b5be41e466" containerID="f89d769e3411b53456c9bf2c337848e3ec8589dd570b8e5cb12e4ec980a1d083" exitCode=0
Jul 12 22:39:30.881522 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:30.881118    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55d0a565-n5khk" event={"ID":"3020733f-d891-46d3-848a-63b5be41e466","Type":"ContainerDied","Data":"f89d769e3411b53456c9bf2c337848e3ec8589dd570b8e5cb12e4ec980a1d083"}
Jul 12 22:39:30.881522 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:30.881195    2560 scope.go:117] "RemoveContainer" containerID="1e46c0d0944ccc302244405ef1cb4ed563a50be87192cd36ccfc5809a00aa0cd"
Jul 12 22:39:30.897909 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:30.897854    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55d0a565-n5khk" podStartSLOduration=17.897838433 podStartE2EDuration="17.897838433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:39:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:39:29.887916327 +0000 UTC m=+1742.201419682" watchObservedRunningTime="2026-07-12 22:39:30.897838433 +0000 UTC m=+1743.211341787"
Jul 12 22:39:30.898036 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:30.898006    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55d0a565-n5khk"]
Jul 12 22:39:32.028443 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:32.028416    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55d0a565-n5khk"
Jul 12 22:39:32.060280 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:32.060254    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3020733f-d891-46d3-848a-63b5be41e466-maas-api-service-ca\") pod \"3020733f-d891-46d3-848a-63b5be41e466\" (UID: \"3020733f-d891-46d3-848a-63b5be41e466\") "
Jul 12 22:39:32.060631 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:32.060599    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3020733f-d891-46d3-848a-63b5be41e466-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3020733f-d891-46d3-848a-63b5be41e466" (UID: "3020733f-d891-46d3-848a-63b5be41e466"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 12 22:39:32.161055 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:32.160955    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3020733f-d891-46d3-848a-63b5be41e466-maas-api-service-ca\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:39:32.894256 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:32.894218    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55d0a565-n5khk" event={"ID":"3020733f-d891-46d3-848a-63b5be41e466","Type":"ContainerDied","Data":"b50dd2d892887f7938921dcdd1ff49e4531fec5d37d1fab29f85e6bffb9b4e2e"}
Jul 12 22:39:32.894433 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:32.894301    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b50dd2d892887f7938921dcdd1ff49e4531fec5d37d1fab29f85e6bffb9b4e2e"
Jul 12 22:39:32.894433 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:32.894229    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55d0a565-n5khk"
Jul 12 22:39:32.923922 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:32.923891    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55d0a565-n5khk"]
Jul 12 22:39:32.927664 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:32.927637    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55d0a565-n5khk"]
Jul 12 22:39:34.246284 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:39:34.246248    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3020733f-d891-46d3-848a-63b5be41e466" path="/var/lib/kubelet/pods/3020733f-d891-46d3-848a-63b5be41e466/volumes"
Jul 12 22:40:03.443791 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:40:03.443760    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55d0a565-6gzd9"]
Jul 12 22:40:03.444349 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:40:03.444107    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3020733f-d891-46d3-848a-63b5be41e466" containerName="revoke-keys"
Jul 12 22:40:03.444349 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:40:03.444118    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3020733f-d891-46d3-848a-63b5be41e466" containerName="revoke-keys"
Jul 12 22:40:03.444349 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:40:03.444130    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3020733f-d891-46d3-848a-63b5be41e466" containerName="revoke-keys"
Jul 12 22:40:03.444349 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:40:03.444136    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3020733f-d891-46d3-848a-63b5be41e466" containerName="revoke-keys"
Jul 12 22:40:03.444349 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:40:03.444145    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3020733f-d891-46d3-848a-63b5be41e466" containerName="revoke-keys"
Jul 12 22:40:03.444349 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:40:03.444152    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3020733f-d891-46d3-848a-63b5be41e466" containerName="revoke-keys"
Jul 12 22:40:03.444349 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:40:03.444166    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ff016c4-cea3-460f-9d4b-d89659b9b257" containerName="istio-proxy"
Jul 12 22:40:03.444349 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:40:03.444173    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ff016c4-cea3-460f-9d4b-d89659b9b257" containerName="istio-proxy"
Jul 12 22:40:03.444349 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:40:03.444179    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11cf1a6b-3e3a-4d82-8d69-04166eeef07f" containerName="revoke-keys"
Jul 12 22:40:03.444349 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:40:03.444184    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="11cf1a6b-3e3a-4d82-8d69-04166eeef07f" containerName="revoke-keys"
Jul 12 22:40:03.444349 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:40:03.444246    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3020733f-d891-46d3-848a-63b5be41e466" containerName="revoke-keys"
Jul 12 22:40:03.444349 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:40:03.444255    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3020733f-d891-46d3-848a-63b5be41e466" containerName="revoke-keys"
Jul 12 22:40:03.447406 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:40:03.447390    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55d0a565-6gzd9"
Jul 12 22:40:03.450185 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:40:03.450146    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 12 22:40:03.450185 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:40:03.450168    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlrdf\""
Jul 12 22:40:03.454229 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:40:03.454203    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55d0a565-6gzd9"]
Jul 12 22:40:03.508786 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:40:03.508753    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1daa9cc-c99c-4d25-9160-1f00d8a1683e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-55d0a565-6gzd9\" (UID: \"e1daa9cc-c99c-4d25-9160-1f00d8a1683e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55d0a565-6gzd9"
Jul 12 22:40:03.609960 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:40:03.609917    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1daa9cc-c99c-4d25-9160-1f00d8a1683e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-55d0a565-6gzd9\" (UID: \"e1daa9cc-c99c-4d25-9160-1f00d8a1683e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55d0a565-6gzd9"
Jul 12 22:40:03.610597 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:40:03.610580    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1daa9cc-c99c-4d25-9160-1f00d8a1683e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-55d0a565-6gzd9\" (UID: \"e1daa9cc-c99c-4d25-9160-1f00d8a1683e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55d0a565-6gzd9"
Jul 12 22:40:03.759497 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:40:03.759406    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55d0a565-6gzd9"
Jul 12 22:40:03.896927 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:40:03.896904    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55d0a565-6gzd9"]
Jul 12 22:40:03.899032 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:40:03.899008    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1daa9cc_c99c_4d25_9160_1f00d8a1683e.slice/crio-86d3f8d1de401a7aae85b40dc9204de79dae61ba3270c662f7a8ad0b7822361f WatchSource:0}: Error finding container 86d3f8d1de401a7aae85b40dc9204de79dae61ba3270c662f7a8ad0b7822361f: Status 404 returned error can't find the container with id 86d3f8d1de401a7aae85b40dc9204de79dae61ba3270c662f7a8ad0b7822361f
Jul 12 22:40:04.052992 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:40:04.052906    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55d0a565-6gzd9" event={"ID":"e1daa9cc-c99c-4d25-9160-1f00d8a1683e","Type":"ContainerStarted","Data":"86d3f8d1de401a7aae85b40dc9204de79dae61ba3270c662f7a8ad0b7822361f"}
Jul 12 22:40:05.059880 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:40:05.059791    2560 generic.go:356] "Generic (PLEG): container finished" podID="e1daa9cc-c99c-4d25-9160-1f00d8a1683e" containerID="4a236a175440e5a63337124daa5da2c8f00178f963bd6126957c2c69c8adc91e" exitCode=0
Jul 12 22:40:05.060243 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:40:05.059883    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55d0a565-6gzd9" event={"ID":"e1daa9cc-c99c-4d25-9160-1f00d8a1683e","Type":"ContainerDied","Data":"4a236a175440e5a63337124daa5da2c8f00178f963bd6126957c2c69c8adc91e"}
Jul 12 22:40:06.211506 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:40:06.211482    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55d0a565-6gzd9"
Jul 12 22:40:06.341091 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:40:06.340995    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1daa9cc-c99c-4d25-9160-1f00d8a1683e-maas-api-service-ca\") pod \"e1daa9cc-c99c-4d25-9160-1f00d8a1683e\" (UID: \"e1daa9cc-c99c-4d25-9160-1f00d8a1683e\") "
Jul 12 22:40:06.341419 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:40:06.341388    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1daa9cc-c99c-4d25-9160-1f00d8a1683e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e1daa9cc-c99c-4d25-9160-1f00d8a1683e" (UID: "e1daa9cc-c99c-4d25-9160-1f00d8a1683e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 12 22:40:06.341636 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:40:06.341616    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1daa9cc-c99c-4d25-9160-1f00d8a1683e-maas-api-service-ca\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:40:07.071214 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:40:07.071183    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55d0a565-6gzd9"
Jul 12 22:40:07.071386 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:40:07.071218    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55d0a565-6gzd9" event={"ID":"e1daa9cc-c99c-4d25-9160-1f00d8a1683e","Type":"ContainerDied","Data":"86d3f8d1de401a7aae85b40dc9204de79dae61ba3270c662f7a8ad0b7822361f"}
Jul 12 22:40:07.071386 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:40:07.071269    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86d3f8d1de401a7aae85b40dc9204de79dae61ba3270c662f7a8ad0b7822361f"
Jul 12 22:40:28.287416 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:40:28.287385    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52pbf_1b1f17bc-a992-4c72-8fdc-a33a3c3649c4/ovn-acl-logging/0.log"
Jul 12 22:40:28.290274 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:40:28.290252    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52pbf_1b1f17bc-a992-4c72-8fdc-a33a3c3649c4/ovn-acl-logging/0.log"
Jul 12 22:42:05.220698 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:05.220617    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"]
Jul 12 22:42:05.221295 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:05.221135    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1daa9cc-c99c-4d25-9160-1f00d8a1683e" containerName="revoke-keys"
Jul 12 22:42:05.221295 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:05.221153    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1daa9cc-c99c-4d25-9160-1f00d8a1683e" containerName="revoke-keys"
Jul 12 22:42:05.221495 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:05.221302    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3020733f-d891-46d3-848a-63b5be41e466" containerName="revoke-keys"
Jul 12 22:42:05.221495 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:05.221316    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1daa9cc-c99c-4d25-9160-1f00d8a1683e" containerName="revoke-keys"
Jul 12 22:42:05.224413 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:05.224393    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"
Jul 12 22:42:05.228051 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:05.228029    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-10dc8c96-openshift-default-dockercfg-6b87m\""
Jul 12 22:42:05.228142 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:05.228043    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 12 22:42:05.255999 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:05.255969    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"]
Jul 12 22:42:05.380250 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:05.380208    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-workload-socket\") pod \"e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9\" (UID: \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\") " pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"
Jul 12 22:42:05.380250 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:05.380251    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-istio-data\") pod \"e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9\" (UID: \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\") " pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"
Jul 12 22:42:05.380491 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:05.380297    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-istiod-ca-cert\") pod \"e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9\" (UID: \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\") " pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"
Jul 12 22:42:05.380491 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:05.380323    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-istio-token\") pod \"e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9\" (UID: \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\") " pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"
Jul 12 22:42:05.380491 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:05.380342    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csbk6\" (UniqueName: \"kubernetes.io/projected/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-kube-api-access-csbk6\") pod \"e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9\" (UID: \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\") " pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"
Jul 12 22:42:05.380491 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:05.380362    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-istio-podinfo\") pod \"e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9\" (UID: \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\") " pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"
Jul 12 22:42:05.380870 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:05.380841    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-workload-certs\") pod \"e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9\" (UID: \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\") " pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"
Jul 12 22:42:05.381117 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:05.381092    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-credential-socket\") pod \"e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9\" (UID: \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\") " pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"
Jul 12 22:42:05.381220 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:05.381155    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-istio-envoy\") pod \"e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9\" (UID: \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\") " pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"
Jul 12 22:42:05.482038 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:05.481948    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-workload-socket\") pod \"e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9\" (UID: \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\") " pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"
Jul 12 22:42:05.482038 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:05.482001    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-istio-data\") pod \"e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9\" (UID: \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\") " pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"
Jul 12 22:42:05.482038 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:05.482027    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-istiod-ca-cert\") pod \"e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9\" (UID: \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\") " pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"
Jul 12 22:42:05.482322 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:05.482043    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-istio-token\") pod \"e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9\" (UID: \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\") " pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"
Jul 12 22:42:05.482322 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:05.482064    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csbk6\" (UniqueName: \"kubernetes.io/projected/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-kube-api-access-csbk6\") pod \"e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9\" (UID: \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\") " pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"
Jul 12 22:42:05.482322 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:05.482089    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-istio-podinfo\") pod \"e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9\" (UID: \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\") " pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"
Jul 12 22:42:05.482322 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:05.482151    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-workload-certs\") pod \"e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9\" (UID: \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\") " pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"
Jul 12 22:42:05.482322 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:05.482186    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-credential-socket\") pod \"e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9\" (UID: \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\") " pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"
Jul 12 22:42:05.482322 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:05.482222    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-istio-envoy\") pod \"e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9\" (UID: \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\") " pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"
Jul 12 22:42:05.482656 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:05.482326    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-workload-socket\") pod \"e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9\" (UID: \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\") " pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"
Jul 12 22:42:05.482656 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:05.482417    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-istio-data\") pod \"e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9\" (UID: \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\") " pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"
Jul 12 22:42:05.482656 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:05.482635    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-workload-certs\") pod \"e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9\" (UID: \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\") " pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"
Jul 12 22:42:05.482790 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:05.482674    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-credential-socket\") pod \"e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9\" (UID: \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\") " pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"
Jul 12 22:42:05.482828 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:05.482798    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-istiod-ca-cert\") pod \"e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9\" (UID: \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\") " pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"
Jul 12 22:42:05.484616 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:05.484596    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-istio-envoy\") pod \"e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9\" (UID: \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\") " pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"
Jul 12 22:42:05.484842 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:05.484823    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-istio-podinfo\") pod \"e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9\" (UID: \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\") " pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"
Jul 12 22:42:05.489954 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:05.489929    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-istio-token\") pod \"e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9\" (UID: \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\") " pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"
Jul 12 22:42:05.490087 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:05.490070    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csbk6\" (UniqueName: \"kubernetes.io/projected/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-kube-api-access-csbk6\") pod \"e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9\" (UID: \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\") " pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"
Jul 12 22:42:05.535455 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:05.535425    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"
Jul 12 22:42:05.684086 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:05.684059    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"]
Jul 12 22:42:05.686869 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:42:05.686840    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbd57940_e1d4_48ac_8b8a_4fbd8a3a5a4b.slice/crio-cab33539f2f769b6849911ed5cf9f981b8f4e9f0c09fa76d743a6ff24fc8a8d3 WatchSource:0}: Error finding container cab33539f2f769b6849911ed5cf9f981b8f4e9f0c09fa76d743a6ff24fc8a8d3: Status 404 returned error can't find the container with id cab33539f2f769b6849911ed5cf9f981b8f4e9f0c09fa76d743a6ff24fc8a8d3
Jul 12 22:42:05.689163 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:05.689124    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:42:05.689293 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:05.689215    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:42:05.689293 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:05.689257    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:42:06.696719 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:06.696682    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9" event={"ID":"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b","Type":"ContainerStarted","Data":"e8ec60b82d5603478173e97670814a8c1d1e5d0ddb35fc717ad7dee409cf4c16"}
Jul 12 22:42:06.697175 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:06.696738    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9" event={"ID":"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b","Type":"ContainerStarted","Data":"cab33539f2f769b6849911ed5cf9f981b8f4e9f0c09fa76d743a6ff24fc8a8d3"}
Jul 12 22:42:06.719405 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:06.719357    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9" podStartSLOduration=1.7193424510000002 podStartE2EDuration="1.719342451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:42:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:42:06.716422697 +0000 UTC m=+1899.029926050" watchObservedRunningTime="2026-07-12 22:42:06.719342451 +0000 UTC m=+1899.032845804"
Jul 12 22:42:07.535780 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:07.535745    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"
Jul 12 22:42:07.541312 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:07.541284    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"
Jul 12 22:42:07.702634 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:07.702603    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"
Jul 12 22:42:07.703691 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:07.703671    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"
Jul 12 22:42:20.720294 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:20.720252    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10dc8c96-gngqr"]
Jul 12 22:42:20.725470 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:20.725446    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10dc8c96-gngqr"
Jul 12 22:42:20.728591 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:20.728565    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 12 22:42:20.728720 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:20.728630    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlrdf\""
Jul 12 22:42:20.735076 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:20.735051    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10dc8c96-gngqr"]
Jul 12 22:42:20.824071 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:20.824032    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8db14621-e88f-4e33-a062-40d12c147f12-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-10dc8c96-gngqr\" (UID: \"8db14621-e88f-4e33-a062-40d12c147f12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10dc8c96-gngqr"
Jul 12 22:42:20.924885 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:20.924849    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8db14621-e88f-4e33-a062-40d12c147f12-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-10dc8c96-gngqr\" (UID: \"8db14621-e88f-4e33-a062-40d12c147f12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10dc8c96-gngqr"
Jul 12 22:42:20.926169 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:20.926136    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8db14621-e88f-4e33-a062-40d12c147f12-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-10dc8c96-gngqr\" (UID: \"8db14621-e88f-4e33-a062-40d12c147f12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10dc8c96-gngqr"
Jul 12 22:42:21.038904 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:21.038818    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10dc8c96-gngqr"
Jul 12 22:42:21.189545 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:21.189502    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10dc8c96-gngqr"]
Jul 12 22:42:21.192009 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:42:21.191981    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8db14621_e88f_4e33_a062_40d12c147f12.slice/crio-476a70645f3e3299c78d7a5443acdc763c3ab0e41e5752ec901ff648f400a801 WatchSource:0}: Error finding container 476a70645f3e3299c78d7a5443acdc763c3ab0e41e5752ec901ff648f400a801: Status 404 returned error can't find the container with id 476a70645f3e3299c78d7a5443acdc763c3ab0e41e5752ec901ff648f400a801
Jul 12 22:42:21.786634 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:21.786545    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10dc8c96-gngqr" event={"ID":"8db14621-e88f-4e33-a062-40d12c147f12","Type":"ContainerStarted","Data":"722639832b1fa3426b5aba0694bce449cfbd75825c1a00716defbd7b30b3dc19"}
Jul 12 22:42:21.786634 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:21.786606    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10dc8c96-gngqr" event={"ID":"8db14621-e88f-4e33-a062-40d12c147f12","Type":"ContainerStarted","Data":"476a70645f3e3299c78d7a5443acdc763c3ab0e41e5752ec901ff648f400a801"}
Jul 12 22:42:21.806960 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:21.806895    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10dc8c96-gngqr" podStartSLOduration=1.806874633 podStartE2EDuration="1.806874633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:42:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:42:21.804962545 +0000 UTC m=+1914.118465894" watchObservedRunningTime="2026-07-12 22:42:21.806874633 +0000 UTC m=+1914.120377990"
Jul 12 22:42:22.703641 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:22.703605    2560 patch_prober.go:28] interesting pod/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Jul 12 22:42:22.703864 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:22.703658    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9" podUID="cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Jul 12 22:42:22.793233 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:22.793196    2560 generic.go:356] "Generic (PLEG): container finished" podID="8db14621-e88f-4e33-a062-40d12c147f12" containerID="722639832b1fa3426b5aba0694bce449cfbd75825c1a00716defbd7b30b3dc19" exitCode=0
Jul 12 22:42:22.793682 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:22.793288    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10dc8c96-gngqr" event={"ID":"8db14621-e88f-4e33-a062-40d12c147f12","Type":"ContainerDied","Data":"722639832b1fa3426b5aba0694bce449cfbd75825c1a00716defbd7b30b3dc19"}
Jul 12 22:42:22.795042 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:22.795016    2560 generic.go:356] "Generic (PLEG): container finished" podID="cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b" containerID="e8ec60b82d5603478173e97670814a8c1d1e5d0ddb35fc717ad7dee409cf4c16" exitCode=0
Jul 12 22:42:22.795200 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:22.795093    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9" event={"ID":"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b","Type":"ContainerDied","Data":"e8ec60b82d5603478173e97670814a8c1d1e5d0ddb35fc717ad7dee409cf4c16"}
Jul 12 22:42:22.795487 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:22.795473    2560 scope.go:117] "RemoveContainer" containerID="e8ec60b82d5603478173e97670814a8c1d1e5d0ddb35fc717ad7dee409cf4c16"
Jul 12 22:42:22.796512 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:22.796485    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:42:22.796670 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:22.796588    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:42:22.796670 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:22.796638    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:42:23.802137 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:23.802098    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9" event={"ID":"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b","Type":"ContainerStarted","Data":"94a446e8618b3ead4457117772f8415a859ea6d8c39c214538a9ac572486845a"}
Jul 12 22:42:23.991865 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:23.991836    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10dc8c96-gngqr"
Jul 12 22:42:24.053437 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:24.053329    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8db14621-e88f-4e33-a062-40d12c147f12-maas-api-service-ca\") pod \"8db14621-e88f-4e33-a062-40d12c147f12\" (UID: \"8db14621-e88f-4e33-a062-40d12c147f12\") "
Jul 12 22:42:24.053778 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:24.053750    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8db14621-e88f-4e33-a062-40d12c147f12-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8db14621-e88f-4e33-a062-40d12c147f12" (UID: "8db14621-e88f-4e33-a062-40d12c147f12"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 12 22:42:24.155020 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:24.154979    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8db14621-e88f-4e33-a062-40d12c147f12-maas-api-service-ca\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:42:24.536656 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:24.536617    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"
Jul 12 22:42:24.536865 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:24.536784    2560 patch_prober.go:28] interesting pod/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Jul 12 22:42:24.536865 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:24.536832    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9" podUID="cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Jul 12 22:42:24.808970 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:24.808889    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10dc8c96-gngqr"
Jul 12 22:42:24.808970 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:24.808923    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10dc8c96-gngqr" event={"ID":"8db14621-e88f-4e33-a062-40d12c147f12","Type":"ContainerDied","Data":"476a70645f3e3299c78d7a5443acdc763c3ab0e41e5752ec901ff648f400a801"}
Jul 12 22:42:24.808970 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:24.808968    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="476a70645f3e3299c78d7a5443acdc763c3ab0e41e5752ec901ff648f400a801"
Jul 12 22:42:25.536639 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:25.536597    2560 patch_prober.go:28] interesting pod/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Jul 12 22:42:25.536848 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:25.536668    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9" podUID="cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Jul 12 22:42:26.536335 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:26.536286    2560 patch_prober.go:28] interesting pod/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Jul 12 22:42:26.536838 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:26.536371    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9" podUID="cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Jul 12 22:42:27.536558 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:27.535848    2560 patch_prober.go:28] interesting pod/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Jul 12 22:42:27.536558 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:27.535936    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9" podUID="cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Jul 12 22:42:28.396627 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:28.396593    2560 scope.go:117] "RemoveContainer" containerID="4c72de6686e1888225c102cf2eef3b13c11c5adb12868dc53491b7351bae6789"
Jul 12 22:42:28.540757 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:28.540730    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"
Jul 12 22:42:28.541253 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:28.541234    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"
Jul 12 22:42:28.542114 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:28.542092    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"
Jul 12 22:42:46.469502 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:46.469467    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"]
Jul 12 22:42:46.470399 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:46.470326    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9" podUID="cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b" containerName="istio-proxy" containerID="cri-o://94a446e8618b3ead4457117772f8415a859ea6d8c39c214538a9ac572486845a" gracePeriod=30
Jul 12 22:42:51.730902 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:51.730875    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"
Jul 12 22:42:51.808137 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:51.808052    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csbk6\" (UniqueName: \"kubernetes.io/projected/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-kube-api-access-csbk6\") pod \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\" (UID: \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\") "
Jul 12 22:42:51.808137 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:51.808091    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-istio-envoy\") pod \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\" (UID: \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\") "
Jul 12 22:42:51.808137 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:51.808115    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-workload-certs\") pod \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\" (UID: \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\") "
Jul 12 22:42:51.808406 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:51.808160    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-istio-podinfo\") pod \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\" (UID: \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\") "
Jul 12 22:42:51.808406 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:51.808179    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-istio-data\") pod \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\" (UID: \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\") "
Jul 12 22:42:51.808406 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:51.808205    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-istio-token\") pod \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\" (UID: \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\") "
Jul 12 22:42:51.808406 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:51.808249    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-istiod-ca-cert\") pod \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\" (UID: \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\") "
Jul 12 22:42:51.808635 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:51.808565    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b" (UID: "cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:42:51.808687 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:51.808658    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b" (UID: "cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 12 22:42:51.808745 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:51.808685    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-credential-socket\") pod \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\" (UID: \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\") "
Jul 12 22:42:51.808807 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:51.808769    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-workload-socket\") pod \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\" (UID: \"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b\") "
Jul 12 22:42:51.808885 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:51.808859    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b" (UID: "cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:42:51.809222 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:51.808987    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b" (UID: "cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:42:51.809490 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:51.809461    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-istio-data" (OuterVolumeSpecName: "istio-data") pod "cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b" (UID: "cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:42:51.809683 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:51.809662    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-istio-data\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:42:51.809729 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:51.809692    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-istiod-ca-cert\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:42:51.809729 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:51.809707    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-credential-socket\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:42:51.809729 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:51.809720    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-workload-socket\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:42:51.809824 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:51.809737    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-workload-certs\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:42:51.810513 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:51.810487    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b" (UID: "cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:42:51.810779 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:51.810761    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-istio-token" (OuterVolumeSpecName: "istio-token") pod "cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b" (UID: "cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 22:42:51.810929 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:51.810909    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b" (UID: "cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 12 22:42:51.810983 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:51.810927    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-kube-api-access-csbk6" (OuterVolumeSpecName: "kube-api-access-csbk6") pod "cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b" (UID: "cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b"). InnerVolumeSpecName "kube-api-access-csbk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 22:42:51.910849 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:51.910815    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csbk6\" (UniqueName: \"kubernetes.io/projected/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-kube-api-access-csbk6\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:42:51.910849 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:51.910844    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-istio-envoy\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:42:51.910849 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:51.910856    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-istio-podinfo\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:42:51.911080 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:51.910868    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b-istio-token\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:42:51.949683 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:51.949648    2560 generic.go:356] "Generic (PLEG): container finished" podID="cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b" containerID="94a446e8618b3ead4457117772f8415a859ea6d8c39c214538a9ac572486845a" exitCode=0
Jul 12 22:42:51.949845 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:51.949711    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"
Jul 12 22:42:51.949845 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:51.949735    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9" event={"ID":"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b","Type":"ContainerDied","Data":"94a446e8618b3ead4457117772f8415a859ea6d8c39c214538a9ac572486845a"}
Jul 12 22:42:51.949845 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:51.949834    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9" event={"ID":"cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b","Type":"ContainerDied","Data":"cab33539f2f769b6849911ed5cf9f981b8f4e9f0c09fa76d743a6ff24fc8a8d3"}
Jul 12 22:42:51.949974 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:51.949865    2560 scope.go:117] "RemoveContainer" containerID="94a446e8618b3ead4457117772f8415a859ea6d8c39c214538a9ac572486845a"
Jul 12 22:42:51.959787 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:51.959771    2560 scope.go:117] "RemoveContainer" containerID="e8ec60b82d5603478173e97670814a8c1d1e5d0ddb35fc717ad7dee409cf4c16"
Jul 12 22:42:51.967958 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:51.967942    2560 scope.go:117] "RemoveContainer" containerID="94a446e8618b3ead4457117772f8415a859ea6d8c39c214538a9ac572486845a"
Jul 12 22:42:51.968206 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:42:51.968188    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94a446e8618b3ead4457117772f8415a859ea6d8c39c214538a9ac572486845a\": container with ID starting with 94a446e8618b3ead4457117772f8415a859ea6d8c39c214538a9ac572486845a not found: ID does not exist" containerID="94a446e8618b3ead4457117772f8415a859ea6d8c39c214538a9ac572486845a"
Jul 12 22:42:51.968246 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:51.968215    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94a446e8618b3ead4457117772f8415a859ea6d8c39c214538a9ac572486845a"} err="failed to get container status \"94a446e8618b3ead4457117772f8415a859ea6d8c39c214538a9ac572486845a\": rpc error: code = NotFound desc = could not find container \"94a446e8618b3ead4457117772f8415a859ea6d8c39c214538a9ac572486845a\": container with ID starting with 94a446e8618b3ead4457117772f8415a859ea6d8c39c214538a9ac572486845a not found: ID does not exist"
Jul 12 22:42:51.968246 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:51.968231    2560 scope.go:117] "RemoveContainer" containerID="e8ec60b82d5603478173e97670814a8c1d1e5d0ddb35fc717ad7dee409cf4c16"
Jul 12 22:42:51.968439 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:42:51.968422    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8ec60b82d5603478173e97670814a8c1d1e5d0ddb35fc717ad7dee409cf4c16\": container with ID starting with e8ec60b82d5603478173e97670814a8c1d1e5d0ddb35fc717ad7dee409cf4c16 not found: ID does not exist" containerID="e8ec60b82d5603478173e97670814a8c1d1e5d0ddb35fc717ad7dee409cf4c16"
Jul 12 22:42:51.968475 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:51.968444    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8ec60b82d5603478173e97670814a8c1d1e5d0ddb35fc717ad7dee409cf4c16"} err="failed to get container status \"e8ec60b82d5603478173e97670814a8c1d1e5d0ddb35fc717ad7dee409cf4c16\": rpc error: code = NotFound desc = could not find container \"e8ec60b82d5603478173e97670814a8c1d1e5d0ddb35fc717ad7dee409cf4c16\": container with ID starting with e8ec60b82d5603478173e97670814a8c1d1e5d0ddb35fc717ad7dee409cf4c16 not found: ID does not exist"
Jul 12 22:42:51.972813 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:51.972790    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"]
Jul 12 22:42:51.976480 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:51.976460    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-10dc8c96-openshift-default-6c67488bcf-k92j9"]
Jul 12 22:42:52.246037 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:42:52.245995    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b" path="/var/lib/kubelet/pods/cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b/volumes"
Jul 12 22:43:06.056033 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:06.056003    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"]
Jul 12 22:43:06.056421 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:06.056374    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b" containerName="istio-proxy"
Jul 12 22:43:06.056421 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:06.056386    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b" containerName="istio-proxy"
Jul 12 22:43:06.056421 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:06.056401    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8db14621-e88f-4e33-a062-40d12c147f12" containerName="revoke-keys"
Jul 12 22:43:06.056421 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:06.056406    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8db14621-e88f-4e33-a062-40d12c147f12" containerName="revoke-keys"
Jul 12 22:43:06.056677 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:06.056508    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b" containerName="istio-proxy"
Jul 12 22:43:06.056677 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:06.056520    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8db14621-e88f-4e33-a062-40d12c147f12" containerName="revoke-keys"
Jul 12 22:43:06.059719 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:06.059698    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"
Jul 12 22:43:06.062474 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:06.062450    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 12 22:43:06.062474 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:06.062469    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3f30527e-openshift-default-dockercfg-gwqkq\""
Jul 12 22:43:06.071122 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:06.071100    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"]
Jul 12 22:43:06.143436 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:06.143397    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eff55a5d-0b24-44c1-a418-a65a49893df5-credential-socket\") pod \"e2e-mt-3f30527e-openshift-default-d64b87999-5v47w\" (UID: \"eff55a5d-0b24-44c1-a418-a65a49893df5\") " pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"
Jul 12 22:43:06.143647 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:06.143449    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eff55a5d-0b24-44c1-a418-a65a49893df5-istiod-ca-cert\") pod \"e2e-mt-3f30527e-openshift-default-d64b87999-5v47w\" (UID: \"eff55a5d-0b24-44c1-a418-a65a49893df5\") " pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"
Jul 12 22:43:06.143647 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:06.143510    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eff55a5d-0b24-44c1-a418-a65a49893df5-istio-podinfo\") pod \"e2e-mt-3f30527e-openshift-default-d64b87999-5v47w\" (UID: \"eff55a5d-0b24-44c1-a418-a65a49893df5\") " pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"
Jul 12 22:43:06.143647 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:06.143584    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eff55a5d-0b24-44c1-a418-a65a49893df5-istio-envoy\") pod \"e2e-mt-3f30527e-openshift-default-d64b87999-5v47w\" (UID: \"eff55a5d-0b24-44c1-a418-a65a49893df5\") " pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"
Jul 12 22:43:06.143647 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:06.143610    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eff55a5d-0b24-44c1-a418-a65a49893df5-istio-token\") pod \"e2e-mt-3f30527e-openshift-default-d64b87999-5v47w\" (UID: \"eff55a5d-0b24-44c1-a418-a65a49893df5\") " pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"
Jul 12 22:43:06.143647 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:06.143636    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eff55a5d-0b24-44c1-a418-a65a49893df5-workload-certs\") pod \"e2e-mt-3f30527e-openshift-default-d64b87999-5v47w\" (UID: \"eff55a5d-0b24-44c1-a418-a65a49893df5\") " pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"
Jul 12 22:43:06.143840 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:06.143681    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eff55a5d-0b24-44c1-a418-a65a49893df5-istio-data\") pod \"e2e-mt-3f30527e-openshift-default-d64b87999-5v47w\" (UID: \"eff55a5d-0b24-44c1-a418-a65a49893df5\") " pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"
Jul 12 22:43:06.143840 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:06.143704    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tvl6\" (UniqueName: \"kubernetes.io/projected/eff55a5d-0b24-44c1-a418-a65a49893df5-kube-api-access-8tvl6\") pod \"e2e-mt-3f30527e-openshift-default-d64b87999-5v47w\" (UID: \"eff55a5d-0b24-44c1-a418-a65a49893df5\") " pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"
Jul 12 22:43:06.143840 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:06.143726    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eff55a5d-0b24-44c1-a418-a65a49893df5-workload-socket\") pod \"e2e-mt-3f30527e-openshift-default-d64b87999-5v47w\" (UID: \"eff55a5d-0b24-44c1-a418-a65a49893df5\") " pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"
Jul 12 22:43:06.244161 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:06.244127    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eff55a5d-0b24-44c1-a418-a65a49893df5-istio-podinfo\") pod \"e2e-mt-3f30527e-openshift-default-d64b87999-5v47w\" (UID: \"eff55a5d-0b24-44c1-a418-a65a49893df5\") " pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"
Jul 12 22:43:06.244360 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:06.244181    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eff55a5d-0b24-44c1-a418-a65a49893df5-istio-envoy\") pod \"e2e-mt-3f30527e-openshift-default-d64b87999-5v47w\" (UID: \"eff55a5d-0b24-44c1-a418-a65a49893df5\") " pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"
Jul 12 22:43:06.244360 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:06.244212    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eff55a5d-0b24-44c1-a418-a65a49893df5-istio-token\") pod \"e2e-mt-3f30527e-openshift-default-d64b87999-5v47w\" (UID: \"eff55a5d-0b24-44c1-a418-a65a49893df5\") " pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"
Jul 12 22:43:06.244360 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:06.244248    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eff55a5d-0b24-44c1-a418-a65a49893df5-workload-certs\") pod \"e2e-mt-3f30527e-openshift-default-d64b87999-5v47w\" (UID: \"eff55a5d-0b24-44c1-a418-a65a49893df5\") " pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"
Jul 12 22:43:06.244360 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:06.244286    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eff55a5d-0b24-44c1-a418-a65a49893df5-istio-data\") pod \"e2e-mt-3f30527e-openshift-default-d64b87999-5v47w\" (UID: \"eff55a5d-0b24-44c1-a418-a65a49893df5\") " pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"
Jul 12 22:43:06.244360 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:06.244321    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tvl6\" (UniqueName: \"kubernetes.io/projected/eff55a5d-0b24-44c1-a418-a65a49893df5-kube-api-access-8tvl6\") pod \"e2e-mt-3f30527e-openshift-default-d64b87999-5v47w\" (UID: \"eff55a5d-0b24-44c1-a418-a65a49893df5\") " pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"
Jul 12 22:43:06.244360 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:06.244352    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eff55a5d-0b24-44c1-a418-a65a49893df5-workload-socket\") pod \"e2e-mt-3f30527e-openshift-default-d64b87999-5v47w\" (UID: \"eff55a5d-0b24-44c1-a418-a65a49893df5\") " pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"
Jul 12 22:43:06.244752 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:06.244590    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eff55a5d-0b24-44c1-a418-a65a49893df5-credential-socket\") pod \"e2e-mt-3f30527e-openshift-default-d64b87999-5v47w\" (UID: \"eff55a5d-0b24-44c1-a418-a65a49893df5\") " pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"
Jul 12 22:43:06.244752 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:06.244662    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eff55a5d-0b24-44c1-a418-a65a49893df5-istiod-ca-cert\") pod \"e2e-mt-3f30527e-openshift-default-d64b87999-5v47w\" (UID: \"eff55a5d-0b24-44c1-a418-a65a49893df5\") " pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"
Jul 12 22:43:06.244857 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:06.244761    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eff55a5d-0b24-44c1-a418-a65a49893df5-workload-socket\") pod \"e2e-mt-3f30527e-openshift-default-d64b87999-5v47w\" (UID: \"eff55a5d-0b24-44c1-a418-a65a49893df5\") " pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"
Jul 12 22:43:06.244918 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:06.244850    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eff55a5d-0b24-44c1-a418-a65a49893df5-workload-certs\") pod \"e2e-mt-3f30527e-openshift-default-d64b87999-5v47w\" (UID: \"eff55a5d-0b24-44c1-a418-a65a49893df5\") " pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"
Jul 12 22:43:06.245037 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:06.244928    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eff55a5d-0b24-44c1-a418-a65a49893df5-credential-socket\") pod \"e2e-mt-3f30527e-openshift-default-d64b87999-5v47w\" (UID: \"eff55a5d-0b24-44c1-a418-a65a49893df5\") " pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"
Jul 12 22:43:06.245226 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:06.245203    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eff55a5d-0b24-44c1-a418-a65a49893df5-istio-data\") pod \"e2e-mt-3f30527e-openshift-default-d64b87999-5v47w\" (UID: \"eff55a5d-0b24-44c1-a418-a65a49893df5\") " pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"
Jul 12 22:43:06.245291 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:06.245257    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eff55a5d-0b24-44c1-a418-a65a49893df5-istiod-ca-cert\") pod \"e2e-mt-3f30527e-openshift-default-d64b87999-5v47w\" (UID: \"eff55a5d-0b24-44c1-a418-a65a49893df5\") " pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"
Jul 12 22:43:06.246818 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:06.246797    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eff55a5d-0b24-44c1-a418-a65a49893df5-istio-envoy\") pod \"e2e-mt-3f30527e-openshift-default-d64b87999-5v47w\" (UID: \"eff55a5d-0b24-44c1-a418-a65a49893df5\") " pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"
Jul 12 22:43:06.247026 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:06.247009    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eff55a5d-0b24-44c1-a418-a65a49893df5-istio-podinfo\") pod \"e2e-mt-3f30527e-openshift-default-d64b87999-5v47w\" (UID: \"eff55a5d-0b24-44c1-a418-a65a49893df5\") " pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"
Jul 12 22:43:06.252044 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:06.252019    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eff55a5d-0b24-44c1-a418-a65a49893df5-istio-token\") pod \"e2e-mt-3f30527e-openshift-default-d64b87999-5v47w\" (UID: \"eff55a5d-0b24-44c1-a418-a65a49893df5\") " pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"
Jul 12 22:43:06.252239 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:06.252221    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tvl6\" (UniqueName: \"kubernetes.io/projected/eff55a5d-0b24-44c1-a418-a65a49893df5-kube-api-access-8tvl6\") pod \"e2e-mt-3f30527e-openshift-default-d64b87999-5v47w\" (UID: \"eff55a5d-0b24-44c1-a418-a65a49893df5\") " pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"
Jul 12 22:43:06.371243 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:06.371208    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"
Jul 12 22:43:06.511301 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:06.511272    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"]
Jul 12 22:43:06.513357 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:43:06.513325    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeff55a5d_0b24_44c1_a418_a65a49893df5.slice/crio-c08190ffbab799e6d5bade00df42ecdbc14c4354ad13c4f44a2f1c4ae72ed4c9 WatchSource:0}: Error finding container c08190ffbab799e6d5bade00df42ecdbc14c4354ad13c4f44a2f1c4ae72ed4c9: Status 404 returned error can't find the container with id c08190ffbab799e6d5bade00df42ecdbc14c4354ad13c4f44a2f1c4ae72ed4c9
Jul 12 22:43:06.515436 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:06.515402    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:43:06.515506 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:06.515468    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:43:06.515506 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:06.515497    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:43:07.032667 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:07.032636    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w" event={"ID":"eff55a5d-0b24-44c1-a418-a65a49893df5","Type":"ContainerStarted","Data":"ab841cdaddc1896a8da3b75ba9efa44ea9bd7ee783a482ed1bd51e6b067a9a0a"}
Jul 12 22:43:07.032851 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:07.032697    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w" event={"ID":"eff55a5d-0b24-44c1-a418-a65a49893df5","Type":"ContainerStarted","Data":"c08190ffbab799e6d5bade00df42ecdbc14c4354ad13c4f44a2f1c4ae72ed4c9"}
Jul 12 22:43:07.055640 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:07.055589    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w" podStartSLOduration=1.05557018 podStartE2EDuration="1.05557018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:43:07.053837315 +0000 UTC m=+1959.367340662" watchObservedRunningTime="2026-07-12 22:43:07.05557018 +0000 UTC m=+1959.369073538"
Jul 12 22:43:07.371859 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:07.371824    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"
Jul 12 22:43:07.377106 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:07.377082    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"
Jul 12 22:43:08.038582 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:08.038526    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"
Jul 12 22:43:08.039470 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:08.039450    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"
Jul 12 22:43:23.919099 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:23.917652    2560 patch_prober.go:28] interesting pod/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": read tcp 10.133.0.2:53056->10.133.0.57:15021: read: connection reset by peer" start-of-body=
Jul 12 22:43:23.919099 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:23.917712    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w" podUID="eff55a5d-0b24-44c1-a418-a65a49893df5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": read tcp 10.133.0.2:53056->10.133.0.57:15021: read: connection reset by peer"
Jul 12 22:43:24.089791 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:24.089753    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-3f30527e/e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq"]
Jul 12 22:43:24.090312 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:24.090290    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b" containerName="istio-proxy"
Jul 12 22:43:24.090312 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:24.090312    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b" containerName="istio-proxy"
Jul 12 22:43:24.090480 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:24.090408    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbd57940-e1d4-48ac-8b8a-4fbd8a3a5a4b" containerName="istio-proxy"
Jul 12 22:43:24.093449 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:24.093432    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3f30527e/e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq"
Jul 12 22:43:24.097246 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:24.097204    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3f30527e\"/\"default-dockercfg-4k9hd\""
Jul 12 22:43:24.097246 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:24.097209    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3f30527e\"/\"kube-root-ca.crt\""
Jul 12 22:43:24.097948 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:24.097930    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3f30527e\"/\"openshift-service-ca.crt\""
Jul 12 22:43:24.098100 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:24.098080    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3f30527e\"/\"e2e-lifecycle-model-3f30527e-kserve-self-signed-certs\""
Jul 12 22:43:24.107345 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:24.107319    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3f30527e/e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq"]
Jul 12 22:43:24.124333 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:24.124296    2560 generic.go:356] "Generic (PLEG): container finished" podID="eff55a5d-0b24-44c1-a418-a65a49893df5" containerID="ab841cdaddc1896a8da3b75ba9efa44ea9bd7ee783a482ed1bd51e6b067a9a0a" exitCode=0
Jul 12 22:43:24.124495 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:24.124362    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w" event={"ID":"eff55a5d-0b24-44c1-a418-a65a49893df5","Type":"ContainerDied","Data":"ab841cdaddc1896a8da3b75ba9efa44ea9bd7ee783a482ed1bd51e6b067a9a0a"}
Jul 12 22:43:24.124809 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:24.124783    2560 scope.go:117] "RemoveContainer" containerID="ab841cdaddc1896a8da3b75ba9efa44ea9bd7ee783a482ed1bd51e6b067a9a0a"
Jul 12 22:43:24.126011 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:24.125977    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:43:24.126107 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:24.126062    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:43:24.126190 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:24.126113    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:43:24.206848 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:24.206813    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn6vp\" (UniqueName: \"kubernetes.io/projected/f15a9c8d-6582-463e-8f98-c38eaa6e0a36-kube-api-access-zn6vp\") pod \"e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq\" (UID: \"f15a9c8d-6582-463e-8f98-c38eaa6e0a36\") " pod="ai-tenant-e2e-mt-3f30527e/e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq"
Jul 12 22:43:24.207007 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:24.206987    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f15a9c8d-6582-463e-8f98-c38eaa6e0a36-tls-certs\") pod \"e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq\" (UID: \"f15a9c8d-6582-463e-8f98-c38eaa6e0a36\") " pod="ai-tenant-e2e-mt-3f30527e/e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq"
Jul 12 22:43:24.207062 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:24.207028    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f15a9c8d-6582-463e-8f98-c38eaa6e0a36-dshm\") pod \"e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq\" (UID: \"f15a9c8d-6582-463e-8f98-c38eaa6e0a36\") " pod="ai-tenant-e2e-mt-3f30527e/e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq"
Jul 12 22:43:24.207137 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:24.207114    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f15a9c8d-6582-463e-8f98-c38eaa6e0a36-model-cache\") pod \"e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq\" (UID: \"f15a9c8d-6582-463e-8f98-c38eaa6e0a36\") " pod="ai-tenant-e2e-mt-3f30527e/e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq"
Jul 12 22:43:24.207199 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:24.207178    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f15a9c8d-6582-463e-8f98-c38eaa6e0a36-home\") pod \"e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq\" (UID: \"f15a9c8d-6582-463e-8f98-c38eaa6e0a36\") " pod="ai-tenant-e2e-mt-3f30527e/e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq"
Jul 12 22:43:24.207256 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:24.207209    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f15a9c8d-6582-463e-8f98-c38eaa6e0a36-tmp-dir\") pod \"e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq\" (UID: \"f15a9c8d-6582-463e-8f98-c38eaa6e0a36\") " pod="ai-tenant-e2e-mt-3f30527e/e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq"
Jul 12 22:43:24.308498 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:24.308463    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn6vp\" (UniqueName: \"kubernetes.io/projected/f15a9c8d-6582-463e-8f98-c38eaa6e0a36-kube-api-access-zn6vp\") pod \"e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq\" (UID: \"f15a9c8d-6582-463e-8f98-c38eaa6e0a36\") " pod="ai-tenant-e2e-mt-3f30527e/e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq"
Jul 12 22:43:24.308702 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:24.308579    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f15a9c8d-6582-463e-8f98-c38eaa6e0a36-tls-certs\") pod \"e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq\" (UID: \"f15a9c8d-6582-463e-8f98-c38eaa6e0a36\") " pod="ai-tenant-e2e-mt-3f30527e/e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq"
Jul 12 22:43:24.308702 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:24.308622    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f15a9c8d-6582-463e-8f98-c38eaa6e0a36-dshm\") pod \"e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq\" (UID: \"f15a9c8d-6582-463e-8f98-c38eaa6e0a36\") " pod="ai-tenant-e2e-mt-3f30527e/e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq"
Jul 12 22:43:24.308702 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:24.308652    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f15a9c8d-6582-463e-8f98-c38eaa6e0a36-model-cache\") pod \"e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq\" (UID: \"f15a9c8d-6582-463e-8f98-c38eaa6e0a36\") " pod="ai-tenant-e2e-mt-3f30527e/e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq"
Jul 12 22:43:24.308702 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:24.308675    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f15a9c8d-6582-463e-8f98-c38eaa6e0a36-home\") pod \"e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq\" (UID: \"f15a9c8d-6582-463e-8f98-c38eaa6e0a36\") " pod="ai-tenant-e2e-mt-3f30527e/e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq"
Jul 12 22:43:24.308702 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:24.308697    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f15a9c8d-6582-463e-8f98-c38eaa6e0a36-tmp-dir\") pod \"e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq\" (UID: \"f15a9c8d-6582-463e-8f98-c38eaa6e0a36\") " pod="ai-tenant-e2e-mt-3f30527e/e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq"
Jul 12 22:43:24.309132 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:24.309110    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f15a9c8d-6582-463e-8f98-c38eaa6e0a36-tmp-dir\") pod \"e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq\" (UID: \"f15a9c8d-6582-463e-8f98-c38eaa6e0a36\") " pod="ai-tenant-e2e-mt-3f30527e/e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq"
Jul 12 22:43:24.309245 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:24.309172    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f15a9c8d-6582-463e-8f98-c38eaa6e0a36-home\") pod \"e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq\" (UID: \"f15a9c8d-6582-463e-8f98-c38eaa6e0a36\") " pod="ai-tenant-e2e-mt-3f30527e/e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq"
Jul 12 22:43:24.309245 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:24.309221    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f15a9c8d-6582-463e-8f98-c38eaa6e0a36-model-cache\") pod \"e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq\" (UID: \"f15a9c8d-6582-463e-8f98-c38eaa6e0a36\") " pod="ai-tenant-e2e-mt-3f30527e/e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq"
Jul 12 22:43:24.311577 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:24.311524    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f15a9c8d-6582-463e-8f98-c38eaa6e0a36-tls-certs\") pod \"e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq\" (UID: \"f15a9c8d-6582-463e-8f98-c38eaa6e0a36\") " pod="ai-tenant-e2e-mt-3f30527e/e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq"
Jul 12 22:43:24.311703 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:24.311682    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f15a9c8d-6582-463e-8f98-c38eaa6e0a36-dshm\") pod \"e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq\" (UID: \"f15a9c8d-6582-463e-8f98-c38eaa6e0a36\") " pod="ai-tenant-e2e-mt-3f30527e/e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq"
Jul 12 22:43:24.318735 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:24.318704    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn6vp\" (UniqueName: \"kubernetes.io/projected/f15a9c8d-6582-463e-8f98-c38eaa6e0a36-kube-api-access-zn6vp\") pod \"e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq\" (UID: \"f15a9c8d-6582-463e-8f98-c38eaa6e0a36\") " pod="ai-tenant-e2e-mt-3f30527e/e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq"
Jul 12 22:43:24.402619 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:24.402581    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3f30527e/e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq"
Jul 12 22:43:24.551134 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:24.551101    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3f30527e/e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq"]
Jul 12 22:43:24.554590 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:43:24.554560    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf15a9c8d_6582_463e_8f98_c38eaa6e0a36.slice/crio-c5307aa988cf99269f17cd66c52c7bca56c3c5d6ca6252141e04a956d0ce6e69 WatchSource:0}: Error finding container c5307aa988cf99269f17cd66c52c7bca56c3c5d6ca6252141e04a956d0ce6e69: Status 404 returned error can't find the container with id c5307aa988cf99269f17cd66c52c7bca56c3c5d6ca6252141e04a956d0ce6e69
Jul 12 22:43:25.135471 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:25.135437    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w" event={"ID":"eff55a5d-0b24-44c1-a418-a65a49893df5","Type":"ContainerStarted","Data":"859a8ee602361a17a7dd758fe39bac05dd2247e008a2c1c0d5a112bdbc490778"}
Jul 12 22:43:25.138984 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:25.138944    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3f30527e/e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq" event={"ID":"f15a9c8d-6582-463e-8f98-c38eaa6e0a36","Type":"ContainerStarted","Data":"13a03735e35ab678b5a7090bf0b392006e7766ed3afb450ff866d8fc48d97f45"}
Jul 12 22:43:25.139130 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:25.138997    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3f30527e/e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq" event={"ID":"f15a9c8d-6582-463e-8f98-c38eaa6e0a36","Type":"ContainerStarted","Data":"c5307aa988cf99269f17cd66c52c7bca56c3c5d6ca6252141e04a956d0ce6e69"}
Jul 12 22:43:25.180323 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:25.180265    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-3f30527e/e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq" podStartSLOduration=1.180250055 podStartE2EDuration="1.180250055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:43:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:43:25.179397901 +0000 UTC m=+1977.492901255" watchObservedRunningTime="2026-07-12 22:43:25.180250055 +0000 UTC m=+1977.493753467"
Jul 12 22:43:25.372238 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:25.372196    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"
Jul 12 22:43:25.372435 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:25.372411    2560 patch_prober.go:28] interesting pod/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Jul 12 22:43:25.372476 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:25.372460    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w" podUID="eff55a5d-0b24-44c1-a418-a65a49893df5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Jul 12 22:43:26.371662 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:26.371627    2560 patch_prober.go:28] interesting pod/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Jul 12 22:43:26.372073 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:26.371696    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w" podUID="eff55a5d-0b24-44c1-a418-a65a49893df5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Jul 12 22:43:27.371677 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:27.371632    2560 patch_prober.go:28] interesting pod/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Jul 12 22:43:27.372096 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:27.371700    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w" podUID="eff55a5d-0b24-44c1-a418-a65a49893df5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Jul 12 22:43:28.372183 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:28.372148    2560 patch_prober.go:28] interesting pod/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Jul 12 22:43:28.372582 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:28.372204    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w" podUID="eff55a5d-0b24-44c1-a418-a65a49893df5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Jul 12 22:43:29.375473 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:29.375443    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"
Jul 12 22:43:29.375870 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:29.375727    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"
Jul 12 22:43:29.376363 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:29.376348    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"
Jul 12 22:43:34.403661 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:34.403615    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-3f30527e/e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq"
Jul 12 22:43:34.404109 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:34.403678    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-3f30527e/e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq"
Jul 12 22:43:34.416518 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:34.416493    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-3f30527e/e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq"
Jul 12 22:43:35.209860 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:35.209826    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-3f30527e/e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq"
Jul 12 22:43:56.935139 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:56.935094    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f30527e-kfzfg"]
Jul 12 22:43:56.939105 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:56.939081    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f30527e-kfzfg"
Jul 12 22:43:56.941867 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:56.941842    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 12 22:43:56.941867 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:56.941862    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlrdf\""
Jul 12 22:43:56.947032 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:56.947005    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f30527e-kfzfg"]
Jul 12 22:43:57.001849 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:57.001806    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f1c44de-eda6-4069-8c1f-2078bdc2e4f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3f30527e-kfzfg\" (UID: \"6f1c44de-eda6-4069-8c1f-2078bdc2e4f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f30527e-kfzfg"
Jul 12 22:43:57.102460 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:57.102416    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f1c44de-eda6-4069-8c1f-2078bdc2e4f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3f30527e-kfzfg\" (UID: \"6f1c44de-eda6-4069-8c1f-2078bdc2e4f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f30527e-kfzfg"
Jul 12 22:43:57.103109 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:57.103088    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f1c44de-eda6-4069-8c1f-2078bdc2e4f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3f30527e-kfzfg\" (UID: \"6f1c44de-eda6-4069-8c1f-2078bdc2e4f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f30527e-kfzfg"
Jul 12 22:43:57.253118 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:57.253031    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f30527e-kfzfg"
Jul 12 22:43:57.404905 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:57.404879    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f30527e-kfzfg"]
Jul 12 22:43:57.407351 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:43:57.407314    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f1c44de_eda6_4069_8c1f_2078bdc2e4f9.slice/crio-185d3955274fcc2e32e09f74077383ae3f75b814a34e1874e2bad1980d5c696e WatchSource:0}: Error finding container 185d3955274fcc2e32e09f74077383ae3f75b814a34e1874e2bad1980d5c696e: Status 404 returned error can't find the container with id 185d3955274fcc2e32e09f74077383ae3f75b814a34e1874e2bad1980d5c696e
Jul 12 22:43:58.329481 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:58.329450    2560 generic.go:356] "Generic (PLEG): container finished" podID="6f1c44de-eda6-4069-8c1f-2078bdc2e4f9" containerID="f93d89ecee39dc59a8e3e72e2963e13124adefa55b8d3870b2fc3de8db83a48b" exitCode=0
Jul 12 22:43:58.329911 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:58.329553    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f30527e-kfzfg" event={"ID":"6f1c44de-eda6-4069-8c1f-2078bdc2e4f9","Type":"ContainerDied","Data":"f93d89ecee39dc59a8e3e72e2963e13124adefa55b8d3870b2fc3de8db83a48b"}
Jul 12 22:43:58.329911 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:58.329616    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f30527e-kfzfg" event={"ID":"6f1c44de-eda6-4069-8c1f-2078bdc2e4f9","Type":"ContainerStarted","Data":"185d3955274fcc2e32e09f74077383ae3f75b814a34e1874e2bad1980d5c696e"}
Jul 12 22:43:59.471325 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:59.471302    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f30527e-kfzfg"
Jul 12 22:43:59.526741 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:59.526696    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f1c44de-eda6-4069-8c1f-2078bdc2e4f9-maas-api-service-ca\") pod \"6f1c44de-eda6-4069-8c1f-2078bdc2e4f9\" (UID: \"6f1c44de-eda6-4069-8c1f-2078bdc2e4f9\") "
Jul 12 22:43:59.527112 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:59.527087    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f1c44de-eda6-4069-8c1f-2078bdc2e4f9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6f1c44de-eda6-4069-8c1f-2078bdc2e4f9" (UID: "6f1c44de-eda6-4069-8c1f-2078bdc2e4f9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 12 22:43:59.628325 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:43:59.628220    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f1c44de-eda6-4069-8c1f-2078bdc2e4f9-maas-api-service-ca\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:44:00.341336 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:44:00.341309    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f30527e-kfzfg"
Jul 12 22:44:00.341502 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:44:00.341334    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3f30527e-kfzfg" event={"ID":"6f1c44de-eda6-4069-8c1f-2078bdc2e4f9","Type":"ContainerDied","Data":"185d3955274fcc2e32e09f74077383ae3f75b814a34e1874e2bad1980d5c696e"}
Jul 12 22:44:00.341502 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:44:00.341388    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="185d3955274fcc2e32e09f74077383ae3f75b814a34e1874e2bad1980d5c696e"
Jul 12 22:44:17.012630 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:44:17.012589    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3f30527e/e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq"]
Jul 12 22:44:28.431491 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:44:28.431451    2560 scope.go:117] "RemoveContainer" containerID="14eb107a04c782d8bf517d434707e222c5c9ba07f98dd4be80b84ba6093145a8"
Jul 12 22:44:32.034583 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:44:32.034493    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-3f30527e/e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq" podUID="f15a9c8d-6582-463e-8f98-c38eaa6e0a36" containerName="main" containerID="cri-o://13a03735e35ab678b5a7090bf0b392006e7766ed3afb450ff866d8fc48d97f45" gracePeriod=45
Jul 12 22:44:32.294374 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:44:32.294314    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3f30527e/e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq"
Jul 12 22:44:32.425835 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:44:32.425795    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f15a9c8d-6582-463e-8f98-c38eaa6e0a36-tmp-dir\") pod \"f15a9c8d-6582-463e-8f98-c38eaa6e0a36\" (UID: \"f15a9c8d-6582-463e-8f98-c38eaa6e0a36\") "
Jul 12 22:44:32.426017 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:44:32.425850    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f15a9c8d-6582-463e-8f98-c38eaa6e0a36-dshm\") pod \"f15a9c8d-6582-463e-8f98-c38eaa6e0a36\" (UID: \"f15a9c8d-6582-463e-8f98-c38eaa6e0a36\") "
Jul 12 22:44:32.426017 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:44:32.425893    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f15a9c8d-6582-463e-8f98-c38eaa6e0a36-home\") pod \"f15a9c8d-6582-463e-8f98-c38eaa6e0a36\" (UID: \"f15a9c8d-6582-463e-8f98-c38eaa6e0a36\") "
Jul 12 22:44:32.426017 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:44:32.425923    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zn6vp\" (UniqueName: \"kubernetes.io/projected/f15a9c8d-6582-463e-8f98-c38eaa6e0a36-kube-api-access-zn6vp\") pod \"f15a9c8d-6582-463e-8f98-c38eaa6e0a36\" (UID: \"f15a9c8d-6582-463e-8f98-c38eaa6e0a36\") "
Jul 12 22:44:32.426017 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:44:32.425971    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f15a9c8d-6582-463e-8f98-c38eaa6e0a36-tls-certs\") pod \"f15a9c8d-6582-463e-8f98-c38eaa6e0a36\" (UID: \"f15a9c8d-6582-463e-8f98-c38eaa6e0a36\") "
Jul 12 22:44:32.426017 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:44:32.426008    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f15a9c8d-6582-463e-8f98-c38eaa6e0a36-model-cache\") pod \"f15a9c8d-6582-463e-8f98-c38eaa6e0a36\" (UID: \"f15a9c8d-6582-463e-8f98-c38eaa6e0a36\") "
Jul 12 22:44:32.426281 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:44:32.426149    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f15a9c8d-6582-463e-8f98-c38eaa6e0a36-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f15a9c8d-6582-463e-8f98-c38eaa6e0a36" (UID: "f15a9c8d-6582-463e-8f98-c38eaa6e0a36"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:44:32.426281 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:44:32.426186    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f15a9c8d-6582-463e-8f98-c38eaa6e0a36-home" (OuterVolumeSpecName: "home") pod "f15a9c8d-6582-463e-8f98-c38eaa6e0a36" (UID: "f15a9c8d-6582-463e-8f98-c38eaa6e0a36"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:44:32.426395 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:44:32.426307    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f15a9c8d-6582-463e-8f98-c38eaa6e0a36-tmp-dir\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:44:32.426395 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:44:32.426325    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f15a9c8d-6582-463e-8f98-c38eaa6e0a36-home\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:44:32.426395 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:44:32.426351    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f15a9c8d-6582-463e-8f98-c38eaa6e0a36-model-cache" (OuterVolumeSpecName: "model-cache") pod "f15a9c8d-6582-463e-8f98-c38eaa6e0a36" (UID: "f15a9c8d-6582-463e-8f98-c38eaa6e0a36"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:44:32.428371 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:44:32.428337    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f15a9c8d-6582-463e-8f98-c38eaa6e0a36-kube-api-access-zn6vp" (OuterVolumeSpecName: "kube-api-access-zn6vp") pod "f15a9c8d-6582-463e-8f98-c38eaa6e0a36" (UID: "f15a9c8d-6582-463e-8f98-c38eaa6e0a36"). InnerVolumeSpecName "kube-api-access-zn6vp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 22:44:32.428371 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:44:32.428350    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f15a9c8d-6582-463e-8f98-c38eaa6e0a36-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f15a9c8d-6582-463e-8f98-c38eaa6e0a36" (UID: "f15a9c8d-6582-463e-8f98-c38eaa6e0a36"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 12 22:44:32.428371 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:44:32.428354    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f15a9c8d-6582-463e-8f98-c38eaa6e0a36-dshm" (OuterVolumeSpecName: "dshm") pod "f15a9c8d-6582-463e-8f98-c38eaa6e0a36" (UID: "f15a9c8d-6582-463e-8f98-c38eaa6e0a36"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:44:32.523438 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:44:32.523405    2560 generic.go:356] "Generic (PLEG): container finished" podID="f15a9c8d-6582-463e-8f98-c38eaa6e0a36" containerID="13a03735e35ab678b5a7090bf0b392006e7766ed3afb450ff866d8fc48d97f45" exitCode=0
Jul 12 22:44:32.523642 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:44:32.523450    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3f30527e/e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq"
Jul 12 22:44:32.523642 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:44:32.523487    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3f30527e/e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq" event={"ID":"f15a9c8d-6582-463e-8f98-c38eaa6e0a36","Type":"ContainerDied","Data":"13a03735e35ab678b5a7090bf0b392006e7766ed3afb450ff866d8fc48d97f45"}
Jul 12 22:44:32.523642 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:44:32.523544    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3f30527e/e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq" event={"ID":"f15a9c8d-6582-463e-8f98-c38eaa6e0a36","Type":"ContainerDied","Data":"c5307aa988cf99269f17cd66c52c7bca56c3c5d6ca6252141e04a956d0ce6e69"}
Jul 12 22:44:32.523642 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:44:32.523561    2560 scope.go:117] "RemoveContainer" containerID="13a03735e35ab678b5a7090bf0b392006e7766ed3afb450ff866d8fc48d97f45"
Jul 12 22:44:32.526804 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:44:32.526779    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f15a9c8d-6582-463e-8f98-c38eaa6e0a36-tls-certs\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:44:32.526804 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:44:32.526806    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f15a9c8d-6582-463e-8f98-c38eaa6e0a36-model-cache\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:44:32.526946 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:44:32.526815    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f15a9c8d-6582-463e-8f98-c38eaa6e0a36-dshm\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:44:32.526946 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:44:32.526824    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn6vp\" (UniqueName: \"kubernetes.io/projected/f15a9c8d-6582-463e-8f98-c38eaa6e0a36-kube-api-access-zn6vp\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:44:32.533289 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:44:32.533269    2560 scope.go:117] "RemoveContainer" containerID="13a03735e35ab678b5a7090bf0b392006e7766ed3afb450ff866d8fc48d97f45"
Jul 12 22:44:32.533622 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:44:32.533604    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13a03735e35ab678b5a7090bf0b392006e7766ed3afb450ff866d8fc48d97f45\": container with ID starting with 13a03735e35ab678b5a7090bf0b392006e7766ed3afb450ff866d8fc48d97f45 not found: ID does not exist" containerID="13a03735e35ab678b5a7090bf0b392006e7766ed3afb450ff866d8fc48d97f45"
Jul 12 22:44:32.533669 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:44:32.533633    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13a03735e35ab678b5a7090bf0b392006e7766ed3afb450ff866d8fc48d97f45"} err="failed to get container status \"13a03735e35ab678b5a7090bf0b392006e7766ed3afb450ff866d8fc48d97f45\": rpc error: code = NotFound desc = could not find container \"13a03735e35ab678b5a7090bf0b392006e7766ed3afb450ff866d8fc48d97f45\": container with ID starting with 13a03735e35ab678b5a7090bf0b392006e7766ed3afb450ff866d8fc48d97f45 not found: ID does not exist"
Jul 12 22:44:32.546125 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:44:32.546061    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3f30527e/e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq"]
Jul 12 22:44:32.549688 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:44:32.549666    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-3f30527e/e2e-lifecycle-model-3f30527e-kserve-7bcb5f79-jn2mq"]
Jul 12 22:44:34.245739 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:44:34.245705    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f15a9c8d-6582-463e-8f98-c38eaa6e0a36" path="/var/lib/kubelet/pods/f15a9c8d-6582-463e-8f98-c38eaa6e0a36/volumes"
Jul 12 22:45:00.147340 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:00.147245    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29731605-kdklg"]
Jul 12 22:45:00.147861 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:00.147704    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f15a9c8d-6582-463e-8f98-c38eaa6e0a36" containerName="main"
Jul 12 22:45:00.147861 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:00.147718    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f15a9c8d-6582-463e-8f98-c38eaa6e0a36" containerName="main"
Jul 12 22:45:00.147861 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:00.147726    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f1c44de-eda6-4069-8c1f-2078bdc2e4f9" containerName="revoke-keys"
Jul 12 22:45:00.147861 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:00.147731    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f1c44de-eda6-4069-8c1f-2078bdc2e4f9" containerName="revoke-keys"
Jul 12 22:45:00.147861 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:00.147797    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f1c44de-eda6-4069-8c1f-2078bdc2e4f9" containerName="revoke-keys"
Jul 12 22:45:00.147861 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:00.147806    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f15a9c8d-6582-463e-8f98-c38eaa6e0a36" containerName="main"
Jul 12 22:45:00.152112 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:00.152094    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731605-kdklg"
Jul 12 22:45:00.173418 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:00.173387    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlrdf\""
Jul 12 22:45:00.176686 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:00.176645    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29731605-kdklg"]
Jul 12 22:45:00.184009 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:00.183982    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731605-kdklg"
Jul 12 22:45:00.326673 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:00.326601    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29731605-kdklg"]
Jul 12 22:45:00.330042 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:45:00.330000    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4ec29fe_5bff_44fc_8821_36f12c05d39f.slice/crio-b914c8c1e2ca903a6e8e6024d7225089eb8d32e766f2ff1a0fe9fe2fa725ece6 WatchSource:0}: Error finding container b914c8c1e2ca903a6e8e6024d7225089eb8d32e766f2ff1a0fe9fe2fa725ece6: Status 404 returned error can't find the container with id b914c8c1e2ca903a6e8e6024d7225089eb8d32e766f2ff1a0fe9fe2fa725ece6
Jul 12 22:45:00.675592 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:00.675553    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731605-kdklg" event={"ID":"d4ec29fe-5bff-44fc-8821-36f12c05d39f","Type":"ContainerStarted","Data":"fb302ecf24a6e3f7f21656ae0d12e4d1c5583d2c06021367b51b1e20aa840b0c"}
Jul 12 22:45:00.675758 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:00.675608    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731605-kdklg" event={"ID":"d4ec29fe-5bff-44fc-8821-36f12c05d39f","Type":"ContainerStarted","Data":"b914c8c1e2ca903a6e8e6024d7225089eb8d32e766f2ff1a0fe9fe2fa725ece6"}
Jul 12 22:45:00.692047 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:00.691989    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731605-kdklg" podStartSLOduration=0.691970446 podStartE2EDuration="691.970446ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:45:00.689769863 +0000 UTC m=+2073.003273216" watchObservedRunningTime="2026-07-12 22:45:00.691970446 +0000 UTC m=+2073.005473797"
Jul 12 22:45:01.681891 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:01.681856    2560 generic.go:356] "Generic (PLEG): container finished" podID="d4ec29fe-5bff-44fc-8821-36f12c05d39f" containerID="fb302ecf24a6e3f7f21656ae0d12e4d1c5583d2c06021367b51b1e20aa840b0c" exitCode=0
Jul 12 22:45:01.682276 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:01.681938    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731605-kdklg" event={"ID":"d4ec29fe-5bff-44fc-8821-36f12c05d39f","Type":"ContainerDied","Data":"fb302ecf24a6e3f7f21656ae0d12e4d1c5583d2c06021367b51b1e20aa840b0c"}
Jul 12 22:45:02.838191 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:02.838163    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731605-kdklg"
Jul 12 22:45:03.694183 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:03.694142    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731605-kdklg" event={"ID":"d4ec29fe-5bff-44fc-8821-36f12c05d39f","Type":"ContainerDied","Data":"b914c8c1e2ca903a6e8e6024d7225089eb8d32e766f2ff1a0fe9fe2fa725ece6"}
Jul 12 22:45:03.694183 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:03.694163    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29731605-kdklg"
Jul 12 22:45:03.694404 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:03.694202    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b914c8c1e2ca903a6e8e6024d7225089eb8d32e766f2ff1a0fe9fe2fa725ece6"
Jul 12 22:45:04.291670 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:04.291636    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"]
Jul 12 22:45:04.292191 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:04.291911    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w" podUID="eff55a5d-0b24-44c1-a418-a65a49893df5" containerName="istio-proxy" containerID="cri-o://859a8ee602361a17a7dd758fe39bac05dd2247e008a2c1c0d5a112bdbc490778" gracePeriod=30
Jul 12 22:45:09.567643 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:09.567610    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"
Jul 12 22:45:09.659108 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:09.659068    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eff55a5d-0b24-44c1-a418-a65a49893df5-workload-socket\") pod \"eff55a5d-0b24-44c1-a418-a65a49893df5\" (UID: \"eff55a5d-0b24-44c1-a418-a65a49893df5\") "
Jul 12 22:45:09.659310 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:09.659127    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eff55a5d-0b24-44c1-a418-a65a49893df5-credential-socket\") pod \"eff55a5d-0b24-44c1-a418-a65a49893df5\" (UID: \"eff55a5d-0b24-44c1-a418-a65a49893df5\") "
Jul 12 22:45:09.659310 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:09.659180    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eff55a5d-0b24-44c1-a418-a65a49893df5-istio-token\") pod \"eff55a5d-0b24-44c1-a418-a65a49893df5\" (UID: \"eff55a5d-0b24-44c1-a418-a65a49893df5\") "
Jul 12 22:45:09.659310 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:09.659209    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eff55a5d-0b24-44c1-a418-a65a49893df5-istiod-ca-cert\") pod \"eff55a5d-0b24-44c1-a418-a65a49893df5\" (UID: \"eff55a5d-0b24-44c1-a418-a65a49893df5\") "
Jul 12 22:45:09.659310 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:09.659257    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eff55a5d-0b24-44c1-a418-a65a49893df5-istio-data\") pod \"eff55a5d-0b24-44c1-a418-a65a49893df5\" (UID: \"eff55a5d-0b24-44c1-a418-a65a49893df5\") "
Jul 12 22:45:09.659310 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:09.659286    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tvl6\" (UniqueName: \"kubernetes.io/projected/eff55a5d-0b24-44c1-a418-a65a49893df5-kube-api-access-8tvl6\") pod \"eff55a5d-0b24-44c1-a418-a65a49893df5\" (UID: \"eff55a5d-0b24-44c1-a418-a65a49893df5\") "
Jul 12 22:45:09.659310 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:09.659287    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff55a5d-0b24-44c1-a418-a65a49893df5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "eff55a5d-0b24-44c1-a418-a65a49893df5" (UID: "eff55a5d-0b24-44c1-a418-a65a49893df5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:45:09.659310 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:09.659312    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eff55a5d-0b24-44c1-a418-a65a49893df5-istio-envoy\") pod \"eff55a5d-0b24-44c1-a418-a65a49893df5\" (UID: \"eff55a5d-0b24-44c1-a418-a65a49893df5\") "
Jul 12 22:45:09.659712 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:09.659350    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eff55a5d-0b24-44c1-a418-a65a49893df5-workload-certs\") pod \"eff55a5d-0b24-44c1-a418-a65a49893df5\" (UID: \"eff55a5d-0b24-44c1-a418-a65a49893df5\") "
Jul 12 22:45:09.659712 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:09.659393    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eff55a5d-0b24-44c1-a418-a65a49893df5-istio-podinfo\") pod \"eff55a5d-0b24-44c1-a418-a65a49893df5\" (UID: \"eff55a5d-0b24-44c1-a418-a65a49893df5\") "
Jul 12 22:45:09.659712 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:09.659419    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff55a5d-0b24-44c1-a418-a65a49893df5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "eff55a5d-0b24-44c1-a418-a65a49893df5" (UID: "eff55a5d-0b24-44c1-a418-a65a49893df5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:45:09.659712 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:09.659697    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eff55a5d-0b24-44c1-a418-a65a49893df5-workload-socket\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:45:09.659922 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:09.659713    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eff55a5d-0b24-44c1-a418-a65a49893df5-credential-socket\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:45:09.659922 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:09.659726    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eff55a5d-0b24-44c1-a418-a65a49893df5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "eff55a5d-0b24-44c1-a418-a65a49893df5" (UID: "eff55a5d-0b24-44c1-a418-a65a49893df5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 12 22:45:09.659922 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:09.659883    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff55a5d-0b24-44c1-a418-a65a49893df5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "eff55a5d-0b24-44c1-a418-a65a49893df5" (UID: "eff55a5d-0b24-44c1-a418-a65a49893df5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:45:09.660133 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:09.660102    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff55a5d-0b24-44c1-a418-a65a49893df5-istio-data" (OuterVolumeSpecName: "istio-data") pod "eff55a5d-0b24-44c1-a418-a65a49893df5" (UID: "eff55a5d-0b24-44c1-a418-a65a49893df5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:45:09.662040 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:09.662008    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eff55a5d-0b24-44c1-a418-a65a49893df5-istio-token" (OuterVolumeSpecName: "istio-token") pod "eff55a5d-0b24-44c1-a418-a65a49893df5" (UID: "eff55a5d-0b24-44c1-a418-a65a49893df5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 22:45:09.662242 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:09.662216    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eff55a5d-0b24-44c1-a418-a65a49893df5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "eff55a5d-0b24-44c1-a418-a65a49893df5" (UID: "eff55a5d-0b24-44c1-a418-a65a49893df5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 12 22:45:09.662342 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:09.662245    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eff55a5d-0b24-44c1-a418-a65a49893df5-kube-api-access-8tvl6" (OuterVolumeSpecName: "kube-api-access-8tvl6") pod "eff55a5d-0b24-44c1-a418-a65a49893df5" (UID: "eff55a5d-0b24-44c1-a418-a65a49893df5"). InnerVolumeSpecName "kube-api-access-8tvl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 22:45:09.662600 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:09.662571    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff55a5d-0b24-44c1-a418-a65a49893df5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "eff55a5d-0b24-44c1-a418-a65a49893df5" (UID: "eff55a5d-0b24-44c1-a418-a65a49893df5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:45:09.732738 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:09.732646    2560 generic.go:356] "Generic (PLEG): container finished" podID="eff55a5d-0b24-44c1-a418-a65a49893df5" containerID="859a8ee602361a17a7dd758fe39bac05dd2247e008a2c1c0d5a112bdbc490778" exitCode=0
Jul 12 22:45:09.732738 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:09.732699    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w" event={"ID":"eff55a5d-0b24-44c1-a418-a65a49893df5","Type":"ContainerDied","Data":"859a8ee602361a17a7dd758fe39bac05dd2247e008a2c1c0d5a112bdbc490778"}
Jul 12 22:45:09.733011 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:09.732717    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"
Jul 12 22:45:09.733011 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:09.732784    2560 scope.go:117] "RemoveContainer" containerID="859a8ee602361a17a7dd758fe39bac05dd2247e008a2c1c0d5a112bdbc490778"
Jul 12 22:45:09.733011 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:09.732770    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w" event={"ID":"eff55a5d-0b24-44c1-a418-a65a49893df5","Type":"ContainerDied","Data":"c08190ffbab799e6d5bade00df42ecdbc14c4354ad13c4f44a2f1c4ae72ed4c9"}
Jul 12 22:45:09.745220 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:09.745196    2560 scope.go:117] "RemoveContainer" containerID="ab841cdaddc1896a8da3b75ba9efa44ea9bd7ee783a482ed1bd51e6b067a9a0a"
Jul 12 22:45:09.755952 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:09.755922    2560 scope.go:117] "RemoveContainer" containerID="859a8ee602361a17a7dd758fe39bac05dd2247e008a2c1c0d5a112bdbc490778"
Jul 12 22:45:09.756375 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:45:09.756336    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"859a8ee602361a17a7dd758fe39bac05dd2247e008a2c1c0d5a112bdbc490778\": container with ID starting with 859a8ee602361a17a7dd758fe39bac05dd2247e008a2c1c0d5a112bdbc490778 not found: ID does not exist" containerID="859a8ee602361a17a7dd758fe39bac05dd2247e008a2c1c0d5a112bdbc490778"
Jul 12 22:45:09.756465 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:09.756392    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"859a8ee602361a17a7dd758fe39bac05dd2247e008a2c1c0d5a112bdbc490778"} err="failed to get container status \"859a8ee602361a17a7dd758fe39bac05dd2247e008a2c1c0d5a112bdbc490778\": rpc error: code = NotFound desc = could not find container \"859a8ee602361a17a7dd758fe39bac05dd2247e008a2c1c0d5a112bdbc490778\": container with ID starting with 859a8ee602361a17a7dd758fe39bac05dd2247e008a2c1c0d5a112bdbc490778 not found: ID does not exist"
Jul 12 22:45:09.756465 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:09.756419    2560 scope.go:117] "RemoveContainer" containerID="ab841cdaddc1896a8da3b75ba9efa44ea9bd7ee783a482ed1bd51e6b067a9a0a"
Jul 12 22:45:09.756789 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:45:09.756765    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab841cdaddc1896a8da3b75ba9efa44ea9bd7ee783a482ed1bd51e6b067a9a0a\": container with ID starting with ab841cdaddc1896a8da3b75ba9efa44ea9bd7ee783a482ed1bd51e6b067a9a0a not found: ID does not exist" containerID="ab841cdaddc1896a8da3b75ba9efa44ea9bd7ee783a482ed1bd51e6b067a9a0a"
Jul 12 22:45:09.756876 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:09.756796    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab841cdaddc1896a8da3b75ba9efa44ea9bd7ee783a482ed1bd51e6b067a9a0a"} err="failed to get container status \"ab841cdaddc1896a8da3b75ba9efa44ea9bd7ee783a482ed1bd51e6b067a9a0a\": rpc error: code = NotFound desc = could not find container \"ab841cdaddc1896a8da3b75ba9efa44ea9bd7ee783a482ed1bd51e6b067a9a0a\": container with ID starting with ab841cdaddc1896a8da3b75ba9efa44ea9bd7ee783a482ed1bd51e6b067a9a0a not found: ID does not exist"
Jul 12 22:45:09.760415 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:09.760393    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eff55a5d-0b24-44c1-a418-a65a49893df5-workload-certs\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:45:09.760507 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:09.760420    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eff55a5d-0b24-44c1-a418-a65a49893df5-istio-podinfo\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:45:09.760507 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:09.760434    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eff55a5d-0b24-44c1-a418-a65a49893df5-istio-token\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:45:09.760507 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:09.760446    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eff55a5d-0b24-44c1-a418-a65a49893df5-istiod-ca-cert\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:45:09.760507 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:09.760458    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eff55a5d-0b24-44c1-a418-a65a49893df5-istio-data\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:45:09.760507 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:09.760488    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tvl6\" (UniqueName: \"kubernetes.io/projected/eff55a5d-0b24-44c1-a418-a65a49893df5-kube-api-access-8tvl6\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:45:09.760507 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:09.760502    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eff55a5d-0b24-44c1-a418-a65a49893df5-istio-envoy\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:45:09.760725 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:09.760666    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"]
Jul 12 22:45:09.768266 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:09.768238    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3f30527e-openshift-default-d64b87999-5v47w"]
Jul 12 22:45:10.248004 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:10.247975    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eff55a5d-0b24-44c1-a418-a65a49893df5" path="/var/lib/kubelet/pods/eff55a5d-0b24-44c1-a418-a65a49893df5/volumes"
Jul 12 22:45:28.319172 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:28.319144    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52pbf_1b1f17bc-a992-4c72-8fdc-a33a3c3649c4/ovn-acl-logging/0.log"
Jul 12 22:45:28.322279 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:28.322257    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52pbf_1b1f17bc-a992-4c72-8fdc-a33a3c3649c4/ovn-acl-logging/0.log"
Jul 12 22:45:58.476139 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.476101    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"]
Jul 12 22:45:58.476806 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.476679    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4ec29fe-5bff-44fc-8821-36f12c05d39f" containerName="cleanup"
Jul 12 22:45:58.476806 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.476699    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4ec29fe-5bff-44fc-8821-36f12c05d39f" containerName="cleanup"
Jul 12 22:45:58.476806 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.476731    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eff55a5d-0b24-44c1-a418-a65a49893df5" containerName="istio-proxy"
Jul 12 22:45:58.476806 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.476742    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="eff55a5d-0b24-44c1-a418-a65a49893df5" containerName="istio-proxy"
Jul 12 22:45:58.476806 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.476754    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eff55a5d-0b24-44c1-a418-a65a49893df5" containerName="istio-proxy"
Jul 12 22:45:58.476806 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.476763    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="eff55a5d-0b24-44c1-a418-a65a49893df5" containerName="istio-proxy"
Jul 12 22:45:58.477100 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.476878    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="eff55a5d-0b24-44c1-a418-a65a49893df5" containerName="istio-proxy"
Jul 12 22:45:58.477100 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.476890    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4ec29fe-5bff-44fc-8821-36f12c05d39f" containerName="cleanup"
Jul 12 22:45:58.480202 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.480182    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"
Jul 12 22:45:58.488200 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.488178    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-30d729-openshift-default-dockercfg-76vmh\""
Jul 12 22:45:58.488307 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.488272    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 12 22:45:58.496835 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.496812    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"]
Jul 12 22:45:58.581827 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.581791    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-istio-data\") pod \"e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd\" (UID: \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\") " pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"
Jul 12 22:45:58.581991 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.581845    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-workload-socket\") pod \"e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd\" (UID: \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\") " pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"
Jul 12 22:45:58.581991 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.581895    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-workload-certs\") pod \"e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd\" (UID: \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\") " pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"
Jul 12 22:45:58.581991 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.581914    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qh55\" (UniqueName: \"kubernetes.io/projected/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-kube-api-access-8qh55\") pod \"e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd\" (UID: \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\") " pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"
Jul 12 22:45:58.582105 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.582012    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-istiod-ca-cert\") pod \"e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd\" (UID: \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\") " pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"
Jul 12 22:45:58.582105 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.582058    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-credential-socket\") pod \"e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd\" (UID: \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\") " pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"
Jul 12 22:45:58.582105 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.582079    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-istio-token\") pod \"e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd\" (UID: \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\") " pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"
Jul 12 22:45:58.582105 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.582098    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-istio-envoy\") pod \"e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd\" (UID: \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\") " pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"
Jul 12 22:45:58.582273 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.582114    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-istio-podinfo\") pod \"e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd\" (UID: \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\") " pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"
Jul 12 22:45:58.683546 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.683507    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-workload-socket\") pod \"e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd\" (UID: \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\") " pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"
Jul 12 22:45:58.683737 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.683597    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-workload-certs\") pod \"e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd\" (UID: \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\") " pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"
Jul 12 22:45:58.683737 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.683622    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qh55\" (UniqueName: \"kubernetes.io/projected/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-kube-api-access-8qh55\") pod \"e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd\" (UID: \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\") " pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"
Jul 12 22:45:58.683737 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.683672    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-istiod-ca-cert\") pod \"e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd\" (UID: \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\") " pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"
Jul 12 22:45:58.683737 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.683731    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-credential-socket\") pod \"e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd\" (UID: \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\") " pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"
Jul 12 22:45:58.683997 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.683757    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-istio-token\") pod \"e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd\" (UID: \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\") " pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"
Jul 12 22:45:58.683997 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.683785    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-istio-envoy\") pod \"e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd\" (UID: \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\") " pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"
Jul 12 22:45:58.683997 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.683805    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-istio-podinfo\") pod \"e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd\" (UID: \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\") " pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"
Jul 12 22:45:58.683997 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.683849    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-istio-data\") pod \"e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd\" (UID: \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\") " pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"
Jul 12 22:45:58.683997 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.683982    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-workload-socket\") pod \"e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd\" (UID: \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\") " pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"
Jul 12 22:45:58.684221 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.684032    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-workload-certs\") pod \"e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd\" (UID: \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\") " pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"
Jul 12 22:45:58.684221 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.684124    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-credential-socket\") pod \"e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd\" (UID: \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\") " pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"
Jul 12 22:45:58.684291 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.684236    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-istio-data\") pod \"e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd\" (UID: \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\") " pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"
Jul 12 22:45:58.684415 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.684390    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-istiod-ca-cert\") pod \"e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd\" (UID: \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\") " pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"
Jul 12 22:45:58.686261 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.686235    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-istio-envoy\") pod \"e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd\" (UID: \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\") " pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"
Jul 12 22:45:58.686469 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.686453    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-istio-podinfo\") pod \"e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd\" (UID: \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\") " pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"
Jul 12 22:45:58.691981 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.691955    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-istio-token\") pod \"e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd\" (UID: \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\") " pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"
Jul 12 22:45:58.692155 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.692138    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qh55\" (UniqueName: \"kubernetes.io/projected/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-kube-api-access-8qh55\") pod \"e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd\" (UID: \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\") " pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"
Jul 12 22:45:58.789844 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.789760    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"
Jul 12 22:45:58.932212 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.932183    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"]
Jul 12 22:45:58.934284 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:45:58.934254    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11f2f481_aa0b_46e8_a5ee_4b14336cc4eb.slice/crio-2d687e14a0e5b20ef6e555827b2433cfced2cb0aa4405e2432f66fdc666d507c WatchSource:0}: Error finding container 2d687e14a0e5b20ef6e555827b2433cfced2cb0aa4405e2432f66fdc666d507c: Status 404 returned error can't find the container with id 2d687e14a0e5b20ef6e555827b2433cfced2cb0aa4405e2432f66fdc666d507c
Jul 12 22:45:58.937067 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.937027    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:45:58.937163 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.937145    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:45:58.937208 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.937191    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:45:58.996370 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:45:58.996343    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd" event={"ID":"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb","Type":"ContainerStarted","Data":"2d687e14a0e5b20ef6e555827b2433cfced2cb0aa4405e2432f66fdc666d507c"}
Jul 12 22:46:00.003502 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:00.003463    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd" event={"ID":"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb","Type":"ContainerStarted","Data":"dd207950ea17e1ee79851b278b48f26a2f2ba56020aa51d49b0e4e76eae005cc"}
Jul 12 22:46:00.024938 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:00.024886    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd" podStartSLOduration=2.02486926 podStartE2EDuration="2.02486926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:45:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:46:00.022671524 +0000 UTC m=+2132.336174880" watchObservedRunningTime="2026-07-12 22:46:00.02486926 +0000 UTC m=+2132.338372614"
Jul 12 22:46:00.789984 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:00.789941    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"
Jul 12 22:46:00.795087 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:00.795060    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"
Jul 12 22:46:01.009076 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:01.009049    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"
Jul 12 22:46:01.010045 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:01.010023    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"
Jul 12 22:46:07.069649 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:07.069613    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"]
Jul 12 22:46:07.070285 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:07.070269    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="eff55a5d-0b24-44c1-a418-a65a49893df5" containerName="istio-proxy"
Jul 12 22:46:07.074011 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:07.073985    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"
Jul 12 22:46:07.076762 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:07.076737    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-58d7a8-openshift-default-dockercfg-blfhg\""
Jul 12 22:46:07.084821 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:07.084794    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"]
Jul 12 22:46:07.160956 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:07.160920    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-istio-data\") pod \"e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc\" (UID: \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\") " pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"
Jul 12 22:46:07.161154 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:07.160974    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-istio-podinfo\") pod \"e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc\" (UID: \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\") " pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"
Jul 12 22:46:07.161154 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:07.161072    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-istio-envoy\") pod \"e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc\" (UID: \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\") " pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"
Jul 12 22:46:07.161154 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:07.161117    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-istiod-ca-cert\") pod \"e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc\" (UID: \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\") " pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"
Jul 12 22:46:07.161284 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:07.161171    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-credential-socket\") pod \"e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc\" (UID: \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\") " pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"
Jul 12 22:46:07.161284 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:07.161209    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-workload-certs\") pod \"e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc\" (UID: \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\") " pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"
Jul 12 22:46:07.161284 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:07.161235    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgnnh\" (UniqueName: \"kubernetes.io/projected/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-kube-api-access-lgnnh\") pod \"e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc\" (UID: \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\") " pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"
Jul 12 22:46:07.161404 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:07.161280    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-workload-socket\") pod \"e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc\" (UID: \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\") " pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"
Jul 12 22:46:07.161404 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:07.161306    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-istio-token\") pod \"e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc\" (UID: \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\") " pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"
Jul 12 22:46:07.262434 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:07.262395    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-workload-socket\") pod \"e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc\" (UID: \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\") " pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"
Jul 12 22:46:07.262434 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:07.262433    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-istio-token\") pod \"e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc\" (UID: \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\") " pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"
Jul 12 22:46:07.262734 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:07.262504    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-istio-data\") pod \"e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc\" (UID: \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\") " pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"
Jul 12 22:46:07.262734 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:07.262574    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-istio-podinfo\") pod \"e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc\" (UID: \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\") " pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"
Jul 12 22:46:07.262734 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:07.262640    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-istio-envoy\") pod \"e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc\" (UID: \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\") " pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"
Jul 12 22:46:07.262734 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:07.262670    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-istiod-ca-cert\") pod \"e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc\" (UID: \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\") " pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"
Jul 12 22:46:07.262734 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:07.262709    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-credential-socket\") pod \"e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc\" (UID: \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\") " pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"
Jul 12 22:46:07.262989 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:07.262746    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-workload-certs\") pod \"e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc\" (UID: \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\") " pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"
Jul 12 22:46:07.262989 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:07.262774    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgnnh\" (UniqueName: \"kubernetes.io/projected/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-kube-api-access-lgnnh\") pod \"e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc\" (UID: \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\") " pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"
Jul 12 22:46:07.262989 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:07.262935    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-workload-socket\") pod \"e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc\" (UID: \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\") " pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"
Jul 12 22:46:07.262989 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:07.262951    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-istio-data\") pod \"e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc\" (UID: \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\") " pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"
Jul 12 22:46:07.263263 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:07.263239    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-workload-certs\") pod \"e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc\" (UID: \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\") " pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"
Jul 12 22:46:07.263326 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:07.263261    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-credential-socket\") pod \"e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc\" (UID: \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\") " pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"
Jul 12 22:46:07.263397 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:07.263378    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-istiod-ca-cert\") pod \"e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc\" (UID: \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\") " pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"
Jul 12 22:46:07.265730 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:07.265685    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-istio-envoy\") pod \"e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc\" (UID: \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\") " pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"
Jul 12 22:46:07.265995 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:07.265979    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-istio-podinfo\") pod \"e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc\" (UID: \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\") " pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"
Jul 12 22:46:07.272434 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:07.272410    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-istio-token\") pod \"e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc\" (UID: \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\") " pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"
Jul 12 22:46:07.272601 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:07.272583    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgnnh\" (UniqueName: \"kubernetes.io/projected/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-kube-api-access-lgnnh\") pod \"e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc\" (UID: \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\") " pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"
Jul 12 22:46:07.385835 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:07.385740    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"
Jul 12 22:46:07.540985 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:07.540951    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"]
Jul 12 22:46:07.541648 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:46:07.541616    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda92a7f95_b8ee_442d_80c6_bfc6261d16a9.slice/crio-6c25705168de41ca53b4d978fe021dd9b36a54edf6e11e4e9fd3594d2666043b WatchSource:0}: Error finding container 6c25705168de41ca53b4d978fe021dd9b36a54edf6e11e4e9fd3594d2666043b: Status 404 returned error can't find the container with id 6c25705168de41ca53b4d978fe021dd9b36a54edf6e11e4e9fd3594d2666043b
Jul 12 22:46:07.544144 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:07.544100    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:46:07.544214 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:07.544198    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:46:07.544260 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:07.544242    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:46:08.050258 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:08.050219    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc" event={"ID":"a92a7f95-b8ee-442d-80c6-bfc6261d16a9","Type":"ContainerStarted","Data":"3cf6d3ae7e18f55890f05880d5e643c6a6ddd85ab9601563bba14489c5542b95"}
Jul 12 22:46:08.050437 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:08.050284    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc" event={"ID":"a92a7f95-b8ee-442d-80c6-bfc6261d16a9","Type":"ContainerStarted","Data":"6c25705168de41ca53b4d978fe021dd9b36a54edf6e11e4e9fd3594d2666043b"}
Jul 12 22:46:08.069881 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:08.069816    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc" podStartSLOduration=1.069795676 podStartE2EDuration="1.069795676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:46:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:46:08.069646348 +0000 UTC m=+2140.383149703" watchObservedRunningTime="2026-07-12 22:46:08.069795676 +0000 UTC m=+2140.383299031"
Jul 12 22:46:08.386833 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:08.386797    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"
Jul 12 22:46:08.392188 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:08.392159    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"
Jul 12 22:46:09.055773 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:09.055743    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"
Jul 12 22:46:09.056967 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:09.056946    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"
Jul 12 22:46:17.049782 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:17.049731    2560 patch_prober.go:28] interesting pod/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 12 22:46:17.050271 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:17.049826    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd" podUID="11f2f481-aa0b-46e8-a5ee-4b14336cc4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 12 22:46:19.114260 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:19.114228    2560 generic.go:356] "Generic (PLEG): container finished" podID="11f2f481-aa0b-46e8-a5ee-4b14336cc4eb" containerID="dd207950ea17e1ee79851b278b48f26a2f2ba56020aa51d49b0e4e76eae005cc" exitCode=0
Jul 12 22:46:19.114699 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:19.114307    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd" event={"ID":"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb","Type":"ContainerDied","Data":"dd207950ea17e1ee79851b278b48f26a2f2ba56020aa51d49b0e4e76eae005cc"}
Jul 12 22:46:19.114741 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:19.114725    2560 scope.go:117] "RemoveContainer" containerID="dd207950ea17e1ee79851b278b48f26a2f2ba56020aa51d49b0e4e76eae005cc"
Jul 12 22:46:19.115732 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:19.115703    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:46:19.115836 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:19.115769    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:46:19.115836 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:19.115800    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:46:20.123421 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:20.122497    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd" event={"ID":"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb","Type":"ContainerStarted","Data":"1f3f96544b5a244b8f4590236aaaf576cf4229a148da200e44a39d2eb007b398"}
Jul 12 22:46:20.790011 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:20.789976    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"
Jul 12 22:46:20.790619 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:20.790162    2560 patch_prober.go:28] interesting pod/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Jul 12 22:46:20.790731 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:20.790651    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd" podUID="11f2f481-aa0b-46e8-a5ee-4b14336cc4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Jul 12 22:46:21.790892 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:21.790851    2560 patch_prober.go:28] interesting pod/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Jul 12 22:46:21.791282 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:21.790914    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd" podUID="11f2f481-aa0b-46e8-a5ee-4b14336cc4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Jul 12 22:46:22.790691 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:22.790649    2560 patch_prober.go:28] interesting pod/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Jul 12 22:46:22.790897 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:22.790709    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd" podUID="11f2f481-aa0b-46e8-a5ee-4b14336cc4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Jul 12 22:46:23.790583 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:23.790512    2560 patch_prober.go:28] interesting pod/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Jul 12 22:46:23.790960 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:23.790604    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd" podUID="11f2f481-aa0b-46e8-a5ee-4b14336cc4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Jul 12 22:46:24.794492 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:24.794409    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"
Jul 12 22:46:24.794953 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:24.794811    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"
Jul 12 22:46:24.795647 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:24.795624    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"
Jul 12 22:46:28.196034 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:28.196001    2560 generic.go:356] "Generic (PLEG): container finished" podID="a92a7f95-b8ee-442d-80c6-bfc6261d16a9" containerID="3cf6d3ae7e18f55890f05880d5e643c6a6ddd85ab9601563bba14489c5542b95" exitCode=0
Jul 12 22:46:28.196444 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:28.196057    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc" event={"ID":"a92a7f95-b8ee-442d-80c6-bfc6261d16a9","Type":"ContainerDied","Data":"3cf6d3ae7e18f55890f05880d5e643c6a6ddd85ab9601563bba14489c5542b95"}
Jul 12 22:46:28.196486 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:28.196450    2560 scope.go:117] "RemoveContainer" containerID="3cf6d3ae7e18f55890f05880d5e643c6a6ddd85ab9601563bba14489c5542b95"
Jul 12 22:46:28.197480 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:28.197449    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:46:28.197555 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:28.197520    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:46:28.197594 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:28.197579    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:46:28.479913 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:28.479816    2560 scope.go:117] "RemoveContainer" containerID="f89d769e3411b53456c9bf2c337848e3ec8589dd570b8e5cb12e4ec980a1d083"
Jul 12 22:46:29.203651 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:29.203618    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc" event={"ID":"a92a7f95-b8ee-442d-80c6-bfc6261d16a9","Type":"ContainerStarted","Data":"7ac354354333a391d0d3cf0349b29022cdfce6115deff2b8422b163610c3fb11"}
Jul 12 22:46:29.386632 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:29.386588    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"
Jul 12 22:46:29.386936 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:29.386910    2560 patch_prober.go:28] interesting pod/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Jul 12 22:46:29.387038 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:29.386967    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc" podUID="a92a7f95-b8ee-442d-80c6-bfc6261d16a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Jul 12 22:46:30.386886 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:30.386850    2560 patch_prober.go:28] interesting pod/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Jul 12 22:46:30.387278 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:30.386909    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc" podUID="a92a7f95-b8ee-442d-80c6-bfc6261d16a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Jul 12 22:46:31.386955 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:31.386918    2560 patch_prober.go:28] interesting pod/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Jul 12 22:46:31.387356 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:31.386974    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc" podUID="a92a7f95-b8ee-442d-80c6-bfc6261d16a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Jul 12 22:46:32.386999 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:32.386962    2560 patch_prober.go:28] interesting pod/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Jul 12 22:46:32.387468 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:32.387039    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc" podUID="a92a7f95-b8ee-442d-80c6-bfc6261d16a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Jul 12 22:46:32.908945 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:32.908907    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-30d729/test-model-30d729-kserve-86765c85ff-k2rxk"]
Jul 12 22:46:32.912961 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:32.912936    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-30d729/test-model-30d729-kserve-86765c85ff-k2rxk"
Jul 12 22:46:32.913498 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:32.913473    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-30d729/test-model-30d729-kserve-86765c85ff-k2rxk"]
Jul 12 22:46:32.916096 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:32.916065    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-30d729\"/\"kube-root-ca.crt\""
Jul 12 22:46:32.917238 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:32.917214    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-30d729\"/\"openshift-service-ca.crt\""
Jul 12 22:46:32.917341 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:32.917220    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-30d729\"/\"test-model-30d729-kserve-self-signed-certs\""
Jul 12 22:46:32.917401 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:32.917385    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-30d729\"/\"default-dockercfg-czhrw\""
Jul 12 22:46:32.999954 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:32.999915    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwlx8\" (UniqueName: \"kubernetes.io/projected/93183570-e5e1-40c9-8f6b-bb001ebd9f1f-kube-api-access-nwlx8\") pod \"test-model-30d729-kserve-86765c85ff-k2rxk\" (UID: \"93183570-e5e1-40c9-8f6b-bb001ebd9f1f\") " pod="ai-tenant-e2e-inf-a-30d729/test-model-30d729-kserve-86765c85ff-k2rxk"
Jul 12 22:46:33.000137 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:32.999963    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93183570-e5e1-40c9-8f6b-bb001ebd9f1f-dshm\") pod \"test-model-30d729-kserve-86765c85ff-k2rxk\" (UID: \"93183570-e5e1-40c9-8f6b-bb001ebd9f1f\") " pod="ai-tenant-e2e-inf-a-30d729/test-model-30d729-kserve-86765c85ff-k2rxk"
Jul 12 22:46:33.000137 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:32.999989    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93183570-e5e1-40c9-8f6b-bb001ebd9f1f-home\") pod \"test-model-30d729-kserve-86765c85ff-k2rxk\" (UID: \"93183570-e5e1-40c9-8f6b-bb001ebd9f1f\") " pod="ai-tenant-e2e-inf-a-30d729/test-model-30d729-kserve-86765c85ff-k2rxk"
Jul 12 22:46:33.000137 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:33.000088    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93183570-e5e1-40c9-8f6b-bb001ebd9f1f-model-cache\") pod \"test-model-30d729-kserve-86765c85ff-k2rxk\" (UID: \"93183570-e5e1-40c9-8f6b-bb001ebd9f1f\") " pod="ai-tenant-e2e-inf-a-30d729/test-model-30d729-kserve-86765c85ff-k2rxk"
Jul 12 22:46:33.000137 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:33.000123    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93183570-e5e1-40c9-8f6b-bb001ebd9f1f-tmp-dir\") pod \"test-model-30d729-kserve-86765c85ff-k2rxk\" (UID: \"93183570-e5e1-40c9-8f6b-bb001ebd9f1f\") " pod="ai-tenant-e2e-inf-a-30d729/test-model-30d729-kserve-86765c85ff-k2rxk"
Jul 12 22:46:33.000289 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:33.000143    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93183570-e5e1-40c9-8f6b-bb001ebd9f1f-tls-certs\") pod \"test-model-30d729-kserve-86765c85ff-k2rxk\" (UID: \"93183570-e5e1-40c9-8f6b-bb001ebd9f1f\") " pod="ai-tenant-e2e-inf-a-30d729/test-model-30d729-kserve-86765c85ff-k2rxk"
Jul 12 22:46:33.101388 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:33.101343    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93183570-e5e1-40c9-8f6b-bb001ebd9f1f-model-cache\") pod \"test-model-30d729-kserve-86765c85ff-k2rxk\" (UID: \"93183570-e5e1-40c9-8f6b-bb001ebd9f1f\") " pod="ai-tenant-e2e-inf-a-30d729/test-model-30d729-kserve-86765c85ff-k2rxk"
Jul 12 22:46:33.101388 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:33.101394    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93183570-e5e1-40c9-8f6b-bb001ebd9f1f-tmp-dir\") pod \"test-model-30d729-kserve-86765c85ff-k2rxk\" (UID: \"93183570-e5e1-40c9-8f6b-bb001ebd9f1f\") " pod="ai-tenant-e2e-inf-a-30d729/test-model-30d729-kserve-86765c85ff-k2rxk"
Jul 12 22:46:33.101683 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:33.101418    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93183570-e5e1-40c9-8f6b-bb001ebd9f1f-tls-certs\") pod \"test-model-30d729-kserve-86765c85ff-k2rxk\" (UID: \"93183570-e5e1-40c9-8f6b-bb001ebd9f1f\") " pod="ai-tenant-e2e-inf-a-30d729/test-model-30d729-kserve-86765c85ff-k2rxk"
Jul 12 22:46:33.101683 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:33.101493    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwlx8\" (UniqueName: \"kubernetes.io/projected/93183570-e5e1-40c9-8f6b-bb001ebd9f1f-kube-api-access-nwlx8\") pod \"test-model-30d729-kserve-86765c85ff-k2rxk\" (UID: \"93183570-e5e1-40c9-8f6b-bb001ebd9f1f\") " pod="ai-tenant-e2e-inf-a-30d729/test-model-30d729-kserve-86765c85ff-k2rxk"
Jul 12 22:46:33.101683 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:33.101526    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93183570-e5e1-40c9-8f6b-bb001ebd9f1f-dshm\") pod \"test-model-30d729-kserve-86765c85ff-k2rxk\" (UID: \"93183570-e5e1-40c9-8f6b-bb001ebd9f1f\") " pod="ai-tenant-e2e-inf-a-30d729/test-model-30d729-kserve-86765c85ff-k2rxk"
Jul 12 22:46:33.101683 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:33.101583    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93183570-e5e1-40c9-8f6b-bb001ebd9f1f-home\") pod \"test-model-30d729-kserve-86765c85ff-k2rxk\" (UID: \"93183570-e5e1-40c9-8f6b-bb001ebd9f1f\") " pod="ai-tenant-e2e-inf-a-30d729/test-model-30d729-kserve-86765c85ff-k2rxk"
Jul 12 22:46:33.101918 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:33.101866    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93183570-e5e1-40c9-8f6b-bb001ebd9f1f-model-cache\") pod \"test-model-30d729-kserve-86765c85ff-k2rxk\" (UID: \"93183570-e5e1-40c9-8f6b-bb001ebd9f1f\") " pod="ai-tenant-e2e-inf-a-30d729/test-model-30d729-kserve-86765c85ff-k2rxk"
Jul 12 22:46:33.101918 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:33.101894    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93183570-e5e1-40c9-8f6b-bb001ebd9f1f-tmp-dir\") pod \"test-model-30d729-kserve-86765c85ff-k2rxk\" (UID: \"93183570-e5e1-40c9-8f6b-bb001ebd9f1f\") " pod="ai-tenant-e2e-inf-a-30d729/test-model-30d729-kserve-86765c85ff-k2rxk"
Jul 12 22:46:33.102043 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:33.101981    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93183570-e5e1-40c9-8f6b-bb001ebd9f1f-home\") pod \"test-model-30d729-kserve-86765c85ff-k2rxk\" (UID: \"93183570-e5e1-40c9-8f6b-bb001ebd9f1f\") " pod="ai-tenant-e2e-inf-a-30d729/test-model-30d729-kserve-86765c85ff-k2rxk"
Jul 12 22:46:33.103960 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:33.103930    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93183570-e5e1-40c9-8f6b-bb001ebd9f1f-dshm\") pod \"test-model-30d729-kserve-86765c85ff-k2rxk\" (UID: \"93183570-e5e1-40c9-8f6b-bb001ebd9f1f\") " pod="ai-tenant-e2e-inf-a-30d729/test-model-30d729-kserve-86765c85ff-k2rxk"
Jul 12 22:46:33.104107 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:33.104089    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93183570-e5e1-40c9-8f6b-bb001ebd9f1f-tls-certs\") pod \"test-model-30d729-kserve-86765c85ff-k2rxk\" (UID: \"93183570-e5e1-40c9-8f6b-bb001ebd9f1f\") " pod="ai-tenant-e2e-inf-a-30d729/test-model-30d729-kserve-86765c85ff-k2rxk"
Jul 12 22:46:33.109692 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:33.109667    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwlx8\" (UniqueName: \"kubernetes.io/projected/93183570-e5e1-40c9-8f6b-bb001ebd9f1f-kube-api-access-nwlx8\") pod \"test-model-30d729-kserve-86765c85ff-k2rxk\" (UID: \"93183570-e5e1-40c9-8f6b-bb001ebd9f1f\") " pod="ai-tenant-e2e-inf-a-30d729/test-model-30d729-kserve-86765c85ff-k2rxk"
Jul 12 22:46:33.248765 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:33.248674    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-30d729/test-model-30d729-kserve-86765c85ff-k2rxk"
Jul 12 22:46:33.388883 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:33.388855    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-30d729/test-model-30d729-kserve-86765c85ff-k2rxk"]
Jul 12 22:46:33.390298 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:33.390273    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"
Jul 12 22:46:33.390703 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:33.390676    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"
Jul 12 22:46:33.391008 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:46:33.390985    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93183570_e5e1_40c9_8f6b_bb001ebd9f1f.slice/crio-eae5c60047df522d2b9b0f500ec462b8cf8fca1ae2ce52fd6548ad193ca364b1 WatchSource:0}: Error finding container eae5c60047df522d2b9b0f500ec462b8cf8fca1ae2ce52fd6548ad193ca364b1: Status 404 returned error can't find the container with id eae5c60047df522d2b9b0f500ec462b8cf8fca1ae2ce52fd6548ad193ca364b1
Jul 12 22:46:33.391363 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:33.391344    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"
Jul 12 22:46:34.233924 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:34.233885    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-30d729/test-model-30d729-kserve-86765c85ff-k2rxk" event={"ID":"93183570-e5e1-40c9-8f6b-bb001ebd9f1f","Type":"ContainerStarted","Data":"3b0d0817b9eb438480cf9e2da534e389b83e19028b3b7351730f8c4ee5decfd6"}
Jul 12 22:46:34.234128 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:34.233945    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-30d729/test-model-30d729-kserve-86765c85ff-k2rxk" event={"ID":"93183570-e5e1-40c9-8f6b-bb001ebd9f1f","Type":"ContainerStarted","Data":"eae5c60047df522d2b9b0f500ec462b8cf8fca1ae2ce52fd6548ad193ca364b1"}
Jul 12 22:46:34.251855 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:34.251804    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-30d729/test-model-30d729-kserve-86765c85ff-k2rxk" podStartSLOduration=2.251788363 podStartE2EDuration="2.251788363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:46:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:46:34.249571488 +0000 UTC m=+2166.563074833" watchObservedRunningTime="2026-07-12 22:46:34.251788363 +0000 UTC m=+2166.565291717"
Jul 12 22:46:43.249742 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:43.249703    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-30d729/test-model-30d729-kserve-86765c85ff-k2rxk"
Jul 12 22:46:43.250329 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:43.249763    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-30d729/test-model-30d729-kserve-86765c85ff-k2rxk"
Jul 12 22:46:43.262598 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:43.262570    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-30d729/test-model-30d729-kserve-86765c85ff-k2rxk"
Jul 12 22:46:43.294242 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:46:43.294216    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-30d729/test-model-30d729-kserve-86765c85ff-k2rxk"
Jul 12 22:47:00.683971 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:00.683932    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-58d7a8/test-model-58d7a8-kserve-97cd44b99-9m2sq"]
Jul 12 22:47:00.688571 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:00.688523    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-58d7a8/test-model-58d7a8-kserve-97cd44b99-9m2sq"
Jul 12 22:47:00.691262 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:00.691235    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-58d7a8\"/\"openshift-service-ca.crt\""
Jul 12 22:47:00.692354 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:00.692332    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-58d7a8\"/\"default-dockercfg-wgtvk\""
Jul 12 22:47:00.692486 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:00.692364    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-58d7a8\"/\"kube-root-ca.crt\""
Jul 12 22:47:00.692579 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:00.692559    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-58d7a8\"/\"test-model-58d7a8-kserve-self-signed-certs\""
Jul 12 22:47:00.696133 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:00.696108    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-58d7a8/test-model-58d7a8-kserve-97cd44b99-9m2sq"]
Jul 12 22:47:00.857137 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:00.857095    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndl62\" (UniqueName: \"kubernetes.io/projected/297eb43f-a9b4-427a-ab79-d68108bc0c98-kube-api-access-ndl62\") pod \"test-model-58d7a8-kserve-97cd44b99-9m2sq\" (UID: \"297eb43f-a9b4-427a-ab79-d68108bc0c98\") " pod="ai-tenant-e2e-inf-b-58d7a8/test-model-58d7a8-kserve-97cd44b99-9m2sq"
Jul 12 22:47:00.857137 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:00.857142    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/297eb43f-a9b4-427a-ab79-d68108bc0c98-model-cache\") pod \"test-model-58d7a8-kserve-97cd44b99-9m2sq\" (UID: \"297eb43f-a9b4-427a-ab79-d68108bc0c98\") " pod="ai-tenant-e2e-inf-b-58d7a8/test-model-58d7a8-kserve-97cd44b99-9m2sq"
Jul 12 22:47:00.857371 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:00.857165    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/297eb43f-a9b4-427a-ab79-d68108bc0c98-tls-certs\") pod \"test-model-58d7a8-kserve-97cd44b99-9m2sq\" (UID: \"297eb43f-a9b4-427a-ab79-d68108bc0c98\") " pod="ai-tenant-e2e-inf-b-58d7a8/test-model-58d7a8-kserve-97cd44b99-9m2sq"
Jul 12 22:47:00.857371 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:00.857269    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/297eb43f-a9b4-427a-ab79-d68108bc0c98-tmp-dir\") pod \"test-model-58d7a8-kserve-97cd44b99-9m2sq\" (UID: \"297eb43f-a9b4-427a-ab79-d68108bc0c98\") " pod="ai-tenant-e2e-inf-b-58d7a8/test-model-58d7a8-kserve-97cd44b99-9m2sq"
Jul 12 22:47:00.857371 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:00.857317    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/297eb43f-a9b4-427a-ab79-d68108bc0c98-home\") pod \"test-model-58d7a8-kserve-97cd44b99-9m2sq\" (UID: \"297eb43f-a9b4-427a-ab79-d68108bc0c98\") " pod="ai-tenant-e2e-inf-b-58d7a8/test-model-58d7a8-kserve-97cd44b99-9m2sq"
Jul 12 22:47:00.857371 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:00.857353    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/297eb43f-a9b4-427a-ab79-d68108bc0c98-dshm\") pod \"test-model-58d7a8-kserve-97cd44b99-9m2sq\" (UID: \"297eb43f-a9b4-427a-ab79-d68108bc0c98\") " pod="ai-tenant-e2e-inf-b-58d7a8/test-model-58d7a8-kserve-97cd44b99-9m2sq"
Jul 12 22:47:00.958225 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:00.958127    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/297eb43f-a9b4-427a-ab79-d68108bc0c98-model-cache\") pod \"test-model-58d7a8-kserve-97cd44b99-9m2sq\" (UID: \"297eb43f-a9b4-427a-ab79-d68108bc0c98\") " pod="ai-tenant-e2e-inf-b-58d7a8/test-model-58d7a8-kserve-97cd44b99-9m2sq"
Jul 12 22:47:00.958225 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:00.958170    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/297eb43f-a9b4-427a-ab79-d68108bc0c98-tls-certs\") pod \"test-model-58d7a8-kserve-97cd44b99-9m2sq\" (UID: \"297eb43f-a9b4-427a-ab79-d68108bc0c98\") " pod="ai-tenant-e2e-inf-b-58d7a8/test-model-58d7a8-kserve-97cd44b99-9m2sq"
Jul 12 22:47:00.958470 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:00.958372    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/297eb43f-a9b4-427a-ab79-d68108bc0c98-tmp-dir\") pod \"test-model-58d7a8-kserve-97cd44b99-9m2sq\" (UID: \"297eb43f-a9b4-427a-ab79-d68108bc0c98\") " pod="ai-tenant-e2e-inf-b-58d7a8/test-model-58d7a8-kserve-97cd44b99-9m2sq"
Jul 12 22:47:00.958470 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:00.958453    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/297eb43f-a9b4-427a-ab79-d68108bc0c98-home\") pod \"test-model-58d7a8-kserve-97cd44b99-9m2sq\" (UID: \"297eb43f-a9b4-427a-ab79-d68108bc0c98\") " pod="ai-tenant-e2e-inf-b-58d7a8/test-model-58d7a8-kserve-97cd44b99-9m2sq"
Jul 12 22:47:00.958627 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:00.958512    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/297eb43f-a9b4-427a-ab79-d68108bc0c98-dshm\") pod \"test-model-58d7a8-kserve-97cd44b99-9m2sq\" (UID: \"297eb43f-a9b4-427a-ab79-d68108bc0c98\") " pod="ai-tenant-e2e-inf-b-58d7a8/test-model-58d7a8-kserve-97cd44b99-9m2sq"
Jul 12 22:47:00.958627 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:00.958590    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndl62\" (UniqueName: \"kubernetes.io/projected/297eb43f-a9b4-427a-ab79-d68108bc0c98-kube-api-access-ndl62\") pod \"test-model-58d7a8-kserve-97cd44b99-9m2sq\" (UID: \"297eb43f-a9b4-427a-ab79-d68108bc0c98\") " pod="ai-tenant-e2e-inf-b-58d7a8/test-model-58d7a8-kserve-97cd44b99-9m2sq"
Jul 12 22:47:00.958627 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:00.958599    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/297eb43f-a9b4-427a-ab79-d68108bc0c98-model-cache\") pod \"test-model-58d7a8-kserve-97cd44b99-9m2sq\" (UID: \"297eb43f-a9b4-427a-ab79-d68108bc0c98\") " pod="ai-tenant-e2e-inf-b-58d7a8/test-model-58d7a8-kserve-97cd44b99-9m2sq"
Jul 12 22:47:00.958763 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:00.958741    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/297eb43f-a9b4-427a-ab79-d68108bc0c98-tmp-dir\") pod \"test-model-58d7a8-kserve-97cd44b99-9m2sq\" (UID: \"297eb43f-a9b4-427a-ab79-d68108bc0c98\") " pod="ai-tenant-e2e-inf-b-58d7a8/test-model-58d7a8-kserve-97cd44b99-9m2sq"
Jul 12 22:47:00.958821 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:00.958804    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/297eb43f-a9b4-427a-ab79-d68108bc0c98-home\") pod \"test-model-58d7a8-kserve-97cd44b99-9m2sq\" (UID: \"297eb43f-a9b4-427a-ab79-d68108bc0c98\") " pod="ai-tenant-e2e-inf-b-58d7a8/test-model-58d7a8-kserve-97cd44b99-9m2sq"
Jul 12 22:47:00.960896 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:00.960861    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/297eb43f-a9b4-427a-ab79-d68108bc0c98-dshm\") pod \"test-model-58d7a8-kserve-97cd44b99-9m2sq\" (UID: \"297eb43f-a9b4-427a-ab79-d68108bc0c98\") " pod="ai-tenant-e2e-inf-b-58d7a8/test-model-58d7a8-kserve-97cd44b99-9m2sq"
Jul 12 22:47:00.961024 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:00.960947    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/297eb43f-a9b4-427a-ab79-d68108bc0c98-tls-certs\") pod \"test-model-58d7a8-kserve-97cd44b99-9m2sq\" (UID: \"297eb43f-a9b4-427a-ab79-d68108bc0c98\") " pod="ai-tenant-e2e-inf-b-58d7a8/test-model-58d7a8-kserve-97cd44b99-9m2sq"
Jul 12 22:47:00.966318 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:00.966297    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndl62\" (UniqueName: \"kubernetes.io/projected/297eb43f-a9b4-427a-ab79-d68108bc0c98-kube-api-access-ndl62\") pod \"test-model-58d7a8-kserve-97cd44b99-9m2sq\" (UID: \"297eb43f-a9b4-427a-ab79-d68108bc0c98\") " pod="ai-tenant-e2e-inf-b-58d7a8/test-model-58d7a8-kserve-97cd44b99-9m2sq"
Jul 12 22:47:00.999219 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:00.999181    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-58d7a8/test-model-58d7a8-kserve-97cd44b99-9m2sq"
Jul 12 22:47:01.143201 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:01.143168    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-58d7a8/test-model-58d7a8-kserve-97cd44b99-9m2sq"]
Jul 12 22:47:01.145166 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:47:01.145141    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod297eb43f_a9b4_427a_ab79_d68108bc0c98.slice/crio-dd73cc7e1d32c9a7ee235ccc100242f7558901f7f2163964ca914a7b48836ce4 WatchSource:0}: Error finding container dd73cc7e1d32c9a7ee235ccc100242f7558901f7f2163964ca914a7b48836ce4: Status 404 returned error can't find the container with id dd73cc7e1d32c9a7ee235ccc100242f7558901f7f2163964ca914a7b48836ce4
Jul 12 22:47:01.385424 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:01.385385    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-58d7a8/test-model-58d7a8-kserve-97cd44b99-9m2sq" event={"ID":"297eb43f-a9b4-427a-ab79-d68108bc0c98","Type":"ContainerStarted","Data":"b3dbc50f501e5e9c255d298cbe11c2c4333c97d04b4a62efcadf355a3097dc8a"}
Jul 12 22:47:01.385424 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:01.385426    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-58d7a8/test-model-58d7a8-kserve-97cd44b99-9m2sq" event={"ID":"297eb43f-a9b4-427a-ab79-d68108bc0c98","Type":"ContainerStarted","Data":"dd73cc7e1d32c9a7ee235ccc100242f7558901f7f2163964ca914a7b48836ce4"}
Jul 12 22:47:01.402198 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:01.402146    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-58d7a8/test-model-58d7a8-kserve-97cd44b99-9m2sq" podStartSLOduration=1.402128297 podStartE2EDuration="1.402128297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:47:01.401191073 +0000 UTC m=+2193.714694426" watchObservedRunningTime="2026-07-12 22:47:01.402128297 +0000 UTC m=+2193.715631651"
Jul 12 22:47:10.999631 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:10.999587    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-58d7a8/test-model-58d7a8-kserve-97cd44b99-9m2sq"
Jul 12 22:47:11.000119 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:10.999649    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-58d7a8/test-model-58d7a8-kserve-97cd44b99-9m2sq"
Jul 12 22:47:11.012082 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:11.012057    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-58d7a8/test-model-58d7a8-kserve-97cd44b99-9m2sq"
Jul 12 22:47:11.461344 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:11.461313    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-58d7a8/test-model-58d7a8-kserve-97cd44b99-9m2sq"
Jul 12 22:47:28.266056 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:28.266024    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-30d729/test-model-30d729-kserve-86765c85ff-k2rxk"]
Jul 12 22:47:29.272163 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:29.272123    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-58d7a8/test-model-58d7a8-kserve-97cd44b99-9m2sq"]
Jul 12 22:47:43.286881 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:43.286822    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-30d729/test-model-30d729-kserve-86765c85ff-k2rxk" podUID="93183570-e5e1-40c9-8f6b-bb001ebd9f1f" containerName="main" containerID="cri-o://3b0d0817b9eb438480cf9e2da534e389b83e19028b3b7351730f8c4ee5decfd6" gracePeriod=45
Jul 12 22:47:43.291330 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:43.291296    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-30d729/test-model-30d729-kserve-86765c85ff-k2rxk" podUID="93183570-e5e1-40c9-8f6b-bb001ebd9f1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": EOF"
Jul 12 22:47:43.543832 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:43.543765    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-30d729/test-model-30d729-kserve-86765c85ff-k2rxk"
Jul 12 22:47:43.627224 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:43.627187    2560 generic.go:356] "Generic (PLEG): container finished" podID="93183570-e5e1-40c9-8f6b-bb001ebd9f1f" containerID="3b0d0817b9eb438480cf9e2da534e389b83e19028b3b7351730f8c4ee5decfd6" exitCode=0
Jul 12 22:47:43.627427 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:43.627232    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-30d729/test-model-30d729-kserve-86765c85ff-k2rxk"
Jul 12 22:47:43.627427 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:43.627248    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-30d729/test-model-30d729-kserve-86765c85ff-k2rxk" event={"ID":"93183570-e5e1-40c9-8f6b-bb001ebd9f1f","Type":"ContainerDied","Data":"3b0d0817b9eb438480cf9e2da534e389b83e19028b3b7351730f8c4ee5decfd6"}
Jul 12 22:47:43.627427 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:43.627289    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-30d729/test-model-30d729-kserve-86765c85ff-k2rxk" event={"ID":"93183570-e5e1-40c9-8f6b-bb001ebd9f1f","Type":"ContainerDied","Data":"eae5c60047df522d2b9b0f500ec462b8cf8fca1ae2ce52fd6548ad193ca364b1"}
Jul 12 22:47:43.627427 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:43.627311    2560 scope.go:117] "RemoveContainer" containerID="3b0d0817b9eb438480cf9e2da534e389b83e19028b3b7351730f8c4ee5decfd6"
Jul 12 22:47:43.638962 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:43.638927    2560 scope.go:117] "RemoveContainer" containerID="3b0d0817b9eb438480cf9e2da534e389b83e19028b3b7351730f8c4ee5decfd6"
Jul 12 22:47:43.639383 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:47:43.639360    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b0d0817b9eb438480cf9e2da534e389b83e19028b3b7351730f8c4ee5decfd6\": container with ID starting with 3b0d0817b9eb438480cf9e2da534e389b83e19028b3b7351730f8c4ee5decfd6 not found: ID does not exist" containerID="3b0d0817b9eb438480cf9e2da534e389b83e19028b3b7351730f8c4ee5decfd6"
Jul 12 22:47:43.639440 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:43.639398    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b0d0817b9eb438480cf9e2da534e389b83e19028b3b7351730f8c4ee5decfd6"} err="failed to get container status \"3b0d0817b9eb438480cf9e2da534e389b83e19028b3b7351730f8c4ee5decfd6\": rpc error: code = NotFound desc = could not find container \"3b0d0817b9eb438480cf9e2da534e389b83e19028b3b7351730f8c4ee5decfd6\": container with ID starting with 3b0d0817b9eb438480cf9e2da534e389b83e19028b3b7351730f8c4ee5decfd6 not found: ID does not exist"
Jul 12 22:47:43.661367 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:43.661345    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93183570-e5e1-40c9-8f6b-bb001ebd9f1f-dshm\") pod \"93183570-e5e1-40c9-8f6b-bb001ebd9f1f\" (UID: \"93183570-e5e1-40c9-8f6b-bb001ebd9f1f\") "
Jul 12 22:47:43.661493 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:43.661411    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwlx8\" (UniqueName: \"kubernetes.io/projected/93183570-e5e1-40c9-8f6b-bb001ebd9f1f-kube-api-access-nwlx8\") pod \"93183570-e5e1-40c9-8f6b-bb001ebd9f1f\" (UID: \"93183570-e5e1-40c9-8f6b-bb001ebd9f1f\") "
Jul 12 22:47:43.661493 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:43.661426    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93183570-e5e1-40c9-8f6b-bb001ebd9f1f-model-cache\") pod \"93183570-e5e1-40c9-8f6b-bb001ebd9f1f\" (UID: \"93183570-e5e1-40c9-8f6b-bb001ebd9f1f\") "
Jul 12 22:47:43.661493 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:43.661459    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93183570-e5e1-40c9-8f6b-bb001ebd9f1f-tls-certs\") pod \"93183570-e5e1-40c9-8f6b-bb001ebd9f1f\" (UID: \"93183570-e5e1-40c9-8f6b-bb001ebd9f1f\") "
Jul 12 22:47:43.661493 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:43.661478    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93183570-e5e1-40c9-8f6b-bb001ebd9f1f-home\") pod \"93183570-e5e1-40c9-8f6b-bb001ebd9f1f\" (UID: \"93183570-e5e1-40c9-8f6b-bb001ebd9f1f\") "
Jul 12 22:47:43.661733 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:43.661522    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93183570-e5e1-40c9-8f6b-bb001ebd9f1f-tmp-dir\") pod \"93183570-e5e1-40c9-8f6b-bb001ebd9f1f\" (UID: \"93183570-e5e1-40c9-8f6b-bb001ebd9f1f\") "
Jul 12 22:47:43.661810 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:43.661773    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93183570-e5e1-40c9-8f6b-bb001ebd9f1f-model-cache" (OuterVolumeSpecName: "model-cache") pod "93183570-e5e1-40c9-8f6b-bb001ebd9f1f" (UID: "93183570-e5e1-40c9-8f6b-bb001ebd9f1f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:47:43.661875 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:43.661790    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93183570-e5e1-40c9-8f6b-bb001ebd9f1f-home" (OuterVolumeSpecName: "home") pod "93183570-e5e1-40c9-8f6b-bb001ebd9f1f" (UID: "93183570-e5e1-40c9-8f6b-bb001ebd9f1f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:47:43.661917 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:43.661872    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93183570-e5e1-40c9-8f6b-bb001ebd9f1f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "93183570-e5e1-40c9-8f6b-bb001ebd9f1f" (UID: "93183570-e5e1-40c9-8f6b-bb001ebd9f1f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:47:43.663782 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:43.663752    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93183570-e5e1-40c9-8f6b-bb001ebd9f1f-dshm" (OuterVolumeSpecName: "dshm") pod "93183570-e5e1-40c9-8f6b-bb001ebd9f1f" (UID: "93183570-e5e1-40c9-8f6b-bb001ebd9f1f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:47:43.663782 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:43.663753    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93183570-e5e1-40c9-8f6b-bb001ebd9f1f-kube-api-access-nwlx8" (OuterVolumeSpecName: "kube-api-access-nwlx8") pod "93183570-e5e1-40c9-8f6b-bb001ebd9f1f" (UID: "93183570-e5e1-40c9-8f6b-bb001ebd9f1f"). InnerVolumeSpecName "kube-api-access-nwlx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 22:47:43.663782 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:43.663775    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93183570-e5e1-40c9-8f6b-bb001ebd9f1f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "93183570-e5e1-40c9-8f6b-bb001ebd9f1f" (UID: "93183570-e5e1-40c9-8f6b-bb001ebd9f1f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 12 22:47:43.762503 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:43.762470    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwlx8\" (UniqueName: \"kubernetes.io/projected/93183570-e5e1-40c9-8f6b-bb001ebd9f1f-kube-api-access-nwlx8\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:47:43.762503 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:43.762498    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93183570-e5e1-40c9-8f6b-bb001ebd9f1f-model-cache\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:47:43.762503 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:43.762508    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93183570-e5e1-40c9-8f6b-bb001ebd9f1f-tls-certs\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:47:43.762754 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:43.762515    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93183570-e5e1-40c9-8f6b-bb001ebd9f1f-home\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:47:43.762754 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:43.762524    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93183570-e5e1-40c9-8f6b-bb001ebd9f1f-tmp-dir\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:47:43.762754 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:43.762557    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93183570-e5e1-40c9-8f6b-bb001ebd9f1f-dshm\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:47:43.949155 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:43.949119    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-30d729/test-model-30d729-kserve-86765c85ff-k2rxk"]
Jul 12 22:47:43.952881 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:43.952856    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-30d729/test-model-30d729-kserve-86765c85ff-k2rxk"]
Jul 12 22:47:44.246796 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:44.246717    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93183570-e5e1-40c9-8f6b-bb001ebd9f1f" path="/var/lib/kubelet/pods/93183570-e5e1-40c9-8f6b-bb001ebd9f1f/volumes"
Jul 12 22:47:44.293314 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:44.293263    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-58d7a8/test-model-58d7a8-kserve-97cd44b99-9m2sq" podUID="297eb43f-a9b4-427a-ab79-d68108bc0c98" containerName="main" containerID="cri-o://b3dbc50f501e5e9c255d298cbe11c2c4333c97d04b4a62efcadf355a3097dc8a" gracePeriod=45
Jul 12 22:47:44.550626 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:44.550599    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-58d7a8/test-model-58d7a8-kserve-97cd44b99-9m2sq"
Jul 12 22:47:44.633772 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:44.633738    2560 generic.go:356] "Generic (PLEG): container finished" podID="297eb43f-a9b4-427a-ab79-d68108bc0c98" containerID="b3dbc50f501e5e9c255d298cbe11c2c4333c97d04b4a62efcadf355a3097dc8a" exitCode=0
Jul 12 22:47:44.633982 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:44.633786    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-58d7a8/test-model-58d7a8-kserve-97cd44b99-9m2sq"
Jul 12 22:47:44.633982 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:44.633831    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-58d7a8/test-model-58d7a8-kserve-97cd44b99-9m2sq" event={"ID":"297eb43f-a9b4-427a-ab79-d68108bc0c98","Type":"ContainerDied","Data":"b3dbc50f501e5e9c255d298cbe11c2c4333c97d04b4a62efcadf355a3097dc8a"}
Jul 12 22:47:44.633982 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:44.633869    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-58d7a8/test-model-58d7a8-kserve-97cd44b99-9m2sq" event={"ID":"297eb43f-a9b4-427a-ab79-d68108bc0c98","Type":"ContainerDied","Data":"dd73cc7e1d32c9a7ee235ccc100242f7558901f7f2163964ca914a7b48836ce4"}
Jul 12 22:47:44.633982 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:44.633890    2560 scope.go:117] "RemoveContainer" containerID="b3dbc50f501e5e9c255d298cbe11c2c4333c97d04b4a62efcadf355a3097dc8a"
Jul 12 22:47:44.642980 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:44.642959    2560 scope.go:117] "RemoveContainer" containerID="b3dbc50f501e5e9c255d298cbe11c2c4333c97d04b4a62efcadf355a3097dc8a"
Jul 12 22:47:44.643260 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:47:44.643243    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3dbc50f501e5e9c255d298cbe11c2c4333c97d04b4a62efcadf355a3097dc8a\": container with ID starting with b3dbc50f501e5e9c255d298cbe11c2c4333c97d04b4a62efcadf355a3097dc8a not found: ID does not exist" containerID="b3dbc50f501e5e9c255d298cbe11c2c4333c97d04b4a62efcadf355a3097dc8a"
Jul 12 22:47:44.643303 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:44.643269    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3dbc50f501e5e9c255d298cbe11c2c4333c97d04b4a62efcadf355a3097dc8a"} err="failed to get container status \"b3dbc50f501e5e9c255d298cbe11c2c4333c97d04b4a62efcadf355a3097dc8a\": rpc error: code = NotFound desc = could not find container \"b3dbc50f501e5e9c255d298cbe11c2c4333c97d04b4a62efcadf355a3097dc8a\": container with ID starting with b3dbc50f501e5e9c255d298cbe11c2c4333c97d04b4a62efcadf355a3097dc8a not found: ID does not exist"
Jul 12 22:47:44.669611 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:44.669578    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/297eb43f-a9b4-427a-ab79-d68108bc0c98-tmp-dir\") pod \"297eb43f-a9b4-427a-ab79-d68108bc0c98\" (UID: \"297eb43f-a9b4-427a-ab79-d68108bc0c98\") "
Jul 12 22:47:44.669744 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:44.669640    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/297eb43f-a9b4-427a-ab79-d68108bc0c98-dshm\") pod \"297eb43f-a9b4-427a-ab79-d68108bc0c98\" (UID: \"297eb43f-a9b4-427a-ab79-d68108bc0c98\") "
Jul 12 22:47:44.669744 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:44.669680    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/297eb43f-a9b4-427a-ab79-d68108bc0c98-home\") pod \"297eb43f-a9b4-427a-ab79-d68108bc0c98\" (UID: \"297eb43f-a9b4-427a-ab79-d68108bc0c98\") "
Jul 12 22:47:44.669852 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:44.669744    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/297eb43f-a9b4-427a-ab79-d68108bc0c98-tls-certs\") pod \"297eb43f-a9b4-427a-ab79-d68108bc0c98\" (UID: \"297eb43f-a9b4-427a-ab79-d68108bc0c98\") "
Jul 12 22:47:44.669852 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:44.669769    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndl62\" (UniqueName: \"kubernetes.io/projected/297eb43f-a9b4-427a-ab79-d68108bc0c98-kube-api-access-ndl62\") pod \"297eb43f-a9b4-427a-ab79-d68108bc0c98\" (UID: \"297eb43f-a9b4-427a-ab79-d68108bc0c98\") "
Jul 12 22:47:44.669852 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:44.669787    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/297eb43f-a9b4-427a-ab79-d68108bc0c98-model-cache\") pod \"297eb43f-a9b4-427a-ab79-d68108bc0c98\" (UID: \"297eb43f-a9b4-427a-ab79-d68108bc0c98\") "
Jul 12 22:47:44.669995 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:44.669855    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297eb43f-a9b4-427a-ab79-d68108bc0c98-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "297eb43f-a9b4-427a-ab79-d68108bc0c98" (UID: "297eb43f-a9b4-427a-ab79-d68108bc0c98"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:47:44.670057 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:44.670011    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297eb43f-a9b4-427a-ab79-d68108bc0c98-home" (OuterVolumeSpecName: "home") pod "297eb43f-a9b4-427a-ab79-d68108bc0c98" (UID: "297eb43f-a9b4-427a-ab79-d68108bc0c98"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:47:44.670105 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:44.670084    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297eb43f-a9b4-427a-ab79-d68108bc0c98-model-cache" (OuterVolumeSpecName: "model-cache") pod "297eb43f-a9b4-427a-ab79-d68108bc0c98" (UID: "297eb43f-a9b4-427a-ab79-d68108bc0c98"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:47:44.670148 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:44.670102    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/297eb43f-a9b4-427a-ab79-d68108bc0c98-home\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:47:44.670148 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:44.670120    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/297eb43f-a9b4-427a-ab79-d68108bc0c98-tmp-dir\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:47:44.672142 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:44.672121    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297eb43f-a9b4-427a-ab79-d68108bc0c98-dshm" (OuterVolumeSpecName: "dshm") pod "297eb43f-a9b4-427a-ab79-d68108bc0c98" (UID: "297eb43f-a9b4-427a-ab79-d68108bc0c98"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:47:44.672227 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:44.672176    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/297eb43f-a9b4-427a-ab79-d68108bc0c98-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "297eb43f-a9b4-427a-ab79-d68108bc0c98" (UID: "297eb43f-a9b4-427a-ab79-d68108bc0c98"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 12 22:47:44.672265 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:44.672217    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/297eb43f-a9b4-427a-ab79-d68108bc0c98-kube-api-access-ndl62" (OuterVolumeSpecName: "kube-api-access-ndl62") pod "297eb43f-a9b4-427a-ab79-d68108bc0c98" (UID: "297eb43f-a9b4-427a-ab79-d68108bc0c98"). InnerVolumeSpecName "kube-api-access-ndl62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 22:47:44.771047 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:44.771005    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/297eb43f-a9b4-427a-ab79-d68108bc0c98-tls-certs\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:47:44.771047 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:44.771044    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndl62\" (UniqueName: \"kubernetes.io/projected/297eb43f-a9b4-427a-ab79-d68108bc0c98-kube-api-access-ndl62\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:47:44.771047 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:44.771058    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/297eb43f-a9b4-427a-ab79-d68108bc0c98-model-cache\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:47:44.771284 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:44.771069    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/297eb43f-a9b4-427a-ab79-d68108bc0c98-dshm\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:47:44.955627 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:44.955591    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-58d7a8/test-model-58d7a8-kserve-97cd44b99-9m2sq"]
Jul 12 22:47:44.959256 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:44.959227    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-58d7a8/test-model-58d7a8-kserve-97cd44b99-9m2sq"]
Jul 12 22:47:46.245925 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:46.245884    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="297eb43f-a9b4-427a-ab79-d68108bc0c98" path="/var/lib/kubelet/pods/297eb43f-a9b4-427a-ab79-d68108bc0c98/volumes"
Jul 12 22:47:55.128051 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:55.127800    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"]
Jul 12 22:47:55.128598 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:47:55.128050    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd" podUID="11f2f481-aa0b-46e8-a5ee-4b14336cc4eb" containerName="istio-proxy" containerID="cri-o://1f3f96544b5a244b8f4590236aaaf576cf4229a148da200e44a39d2eb007b398" gracePeriod=30
Jul 12 22:48:00.394561 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:00.394484    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"
Jul 12 22:48:00.405342 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:00.405318    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-istio-envoy\") pod \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\" (UID: \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\") "
Jul 12 22:48:00.405428 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:00.405353    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qh55\" (UniqueName: \"kubernetes.io/projected/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-kube-api-access-8qh55\") pod \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\" (UID: \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\") "
Jul 12 22:48:00.405428 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:00.405371    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-istio-podinfo\") pod \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\" (UID: \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\") "
Jul 12 22:48:00.405428 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:00.405413    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-istio-data\") pod \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\" (UID: \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\") "
Jul 12 22:48:00.405583 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:00.405437    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-workload-socket\") pod \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\" (UID: \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\") "
Jul 12 22:48:00.405583 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:00.405475    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-workload-certs\") pod \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\" (UID: \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\") "
Jul 12 22:48:00.405583 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:00.405498    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-credential-socket\") pod \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\" (UID: \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\") "
Jul 12 22:48:00.405793 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:00.405619    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-istio-token\") pod \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\" (UID: \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\") "
Jul 12 22:48:00.405793 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:00.405651    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-istiod-ca-cert\") pod \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\" (UID: \"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb\") "
Jul 12 22:48:00.405793 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:00.405776    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "11f2f481-aa0b-46e8-a5ee-4b14336cc4eb" (UID: "11f2f481-aa0b-46e8-a5ee-4b14336cc4eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:48:00.405952 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:00.405803    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "11f2f481-aa0b-46e8-a5ee-4b14336cc4eb" (UID: "11f2f481-aa0b-46e8-a5ee-4b14336cc4eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:48:00.405952 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:00.405926    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "11f2f481-aa0b-46e8-a5ee-4b14336cc4eb" (UID: "11f2f481-aa0b-46e8-a5ee-4b14336cc4eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:48:00.406177 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:00.406114    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-workload-socket\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:48:00.406177 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:00.406139    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-workload-certs\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:48:00.406177 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:00.406153    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-credential-socket\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:48:00.406432 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:00.406263    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "11f2f481-aa0b-46e8-a5ee-4b14336cc4eb" (UID: "11f2f481-aa0b-46e8-a5ee-4b14336cc4eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 12 22:48:00.406655 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:00.406602    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-istio-data" (OuterVolumeSpecName: "istio-data") pod "11f2f481-aa0b-46e8-a5ee-4b14336cc4eb" (UID: "11f2f481-aa0b-46e8-a5ee-4b14336cc4eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:48:00.407813 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:00.407788    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "11f2f481-aa0b-46e8-a5ee-4b14336cc4eb" (UID: "11f2f481-aa0b-46e8-a5ee-4b14336cc4eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:48:00.407813 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:00.407792    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-kube-api-access-8qh55" (OuterVolumeSpecName: "kube-api-access-8qh55") pod "11f2f481-aa0b-46e8-a5ee-4b14336cc4eb" (UID: "11f2f481-aa0b-46e8-a5ee-4b14336cc4eb"). InnerVolumeSpecName "kube-api-access-8qh55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 22:48:00.407963 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:00.407946    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-istio-token" (OuterVolumeSpecName: "istio-token") pod "11f2f481-aa0b-46e8-a5ee-4b14336cc4eb" (UID: "11f2f481-aa0b-46e8-a5ee-4b14336cc4eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 22:48:00.408010 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:00.407996    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "11f2f481-aa0b-46e8-a5ee-4b14336cc4eb" (UID: "11f2f481-aa0b-46e8-a5ee-4b14336cc4eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 12 22:48:00.507064 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:00.507027    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-istio-data\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:48:00.507064 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:00.507058    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-istio-token\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:48:00.507064 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:00.507067    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-istiod-ca-cert\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:48:00.507064 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:00.507075    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-istio-envoy\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:48:00.507344 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:00.507083    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qh55\" (UniqueName: \"kubernetes.io/projected/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-kube-api-access-8qh55\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:48:00.507344 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:00.507092    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb-istio-podinfo\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:48:00.730204 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:00.730100    2560 generic.go:356] "Generic (PLEG): container finished" podID="11f2f481-aa0b-46e8-a5ee-4b14336cc4eb" containerID="1f3f96544b5a244b8f4590236aaaf576cf4229a148da200e44a39d2eb007b398" exitCode=0
Jul 12 22:48:00.730204 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:00.730170    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd" event={"ID":"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb","Type":"ContainerDied","Data":"1f3f96544b5a244b8f4590236aaaf576cf4229a148da200e44a39d2eb007b398"}
Jul 12 22:48:00.730430 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:00.730261    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd" event={"ID":"11f2f481-aa0b-46e8-a5ee-4b14336cc4eb","Type":"ContainerDied","Data":"2d687e14a0e5b20ef6e555827b2433cfced2cb0aa4405e2432f66fdc666d507c"}
Jul 12 22:48:00.730430 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:00.730183    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"
Jul 12 22:48:00.730430 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:00.730285    2560 scope.go:117] "RemoveContainer" containerID="1f3f96544b5a244b8f4590236aaaf576cf4229a148da200e44a39d2eb007b398"
Jul 12 22:48:00.739954 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:00.739936    2560 scope.go:117] "RemoveContainer" containerID="dd207950ea17e1ee79851b278b48f26a2f2ba56020aa51d49b0e4e76eae005cc"
Jul 12 22:48:00.748974 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:00.748950    2560 scope.go:117] "RemoveContainer" containerID="1f3f96544b5a244b8f4590236aaaf576cf4229a148da200e44a39d2eb007b398"
Jul 12 22:48:00.749315 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:48:00.749284    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f3f96544b5a244b8f4590236aaaf576cf4229a148da200e44a39d2eb007b398\": container with ID starting with 1f3f96544b5a244b8f4590236aaaf576cf4229a148da200e44a39d2eb007b398 not found: ID does not exist" containerID="1f3f96544b5a244b8f4590236aaaf576cf4229a148da200e44a39d2eb007b398"
Jul 12 22:48:00.749425 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:00.749326    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f3f96544b5a244b8f4590236aaaf576cf4229a148da200e44a39d2eb007b398"} err="failed to get container status \"1f3f96544b5a244b8f4590236aaaf576cf4229a148da200e44a39d2eb007b398\": rpc error: code = NotFound desc = could not find container \"1f3f96544b5a244b8f4590236aaaf576cf4229a148da200e44a39d2eb007b398\": container with ID starting with 1f3f96544b5a244b8f4590236aaaf576cf4229a148da200e44a39d2eb007b398 not found: ID does not exist"
Jul 12 22:48:00.749425 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:00.749352    2560 scope.go:117] "RemoveContainer" containerID="dd207950ea17e1ee79851b278b48f26a2f2ba56020aa51d49b0e4e76eae005cc"
Jul 12 22:48:00.749775 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:48:00.749753    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd207950ea17e1ee79851b278b48f26a2f2ba56020aa51d49b0e4e76eae005cc\": container with ID starting with dd207950ea17e1ee79851b278b48f26a2f2ba56020aa51d49b0e4e76eae005cc not found: ID does not exist" containerID="dd207950ea17e1ee79851b278b48f26a2f2ba56020aa51d49b0e4e76eae005cc"
Jul 12 22:48:00.749873 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:00.749778    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd207950ea17e1ee79851b278b48f26a2f2ba56020aa51d49b0e4e76eae005cc"} err="failed to get container status \"dd207950ea17e1ee79851b278b48f26a2f2ba56020aa51d49b0e4e76eae005cc\": rpc error: code = NotFound desc = could not find container \"dd207950ea17e1ee79851b278b48f26a2f2ba56020aa51d49b0e4e76eae005cc\": container with ID starting with dd207950ea17e1ee79851b278b48f26a2f2ba56020aa51d49b0e4e76eae005cc not found: ID does not exist"
Jul 12 22:48:00.752400 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:00.752378    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"]
Jul 12 22:48:00.757324 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:00.757303    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-30d729-openshift-default-7fdf5c5c79-fdlsd"]
Jul 12 22:48:02.246877 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:02.246842    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11f2f481-aa0b-46e8-a5ee-4b14336cc4eb" path="/var/lib/kubelet/pods/11f2f481-aa0b-46e8-a5ee-4b14336cc4eb/volumes"
Jul 12 22:48:21.229457 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:21.229418    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"]
Jul 12 22:48:21.229929 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:21.229694    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc" podUID="a92a7f95-b8ee-442d-80c6-bfc6261d16a9" containerName="istio-proxy" containerID="cri-o://7ac354354333a391d0d3cf0349b29022cdfce6115deff2b8422b163610c3fb11" gracePeriod=30
Jul 12 22:48:22.116486 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.116452    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"]
Jul 12 22:48:22.117051 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.117022    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11f2f481-aa0b-46e8-a5ee-4b14336cc4eb" containerName="istio-proxy"
Jul 12 22:48:22.117051 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.117044    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="11f2f481-aa0b-46e8-a5ee-4b14336cc4eb" containerName="istio-proxy"
Jul 12 22:48:22.117245 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.117065    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93183570-e5e1-40c9-8f6b-bb001ebd9f1f" containerName="main"
Jul 12 22:48:22.117245 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.117071    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="93183570-e5e1-40c9-8f6b-bb001ebd9f1f" containerName="main"
Jul 12 22:48:22.117245 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.117079    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="297eb43f-a9b4-427a-ab79-d68108bc0c98" containerName="main"
Jul 12 22:48:22.117245 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.117085    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="297eb43f-a9b4-427a-ab79-d68108bc0c98" containerName="main"
Jul 12 22:48:22.117245 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.117143    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="11f2f481-aa0b-46e8-a5ee-4b14336cc4eb" containerName="istio-proxy"
Jul 12 22:48:22.117245 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.117151    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="297eb43f-a9b4-427a-ab79-d68108bc0c98" containerName="main"
Jul 12 22:48:22.117245 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.117157    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="93183570-e5e1-40c9-8f6b-bb001ebd9f1f" containerName="main"
Jul 12 22:48:22.117245 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.117162    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="11f2f481-aa0b-46e8-a5ee-4b14336cc4eb" containerName="istio-proxy"
Jul 12 22:48:22.120616 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.120593    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"
Jul 12 22:48:22.123295 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.123275    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-8a9e41-openshift-default-dockercfg-grfwq\""
Jul 12 22:48:22.130613 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.130585    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"]
Jul 12 22:48:22.205968 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.205929    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b34640-2efb-481f-8a20-d9f5be8bad4a-workload-socket\") pod \"e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc\" (UID: \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\") " pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"
Jul 12 22:48:22.206134 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.206040    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3b34640-2efb-481f-8a20-d9f5be8bad4a-istio-podinfo\") pod \"e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc\" (UID: \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\") " pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"
Jul 12 22:48:22.206134 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.206080    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3b34640-2efb-481f-8a20-d9f5be8bad4a-workload-certs\") pod \"e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc\" (UID: \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\") " pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"
Jul 12 22:48:22.206134 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.206106    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3b34640-2efb-481f-8a20-d9f5be8bad4a-istiod-ca-cert\") pod \"e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc\" (UID: \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\") " pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"
Jul 12 22:48:22.206274 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.206144    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b34640-2efb-481f-8a20-d9f5be8bad4a-credential-socket\") pod \"e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc\" (UID: \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\") " pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"
Jul 12 22:48:22.206274 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.206179    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3b34640-2efb-481f-8a20-d9f5be8bad4a-istio-envoy\") pod \"e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc\" (UID: \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\") " pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"
Jul 12 22:48:22.206274 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.206206    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3b34640-2efb-481f-8a20-d9f5be8bad4a-istio-token\") pod \"e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc\" (UID: \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\") " pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"
Jul 12 22:48:22.206274 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.206230    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2dqr\" (UniqueName: \"kubernetes.io/projected/c3b34640-2efb-481f-8a20-d9f5be8bad4a-kube-api-access-q2dqr\") pod \"e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc\" (UID: \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\") " pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"
Jul 12 22:48:22.206404 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.206299    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3b34640-2efb-481f-8a20-d9f5be8bad4a-istio-data\") pod \"e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc\" (UID: \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\") " pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"
Jul 12 22:48:22.307077 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.307042    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3b34640-2efb-481f-8a20-d9f5be8bad4a-istio-data\") pod \"e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc\" (UID: \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\") " pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"
Jul 12 22:48:22.307567 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.307095    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b34640-2efb-481f-8a20-d9f5be8bad4a-workload-socket\") pod \"e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc\" (UID: \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\") " pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"
Jul 12 22:48:22.307567 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.307279    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3b34640-2efb-481f-8a20-d9f5be8bad4a-istio-podinfo\") pod \"e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc\" (UID: \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\") " pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"
Jul 12 22:48:22.307567 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.307332    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3b34640-2efb-481f-8a20-d9f5be8bad4a-workload-certs\") pod \"e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc\" (UID: \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\") " pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"
Jul 12 22:48:22.307567 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.307354    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3b34640-2efb-481f-8a20-d9f5be8bad4a-istiod-ca-cert\") pod \"e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc\" (UID: \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\") " pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"
Jul 12 22:48:22.307567 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.307395    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b34640-2efb-481f-8a20-d9f5be8bad4a-credential-socket\") pod \"e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc\" (UID: \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\") " pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"
Jul 12 22:48:22.307567 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.307421    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3b34640-2efb-481f-8a20-d9f5be8bad4a-istio-envoy\") pod \"e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc\" (UID: \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\") " pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"
Jul 12 22:48:22.307567 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.307450    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3b34640-2efb-481f-8a20-d9f5be8bad4a-istio-token\") pod \"e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc\" (UID: \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\") " pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"
Jul 12 22:48:22.307567 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.307475    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2dqr\" (UniqueName: \"kubernetes.io/projected/c3b34640-2efb-481f-8a20-d9f5be8bad4a-kube-api-access-q2dqr\") pod \"e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc\" (UID: \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\") " pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"
Jul 12 22:48:22.307567 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.307486    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3b34640-2efb-481f-8a20-d9f5be8bad4a-istio-data\") pod \"e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc\" (UID: \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\") " pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"
Jul 12 22:48:22.307979 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.307581    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b34640-2efb-481f-8a20-d9f5be8bad4a-workload-socket\") pod \"e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc\" (UID: \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\") " pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"
Jul 12 22:48:22.307979 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.307719    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3b34640-2efb-481f-8a20-d9f5be8bad4a-workload-certs\") pod \"e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc\" (UID: \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\") " pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"
Jul 12 22:48:22.307979 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.307746    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b34640-2efb-481f-8a20-d9f5be8bad4a-credential-socket\") pod \"e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc\" (UID: \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\") " pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"
Jul 12 22:48:22.308272 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.308254    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3b34640-2efb-481f-8a20-d9f5be8bad4a-istiod-ca-cert\") pod \"e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc\" (UID: \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\") " pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"
Jul 12 22:48:22.309824 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.309807    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3b34640-2efb-481f-8a20-d9f5be8bad4a-istio-envoy\") pod \"e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc\" (UID: \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\") " pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"
Jul 12 22:48:22.310084 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.310063    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3b34640-2efb-481f-8a20-d9f5be8bad4a-istio-podinfo\") pod \"e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc\" (UID: \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\") " pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"
Jul 12 22:48:22.317545 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.317500    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3b34640-2efb-481f-8a20-d9f5be8bad4a-istio-token\") pod \"e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc\" (UID: \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\") " pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"
Jul 12 22:48:22.317672 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.317623    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2dqr\" (UniqueName: \"kubernetes.io/projected/c3b34640-2efb-481f-8a20-d9f5be8bad4a-kube-api-access-q2dqr\") pod \"e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc\" (UID: \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\") " pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"
Jul 12 22:48:22.432707 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.432616    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"
Jul 12 22:48:22.578090 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.578059    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"]
Jul 12 22:48:22.579782 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:48:22.579758    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3b34640_2efb_481f_8a20_d9f5be8bad4a.slice/crio-d74b98ede2613be3ef6d901f92adfb8017b24626ab9bd53adbad3cea54924a32 WatchSource:0}: Error finding container d74b98ede2613be3ef6d901f92adfb8017b24626ab9bd53adbad3cea54924a32: Status 404 returned error can't find the container with id d74b98ede2613be3ef6d901f92adfb8017b24626ab9bd53adbad3cea54924a32
Jul 12 22:48:22.582187 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.582145    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:48:22.582270 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.582243    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:48:22.582331 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.582314    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:48:22.850523 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.850486    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc" event={"ID":"c3b34640-2efb-481f-8a20-d9f5be8bad4a","Type":"ContainerStarted","Data":"5cd6f8dce3edb2c84db60252b2115aa762c439cfd4d80adbffcf52b6348a688d"}
Jul 12 22:48:22.850728 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.850554    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc" event={"ID":"c3b34640-2efb-481f-8a20-d9f5be8bad4a","Type":"ContainerStarted","Data":"d74b98ede2613be3ef6d901f92adfb8017b24626ab9bd53adbad3cea54924a32"}
Jul 12 22:48:22.892431 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:22.892361    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc" podStartSLOduration=0.892338261 podStartE2EDuration="892.338261ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:48:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:48:22.890731401 +0000 UTC m=+2275.204234754" watchObservedRunningTime="2026-07-12 22:48:22.892338261 +0000 UTC m=+2275.205841656"
Jul 12 22:48:23.433447 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:23.433411    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"
Jul 12 22:48:23.438986 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:23.438958    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"
Jul 12 22:48:23.856650 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:23.856617    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"
Jul 12 22:48:23.857679 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:23.857662    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"
Jul 12 22:48:26.493316 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:26.493291    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"
Jul 12 22:48:26.649521 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:26.649490    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-workload-certs\") pod \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\" (UID: \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\") "
Jul 12 22:48:26.649721 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:26.649554    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-istio-podinfo\") pod \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\" (UID: \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\") "
Jul 12 22:48:26.649721 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:26.649580    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-istio-data\") pod \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\" (UID: \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\") "
Jul 12 22:48:26.649721 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:26.649604    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-istio-envoy\") pod \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\" (UID: \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\") "
Jul 12 22:48:26.649721 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:26.649620    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgnnh\" (UniqueName: \"kubernetes.io/projected/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-kube-api-access-lgnnh\") pod \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\" (UID: \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\") "
Jul 12 22:48:26.649721 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:26.649656    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-istio-token\") pod \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\" (UID: \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\") "
Jul 12 22:48:26.649721 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:26.649682    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-workload-socket\") pod \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\" (UID: \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\") "
Jul 12 22:48:26.649721 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:26.649707    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-istiod-ca-cert\") pod \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\" (UID: \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\") "
Jul 12 22:48:26.650079 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:26.649765    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-credential-socket\") pod \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\" (UID: \"a92a7f95-b8ee-442d-80c6-bfc6261d16a9\") "
Jul 12 22:48:26.650079 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:26.649794    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a92a7f95-b8ee-442d-80c6-bfc6261d16a9" (UID: "a92a7f95-b8ee-442d-80c6-bfc6261d16a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:48:26.650079 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:26.649978    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-workload-certs\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:48:26.650079 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:26.650063    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a92a7f95-b8ee-442d-80c6-bfc6261d16a9" (UID: "a92a7f95-b8ee-442d-80c6-bfc6261d16a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:48:26.650305 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:26.650195    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a92a7f95-b8ee-442d-80c6-bfc6261d16a9" (UID: "a92a7f95-b8ee-442d-80c6-bfc6261d16a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:48:26.650305 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:26.650231    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a92a7f95-b8ee-442d-80c6-bfc6261d16a9" (UID: "a92a7f95-b8ee-442d-80c6-bfc6261d16a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 12 22:48:26.650423 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:26.650362    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-istio-data" (OuterVolumeSpecName: "istio-data") pod "a92a7f95-b8ee-442d-80c6-bfc6261d16a9" (UID: "a92a7f95-b8ee-442d-80c6-bfc6261d16a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:48:26.652074 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:26.652052    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-istio-token" (OuterVolumeSpecName: "istio-token") pod "a92a7f95-b8ee-442d-80c6-bfc6261d16a9" (UID: "a92a7f95-b8ee-442d-80c6-bfc6261d16a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 22:48:26.652281 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:26.652263    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a92a7f95-b8ee-442d-80c6-bfc6261d16a9" (UID: "a92a7f95-b8ee-442d-80c6-bfc6261d16a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:48:26.652620 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:26.652605    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a92a7f95-b8ee-442d-80c6-bfc6261d16a9" (UID: "a92a7f95-b8ee-442d-80c6-bfc6261d16a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 12 22:48:26.652755 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:26.652613    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-kube-api-access-lgnnh" (OuterVolumeSpecName: "kube-api-access-lgnnh") pod "a92a7f95-b8ee-442d-80c6-bfc6261d16a9" (UID: "a92a7f95-b8ee-442d-80c6-bfc6261d16a9"). InnerVolumeSpecName "kube-api-access-lgnnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 22:48:26.751353 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:26.751300    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-workload-socket\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:48:26.751353 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:26.751347    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-istiod-ca-cert\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:48:26.751353 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:26.751356    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-credential-socket\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:48:26.751353 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:26.751366    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-istio-podinfo\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:48:26.751654 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:26.751375    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-istio-data\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:48:26.751654 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:26.751382    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-istio-envoy\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:48:26.751654 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:26.751390    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgnnh\" (UniqueName: \"kubernetes.io/projected/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-kube-api-access-lgnnh\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:48:26.751654 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:26.751399    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a92a7f95-b8ee-442d-80c6-bfc6261d16a9-istio-token\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:48:26.874623 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:26.874577    2560 generic.go:356] "Generic (PLEG): container finished" podID="a92a7f95-b8ee-442d-80c6-bfc6261d16a9" containerID="7ac354354333a391d0d3cf0349b29022cdfce6115deff2b8422b163610c3fb11" exitCode=0
Jul 12 22:48:26.874623 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:26.874627    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"
Jul 12 22:48:26.874878 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:26.874665    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc" event={"ID":"a92a7f95-b8ee-442d-80c6-bfc6261d16a9","Type":"ContainerDied","Data":"7ac354354333a391d0d3cf0349b29022cdfce6115deff2b8422b163610c3fb11"}
Jul 12 22:48:26.874878 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:26.874722    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc" event={"ID":"a92a7f95-b8ee-442d-80c6-bfc6261d16a9","Type":"ContainerDied","Data":"6c25705168de41ca53b4d978fe021dd9b36a54edf6e11e4e9fd3594d2666043b"}
Jul 12 22:48:26.874878 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:26.874745    2560 scope.go:117] "RemoveContainer" containerID="7ac354354333a391d0d3cf0349b29022cdfce6115deff2b8422b163610c3fb11"
Jul 12 22:48:26.884351 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:26.884334    2560 scope.go:117] "RemoveContainer" containerID="3cf6d3ae7e18f55890f05880d5e643c6a6ddd85ab9601563bba14489c5542b95"
Jul 12 22:48:26.893041 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:26.893017    2560 scope.go:117] "RemoveContainer" containerID="7ac354354333a391d0d3cf0349b29022cdfce6115deff2b8422b163610c3fb11"
Jul 12 22:48:26.893333 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:48:26.893312    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ac354354333a391d0d3cf0349b29022cdfce6115deff2b8422b163610c3fb11\": container with ID starting with 7ac354354333a391d0d3cf0349b29022cdfce6115deff2b8422b163610c3fb11 not found: ID does not exist" containerID="7ac354354333a391d0d3cf0349b29022cdfce6115deff2b8422b163610c3fb11"
Jul 12 22:48:26.893470 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:26.893346    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ac354354333a391d0d3cf0349b29022cdfce6115deff2b8422b163610c3fb11"} err="failed to get container status \"7ac354354333a391d0d3cf0349b29022cdfce6115deff2b8422b163610c3fb11\": rpc error: code = NotFound desc = could not find container \"7ac354354333a391d0d3cf0349b29022cdfce6115deff2b8422b163610c3fb11\": container with ID starting with 7ac354354333a391d0d3cf0349b29022cdfce6115deff2b8422b163610c3fb11 not found: ID does not exist"
Jul 12 22:48:26.893470 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:26.893371    2560 scope.go:117] "RemoveContainer" containerID="3cf6d3ae7e18f55890f05880d5e643c6a6ddd85ab9601563bba14489c5542b95"
Jul 12 22:48:26.893689 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:48:26.893665    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cf6d3ae7e18f55890f05880d5e643c6a6ddd85ab9601563bba14489c5542b95\": container with ID starting with 3cf6d3ae7e18f55890f05880d5e643c6a6ddd85ab9601563bba14489c5542b95 not found: ID does not exist" containerID="3cf6d3ae7e18f55890f05880d5e643c6a6ddd85ab9601563bba14489c5542b95"
Jul 12 22:48:26.893738 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:26.893700    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cf6d3ae7e18f55890f05880d5e643c6a6ddd85ab9601563bba14489c5542b95"} err="failed to get container status \"3cf6d3ae7e18f55890f05880d5e643c6a6ddd85ab9601563bba14489c5542b95\": rpc error: code = NotFound desc = could not find container \"3cf6d3ae7e18f55890f05880d5e643c6a6ddd85ab9601563bba14489c5542b95\": container with ID starting with 3cf6d3ae7e18f55890f05880d5e643c6a6ddd85ab9601563bba14489c5542b95 not found: ID does not exist"
Jul 12 22:48:26.896306 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:26.896285    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"]
Jul 12 22:48:26.900015 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:26.899960    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-58d7a8-openshift-default-86cc9d66df-pt9tc"]
Jul 12 22:48:28.246876 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:28.246833    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a92a7f95-b8ee-442d-80c6-bfc6261d16a9" path="/var/lib/kubelet/pods/a92a7f95-b8ee-442d-80c6-bfc6261d16a9/volumes"
Jul 12 22:48:39.648503 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:39.647023    2560 patch_prober.go:28] interesting pod/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": read tcp 10.133.0.2:34806->10.133.0.65:15021: read: connection reset by peer" start-of-body=
Jul 12 22:48:39.648503 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:39.647103    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc" podUID="c3b34640-2efb-481f-8a20-d9f5be8bad4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": read tcp 10.133.0.2:34806->10.133.0.65:15021: read: connection reset by peer"
Jul 12 22:48:39.956076 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:39.955968    2560 generic.go:356] "Generic (PLEG): container finished" podID="c3b34640-2efb-481f-8a20-d9f5be8bad4a" containerID="5cd6f8dce3edb2c84db60252b2115aa762c439cfd4d80adbffcf52b6348a688d" exitCode=0
Jul 12 22:48:39.956076 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:39.956040    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc" event={"ID":"c3b34640-2efb-481f-8a20-d9f5be8bad4a","Type":"ContainerDied","Data":"5cd6f8dce3edb2c84db60252b2115aa762c439cfd4d80adbffcf52b6348a688d"}
Jul 12 22:48:39.956478 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:39.956462    2560 scope.go:117] "RemoveContainer" containerID="5cd6f8dce3edb2c84db60252b2115aa762c439cfd4d80adbffcf52b6348a688d"
Jul 12 22:48:39.957509 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:39.957481    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:48:39.957601 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:39.957582    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:48:39.957648 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:39.957633    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:48:40.963010 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:40.962971    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc" event={"ID":"c3b34640-2efb-481f-8a20-d9f5be8bad4a","Type":"ContainerStarted","Data":"24484f679f652aac2c2ef3e8b27b91a96e225e558aaa53030c3503d33b27721f"}
Jul 12 22:48:41.433233 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:41.432802    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"
Jul 12 22:48:41.433445 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:41.433247    2560 patch_prober.go:28] interesting pod/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Jul 12 22:48:41.433445 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:41.433299    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc" podUID="c3b34640-2efb-481f-8a20-d9f5be8bad4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Jul 12 22:48:42.433674 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:42.433615    2560 patch_prober.go:28] interesting pod/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Jul 12 22:48:42.434139 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:42.433722    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc" podUID="c3b34640-2efb-481f-8a20-d9f5be8bad4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Jul 12 22:48:43.433378 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:43.433341    2560 patch_prober.go:28] interesting pod/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Jul 12 22:48:43.433687 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:43.433659    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc" podUID="c3b34640-2efb-481f-8a20-d9f5be8bad4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Jul 12 22:48:44.433257 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:44.433212    2560 patch_prober.go:28] interesting pod/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Jul 12 22:48:44.433777 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:44.433292    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc" podUID="c3b34640-2efb-481f-8a20-d9f5be8bad4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Jul 12 22:48:45.437300 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:45.437266    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"
Jul 12 22:48:45.437706 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:45.437560    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"
Jul 12 22:48:45.438184 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:45.438169    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"
Jul 12 22:48:45.523628 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:45.523591    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8a9e41-qcgnt"]
Jul 12 22:48:45.524198 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:45.524182    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a92a7f95-b8ee-442d-80c6-bfc6261d16a9" containerName="istio-proxy"
Jul 12 22:48:45.524252 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:45.524202    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a92a7f95-b8ee-442d-80c6-bfc6261d16a9" containerName="istio-proxy"
Jul 12 22:48:45.524252 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:45.524223    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11f2f481-aa0b-46e8-a5ee-4b14336cc4eb" containerName="istio-proxy"
Jul 12 22:48:45.524252 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:45.524232    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="11f2f481-aa0b-46e8-a5ee-4b14336cc4eb" containerName="istio-proxy"
Jul 12 22:48:45.524355 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:45.524344    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a92a7f95-b8ee-442d-80c6-bfc6261d16a9" containerName="istio-proxy"
Jul 12 22:48:45.528289 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:45.528267    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8a9e41-qcgnt"
Jul 12 22:48:45.530865 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:45.530833    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 12 22:48:45.531116 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:45.531094    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlrdf\""
Jul 12 22:48:45.532972 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:45.532949    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8a9e41-qcgnt"]
Jul 12 22:48:45.632215 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:45.632177    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d928aaf-ed14-4141-9689-2bef6c82893b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8a9e41-qcgnt\" (UID: \"9d928aaf-ed14-4141-9689-2bef6c82893b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8a9e41-qcgnt"
Jul 12 22:48:45.733463 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:45.733363    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d928aaf-ed14-4141-9689-2bef6c82893b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8a9e41-qcgnt\" (UID: \"9d928aaf-ed14-4141-9689-2bef6c82893b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8a9e41-qcgnt"
Jul 12 22:48:45.734125 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:45.734101    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d928aaf-ed14-4141-9689-2bef6c82893b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8a9e41-qcgnt\" (UID: \"9d928aaf-ed14-4141-9689-2bef6c82893b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8a9e41-qcgnt"
Jul 12 22:48:45.843264 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:45.843222    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8a9e41-qcgnt"
Jul 12 22:48:45.983087 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:45.983046    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8a9e41-qcgnt"]
Jul 12 22:48:45.983643 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:48:45.983577    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d928aaf_ed14_4141_9689_2bef6c82893b.slice/crio-98ce335580dcf6573f12547b0006db9fc99e758b7781e0a48df3016bdcaaab5c WatchSource:0}: Error finding container 98ce335580dcf6573f12547b0006db9fc99e758b7781e0a48df3016bdcaaab5c: Status 404 returned error can't find the container with id 98ce335580dcf6573f12547b0006db9fc99e758b7781e0a48df3016bdcaaab5c
Jul 12 22:48:45.993580 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:45.993552    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8a9e41-qcgnt" event={"ID":"9d928aaf-ed14-4141-9689-2bef6c82893b","Type":"ContainerStarted","Data":"98ce335580dcf6573f12547b0006db9fc99e758b7781e0a48df3016bdcaaab5c"}
Jul 12 22:48:47.001151 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:47.001067    2560 generic.go:356] "Generic (PLEG): container finished" podID="9d928aaf-ed14-4141-9689-2bef6c82893b" containerID="7bbbd93521482c03a01386cd03fcf5d5cc3701de35d9c9631908239af0657e94" exitCode=0
Jul 12 22:48:47.001151 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:47.001123    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8a9e41-qcgnt" event={"ID":"9d928aaf-ed14-4141-9689-2bef6c82893b","Type":"ContainerDied","Data":"7bbbd93521482c03a01386cd03fcf5d5cc3701de35d9c9631908239af0657e94"}
Jul 12 22:48:48.143466 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:48.143433    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8a9e41-qcgnt"
Jul 12 22:48:48.259860 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:48.259766    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d928aaf-ed14-4141-9689-2bef6c82893b-maas-api-service-ca\") pod \"9d928aaf-ed14-4141-9689-2bef6c82893b\" (UID: \"9d928aaf-ed14-4141-9689-2bef6c82893b\") "
Jul 12 22:48:48.260130 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:48.260109    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d928aaf-ed14-4141-9689-2bef6c82893b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9d928aaf-ed14-4141-9689-2bef6c82893b" (UID: "9d928aaf-ed14-4141-9689-2bef6c82893b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 12 22:48:48.360804 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:48.360769    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d928aaf-ed14-4141-9689-2bef6c82893b-maas-api-service-ca\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:48:49.012316 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:49.012278    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8a9e41-qcgnt" event={"ID":"9d928aaf-ed14-4141-9689-2bef6c82893b","Type":"ContainerDied","Data":"98ce335580dcf6573f12547b0006db9fc99e758b7781e0a48df3016bdcaaab5c"}
Jul 12 22:48:49.012316 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:49.012313    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8a9e41-qcgnt"
Jul 12 22:48:49.012557 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:48:49.012338    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98ce335580dcf6573f12547b0006db9fc99e758b7781e0a48df3016bdcaaab5c"
Jul 12 22:49:11.268427 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:11.268392    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"]
Jul 12 22:49:11.269044 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:11.268620    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc" podUID="c3b34640-2efb-481f-8a20-d9f5be8bad4a" containerName="istio-proxy" containerID="cri-o://24484f679f652aac2c2ef3e8b27b91a96e225e558aaa53030c3503d33b27721f" gracePeriod=30
Jul 12 22:49:11.642662 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:11.642618    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-189e5d-vk2vj"]
Jul 12 22:49:11.643076 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:11.643058    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a92a7f95-b8ee-442d-80c6-bfc6261d16a9" containerName="istio-proxy"
Jul 12 22:49:11.643169 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:11.643079    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a92a7f95-b8ee-442d-80c6-bfc6261d16a9" containerName="istio-proxy"
Jul 12 22:49:11.643169 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:11.643105    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d928aaf-ed14-4141-9689-2bef6c82893b" containerName="revoke-keys"
Jul 12 22:49:11.643169 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:11.643113    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d928aaf-ed14-4141-9689-2bef6c82893b" containerName="revoke-keys"
Jul 12 22:49:11.643330 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:11.643206    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a92a7f95-b8ee-442d-80c6-bfc6261d16a9" containerName="istio-proxy"
Jul 12 22:49:11.643330 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:11.643223    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d928aaf-ed14-4141-9689-2bef6c82893b" containerName="revoke-keys"
Jul 12 22:49:11.646489 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:11.646468    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-189e5d-vk2vj"
Jul 12 22:49:11.649139 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:11.649119    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 12 22:49:11.649245 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:11.649181    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xlrdf\""
Jul 12 22:49:11.652946 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:11.652925    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-189e5d-vk2vj"]
Jul 12 22:49:11.784557 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:11.784493    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7aaf1b0-9be1-46a6-ac1e-3a170e5d79b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-189e5d-vk2vj\" (UID: \"d7aaf1b0-9be1-46a6-ac1e-3a170e5d79b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-189e5d-vk2vj"
Jul 12 22:49:11.885887 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:11.885846    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7aaf1b0-9be1-46a6-ac1e-3a170e5d79b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-189e5d-vk2vj\" (UID: \"d7aaf1b0-9be1-46a6-ac1e-3a170e5d79b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-189e5d-vk2vj"
Jul 12 22:49:11.886493 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:11.886464    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7aaf1b0-9be1-46a6-ac1e-3a170e5d79b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-189e5d-vk2vj\" (UID: \"d7aaf1b0-9be1-46a6-ac1e-3a170e5d79b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-189e5d-vk2vj"
Jul 12 22:49:11.958356 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:11.958278    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-189e5d-vk2vj"
Jul 12 22:49:12.117675 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:12.117648    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-189e5d-vk2vj"]
Jul 12 22:49:12.121061 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:49:12.121033    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7aaf1b0_9be1_46a6_ac1e_3a170e5d79b8.slice/crio-2987271a257d19030178b5a018ab405dc5cd9f48f075ed76d8661f51879af22d WatchSource:0}: Error finding container 2987271a257d19030178b5a018ab405dc5cd9f48f075ed76d8661f51879af22d: Status 404 returned error can't find the container with id 2987271a257d19030178b5a018ab405dc5cd9f48f075ed76d8661f51879af22d
Jul 12 22:49:12.149071 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:12.149040    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-189e5d-vk2vj" event={"ID":"d7aaf1b0-9be1-46a6-ac1e-3a170e5d79b8","Type":"ContainerStarted","Data":"2987271a257d19030178b5a018ab405dc5cd9f48f075ed76d8661f51879af22d"}
Jul 12 22:49:13.156606 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:13.156484    2560 generic.go:356] "Generic (PLEG): container finished" podID="d7aaf1b0-9be1-46a6-ac1e-3a170e5d79b8" containerID="531f3c9575eaa7c9adf66b292de36e555c3bc20c19001feec77fd287b3cdfb18" exitCode=0
Jul 12 22:49:13.156606 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:13.156576    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-189e5d-vk2vj" event={"ID":"d7aaf1b0-9be1-46a6-ac1e-3a170e5d79b8","Type":"ContainerDied","Data":"531f3c9575eaa7c9adf66b292de36e555c3bc20c19001feec77fd287b3cdfb18"}
Jul 12 22:49:14.299245 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:14.299222    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-189e5d-vk2vj"
Jul 12 22:49:14.411406 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:14.411367    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7aaf1b0-9be1-46a6-ac1e-3a170e5d79b8-maas-api-service-ca\") pod \"d7aaf1b0-9be1-46a6-ac1e-3a170e5d79b8\" (UID: \"d7aaf1b0-9be1-46a6-ac1e-3a170e5d79b8\") "
Jul 12 22:49:14.411830 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:14.411745    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7aaf1b0-9be1-46a6-ac1e-3a170e5d79b8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d7aaf1b0-9be1-46a6-ac1e-3a170e5d79b8" (UID: "d7aaf1b0-9be1-46a6-ac1e-3a170e5d79b8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 12 22:49:14.411968 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:14.411831    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7aaf1b0-9be1-46a6-ac1e-3a170e5d79b8-maas-api-service-ca\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:49:15.175895 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:15.175856    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-189e5d-vk2vj" event={"ID":"d7aaf1b0-9be1-46a6-ac1e-3a170e5d79b8","Type":"ContainerDied","Data":"2987271a257d19030178b5a018ab405dc5cd9f48f075ed76d8661f51879af22d"}
Jul 12 22:49:15.176046 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:15.175944    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2987271a257d19030178b5a018ab405dc5cd9f48f075ed76d8661f51879af22d"
Jul 12 22:49:15.176046 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:15.175871    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-189e5d-vk2vj"
Jul 12 22:49:15.439962 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:15.439874    2560 patch_prober.go:28] interesting pod/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 12 22:49:15.439962 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:15.439937    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc" podUID="c3b34640-2efb-481f-8a20-d9f5be8bad4a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 12 22:49:16.536370 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:16.536304    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"
Jul 12 22:49:16.630252 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:16.630214    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3b34640-2efb-481f-8a20-d9f5be8bad4a-istio-token\") pod \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\" (UID: \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\") "
Jul 12 22:49:16.630426 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:16.630264    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2dqr\" (UniqueName: \"kubernetes.io/projected/c3b34640-2efb-481f-8a20-d9f5be8bad4a-kube-api-access-q2dqr\") pod \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\" (UID: \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\") "
Jul 12 22:49:16.630426 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:16.630314    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3b34640-2efb-481f-8a20-d9f5be8bad4a-workload-certs\") pod \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\" (UID: \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\") "
Jul 12 22:49:16.630426 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:16.630344    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3b34640-2efb-481f-8a20-d9f5be8bad4a-istiod-ca-cert\") pod \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\" (UID: \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\") "
Jul 12 22:49:16.630426 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:16.630380    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b34640-2efb-481f-8a20-d9f5be8bad4a-credential-socket\") pod \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\" (UID: \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\") "
Jul 12 22:49:16.630426 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:16.630408    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b34640-2efb-481f-8a20-d9f5be8bad4a-workload-socket\") pod \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\" (UID: \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\") "
Jul 12 22:49:16.630726 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:16.630431    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3b34640-2efb-481f-8a20-d9f5be8bad4a-istio-data\") pod \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\" (UID: \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\") "
Jul 12 22:49:16.630726 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:16.630523    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3b34640-2efb-481f-8a20-d9f5be8bad4a-istio-podinfo\") pod \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\" (UID: \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\") "
Jul 12 22:49:16.630726 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:16.630581    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3b34640-2efb-481f-8a20-d9f5be8bad4a-istio-envoy\") pod \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\" (UID: \"c3b34640-2efb-481f-8a20-d9f5be8bad4a\") "
Jul 12 22:49:16.630726 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:16.630679    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b34640-2efb-481f-8a20-d9f5be8bad4a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c3b34640-2efb-481f-8a20-d9f5be8bad4a" (UID: "c3b34640-2efb-481f-8a20-d9f5be8bad4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:49:16.630927 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:16.630722    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b34640-2efb-481f-8a20-d9f5be8bad4a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c3b34640-2efb-481f-8a20-d9f5be8bad4a" (UID: "c3b34640-2efb-481f-8a20-d9f5be8bad4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:49:16.630927 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:16.630909    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3b34640-2efb-481f-8a20-d9f5be8bad4a-workload-certs\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:49:16.631029 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:16.630928    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b34640-2efb-481f-8a20-d9f5be8bad4a-credential-socket\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:49:16.631552 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:16.631505    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3b34640-2efb-481f-8a20-d9f5be8bad4a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c3b34640-2efb-481f-8a20-d9f5be8bad4a" (UID: "c3b34640-2efb-481f-8a20-d9f5be8bad4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 12 22:49:16.631678 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:16.631602    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b34640-2efb-481f-8a20-d9f5be8bad4a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c3b34640-2efb-481f-8a20-d9f5be8bad4a" (UID: "c3b34640-2efb-481f-8a20-d9f5be8bad4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:49:16.631798 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:16.631770    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b34640-2efb-481f-8a20-d9f5be8bad4a-istio-data" (OuterVolumeSpecName: "istio-data") pod "c3b34640-2efb-481f-8a20-d9f5be8bad4a" (UID: "c3b34640-2efb-481f-8a20-d9f5be8bad4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:49:16.632961 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:16.632922    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3b34640-2efb-481f-8a20-d9f5be8bad4a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c3b34640-2efb-481f-8a20-d9f5be8bad4a" (UID: "c3b34640-2efb-481f-8a20-d9f5be8bad4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 12 22:49:16.633057 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:16.633006    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3b34640-2efb-481f-8a20-d9f5be8bad4a-kube-api-access-q2dqr" (OuterVolumeSpecName: "kube-api-access-q2dqr") pod "c3b34640-2efb-481f-8a20-d9f5be8bad4a" (UID: "c3b34640-2efb-481f-8a20-d9f5be8bad4a"). InnerVolumeSpecName "kube-api-access-q2dqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 22:49:16.633057 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:16.633016    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3b34640-2efb-481f-8a20-d9f5be8bad4a-istio-token" (OuterVolumeSpecName: "istio-token") pod "c3b34640-2efb-481f-8a20-d9f5be8bad4a" (UID: "c3b34640-2efb-481f-8a20-d9f5be8bad4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 22:49:16.633057 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:16.633027    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b34640-2efb-481f-8a20-d9f5be8bad4a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c3b34640-2efb-481f-8a20-d9f5be8bad4a" (UID: "c3b34640-2efb-481f-8a20-d9f5be8bad4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:49:16.731408 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:16.731366    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3b34640-2efb-481f-8a20-d9f5be8bad4a-istio-podinfo\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:49:16.731408 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:16.731403    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3b34640-2efb-481f-8a20-d9f5be8bad4a-istio-envoy\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:49:16.731408 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:16.731412    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3b34640-2efb-481f-8a20-d9f5be8bad4a-istio-token\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:49:16.731671 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:16.731420    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2dqr\" (UniqueName: \"kubernetes.io/projected/c3b34640-2efb-481f-8a20-d9f5be8bad4a-kube-api-access-q2dqr\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:49:16.731671 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:16.731430    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3b34640-2efb-481f-8a20-d9f5be8bad4a-istiod-ca-cert\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:49:16.731671 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:16.731438    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b34640-2efb-481f-8a20-d9f5be8bad4a-workload-socket\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:49:16.731671 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:16.731446    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3b34640-2efb-481f-8a20-d9f5be8bad4a-istio-data\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:49:17.188352 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:17.188321    2560 generic.go:356] "Generic (PLEG): container finished" podID="c3b34640-2efb-481f-8a20-d9f5be8bad4a" containerID="24484f679f652aac2c2ef3e8b27b91a96e225e558aaa53030c3503d33b27721f" exitCode=0
Jul 12 22:49:17.188352 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:17.188357    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc" event={"ID":"c3b34640-2efb-481f-8a20-d9f5be8bad4a","Type":"ContainerDied","Data":"24484f679f652aac2c2ef3e8b27b91a96e225e558aaa53030c3503d33b27721f"}
Jul 12 22:49:17.188614 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:17.188371    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"
Jul 12 22:49:17.188614 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:17.188402    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc" event={"ID":"c3b34640-2efb-481f-8a20-d9f5be8bad4a","Type":"ContainerDied","Data":"d74b98ede2613be3ef6d901f92adfb8017b24626ab9bd53adbad3cea54924a32"}
Jul 12 22:49:17.188614 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:17.188429    2560 scope.go:117] "RemoveContainer" containerID="24484f679f652aac2c2ef3e8b27b91a96e225e558aaa53030c3503d33b27721f"
Jul 12 22:49:17.198812 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:17.198793    2560 scope.go:117] "RemoveContainer" containerID="5cd6f8dce3edb2c84db60252b2115aa762c439cfd4d80adbffcf52b6348a688d"
Jul 12 22:49:17.207364 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:17.207343    2560 scope.go:117] "RemoveContainer" containerID="24484f679f652aac2c2ef3e8b27b91a96e225e558aaa53030c3503d33b27721f"
Jul 12 22:49:17.207680 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:49:17.207656    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24484f679f652aac2c2ef3e8b27b91a96e225e558aaa53030c3503d33b27721f\": container with ID starting with 24484f679f652aac2c2ef3e8b27b91a96e225e558aaa53030c3503d33b27721f not found: ID does not exist" containerID="24484f679f652aac2c2ef3e8b27b91a96e225e558aaa53030c3503d33b27721f"
Jul 12 22:49:17.207772 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:17.207686    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24484f679f652aac2c2ef3e8b27b91a96e225e558aaa53030c3503d33b27721f"} err="failed to get container status \"24484f679f652aac2c2ef3e8b27b91a96e225e558aaa53030c3503d33b27721f\": rpc error: code = NotFound desc = could not find container \"24484f679f652aac2c2ef3e8b27b91a96e225e558aaa53030c3503d33b27721f\": container with ID starting with 24484f679f652aac2c2ef3e8b27b91a96e225e558aaa53030c3503d33b27721f not found: ID does not exist"
Jul 12 22:49:17.207772 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:17.207703    2560 scope.go:117] "RemoveContainer" containerID="5cd6f8dce3edb2c84db60252b2115aa762c439cfd4d80adbffcf52b6348a688d"
Jul 12 22:49:17.208025 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:49:17.208006    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cd6f8dce3edb2c84db60252b2115aa762c439cfd4d80adbffcf52b6348a688d\": container with ID starting with 5cd6f8dce3edb2c84db60252b2115aa762c439cfd4d80adbffcf52b6348a688d not found: ID does not exist" containerID="5cd6f8dce3edb2c84db60252b2115aa762c439cfd4d80adbffcf52b6348a688d"
Jul 12 22:49:17.208086 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:17.208034    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cd6f8dce3edb2c84db60252b2115aa762c439cfd4d80adbffcf52b6348a688d"} err="failed to get container status \"5cd6f8dce3edb2c84db60252b2115aa762c439cfd4d80adbffcf52b6348a688d\": rpc error: code = NotFound desc = could not find container \"5cd6f8dce3edb2c84db60252b2115aa762c439cfd4d80adbffcf52b6348a688d\": container with ID starting with 5cd6f8dce3edb2c84db60252b2115aa762c439cfd4d80adbffcf52b6348a688d not found: ID does not exist"
Jul 12 22:49:17.210877 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:17.210855    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"]
Jul 12 22:49:17.215064 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:17.215045    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-8a9e41-openshift-default-6ff6f96d7f-7kqhc"]
Jul 12 22:49:18.246885 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:18.246852    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3b34640-2efb-481f-8a20-d9f5be8bad4a" path="/var/lib/kubelet/pods/c3b34640-2efb-481f-8a20-d9f5be8bad4a/volumes"
Jul 12 22:49:38.380893 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.380859    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"]
Jul 12 22:49:38.381250 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.381221    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3b34640-2efb-481f-8a20-d9f5be8bad4a" containerName="istio-proxy"
Jul 12 22:49:38.381250 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.381231    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3b34640-2efb-481f-8a20-d9f5be8bad4a" containerName="istio-proxy"
Jul 12 22:49:38.381250 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.381241    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3b34640-2efb-481f-8a20-d9f5be8bad4a" containerName="istio-proxy"
Jul 12 22:49:38.381250 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.381247    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3b34640-2efb-481f-8a20-d9f5be8bad4a" containerName="istio-proxy"
Jul 12 22:49:38.381395 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.381264    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7aaf1b0-9be1-46a6-ac1e-3a170e5d79b8" containerName="revoke-keys"
Jul 12 22:49:38.381395 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.381271    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7aaf1b0-9be1-46a6-ac1e-3a170e5d79b8" containerName="revoke-keys"
Jul 12 22:49:38.381395 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.381329    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3b34640-2efb-481f-8a20-d9f5be8bad4a" containerName="istio-proxy"
Jul 12 22:49:38.381395 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.381337    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7aaf1b0-9be1-46a6-ac1e-3a170e5d79b8" containerName="revoke-keys"
Jul 12 22:49:38.384515 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.384499    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"
Jul 12 22:49:38.387133 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.387102    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 12 22:49:38.387133 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.387132    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-e321a4-openshift-default-dockercfg-5g5hc\""
Jul 12 22:49:38.397623 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.397597    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"]
Jul 12 22:49:38.408970 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.408940    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3697d60a-84d8-4667-9298-40b51c9a26f9-istiod-ca-cert\") pod \"e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5\" (UID: \"3697d60a-84d8-4667-9298-40b51c9a26f9\") " pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"
Jul 12 22:49:38.409121 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.408977    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3697d60a-84d8-4667-9298-40b51c9a26f9-workload-socket\") pod \"e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5\" (UID: \"3697d60a-84d8-4667-9298-40b51c9a26f9\") " pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"
Jul 12 22:49:38.409121 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.409007    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3697d60a-84d8-4667-9298-40b51c9a26f9-istio-envoy\") pod \"e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5\" (UID: \"3697d60a-84d8-4667-9298-40b51c9a26f9\") " pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"
Jul 12 22:49:38.409121 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.409069    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3697d60a-84d8-4667-9298-40b51c9a26f9-istio-podinfo\") pod \"e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5\" (UID: \"3697d60a-84d8-4667-9298-40b51c9a26f9\") " pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"
Jul 12 22:49:38.409340 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.409135    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3697d60a-84d8-4667-9298-40b51c9a26f9-credential-socket\") pod \"e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5\" (UID: \"3697d60a-84d8-4667-9298-40b51c9a26f9\") " pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"
Jul 12 22:49:38.409340 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.409188    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3697d60a-84d8-4667-9298-40b51c9a26f9-workload-certs\") pod \"e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5\" (UID: \"3697d60a-84d8-4667-9298-40b51c9a26f9\") " pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"
Jul 12 22:49:38.409340 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.409228    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3697d60a-84d8-4667-9298-40b51c9a26f9-istio-data\") pod \"e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5\" (UID: \"3697d60a-84d8-4667-9298-40b51c9a26f9\") " pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"
Jul 12 22:49:38.409340 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.409263    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3697d60a-84d8-4667-9298-40b51c9a26f9-istio-token\") pod \"e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5\" (UID: \"3697d60a-84d8-4667-9298-40b51c9a26f9\") " pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"
Jul 12 22:49:38.409340 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.409281    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjnvx\" (UniqueName: \"kubernetes.io/projected/3697d60a-84d8-4667-9298-40b51c9a26f9-kube-api-access-gjnvx\") pod \"e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5\" (UID: \"3697d60a-84d8-4667-9298-40b51c9a26f9\") " pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"
Jul 12 22:49:38.510645 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.510609    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3697d60a-84d8-4667-9298-40b51c9a26f9-istio-podinfo\") pod \"e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5\" (UID: \"3697d60a-84d8-4667-9298-40b51c9a26f9\") " pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"
Jul 12 22:49:38.510853 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.510666    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3697d60a-84d8-4667-9298-40b51c9a26f9-credential-socket\") pod \"e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5\" (UID: \"3697d60a-84d8-4667-9298-40b51c9a26f9\") " pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"
Jul 12 22:49:38.510853 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.510708    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3697d60a-84d8-4667-9298-40b51c9a26f9-workload-certs\") pod \"e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5\" (UID: \"3697d60a-84d8-4667-9298-40b51c9a26f9\") " pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"
Jul 12 22:49:38.510853 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.510834    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3697d60a-84d8-4667-9298-40b51c9a26f9-istio-data\") pod \"e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5\" (UID: \"3697d60a-84d8-4667-9298-40b51c9a26f9\") " pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"
Jul 12 22:49:38.511033 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.510876    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3697d60a-84d8-4667-9298-40b51c9a26f9-istio-token\") pod \"e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5\" (UID: \"3697d60a-84d8-4667-9298-40b51c9a26f9\") " pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"
Jul 12 22:49:38.511033 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.510900    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjnvx\" (UniqueName: \"kubernetes.io/projected/3697d60a-84d8-4667-9298-40b51c9a26f9-kube-api-access-gjnvx\") pod \"e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5\" (UID: \"3697d60a-84d8-4667-9298-40b51c9a26f9\") " pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"
Jul 12 22:49:38.511033 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.510929    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3697d60a-84d8-4667-9298-40b51c9a26f9-istiod-ca-cert\") pod \"e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5\" (UID: \"3697d60a-84d8-4667-9298-40b51c9a26f9\") " pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"
Jul 12 22:49:38.511033 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.510960    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3697d60a-84d8-4667-9298-40b51c9a26f9-workload-socket\") pod \"e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5\" (UID: \"3697d60a-84d8-4667-9298-40b51c9a26f9\") " pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"
Jul 12 22:49:38.511033 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.511011    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3697d60a-84d8-4667-9298-40b51c9a26f9-istio-envoy\") pod \"e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5\" (UID: \"3697d60a-84d8-4667-9298-40b51c9a26f9\") " pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"
Jul 12 22:49:38.511279 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.511186    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3697d60a-84d8-4667-9298-40b51c9a26f9-workload-certs\") pod \"e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5\" (UID: \"3697d60a-84d8-4667-9298-40b51c9a26f9\") " pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"
Jul 12 22:49:38.511316 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.511292    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3697d60a-84d8-4667-9298-40b51c9a26f9-istio-data\") pod \"e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5\" (UID: \"3697d60a-84d8-4667-9298-40b51c9a26f9\") " pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"
Jul 12 22:49:38.511411 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.511387    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3697d60a-84d8-4667-9298-40b51c9a26f9-credential-socket\") pod \"e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5\" (UID: \"3697d60a-84d8-4667-9298-40b51c9a26f9\") " pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"
Jul 12 22:49:38.511455 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.511387    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3697d60a-84d8-4667-9298-40b51c9a26f9-workload-socket\") pod \"e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5\" (UID: \"3697d60a-84d8-4667-9298-40b51c9a26f9\") " pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"
Jul 12 22:49:38.511766 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.511746    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3697d60a-84d8-4667-9298-40b51c9a26f9-istiod-ca-cert\") pod \"e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5\" (UID: \"3697d60a-84d8-4667-9298-40b51c9a26f9\") " pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"
Jul 12 22:49:38.513406 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.513379    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3697d60a-84d8-4667-9298-40b51c9a26f9-istio-envoy\") pod \"e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5\" (UID: \"3697d60a-84d8-4667-9298-40b51c9a26f9\") " pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"
Jul 12 22:49:38.513502 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.513491    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3697d60a-84d8-4667-9298-40b51c9a26f9-istio-podinfo\") pod \"e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5\" (UID: \"3697d60a-84d8-4667-9298-40b51c9a26f9\") " pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"
Jul 12 22:49:38.518736 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.518707    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3697d60a-84d8-4667-9298-40b51c9a26f9-istio-token\") pod \"e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5\" (UID: \"3697d60a-84d8-4667-9298-40b51c9a26f9\") " pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"
Jul 12 22:49:38.518834 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.518811    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjnvx\" (UniqueName: \"kubernetes.io/projected/3697d60a-84d8-4667-9298-40b51c9a26f9-kube-api-access-gjnvx\") pod \"e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5\" (UID: \"3697d60a-84d8-4667-9298-40b51c9a26f9\") " pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"
Jul 12 22:49:38.696502 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.696407    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"
Jul 12 22:49:38.839829 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.839688    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"]
Jul 12 22:49:38.841347 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:49:38.841317    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3697d60a_84d8_4667_9298_40b51c9a26f9.slice/crio-8c490d9ed1e25c1bfd239e69ea2e568ad36d38ebf69a69e997b94803179a9cef WatchSource:0}: Error finding container 8c490d9ed1e25c1bfd239e69ea2e568ad36d38ebf69a69e997b94803179a9cef: Status 404 returned error can't find the container with id 8c490d9ed1e25c1bfd239e69ea2e568ad36d38ebf69a69e997b94803179a9cef
Jul 12 22:49:38.843394 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.843361    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:49:38.843487 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.843431    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:49:38.843487 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:38.843460    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:49:39.313911 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:39.313878    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5" event={"ID":"3697d60a-84d8-4667-9298-40b51c9a26f9","Type":"ContainerStarted","Data":"8240ea4fcd4635aa50215f46fa8ebe87823e2f9b389d493ed54f4d574755f226"}
Jul 12 22:49:39.314106 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:39.313935    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5" event={"ID":"3697d60a-84d8-4667-9298-40b51c9a26f9","Type":"ContainerStarted","Data":"8c490d9ed1e25c1bfd239e69ea2e568ad36d38ebf69a69e997b94803179a9cef"}
Jul 12 22:49:39.335443 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:39.335388    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5" podStartSLOduration=1.3353684000000001 podStartE2EDuration="1.3353684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:49:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:49:39.334677197 +0000 UTC m=+2351.648180552" watchObservedRunningTime="2026-07-12 22:49:39.3353684 +0000 UTC m=+2351.648871755"
Jul 12 22:49:39.696982 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:39.696947    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"
Jul 12 22:49:39.701939 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:39.701911    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"
Jul 12 22:49:40.319136 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:40.319104    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"
Jul 12 22:49:40.320111 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:40.320088    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"
Jul 12 22:49:46.395875 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:46.395840    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"]
Jul 12 22:49:46.396577 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:46.396554    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3b34640-2efb-481f-8a20-d9f5be8bad4a" containerName="istio-proxy"
Jul 12 22:49:46.401272 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:46.401252    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"
Jul 12 22:49:46.403956 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:46.403937    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-ecdd13-openshift-default-dockercfg-44nbs\""
Jul 12 22:49:46.411453 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:46.411425    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"]
Jul 12 22:49:46.481134 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:46.481093    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-workload-certs\") pod \"e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6\" (UID: \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\") " pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"
Jul 12 22:49:46.481134 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:46.481135    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-credential-socket\") pod \"e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6\" (UID: \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\") " pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"
Jul 12 22:49:46.481342 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:46.481161    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-istio-token\") pod \"e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6\" (UID: \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\") " pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"
Jul 12 22:49:46.481342 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:46.481251    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-istio-data\") pod \"e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6\" (UID: \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\") " pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"
Jul 12 22:49:46.481342 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:46.481300    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-istio-envoy\") pod \"e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6\" (UID: \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\") " pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"
Jul 12 22:49:46.481449 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:46.481346    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-workload-socket\") pod \"e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6\" (UID: \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\") " pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"
Jul 12 22:49:46.481449 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:46.481365    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-istiod-ca-cert\") pod \"e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6\" (UID: \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\") " pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"
Jul 12 22:49:46.481449 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:46.481424    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bskfs\" (UniqueName: \"kubernetes.io/projected/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-kube-api-access-bskfs\") pod \"e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6\" (UID: \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\") " pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"
Jul 12 22:49:46.481588 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:46.481465    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-istio-podinfo\") pod \"e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6\" (UID: \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\") " pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"
Jul 12 22:49:46.582410 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:46.582372    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-istio-envoy\") pod \"e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6\" (UID: \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\") " pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"
Jul 12 22:49:46.582410 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:46.582413    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-workload-socket\") pod \"e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6\" (UID: \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\") " pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"
Jul 12 22:49:46.582692 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:46.582436    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-istiod-ca-cert\") pod \"e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6\" (UID: \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\") " pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"
Jul 12 22:49:46.582692 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:46.582487    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bskfs\" (UniqueName: \"kubernetes.io/projected/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-kube-api-access-bskfs\") pod \"e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6\" (UID: \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\") " pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"
Jul 12 22:49:46.582692 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:46.582563    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-istio-podinfo\") pod \"e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6\" (UID: \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\") " pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"
Jul 12 22:49:46.582692 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:46.582612    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-workload-certs\") pod \"e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6\" (UID: \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\") " pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"
Jul 12 22:49:46.582692 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:46.582639    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-credential-socket\") pod \"e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6\" (UID: \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\") " pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"
Jul 12 22:49:46.582692 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:46.582667    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-istio-token\") pod \"e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6\" (UID: \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\") " pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"
Jul 12 22:49:46.582996 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:46.582703    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-istio-data\") pod \"e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6\" (UID: \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\") " pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"
Jul 12 22:49:46.582996 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:46.582885    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-workload-socket\") pod \"e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6\" (UID: \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\") " pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"
Jul 12 22:49:46.583208 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:46.583179    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-workload-certs\") pod \"e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6\" (UID: \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\") " pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"
Jul 12 22:49:46.583562 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:46.583179    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-credential-socket\") pod \"e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6\" (UID: \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\") " pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"
Jul 12 22:49:46.583726 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:46.583589    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-istio-data\") pod \"e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6\" (UID: \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\") " pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"
Jul 12 22:49:46.583838 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:46.583596    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-istiod-ca-cert\") pod \"e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6\" (UID: \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\") " pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"
Jul 12 22:49:46.585159 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:46.585137    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-istio-envoy\") pod \"e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6\" (UID: \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\") " pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"
Jul 12 22:49:46.585485 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:46.585460    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-istio-podinfo\") pod \"e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6\" (UID: \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\") " pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"
Jul 12 22:49:46.591797 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:46.591775    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-istio-token\") pod \"e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6\" (UID: \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\") " pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"
Jul 12 22:49:46.591892 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:46.591833    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bskfs\" (UniqueName: \"kubernetes.io/projected/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-kube-api-access-bskfs\") pod \"e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6\" (UID: \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\") " pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"
Jul 12 22:49:46.713344 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:46.713225    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"
Jul 12 22:49:46.866229 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:46.866193    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"]
Jul 12 22:49:46.868913 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:49:46.868880    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b95dbe7_f37a_45ab_a0ee_f1a270b95f81.slice/crio-3661cffa9c8fbdfdcb94adad6b6a6fbada139b59ea725e23ae8ea83f7aaf7fe1 WatchSource:0}: Error finding container 3661cffa9c8fbdfdcb94adad6b6a6fbada139b59ea725e23ae8ea83f7aaf7fe1: Status 404 returned error can't find the container with id 3661cffa9c8fbdfdcb94adad6b6a6fbada139b59ea725e23ae8ea83f7aaf7fe1
Jul 12 22:49:46.871198 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:46.871155    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:49:46.871278 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:46.871252    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:49:46.871318 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:46.871297    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:49:47.358831 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:47.358788    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6" event={"ID":"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81","Type":"ContainerStarted","Data":"158014843cb2b979de0474d3f87549b01c34fda43048dda0e60b2360296869db"}
Jul 12 22:49:47.359046 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:47.358846    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6" event={"ID":"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81","Type":"ContainerStarted","Data":"3661cffa9c8fbdfdcb94adad6b6a6fbada139b59ea725e23ae8ea83f7aaf7fe1"}
Jul 12 22:49:47.380745 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:47.380688    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6" podStartSLOduration=1.3806670890000001 podStartE2EDuration="1.380667089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:49:47.377753157 +0000 UTC m=+2359.691256524" watchObservedRunningTime="2026-07-12 22:49:47.380667089 +0000 UTC m=+2359.694170442"
Jul 12 22:49:47.713681 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:47.713588    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"
Jul 12 22:49:47.714810 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:47.714780    2560 patch_prober.go:28] interesting pod/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Jul 12 22:49:47.714939 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:47.714851    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6" podUID="4b95dbe7-f37a-45ab-a0ee-f1a270b95f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Jul 12 22:49:48.717660 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:48.717622    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"
Jul 12 22:49:49.371834 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:49.371802    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"
Jul 12 22:49:49.372772 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:49.372750    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"
Jul 12 22:49:55.323558 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:55.320451    2560 patch_prober.go:28] interesting pod/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Jul 12 22:49:55.323558 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:55.320555    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5" podUID="3697d60a-84d8-4667-9298-40b51c9a26f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Jul 12 22:49:55.407806 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:55.407768    2560 generic.go:356] "Generic (PLEG): container finished" podID="3697d60a-84d8-4667-9298-40b51c9a26f9" containerID="8240ea4fcd4635aa50215f46fa8ebe87823e2f9b389d493ed54f4d574755f226" exitCode=0
Jul 12 22:49:55.408006 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:55.407841    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5" event={"ID":"3697d60a-84d8-4667-9298-40b51c9a26f9","Type":"ContainerDied","Data":"8240ea4fcd4635aa50215f46fa8ebe87823e2f9b389d493ed54f4d574755f226"}
Jul 12 22:49:55.408264 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:55.408248    2560 scope.go:117] "RemoveContainer" containerID="8240ea4fcd4635aa50215f46fa8ebe87823e2f9b389d493ed54f4d574755f226"
Jul 12 22:49:55.409205 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:55.409176    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:49:55.409265 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:55.409245    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:49:55.409304 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:55.409289    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:49:56.414891 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:56.414853    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5" event={"ID":"3697d60a-84d8-4667-9298-40b51c9a26f9","Type":"ContainerStarted","Data":"691d2f3d800c7b9cf40914d3d53c815e42dbee40c4f6e4818f65b8d72ee9b302"}
Jul 12 22:49:56.696951 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:56.696856    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"
Jul 12 22:49:56.697206 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:56.697171    2560 patch_prober.go:28] interesting pod/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Jul 12 22:49:56.697317 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:56.697235    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5" podUID="3697d60a-84d8-4667-9298-40b51c9a26f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Jul 12 22:49:57.697392 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:57.697344    2560 patch_prober.go:28] interesting pod/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Jul 12 22:49:57.697882 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:57.697418    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5" podUID="3697d60a-84d8-4667-9298-40b51c9a26f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Jul 12 22:49:58.697457 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:58.697412    2560 patch_prober.go:28] interesting pod/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Jul 12 22:49:58.697971 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:58.697473    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5" podUID="3697d60a-84d8-4667-9298-40b51c9a26f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Jul 12 22:49:59.697180 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:59.697134    2560 patch_prober.go:28] interesting pod/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Jul 12 22:49:59.697334 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:49:59.697201    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5" podUID="3697d60a-84d8-4667-9298-40b51c9a26f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Jul 12 22:50:00.697178 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:00.697132    2560 patch_prober.go:28] interesting pod/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Jul 12 22:50:00.697711 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:00.697198    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5" podUID="3697d60a-84d8-4667-9298-40b51c9a26f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Jul 12 22:50:01.700689 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:01.700658    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"
Jul 12 22:50:01.701147 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:01.701124    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"
Jul 12 22:50:01.701885 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:01.701867    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"
Jul 12 22:50:02.461841 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:02.461768    2560 generic.go:356] "Generic (PLEG): container finished" podID="4b95dbe7-f37a-45ab-a0ee-f1a270b95f81" containerID="158014843cb2b979de0474d3f87549b01c34fda43048dda0e60b2360296869db" exitCode=0
Jul 12 22:50:02.462023 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:02.461841    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6" event={"ID":"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81","Type":"ContainerDied","Data":"158014843cb2b979de0474d3f87549b01c34fda43048dda0e60b2360296869db"}
Jul 12 22:50:02.462398 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:02.462381    2560 scope.go:117] "RemoveContainer" containerID="158014843cb2b979de0474d3f87549b01c34fda43048dda0e60b2360296869db"
Jul 12 22:50:02.463458 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:02.463430    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:50:02.463510 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:02.463500    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:50:02.463564 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:02.463555    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 12 22:50:03.470366 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:03.470324    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6" event={"ID":"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81","Type":"ContainerStarted","Data":"759fce98424c7f82071483773a2d6c1d132e00d981d34955e51c118dba740081"}
Jul 12 22:50:03.714138 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:03.714104    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"
Jul 12 22:50:03.715292 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:03.715266    2560 patch_prober.go:28] interesting pod/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Jul 12 22:50:03.715410 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:03.715313    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6" podUID="4b95dbe7-f37a-45ab-a0ee-f1a270b95f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Jul 12 22:50:04.372011 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:04.371976    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"
Jul 12 22:50:04.714292 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:04.714199    2560 patch_prober.go:28] interesting pod/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Jul 12 22:50:04.714292 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:04.714258    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6" podUID="4b95dbe7-f37a-45ab-a0ee-f1a270b95f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Jul 12 22:50:05.714270 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:05.714226    2560 patch_prober.go:28] interesting pod/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Jul 12 22:50:05.714459 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:05.714286    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6" podUID="4b95dbe7-f37a-45ab-a0ee-f1a270b95f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Jul 12 22:50:06.713833 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:06.713792    2560 patch_prober.go:28] interesting pod/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Jul 12 22:50:06.714024 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:06.713852    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6" podUID="4b95dbe7-f37a-45ab-a0ee-f1a270b95f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Jul 12 22:50:07.717373 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:07.717342    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"
Jul 12 22:50:07.718353 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:07.718334    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"
Jul 12 22:50:12.581800 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:12.581761    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-e321a4/auth-test-model-e321a4-kserve-58d8788b59-nmzp9"]
Jul 12 22:50:12.586177 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:12.586157    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e321a4/auth-test-model-e321a4-kserve-58d8788b59-nmzp9"
Jul 12 22:50:12.589069 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:12.589043    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-e321a4\"/\"kube-root-ca.crt\""
Jul 12 22:50:12.590454 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:12.590427    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e321a4\"/\"default-dockercfg-fhs2q\""
Jul 12 22:50:12.590610 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:12.590429    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e321a4\"/\"auth-test-model-e321a4-kserve-self-signed-certs\""
Jul 12 22:50:12.590610 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:12.590436    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-e321a4\"/\"openshift-service-ca.crt\""
Jul 12 22:50:12.593224 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:12.593199    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e321a4/auth-test-model-e321a4-kserve-58d8788b59-nmzp9"]
Jul 12 22:50:12.754412 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:12.754376    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nflsw\" (UniqueName: \"kubernetes.io/projected/832aed83-bd89-432f-bdc5-bea0994f6a21-kube-api-access-nflsw\") pod \"auth-test-model-e321a4-kserve-58d8788b59-nmzp9\" (UID: \"832aed83-bd89-432f-bdc5-bea0994f6a21\") " pod="ai-tenant-e2e-shared-a-e321a4/auth-test-model-e321a4-kserve-58d8788b59-nmzp9"
Jul 12 22:50:12.754412 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:12.754413    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/832aed83-bd89-432f-bdc5-bea0994f6a21-home\") pod \"auth-test-model-e321a4-kserve-58d8788b59-nmzp9\" (UID: \"832aed83-bd89-432f-bdc5-bea0994f6a21\") " pod="ai-tenant-e2e-shared-a-e321a4/auth-test-model-e321a4-kserve-58d8788b59-nmzp9"
Jul 12 22:50:12.754643 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:12.754431    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/832aed83-bd89-432f-bdc5-bea0994f6a21-tls-certs\") pod \"auth-test-model-e321a4-kserve-58d8788b59-nmzp9\" (UID: \"832aed83-bd89-432f-bdc5-bea0994f6a21\") " pod="ai-tenant-e2e-shared-a-e321a4/auth-test-model-e321a4-kserve-58d8788b59-nmzp9"
Jul 12 22:50:12.754643 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:12.754573    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/832aed83-bd89-432f-bdc5-bea0994f6a21-tmp-dir\") pod \"auth-test-model-e321a4-kserve-58d8788b59-nmzp9\" (UID: \"832aed83-bd89-432f-bdc5-bea0994f6a21\") " pod="ai-tenant-e2e-shared-a-e321a4/auth-test-model-e321a4-kserve-58d8788b59-nmzp9"
Jul 12 22:50:12.754643 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:12.754627    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/832aed83-bd89-432f-bdc5-bea0994f6a21-model-cache\") pod \"auth-test-model-e321a4-kserve-58d8788b59-nmzp9\" (UID: \"832aed83-bd89-432f-bdc5-bea0994f6a21\") " pod="ai-tenant-e2e-shared-a-e321a4/auth-test-model-e321a4-kserve-58d8788b59-nmzp9"
Jul 12 22:50:12.754744 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:12.754669    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/832aed83-bd89-432f-bdc5-bea0994f6a21-dshm\") pod \"auth-test-model-e321a4-kserve-58d8788b59-nmzp9\" (UID: \"832aed83-bd89-432f-bdc5-bea0994f6a21\") " pod="ai-tenant-e2e-shared-a-e321a4/auth-test-model-e321a4-kserve-58d8788b59-nmzp9"
Jul 12 22:50:12.855364 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:12.855276    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/832aed83-bd89-432f-bdc5-bea0994f6a21-model-cache\") pod \"auth-test-model-e321a4-kserve-58d8788b59-nmzp9\" (UID: \"832aed83-bd89-432f-bdc5-bea0994f6a21\") " pod="ai-tenant-e2e-shared-a-e321a4/auth-test-model-e321a4-kserve-58d8788b59-nmzp9"
Jul 12 22:50:12.855364 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:12.855327    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/832aed83-bd89-432f-bdc5-bea0994f6a21-dshm\") pod \"auth-test-model-e321a4-kserve-58d8788b59-nmzp9\" (UID: \"832aed83-bd89-432f-bdc5-bea0994f6a21\") " pod="ai-tenant-e2e-shared-a-e321a4/auth-test-model-e321a4-kserve-58d8788b59-nmzp9"
Jul 12 22:50:12.855645 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:12.855376    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nflsw\" (UniqueName: \"kubernetes.io/projected/832aed83-bd89-432f-bdc5-bea0994f6a21-kube-api-access-nflsw\") pod \"auth-test-model-e321a4-kserve-58d8788b59-nmzp9\" (UID: \"832aed83-bd89-432f-bdc5-bea0994f6a21\") " pod="ai-tenant-e2e-shared-a-e321a4/auth-test-model-e321a4-kserve-58d8788b59-nmzp9"
Jul 12 22:50:12.855645 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:12.855395    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/832aed83-bd89-432f-bdc5-bea0994f6a21-home\") pod \"auth-test-model-e321a4-kserve-58d8788b59-nmzp9\" (UID: \"832aed83-bd89-432f-bdc5-bea0994f6a21\") " pod="ai-tenant-e2e-shared-a-e321a4/auth-test-model-e321a4-kserve-58d8788b59-nmzp9"
Jul 12 22:50:12.855645 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:12.855412    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/832aed83-bd89-432f-bdc5-bea0994f6a21-tls-certs\") pod \"auth-test-model-e321a4-kserve-58d8788b59-nmzp9\" (UID: \"832aed83-bd89-432f-bdc5-bea0994f6a21\") " pod="ai-tenant-e2e-shared-a-e321a4/auth-test-model-e321a4-kserve-58d8788b59-nmzp9"
Jul 12 22:50:12.855645 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:12.855433    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/832aed83-bd89-432f-bdc5-bea0994f6a21-tmp-dir\") pod \"auth-test-model-e321a4-kserve-58d8788b59-nmzp9\" (UID: \"832aed83-bd89-432f-bdc5-bea0994f6a21\") " pod="ai-tenant-e2e-shared-a-e321a4/auth-test-model-e321a4-kserve-58d8788b59-nmzp9"
Jul 12 22:50:12.855860 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:12.855777    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/832aed83-bd89-432f-bdc5-bea0994f6a21-model-cache\") pod \"auth-test-model-e321a4-kserve-58d8788b59-nmzp9\" (UID: \"832aed83-bd89-432f-bdc5-bea0994f6a21\") " pod="ai-tenant-e2e-shared-a-e321a4/auth-test-model-e321a4-kserve-58d8788b59-nmzp9"
Jul 12 22:50:12.855860 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:12.855834    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/832aed83-bd89-432f-bdc5-bea0994f6a21-tmp-dir\") pod \"auth-test-model-e321a4-kserve-58d8788b59-nmzp9\" (UID: \"832aed83-bd89-432f-bdc5-bea0994f6a21\") " pod="ai-tenant-e2e-shared-a-e321a4/auth-test-model-e321a4-kserve-58d8788b59-nmzp9"
Jul 12 22:50:12.855980 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:12.855868    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/832aed83-bd89-432f-bdc5-bea0994f6a21-home\") pod \"auth-test-model-e321a4-kserve-58d8788b59-nmzp9\" (UID: \"832aed83-bd89-432f-bdc5-bea0994f6a21\") " pod="ai-tenant-e2e-shared-a-e321a4/auth-test-model-e321a4-kserve-58d8788b59-nmzp9"
Jul 12 22:50:12.857769 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:12.857746    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/832aed83-bd89-432f-bdc5-bea0994f6a21-dshm\") pod \"auth-test-model-e321a4-kserve-58d8788b59-nmzp9\" (UID: \"832aed83-bd89-432f-bdc5-bea0994f6a21\") " pod="ai-tenant-e2e-shared-a-e321a4/auth-test-model-e321a4-kserve-58d8788b59-nmzp9"
Jul 12 22:50:12.858157 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:12.858135    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/832aed83-bd89-432f-bdc5-bea0994f6a21-tls-certs\") pod \"auth-test-model-e321a4-kserve-58d8788b59-nmzp9\" (UID: \"832aed83-bd89-432f-bdc5-bea0994f6a21\") " pod="ai-tenant-e2e-shared-a-e321a4/auth-test-model-e321a4-kserve-58d8788b59-nmzp9"
Jul 12 22:50:12.864383 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:12.864361    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nflsw\" (UniqueName: \"kubernetes.io/projected/832aed83-bd89-432f-bdc5-bea0994f6a21-kube-api-access-nflsw\") pod \"auth-test-model-e321a4-kserve-58d8788b59-nmzp9\" (UID: \"832aed83-bd89-432f-bdc5-bea0994f6a21\") " pod="ai-tenant-e2e-shared-a-e321a4/auth-test-model-e321a4-kserve-58d8788b59-nmzp9"
Jul 12 22:50:12.897187 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:12.897142    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e321a4/auth-test-model-e321a4-kserve-58d8788b59-nmzp9"
Jul 12 22:50:13.042966 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:13.042937    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e321a4/auth-test-model-e321a4-kserve-58d8788b59-nmzp9"]
Jul 12 22:50:13.044399 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:50:13.044371    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod832aed83_bd89_432f_bdc5_bea0994f6a21.slice/crio-70c51250f8426f419b6d05940aadeeb3da1ac5025d2a4d0f8bfbd0dfa2ed6f97 WatchSource:0}: Error finding container 70c51250f8426f419b6d05940aadeeb3da1ac5025d2a4d0f8bfbd0dfa2ed6f97: Status 404 returned error can't find the container with id 70c51250f8426f419b6d05940aadeeb3da1ac5025d2a4d0f8bfbd0dfa2ed6f97
Jul 12 22:50:13.527204 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:13.527165    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e321a4/auth-test-model-e321a4-kserve-58d8788b59-nmzp9" event={"ID":"832aed83-bd89-432f-bdc5-bea0994f6a21","Type":"ContainerStarted","Data":"faf2b6f867aa36e674d3cbfe73c612ce61263ad3ae0d611f68bfcabd645bb6e7"}
Jul 12 22:50:13.527374 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:13.527211    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e321a4/auth-test-model-e321a4-kserve-58d8788b59-nmzp9" event={"ID":"832aed83-bd89-432f-bdc5-bea0994f6a21","Type":"ContainerStarted","Data":"70c51250f8426f419b6d05940aadeeb3da1ac5025d2a4d0f8bfbd0dfa2ed6f97"}
Jul 12 22:50:13.543757 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:13.543706    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-e321a4/auth-test-model-e321a4-kserve-58d8788b59-nmzp9" podStartSLOduration=1.543688044 podStartE2EDuration="1.543688044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:50:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:50:13.54337029 +0000 UTC m=+2385.856873643" watchObservedRunningTime="2026-07-12 22:50:13.543688044 +0000 UTC m=+2385.857191398"
Jul 12 22:50:22.897694 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:22.897653    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-e321a4/auth-test-model-e321a4-kserve-58d8788b59-nmzp9"
Jul 12 22:50:22.898219 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:22.897719    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-e321a4/auth-test-model-e321a4-kserve-58d8788b59-nmzp9"
Jul 12 22:50:22.910220 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:22.910195    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-e321a4/auth-test-model-e321a4-kserve-58d8788b59-nmzp9"
Jul 12 22:50:23.598870 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:23.598841    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-e321a4/auth-test-model-e321a4-kserve-58d8788b59-nmzp9"
Jul 12 22:50:28.351108 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:28.351078    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52pbf_1b1f17bc-a992-4c72-8fdc-a33a3c3649c4/ovn-acl-logging/0.log"
Jul 12 22:50:28.354769 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:28.354749    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52pbf_1b1f17bc-a992-4c72-8fdc-a33a3c3649c4/ovn-acl-logging/0.log"
Jul 12 22:50:47.867439 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:47.867400    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-ecdd13/auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95"]
Jul 12 22:50:47.871327 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:47.871308    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ecdd13/auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95"
Jul 12 22:50:47.874332 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:47.874307    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ecdd13\"/\"openshift-service-ca.crt\""
Jul 12 22:50:47.875754 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:47.875732    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ecdd13\"/\"kube-root-ca.crt\""
Jul 12 22:50:47.875754 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:47.875742    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ecdd13\"/\"auth-test-model-ecdd13-kserve-self-signed-certs\""
Jul 12 22:50:47.875977 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:47.875739    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ecdd13\"/\"default-dockercfg-2vrrp\""
Jul 12 22:50:47.880907 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:47.880881    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ecdd13/auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95"]
Jul 12 22:50:47.970675 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:47.970637    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e074b79-9415-45a2-814f-64bfd681db68-home\") pod \"auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95\" (UID: \"8e074b79-9415-45a2-814f-64bfd681db68\") " pod="ai-tenant-e2e-shared-b-ecdd13/auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95"
Jul 12 22:50:47.970853 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:47.970687    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e074b79-9415-45a2-814f-64bfd681db68-tls-certs\") pod \"auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95\" (UID: \"8e074b79-9415-45a2-814f-64bfd681db68\") " pod="ai-tenant-e2e-shared-b-ecdd13/auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95"
Jul 12 22:50:47.970853 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:47.970746    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e074b79-9415-45a2-814f-64bfd681db68-model-cache\") pod \"auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95\" (UID: \"8e074b79-9415-45a2-814f-64bfd681db68\") " pod="ai-tenant-e2e-shared-b-ecdd13/auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95"
Jul 12 22:50:47.970853 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:47.970798    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e074b79-9415-45a2-814f-64bfd681db68-dshm\") pod \"auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95\" (UID: \"8e074b79-9415-45a2-814f-64bfd681db68\") " pod="ai-tenant-e2e-shared-b-ecdd13/auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95"
Jul 12 22:50:47.970853 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:47.970829    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e074b79-9415-45a2-814f-64bfd681db68-tmp-dir\") pod \"auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95\" (UID: \"8e074b79-9415-45a2-814f-64bfd681db68\") " pod="ai-tenant-e2e-shared-b-ecdd13/auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95"
Jul 12 22:50:47.970999 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:47.970875    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgc5c\" (UniqueName: \"kubernetes.io/projected/8e074b79-9415-45a2-814f-64bfd681db68-kube-api-access-sgc5c\") pod \"auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95\" (UID: \"8e074b79-9415-45a2-814f-64bfd681db68\") " pod="ai-tenant-e2e-shared-b-ecdd13/auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95"
Jul 12 22:50:48.072328 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:48.072274    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e074b79-9415-45a2-814f-64bfd681db68-tls-certs\") pod \"auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95\" (UID: \"8e074b79-9415-45a2-814f-64bfd681db68\") " pod="ai-tenant-e2e-shared-b-ecdd13/auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95"
Jul 12 22:50:48.072328 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:48.072332    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e074b79-9415-45a2-814f-64bfd681db68-model-cache\") pod \"auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95\" (UID: \"8e074b79-9415-45a2-814f-64bfd681db68\") " pod="ai-tenant-e2e-shared-b-ecdd13/auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95"
Jul 12 22:50:48.072606 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:48.072367    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e074b79-9415-45a2-814f-64bfd681db68-dshm\") pod \"auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95\" (UID: \"8e074b79-9415-45a2-814f-64bfd681db68\") " pod="ai-tenant-e2e-shared-b-ecdd13/auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95"
Jul 12 22:50:48.072606 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:48.072393    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e074b79-9415-45a2-814f-64bfd681db68-tmp-dir\") pod \"auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95\" (UID: \"8e074b79-9415-45a2-814f-64bfd681db68\") " pod="ai-tenant-e2e-shared-b-ecdd13/auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95"
Jul 12 22:50:48.072606 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:48.072426    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgc5c\" (UniqueName: \"kubernetes.io/projected/8e074b79-9415-45a2-814f-64bfd681db68-kube-api-access-sgc5c\") pod \"auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95\" (UID: \"8e074b79-9415-45a2-814f-64bfd681db68\") " pod="ai-tenant-e2e-shared-b-ecdd13/auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95"
Jul 12 22:50:48.072606 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:48.072520    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e074b79-9415-45a2-814f-64bfd681db68-home\") pod \"auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95\" (UID: \"8e074b79-9415-45a2-814f-64bfd681db68\") " pod="ai-tenant-e2e-shared-b-ecdd13/auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95"
Jul 12 22:50:48.072929 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:48.072906    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e074b79-9415-45a2-814f-64bfd681db68-model-cache\") pod \"auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95\" (UID: \"8e074b79-9415-45a2-814f-64bfd681db68\") " pod="ai-tenant-e2e-shared-b-ecdd13/auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95"
Jul 12 22:50:48.073035 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:48.072956    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e074b79-9415-45a2-814f-64bfd681db68-tmp-dir\") pod \"auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95\" (UID: \"8e074b79-9415-45a2-814f-64bfd681db68\") " pod="ai-tenant-e2e-shared-b-ecdd13/auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95"
Jul 12 22:50:48.073035 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:48.072983    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e074b79-9415-45a2-814f-64bfd681db68-home\") pod \"auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95\" (UID: \"8e074b79-9415-45a2-814f-64bfd681db68\") " pod="ai-tenant-e2e-shared-b-ecdd13/auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95"
Jul 12 22:50:48.074844 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:48.074825    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e074b79-9415-45a2-814f-64bfd681db68-dshm\") pod \"auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95\" (UID: \"8e074b79-9415-45a2-814f-64bfd681db68\") " pod="ai-tenant-e2e-shared-b-ecdd13/auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95"
Jul 12 22:50:48.075056 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:48.075040    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e074b79-9415-45a2-814f-64bfd681db68-tls-certs\") pod \"auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95\" (UID: \"8e074b79-9415-45a2-814f-64bfd681db68\") " pod="ai-tenant-e2e-shared-b-ecdd13/auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95"
Jul 12 22:50:48.080459 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:48.080430    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgc5c\" (UniqueName: \"kubernetes.io/projected/8e074b79-9415-45a2-814f-64bfd681db68-kube-api-access-sgc5c\") pod \"auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95\" (UID: \"8e074b79-9415-45a2-814f-64bfd681db68\") " pod="ai-tenant-e2e-shared-b-ecdd13/auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95"
Jul 12 22:50:48.188127 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:48.188033    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ecdd13/auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95"
Jul 12 22:50:48.332484 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:48.332449    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ecdd13/auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95"]
Jul 12 22:50:48.334341 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:50:48.334317    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e074b79_9415_45a2_814f_64bfd681db68.slice/crio-a493235b6b77d099ead50534d2b4d721bca41bccedc37fd70a9084bdf72a64ec WatchSource:0}: Error finding container a493235b6b77d099ead50534d2b4d721bca41bccedc37fd70a9084bdf72a64ec: Status 404 returned error can't find the container with id a493235b6b77d099ead50534d2b4d721bca41bccedc37fd70a9084bdf72a64ec
Jul 12 22:50:48.724811 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:48.724756    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ecdd13/auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95" event={"ID":"8e074b79-9415-45a2-814f-64bfd681db68","Type":"ContainerStarted","Data":"3dbde8331ba6217a19b6bdfa00ebb7ae00b5920ad2a9931bcc87dfc794548cbc"}
Jul 12 22:50:48.724811 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:48.724810    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ecdd13/auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95" event={"ID":"8e074b79-9415-45a2-814f-64bfd681db68","Type":"ContainerStarted","Data":"a493235b6b77d099ead50534d2b4d721bca41bccedc37fd70a9084bdf72a64ec"}
Jul 12 22:50:48.742247 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:48.742194    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-ecdd13/auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95" podStartSLOduration=1.742177605 podStartE2EDuration="1.742177605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:50:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:50:48.740282194 +0000 UTC m=+2421.053785548" watchObservedRunningTime="2026-07-12 22:50:48.742177605 +0000 UTC m=+2421.055680996"
Jul 12 22:50:58.188709 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:58.188672    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-ecdd13/auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95"
Jul 12 22:50:58.189203 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:58.188721    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-ecdd13/auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95"
Jul 12 22:50:58.201622 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:58.201591    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-ecdd13/auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95"
Jul 12 22:50:58.791476 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:50:58.791449    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-ecdd13/auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95"
Jul 12 22:52:55.358475 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:52:55.358380    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e321a4/auth-test-model-e321a4-kserve-58d8788b59-nmzp9"]
Jul 12 22:52:56.369027 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:52:56.368988    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ecdd13/auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95"]
Jul 12 22:53:10.380328 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:10.380283    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-e321a4/auth-test-model-e321a4-kserve-58d8788b59-nmzp9" podUID="832aed83-bd89-432f-bdc5-bea0994f6a21" containerName="main" containerID="cri-o://faf2b6f867aa36e674d3cbfe73c612ce61263ad3ae0d611f68bfcabd645bb6e7" gracePeriod=45
Jul 12 22:53:10.522235 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:10.522192    2560 generic.go:356] "Generic (PLEG): container finished" podID="832aed83-bd89-432f-bdc5-bea0994f6a21" containerID="faf2b6f867aa36e674d3cbfe73c612ce61263ad3ae0d611f68bfcabd645bb6e7" exitCode=0
Jul 12 22:53:10.522414 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:10.522271    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e321a4/auth-test-model-e321a4-kserve-58d8788b59-nmzp9" event={"ID":"832aed83-bd89-432f-bdc5-bea0994f6a21","Type":"ContainerDied","Data":"faf2b6f867aa36e674d3cbfe73c612ce61263ad3ae0d611f68bfcabd645bb6e7"}
Jul 12 22:53:10.640198 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:10.640105    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e321a4/auth-test-model-e321a4-kserve-58d8788b59-nmzp9"
Jul 12 22:53:10.811729 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:10.811683    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/832aed83-bd89-432f-bdc5-bea0994f6a21-tmp-dir\") pod \"832aed83-bd89-432f-bdc5-bea0994f6a21\" (UID: \"832aed83-bd89-432f-bdc5-bea0994f6a21\") "
Jul 12 22:53:10.811729 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:10.811732    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/832aed83-bd89-432f-bdc5-bea0994f6a21-model-cache\") pod \"832aed83-bd89-432f-bdc5-bea0994f6a21\" (UID: \"832aed83-bd89-432f-bdc5-bea0994f6a21\") "
Jul 12 22:53:10.811995 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:10.811824    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/832aed83-bd89-432f-bdc5-bea0994f6a21-dshm\") pod \"832aed83-bd89-432f-bdc5-bea0994f6a21\" (UID: \"832aed83-bd89-432f-bdc5-bea0994f6a21\") "
Jul 12 22:53:10.811995 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:10.811851    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/832aed83-bd89-432f-bdc5-bea0994f6a21-tls-certs\") pod \"832aed83-bd89-432f-bdc5-bea0994f6a21\" (UID: \"832aed83-bd89-432f-bdc5-bea0994f6a21\") "
Jul 12 22:53:10.811995 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:10.811877    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nflsw\" (UniqueName: \"kubernetes.io/projected/832aed83-bd89-432f-bdc5-bea0994f6a21-kube-api-access-nflsw\") pod \"832aed83-bd89-432f-bdc5-bea0994f6a21\" (UID: \"832aed83-bd89-432f-bdc5-bea0994f6a21\") "
Jul 12 22:53:10.811995 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:10.811906    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/832aed83-bd89-432f-bdc5-bea0994f6a21-home\") pod \"832aed83-bd89-432f-bdc5-bea0994f6a21\" (UID: \"832aed83-bd89-432f-bdc5-bea0994f6a21\") "
Jul 12 22:53:10.811995 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:10.811972    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/832aed83-bd89-432f-bdc5-bea0994f6a21-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "832aed83-bd89-432f-bdc5-bea0994f6a21" (UID: "832aed83-bd89-432f-bdc5-bea0994f6a21"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:53:10.812233 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:10.812021    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/832aed83-bd89-432f-bdc5-bea0994f6a21-model-cache" (OuterVolumeSpecName: "model-cache") pod "832aed83-bd89-432f-bdc5-bea0994f6a21" (UID: "832aed83-bd89-432f-bdc5-bea0994f6a21"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:53:10.812233 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:10.812198    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/832aed83-bd89-432f-bdc5-bea0994f6a21-home" (OuterVolumeSpecName: "home") pod "832aed83-bd89-432f-bdc5-bea0994f6a21" (UID: "832aed83-bd89-432f-bdc5-bea0994f6a21"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:53:10.812330 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:10.812289    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/832aed83-bd89-432f-bdc5-bea0994f6a21-tmp-dir\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:53:10.812330 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:10.812310    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/832aed83-bd89-432f-bdc5-bea0994f6a21-model-cache\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:53:10.814306 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:10.814280    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/832aed83-bd89-432f-bdc5-bea0994f6a21-dshm" (OuterVolumeSpecName: "dshm") pod "832aed83-bd89-432f-bdc5-bea0994f6a21" (UID: "832aed83-bd89-432f-bdc5-bea0994f6a21"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:53:10.814450 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:10.814341    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/832aed83-bd89-432f-bdc5-bea0994f6a21-kube-api-access-nflsw" (OuterVolumeSpecName: "kube-api-access-nflsw") pod "832aed83-bd89-432f-bdc5-bea0994f6a21" (UID: "832aed83-bd89-432f-bdc5-bea0994f6a21"). InnerVolumeSpecName "kube-api-access-nflsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 22:53:10.814450 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:10.814437    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/832aed83-bd89-432f-bdc5-bea0994f6a21-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "832aed83-bd89-432f-bdc5-bea0994f6a21" (UID: "832aed83-bd89-432f-bdc5-bea0994f6a21"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 12 22:53:10.912970 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:10.912885    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/832aed83-bd89-432f-bdc5-bea0994f6a21-dshm\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:53:10.912970 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:10.912914    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/832aed83-bd89-432f-bdc5-bea0994f6a21-tls-certs\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:53:10.912970 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:10.912926    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nflsw\" (UniqueName: \"kubernetes.io/projected/832aed83-bd89-432f-bdc5-bea0994f6a21-kube-api-access-nflsw\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:53:10.912970 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:10.912935    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/832aed83-bd89-432f-bdc5-bea0994f6a21-home\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:53:11.390456 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:11.390404    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-ecdd13/auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95" podUID="8e074b79-9415-45a2-814f-64bfd681db68" containerName="main" containerID="cri-o://3dbde8331ba6217a19b6bdfa00ebb7ae00b5920ad2a9931bcc87dfc794548cbc" gracePeriod=45
Jul 12 22:53:11.532398 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:11.532362    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e321a4/auth-test-model-e321a4-kserve-58d8788b59-nmzp9"
Jul 12 22:53:11.532398 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:11.532384    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e321a4/auth-test-model-e321a4-kserve-58d8788b59-nmzp9" event={"ID":"832aed83-bd89-432f-bdc5-bea0994f6a21","Type":"ContainerDied","Data":"70c51250f8426f419b6d05940aadeeb3da1ac5025d2a4d0f8bfbd0dfa2ed6f97"}
Jul 12 22:53:11.532690 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:11.532439    2560 scope.go:117] "RemoveContainer" containerID="faf2b6f867aa36e674d3cbfe73c612ce61263ad3ae0d611f68bfcabd645bb6e7"
Jul 12 22:53:11.534456 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:11.534396    2560 generic.go:356] "Generic (PLEG): container finished" podID="8e074b79-9415-45a2-814f-64bfd681db68" containerID="3dbde8331ba6217a19b6bdfa00ebb7ae00b5920ad2a9931bcc87dfc794548cbc" exitCode=0
Jul 12 22:53:11.534456 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:11.534439    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ecdd13/auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95" event={"ID":"8e074b79-9415-45a2-814f-64bfd681db68","Type":"ContainerDied","Data":"3dbde8331ba6217a19b6bdfa00ebb7ae00b5920ad2a9931bcc87dfc794548cbc"}
Jul 12 22:53:11.557352 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:11.557316    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e321a4/auth-test-model-e321a4-kserve-58d8788b59-nmzp9"]
Jul 12 22:53:11.560309 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:11.560279    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-e321a4/auth-test-model-e321a4-kserve-58d8788b59-nmzp9"]
Jul 12 22:53:11.657895 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:11.657822    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ecdd13/auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95"
Jul 12 22:53:11.821978 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:11.821949    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e074b79-9415-45a2-814f-64bfd681db68-model-cache\") pod \"8e074b79-9415-45a2-814f-64bfd681db68\" (UID: \"8e074b79-9415-45a2-814f-64bfd681db68\") "
Jul 12 22:53:11.822187 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:11.822055    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgc5c\" (UniqueName: \"kubernetes.io/projected/8e074b79-9415-45a2-814f-64bfd681db68-kube-api-access-sgc5c\") pod \"8e074b79-9415-45a2-814f-64bfd681db68\" (UID: \"8e074b79-9415-45a2-814f-64bfd681db68\") "
Jul 12 22:53:11.822187 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:11.822077    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e074b79-9415-45a2-814f-64bfd681db68-tls-certs\") pod \"8e074b79-9415-45a2-814f-64bfd681db68\" (UID: \"8e074b79-9415-45a2-814f-64bfd681db68\") "
Jul 12 22:53:11.822187 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:11.822113    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e074b79-9415-45a2-814f-64bfd681db68-home\") pod \"8e074b79-9415-45a2-814f-64bfd681db68\" (UID: \"8e074b79-9415-45a2-814f-64bfd681db68\") "
Jul 12 22:53:11.822187 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:11.822127    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e074b79-9415-45a2-814f-64bfd681db68-dshm\") pod \"8e074b79-9415-45a2-814f-64bfd681db68\" (UID: \"8e074b79-9415-45a2-814f-64bfd681db68\") "
Jul 12 22:53:11.822187 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:11.822152    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e074b79-9415-45a2-814f-64bfd681db68-tmp-dir\") pod \"8e074b79-9415-45a2-814f-64bfd681db68\" (UID: \"8e074b79-9415-45a2-814f-64bfd681db68\") "
Jul 12 22:53:11.822453 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:11.822355    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e074b79-9415-45a2-814f-64bfd681db68-model-cache" (OuterVolumeSpecName: "model-cache") pod "8e074b79-9415-45a2-814f-64bfd681db68" (UID: "8e074b79-9415-45a2-814f-64bfd681db68"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:53:11.822791 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:11.822764    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e074b79-9415-45a2-814f-64bfd681db68-home" (OuterVolumeSpecName: "home") pod "8e074b79-9415-45a2-814f-64bfd681db68" (UID: "8e074b79-9415-45a2-814f-64bfd681db68"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:53:11.823086 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:11.822786    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e074b79-9415-45a2-814f-64bfd681db68-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8e074b79-9415-45a2-814f-64bfd681db68" (UID: "8e074b79-9415-45a2-814f-64bfd681db68"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:53:11.824643 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:11.824617    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e074b79-9415-45a2-814f-64bfd681db68-dshm" (OuterVolumeSpecName: "dshm") pod "8e074b79-9415-45a2-814f-64bfd681db68" (UID: "8e074b79-9415-45a2-814f-64bfd681db68"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:53:11.825071 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:11.825050    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e074b79-9415-45a2-814f-64bfd681db68-kube-api-access-sgc5c" (OuterVolumeSpecName: "kube-api-access-sgc5c") pod "8e074b79-9415-45a2-814f-64bfd681db68" (UID: "8e074b79-9415-45a2-814f-64bfd681db68"). InnerVolumeSpecName "kube-api-access-sgc5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 22:53:11.825125 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:11.825072    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e074b79-9415-45a2-814f-64bfd681db68-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8e074b79-9415-45a2-814f-64bfd681db68" (UID: "8e074b79-9415-45a2-814f-64bfd681db68"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 12 22:53:11.923144 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:11.923052    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgc5c\" (UniqueName: \"kubernetes.io/projected/8e074b79-9415-45a2-814f-64bfd681db68-kube-api-access-sgc5c\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:53:11.923144 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:11.923091    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e074b79-9415-45a2-814f-64bfd681db68-tls-certs\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:53:11.923144 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:11.923107    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e074b79-9415-45a2-814f-64bfd681db68-home\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:53:11.923144 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:11.923119    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e074b79-9415-45a2-814f-64bfd681db68-dshm\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:53:11.923144 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:11.923131    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e074b79-9415-45a2-814f-64bfd681db68-tmp-dir\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:53:11.923144 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:11.923145    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e074b79-9415-45a2-814f-64bfd681db68-model-cache\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:53:12.245734 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:12.245652    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="832aed83-bd89-432f-bdc5-bea0994f6a21" path="/var/lib/kubelet/pods/832aed83-bd89-432f-bdc5-bea0994f6a21/volumes"
Jul 12 22:53:12.541274 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:12.541192    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ecdd13/auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95"
Jul 12 22:53:12.541274 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:12.541224    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ecdd13/auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95" event={"ID":"8e074b79-9415-45a2-814f-64bfd681db68","Type":"ContainerDied","Data":"a493235b6b77d099ead50534d2b4d721bca41bccedc37fd70a9084bdf72a64ec"}
Jul 12 22:53:12.541802 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:12.541280    2560 scope.go:117] "RemoveContainer" containerID="3dbde8331ba6217a19b6bdfa00ebb7ae00b5920ad2a9931bcc87dfc794548cbc"
Jul 12 22:53:12.559831 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:12.559801    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ecdd13/auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95"]
Jul 12 22:53:12.562408 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:12.562386    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-ecdd13/auth-test-model-ecdd13-kserve-555d8bb8f9-nvp95"]
Jul 12 22:53:14.246754 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:14.246718    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e074b79-9415-45a2-814f-64bfd681db68" path="/var/lib/kubelet/pods/8e074b79-9415-45a2-814f-64bfd681db68/volumes"
Jul 12 22:53:34.479068 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:34.479031    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-ecdd13/sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4"]
Jul 12 22:53:34.479704 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:34.479685    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="832aed83-bd89-432f-bdc5-bea0994f6a21" containerName="main"
Jul 12 22:53:34.479704 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:34.479707    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="832aed83-bd89-432f-bdc5-bea0994f6a21" containerName="main"
Jul 12 22:53:34.479864 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:34.479718    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e074b79-9415-45a2-814f-64bfd681db68" containerName="main"
Jul 12 22:53:34.479864 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:34.479726    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e074b79-9415-45a2-814f-64bfd681db68" containerName="main"
Jul 12 22:53:34.479864 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:34.479849    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="832aed83-bd89-432f-bdc5-bea0994f6a21" containerName="main"
Jul 12 22:53:34.480026 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:34.479869    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e074b79-9415-45a2-814f-64bfd681db68" containerName="main"
Jul 12 22:53:34.482957 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:34.482937    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ecdd13/sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4"
Jul 12 22:53:34.486903 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:34.486858    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ecdd13\"/\"openshift-service-ca.crt\""
Jul 12 22:53:34.487176 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:34.487156    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ecdd13\"/\"sub-test-model-ecdd13-kserve-self-signed-certs\""
Jul 12 22:53:34.487282 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:34.487076    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ecdd13\"/\"kube-root-ca.crt\""
Jul 12 22:53:34.487367 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:34.487111    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ecdd13\"/\"default-dockercfg-2vrrp\""
Jul 12 22:53:34.490770 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:34.490751    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ecdd13/sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4"]
Jul 12 22:53:34.634128 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:34.634087    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0d0dd9e-51e1-4482-9cfd-18074850bef8-model-cache\") pod \"sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4\" (UID: \"f0d0dd9e-51e1-4482-9cfd-18074850bef8\") " pod="ai-tenant-e2e-shared-b-ecdd13/sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4"
Jul 12 22:53:34.634447 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:34.634411    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0d0dd9e-51e1-4482-9cfd-18074850bef8-home\") pod \"sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4\" (UID: \"f0d0dd9e-51e1-4482-9cfd-18074850bef8\") " pod="ai-tenant-e2e-shared-b-ecdd13/sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4"
Jul 12 22:53:34.634583 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:34.634474    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0d0dd9e-51e1-4482-9cfd-18074850bef8-tls-certs\") pod \"sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4\" (UID: \"f0d0dd9e-51e1-4482-9cfd-18074850bef8\") " pod="ai-tenant-e2e-shared-b-ecdd13/sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4"
Jul 12 22:53:34.634583 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:34.634524    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0d0dd9e-51e1-4482-9cfd-18074850bef8-dshm\") pod \"sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4\" (UID: \"f0d0dd9e-51e1-4482-9cfd-18074850bef8\") " pod="ai-tenant-e2e-shared-b-ecdd13/sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4"
Jul 12 22:53:34.634719 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:34.634592    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99qb7\" (UniqueName: \"kubernetes.io/projected/f0d0dd9e-51e1-4482-9cfd-18074850bef8-kube-api-access-99qb7\") pod \"sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4\" (UID: \"f0d0dd9e-51e1-4482-9cfd-18074850bef8\") " pod="ai-tenant-e2e-shared-b-ecdd13/sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4"
Jul 12 22:53:34.634719 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:34.634633    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0d0dd9e-51e1-4482-9cfd-18074850bef8-tmp-dir\") pod \"sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4\" (UID: \"f0d0dd9e-51e1-4482-9cfd-18074850bef8\") " pod="ai-tenant-e2e-shared-b-ecdd13/sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4"
Jul 12 22:53:34.735579 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:34.735463    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0d0dd9e-51e1-4482-9cfd-18074850bef8-model-cache\") pod \"sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4\" (UID: \"f0d0dd9e-51e1-4482-9cfd-18074850bef8\") " pod="ai-tenant-e2e-shared-b-ecdd13/sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4"
Jul 12 22:53:34.735579 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:34.735509    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0d0dd9e-51e1-4482-9cfd-18074850bef8-home\") pod \"sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4\" (UID: \"f0d0dd9e-51e1-4482-9cfd-18074850bef8\") " pod="ai-tenant-e2e-shared-b-ecdd13/sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4"
Jul 12 22:53:34.735579 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:34.735560    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0d0dd9e-51e1-4482-9cfd-18074850bef8-tls-certs\") pod \"sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4\" (UID: \"f0d0dd9e-51e1-4482-9cfd-18074850bef8\") " pod="ai-tenant-e2e-shared-b-ecdd13/sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4"
Jul 12 22:53:34.735877 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:34.735586    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0d0dd9e-51e1-4482-9cfd-18074850bef8-dshm\") pod \"sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4\" (UID: \"f0d0dd9e-51e1-4482-9cfd-18074850bef8\") " pod="ai-tenant-e2e-shared-b-ecdd13/sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4"
Jul 12 22:53:34.735877 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:34.735613    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99qb7\" (UniqueName: \"kubernetes.io/projected/f0d0dd9e-51e1-4482-9cfd-18074850bef8-kube-api-access-99qb7\") pod \"sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4\" (UID: \"f0d0dd9e-51e1-4482-9cfd-18074850bef8\") " pod="ai-tenant-e2e-shared-b-ecdd13/sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4"
Jul 12 22:53:34.735877 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:34.735638    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0d0dd9e-51e1-4482-9cfd-18074850bef8-tmp-dir\") pod \"sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4\" (UID: \"f0d0dd9e-51e1-4482-9cfd-18074850bef8\") " pod="ai-tenant-e2e-shared-b-ecdd13/sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4"
Jul 12 22:53:34.736496 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:34.735962    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0d0dd9e-51e1-4482-9cfd-18074850bef8-model-cache\") pod \"sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4\" (UID: \"f0d0dd9e-51e1-4482-9cfd-18074850bef8\") " pod="ai-tenant-e2e-shared-b-ecdd13/sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4"
Jul 12 22:53:34.736496 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:34.735977    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0d0dd9e-51e1-4482-9cfd-18074850bef8-home\") pod \"sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4\" (UID: \"f0d0dd9e-51e1-4482-9cfd-18074850bef8\") " pod="ai-tenant-e2e-shared-b-ecdd13/sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4"
Jul 12 22:53:34.736496 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:34.736081    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0d0dd9e-51e1-4482-9cfd-18074850bef8-tmp-dir\") pod \"sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4\" (UID: \"f0d0dd9e-51e1-4482-9cfd-18074850bef8\") " pod="ai-tenant-e2e-shared-b-ecdd13/sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4"
Jul 12 22:53:34.738099 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:34.738077    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0d0dd9e-51e1-4482-9cfd-18074850bef8-dshm\") pod \"sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4\" (UID: \"f0d0dd9e-51e1-4482-9cfd-18074850bef8\") " pod="ai-tenant-e2e-shared-b-ecdd13/sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4"
Jul 12 22:53:34.738394 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:34.738373    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0d0dd9e-51e1-4482-9cfd-18074850bef8-tls-certs\") pod \"sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4\" (UID: \"f0d0dd9e-51e1-4482-9cfd-18074850bef8\") " pod="ai-tenant-e2e-shared-b-ecdd13/sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4"
Jul 12 22:53:34.743203 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:34.743180    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99qb7\" (UniqueName: \"kubernetes.io/projected/f0d0dd9e-51e1-4482-9cfd-18074850bef8-kube-api-access-99qb7\") pod \"sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4\" (UID: \"f0d0dd9e-51e1-4482-9cfd-18074850bef8\") " pod="ai-tenant-e2e-shared-b-ecdd13/sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4"
Jul 12 22:53:34.795422 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:34.795383    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ecdd13/sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4"
Jul 12 22:53:34.933110 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:34.933078    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ecdd13/sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4"]
Jul 12 22:53:34.934766 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:53:34.934740    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0d0dd9e_51e1_4482_9cfd_18074850bef8.slice/crio-eefcbe3ca7b516f13d7a4b1980d821186a218320635a5048590a6d8ede27ee7c WatchSource:0}: Error finding container eefcbe3ca7b516f13d7a4b1980d821186a218320635a5048590a6d8ede27ee7c: Status 404 returned error can't find the container with id eefcbe3ca7b516f13d7a4b1980d821186a218320635a5048590a6d8ede27ee7c
Jul 12 22:53:35.677604 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:35.677570    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ecdd13/sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4" event={"ID":"f0d0dd9e-51e1-4482-9cfd-18074850bef8","Type":"ContainerStarted","Data":"217f488f532598cbc47b2885adfd7483bdfa35c1bef85609a77c7bdeef2699f6"}
Jul 12 22:53:35.677604 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:35.677611    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ecdd13/sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4" event={"ID":"f0d0dd9e-51e1-4482-9cfd-18074850bef8","Type":"ContainerStarted","Data":"eefcbe3ca7b516f13d7a4b1980d821186a218320635a5048590a6d8ede27ee7c"}
Jul 12 22:53:35.693179 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:35.693129    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-ecdd13/sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4" podStartSLOduration=1.693111891 podStartE2EDuration="1.693111891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:53:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:53:35.692848905 +0000 UTC m=+2588.006352261" watchObservedRunningTime="2026-07-12 22:53:35.693111891 +0000 UTC m=+2588.006615246"
Jul 12 22:53:44.795984 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:44.795933    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-ecdd13/sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4"
Jul 12 22:53:44.796443 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:44.795996    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-ecdd13/sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4"
Jul 12 22:53:44.808796 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:44.808769    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-ecdd13/sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4"
Jul 12 22:53:45.744121 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:53:45.744093    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-ecdd13/sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4"
Jul 12 22:54:31.963442 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:54:31.963402    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ecdd13/sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4"]
Jul 12 22:54:46.985304 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:54:46.985231    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-ecdd13/sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4" podUID="f0d0dd9e-51e1-4482-9cfd-18074850bef8" containerName="main" containerID="cri-o://217f488f532598cbc47b2885adfd7483bdfa35c1bef85609a77c7bdeef2699f6" gracePeriod=45
Jul 12 22:54:47.249471 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:54:47.249401    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ecdd13/sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4"
Jul 12 22:54:47.396688 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:54:47.396645    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0d0dd9e-51e1-4482-9cfd-18074850bef8-home\") pod \"f0d0dd9e-51e1-4482-9cfd-18074850bef8\" (UID: \"f0d0dd9e-51e1-4482-9cfd-18074850bef8\") "
Jul 12 22:54:47.396688 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:54:47.396694    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0d0dd9e-51e1-4482-9cfd-18074850bef8-tmp-dir\") pod \"f0d0dd9e-51e1-4482-9cfd-18074850bef8\" (UID: \"f0d0dd9e-51e1-4482-9cfd-18074850bef8\") "
Jul 12 22:54:47.396954 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:54:47.396731    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0d0dd9e-51e1-4482-9cfd-18074850bef8-model-cache\") pod \"f0d0dd9e-51e1-4482-9cfd-18074850bef8\" (UID: \"f0d0dd9e-51e1-4482-9cfd-18074850bef8\") "
Jul 12 22:54:47.396954 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:54:47.396755    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0d0dd9e-51e1-4482-9cfd-18074850bef8-tls-certs\") pod \"f0d0dd9e-51e1-4482-9cfd-18074850bef8\" (UID: \"f0d0dd9e-51e1-4482-9cfd-18074850bef8\") "
Jul 12 22:54:47.396954 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:54:47.396848    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0d0dd9e-51e1-4482-9cfd-18074850bef8-dshm\") pod \"f0d0dd9e-51e1-4482-9cfd-18074850bef8\" (UID: \"f0d0dd9e-51e1-4482-9cfd-18074850bef8\") "
Jul 12 22:54:47.396954 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:54:47.396872    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99qb7\" (UniqueName: \"kubernetes.io/projected/f0d0dd9e-51e1-4482-9cfd-18074850bef8-kube-api-access-99qb7\") pod \"f0d0dd9e-51e1-4482-9cfd-18074850bef8\" (UID: \"f0d0dd9e-51e1-4482-9cfd-18074850bef8\") "
Jul 12 22:54:47.396954 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:54:47.396935    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d0dd9e-51e1-4482-9cfd-18074850bef8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f0d0dd9e-51e1-4482-9cfd-18074850bef8" (UID: "f0d0dd9e-51e1-4482-9cfd-18074850bef8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:54:47.397219 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:54:47.396985    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d0dd9e-51e1-4482-9cfd-18074850bef8-home" (OuterVolumeSpecName: "home") pod "f0d0dd9e-51e1-4482-9cfd-18074850bef8" (UID: "f0d0dd9e-51e1-4482-9cfd-18074850bef8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:54:47.397219 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:54:47.397071    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d0dd9e-51e1-4482-9cfd-18074850bef8-model-cache" (OuterVolumeSpecName: "model-cache") pod "f0d0dd9e-51e1-4482-9cfd-18074850bef8" (UID: "f0d0dd9e-51e1-4482-9cfd-18074850bef8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:54:47.397365 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:54:47.397345    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0d0dd9e-51e1-4482-9cfd-18074850bef8-home\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:54:47.397437 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:54:47.397368    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0d0dd9e-51e1-4482-9cfd-18074850bef8-tmp-dir\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:54:47.397437 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:54:47.397382    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0d0dd9e-51e1-4482-9cfd-18074850bef8-model-cache\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:54:47.399106 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:54:47.399077    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d0dd9e-51e1-4482-9cfd-18074850bef8-dshm" (OuterVolumeSpecName: "dshm") pod "f0d0dd9e-51e1-4482-9cfd-18074850bef8" (UID: "f0d0dd9e-51e1-4482-9cfd-18074850bef8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:54:47.399776 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:54:47.399760    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0d0dd9e-51e1-4482-9cfd-18074850bef8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f0d0dd9e-51e1-4482-9cfd-18074850bef8" (UID: "f0d0dd9e-51e1-4482-9cfd-18074850bef8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 12 22:54:47.399842 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:54:47.399800    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0d0dd9e-51e1-4482-9cfd-18074850bef8-kube-api-access-99qb7" (OuterVolumeSpecName: "kube-api-access-99qb7") pod "f0d0dd9e-51e1-4482-9cfd-18074850bef8" (UID: "f0d0dd9e-51e1-4482-9cfd-18074850bef8"). InnerVolumeSpecName "kube-api-access-99qb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 22:54:47.498325 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:54:47.498288    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0d0dd9e-51e1-4482-9cfd-18074850bef8-dshm\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:54:47.498325 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:54:47.498321    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99qb7\" (UniqueName: \"kubernetes.io/projected/f0d0dd9e-51e1-4482-9cfd-18074850bef8-kube-api-access-99qb7\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:54:47.498325 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:54:47.498334    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0d0dd9e-51e1-4482-9cfd-18074850bef8-tls-certs\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:54:48.072159 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:54:48.072123    2560 generic.go:356] "Generic (PLEG): container finished" podID="f0d0dd9e-51e1-4482-9cfd-18074850bef8" containerID="217f488f532598cbc47b2885adfd7483bdfa35c1bef85609a77c7bdeef2699f6" exitCode=0
Jul 12 22:54:48.072648 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:54:48.072165    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ecdd13/sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4"
Jul 12 22:54:48.072648 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:54:48.072213    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ecdd13/sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4" event={"ID":"f0d0dd9e-51e1-4482-9cfd-18074850bef8","Type":"ContainerDied","Data":"217f488f532598cbc47b2885adfd7483bdfa35c1bef85609a77c7bdeef2699f6"}
Jul 12 22:54:48.072648 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:54:48.072262    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ecdd13/sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4" event={"ID":"f0d0dd9e-51e1-4482-9cfd-18074850bef8","Type":"ContainerDied","Data":"eefcbe3ca7b516f13d7a4b1980d821186a218320635a5048590a6d8ede27ee7c"}
Jul 12 22:54:48.072648 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:54:48.072285    2560 scope.go:117] "RemoveContainer" containerID="217f488f532598cbc47b2885adfd7483bdfa35c1bef85609a77c7bdeef2699f6"
Jul 12 22:54:48.081843 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:54:48.081820    2560 scope.go:117] "RemoveContainer" containerID="217f488f532598cbc47b2885adfd7483bdfa35c1bef85609a77c7bdeef2699f6"
Jul 12 22:54:48.082114 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:54:48.082094    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"217f488f532598cbc47b2885adfd7483bdfa35c1bef85609a77c7bdeef2699f6\": container with ID starting with 217f488f532598cbc47b2885adfd7483bdfa35c1bef85609a77c7bdeef2699f6 not found: ID does not exist" containerID="217f488f532598cbc47b2885adfd7483bdfa35c1bef85609a77c7bdeef2699f6"
Jul 12 22:54:48.082182 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:54:48.082122    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"217f488f532598cbc47b2885adfd7483bdfa35c1bef85609a77c7bdeef2699f6"} err="failed to get container status \"217f488f532598cbc47b2885adfd7483bdfa35c1bef85609a77c7bdeef2699f6\": rpc error: code = NotFound desc = could not find container \"217f488f532598cbc47b2885adfd7483bdfa35c1bef85609a77c7bdeef2699f6\": container with ID starting with 217f488f532598cbc47b2885adfd7483bdfa35c1bef85609a77c7bdeef2699f6 not found: ID does not exist"
Jul 12 22:54:48.094160 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:54:48.094123    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ecdd13/sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4"]
Jul 12 22:54:48.096264 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:54:48.096247    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-ecdd13/sub-test-model-ecdd13-kserve-67fd7c9c8c-ssrs4"]
Jul 12 22:54:48.247705 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:54:48.247671    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0d0dd9e-51e1-4482-9cfd-18074850bef8" path="/var/lib/kubelet/pods/f0d0dd9e-51e1-4482-9cfd-18074850bef8/volumes"
Jul 12 22:55:00.780742 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:00.780706    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-e321a4/rate-test-model-e321a4-kserve-59c8499d47-5btlv"]
Jul 12 22:55:00.781149 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:00.781136    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0d0dd9e-51e1-4482-9cfd-18074850bef8" containerName="main"
Jul 12 22:55:00.781214 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:00.781151    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0d0dd9e-51e1-4482-9cfd-18074850bef8" containerName="main"
Jul 12 22:55:00.781257 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:00.781226    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0d0dd9e-51e1-4482-9cfd-18074850bef8" containerName="main"
Jul 12 22:55:00.785708 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:00.785688    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e321a4/rate-test-model-e321a4-kserve-59c8499d47-5btlv"
Jul 12 22:55:00.789897 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:00.789875    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-e321a4\"/\"kube-root-ca.crt\""
Jul 12 22:55:00.790215 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:00.789955    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e321a4\"/\"default-dockercfg-fhs2q\""
Jul 12 22:55:00.790332 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:00.790037    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e321a4\"/\"rate-test-model-e321a4-kserve-self-signed-certs\""
Jul 12 22:55:00.790332 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:00.790119    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-e321a4\"/\"openshift-service-ca.crt\""
Jul 12 22:55:00.794048 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:00.793975    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e321a4/rate-test-model-e321a4-kserve-59c8499d47-5btlv"]
Jul 12 22:55:00.809661 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:00.809629    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efcdf6e6-b6c2-4562-b2ae-1498db062c36-tmp-dir\") pod \"rate-test-model-e321a4-kserve-59c8499d47-5btlv\" (UID: \"efcdf6e6-b6c2-4562-b2ae-1498db062c36\") " pod="ai-tenant-e2e-shared-a-e321a4/rate-test-model-e321a4-kserve-59c8499d47-5btlv"
Jul 12 22:55:00.809792 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:00.809677    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghmfn\" (UniqueName: \"kubernetes.io/projected/efcdf6e6-b6c2-4562-b2ae-1498db062c36-kube-api-access-ghmfn\") pod \"rate-test-model-e321a4-kserve-59c8499d47-5btlv\" (UID: \"efcdf6e6-b6c2-4562-b2ae-1498db062c36\") " pod="ai-tenant-e2e-shared-a-e321a4/rate-test-model-e321a4-kserve-59c8499d47-5btlv"
Jul 12 22:55:00.809845 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:00.809784    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efcdf6e6-b6c2-4562-b2ae-1498db062c36-tls-certs\") pod \"rate-test-model-e321a4-kserve-59c8499d47-5btlv\" (UID: \"efcdf6e6-b6c2-4562-b2ae-1498db062c36\") " pod="ai-tenant-e2e-shared-a-e321a4/rate-test-model-e321a4-kserve-59c8499d47-5btlv"
Jul 12 22:55:00.809845 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:00.809814    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efcdf6e6-b6c2-4562-b2ae-1498db062c36-model-cache\") pod \"rate-test-model-e321a4-kserve-59c8499d47-5btlv\" (UID: \"efcdf6e6-b6c2-4562-b2ae-1498db062c36\") " pod="ai-tenant-e2e-shared-a-e321a4/rate-test-model-e321a4-kserve-59c8499d47-5btlv"
Jul 12 22:55:00.809932 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:00.809850    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efcdf6e6-b6c2-4562-b2ae-1498db062c36-dshm\") pod \"rate-test-model-e321a4-kserve-59c8499d47-5btlv\" (UID: \"efcdf6e6-b6c2-4562-b2ae-1498db062c36\") " pod="ai-tenant-e2e-shared-a-e321a4/rate-test-model-e321a4-kserve-59c8499d47-5btlv"
Jul 12 22:55:00.809932 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:00.809875    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efcdf6e6-b6c2-4562-b2ae-1498db062c36-home\") pod \"rate-test-model-e321a4-kserve-59c8499d47-5btlv\" (UID: \"efcdf6e6-b6c2-4562-b2ae-1498db062c36\") " pod="ai-tenant-e2e-shared-a-e321a4/rate-test-model-e321a4-kserve-59c8499d47-5btlv"
Jul 12 22:55:00.910560 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:00.910497    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efcdf6e6-b6c2-4562-b2ae-1498db062c36-tmp-dir\") pod \"rate-test-model-e321a4-kserve-59c8499d47-5btlv\" (UID: \"efcdf6e6-b6c2-4562-b2ae-1498db062c36\") " pod="ai-tenant-e2e-shared-a-e321a4/rate-test-model-e321a4-kserve-59c8499d47-5btlv"
Jul 12 22:55:00.910741 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:00.910590    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghmfn\" (UniqueName: \"kubernetes.io/projected/efcdf6e6-b6c2-4562-b2ae-1498db062c36-kube-api-access-ghmfn\") pod \"rate-test-model-e321a4-kserve-59c8499d47-5btlv\" (UID: \"efcdf6e6-b6c2-4562-b2ae-1498db062c36\") " pod="ai-tenant-e2e-shared-a-e321a4/rate-test-model-e321a4-kserve-59c8499d47-5btlv"
Jul 12 22:55:00.910741 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:00.910651    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efcdf6e6-b6c2-4562-b2ae-1498db062c36-tls-certs\") pod \"rate-test-model-e321a4-kserve-59c8499d47-5btlv\" (UID: \"efcdf6e6-b6c2-4562-b2ae-1498db062c36\") " pod="ai-tenant-e2e-shared-a-e321a4/rate-test-model-e321a4-kserve-59c8499d47-5btlv"
Jul 12 22:55:00.910741 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:00.910681    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efcdf6e6-b6c2-4562-b2ae-1498db062c36-model-cache\") pod \"rate-test-model-e321a4-kserve-59c8499d47-5btlv\" (UID: \"efcdf6e6-b6c2-4562-b2ae-1498db062c36\") " pod="ai-tenant-e2e-shared-a-e321a4/rate-test-model-e321a4-kserve-59c8499d47-5btlv"
Jul 12 22:55:00.910741 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:00.910719    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efcdf6e6-b6c2-4562-b2ae-1498db062c36-dshm\") pod \"rate-test-model-e321a4-kserve-59c8499d47-5btlv\" (UID: \"efcdf6e6-b6c2-4562-b2ae-1498db062c36\") " pod="ai-tenant-e2e-shared-a-e321a4/rate-test-model-e321a4-kserve-59c8499d47-5btlv"
Jul 12 22:55:00.910965 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:00.910743    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efcdf6e6-b6c2-4562-b2ae-1498db062c36-home\") pod \"rate-test-model-e321a4-kserve-59c8499d47-5btlv\" (UID: \"efcdf6e6-b6c2-4562-b2ae-1498db062c36\") " pod="ai-tenant-e2e-shared-a-e321a4/rate-test-model-e321a4-kserve-59c8499d47-5btlv"
Jul 12 22:55:00.910965 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:00.910931    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efcdf6e6-b6c2-4562-b2ae-1498db062c36-tmp-dir\") pod \"rate-test-model-e321a4-kserve-59c8499d47-5btlv\" (UID: \"efcdf6e6-b6c2-4562-b2ae-1498db062c36\") " pod="ai-tenant-e2e-shared-a-e321a4/rate-test-model-e321a4-kserve-59c8499d47-5btlv"
Jul 12 22:55:00.911109 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:00.911086    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efcdf6e6-b6c2-4562-b2ae-1498db062c36-model-cache\") pod \"rate-test-model-e321a4-kserve-59c8499d47-5btlv\" (UID: \"efcdf6e6-b6c2-4562-b2ae-1498db062c36\") " pod="ai-tenant-e2e-shared-a-e321a4/rate-test-model-e321a4-kserve-59c8499d47-5btlv"
Jul 12 22:55:00.911168 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:00.911110    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efcdf6e6-b6c2-4562-b2ae-1498db062c36-home\") pod \"rate-test-model-e321a4-kserve-59c8499d47-5btlv\" (UID: \"efcdf6e6-b6c2-4562-b2ae-1498db062c36\") " pod="ai-tenant-e2e-shared-a-e321a4/rate-test-model-e321a4-kserve-59c8499d47-5btlv"
Jul 12 22:55:00.913088 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:00.913059    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efcdf6e6-b6c2-4562-b2ae-1498db062c36-dshm\") pod \"rate-test-model-e321a4-kserve-59c8499d47-5btlv\" (UID: \"efcdf6e6-b6c2-4562-b2ae-1498db062c36\") " pod="ai-tenant-e2e-shared-a-e321a4/rate-test-model-e321a4-kserve-59c8499d47-5btlv"
Jul 12 22:55:00.913382 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:00.913361    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efcdf6e6-b6c2-4562-b2ae-1498db062c36-tls-certs\") pod \"rate-test-model-e321a4-kserve-59c8499d47-5btlv\" (UID: \"efcdf6e6-b6c2-4562-b2ae-1498db062c36\") " pod="ai-tenant-e2e-shared-a-e321a4/rate-test-model-e321a4-kserve-59c8499d47-5btlv"
Jul 12 22:55:00.918191 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:00.918167    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghmfn\" (UniqueName: \"kubernetes.io/projected/efcdf6e6-b6c2-4562-b2ae-1498db062c36-kube-api-access-ghmfn\") pod \"rate-test-model-e321a4-kserve-59c8499d47-5btlv\" (UID: \"efcdf6e6-b6c2-4562-b2ae-1498db062c36\") " pod="ai-tenant-e2e-shared-a-e321a4/rate-test-model-e321a4-kserve-59c8499d47-5btlv"
Jul 12 22:55:01.096806 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:01.096755    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e321a4/rate-test-model-e321a4-kserve-59c8499d47-5btlv"
Jul 12 22:55:01.241199 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:01.241172    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e321a4/rate-test-model-e321a4-kserve-59c8499d47-5btlv"]
Jul 12 22:55:01.243352 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:55:01.243325    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podefcdf6e6_b6c2_4562_b2ae_1498db062c36.slice/crio-1c21f0c90a6cc72dbf6d222e8abe8fa00a431f1adb4d1c83a73b8cb977e3328c WatchSource:0}: Error finding container 1c21f0c90a6cc72dbf6d222e8abe8fa00a431f1adb4d1c83a73b8cb977e3328c: Status 404 returned error can't find the container with id 1c21f0c90a6cc72dbf6d222e8abe8fa00a431f1adb4d1c83a73b8cb977e3328c
Jul 12 22:55:02.151771 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:02.151733    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e321a4/rate-test-model-e321a4-kserve-59c8499d47-5btlv" event={"ID":"efcdf6e6-b6c2-4562-b2ae-1498db062c36","Type":"ContainerStarted","Data":"cf7621ac6f876886937acfa34fc9e9a18433054e0c62aa3d863887c5799635f8"}
Jul 12 22:55:02.152262 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:02.151780    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e321a4/rate-test-model-e321a4-kserve-59c8499d47-5btlv" event={"ID":"efcdf6e6-b6c2-4562-b2ae-1498db062c36","Type":"ContainerStarted","Data":"1c21f0c90a6cc72dbf6d222e8abe8fa00a431f1adb4d1c83a73b8cb977e3328c"}
Jul 12 22:55:02.169633 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:02.169577    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-e321a4/rate-test-model-e321a4-kserve-59c8499d47-5btlv" podStartSLOduration=2.169558667 podStartE2EDuration="2.169558667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:55:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:55:02.16699232 +0000 UTC m=+2674.480495676" watchObservedRunningTime="2026-07-12 22:55:02.169558667 +0000 UTC m=+2674.483062015"
Jul 12 22:55:11.097899 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:11.097861    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-e321a4/rate-test-model-e321a4-kserve-59c8499d47-5btlv"
Jul 12 22:55:11.098349 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:11.097916    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-e321a4/rate-test-model-e321a4-kserve-59c8499d47-5btlv"
Jul 12 22:55:11.110407 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:11.110381    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-e321a4/rate-test-model-e321a4-kserve-59c8499d47-5btlv"
Jul 12 22:55:11.211974 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:11.211947    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-e321a4/rate-test-model-e321a4-kserve-59c8499d47-5btlv"
Jul 12 22:55:23.269374 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:23.269290    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-ecdd13/rate-test-model-ecdd13-kserve-59697cf67-nnxdp"]
Jul 12 22:55:23.282560 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:23.282488    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ecdd13/rate-test-model-ecdd13-kserve-59697cf67-nnxdp"
Jul 12 22:55:23.283237 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:23.283206    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ecdd13/rate-test-model-ecdd13-kserve-59697cf67-nnxdp"]
Jul 12 22:55:23.285896 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:23.285867    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ecdd13\"/\"kube-root-ca.crt\""
Jul 12 22:55:23.286161 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:23.286143    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ecdd13\"/\"rate-test-model-ecdd13-kserve-self-signed-certs\""
Jul 12 22:55:23.287231 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:23.287186    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ecdd13\"/\"default-dockercfg-2vrrp\""
Jul 12 22:55:23.287380 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:23.287341    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ecdd13\"/\"openshift-service-ca.crt\""
Jul 12 22:55:23.424173 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:23.424137    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce16b97f-a054-4176-9a44-19c5755c4fbe-tls-certs\") pod \"rate-test-model-ecdd13-kserve-59697cf67-nnxdp\" (UID: \"ce16b97f-a054-4176-9a44-19c5755c4fbe\") " pod="ai-tenant-e2e-shared-b-ecdd13/rate-test-model-ecdd13-kserve-59697cf67-nnxdp"
Jul 12 22:55:23.424363 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:23.424216    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce16b97f-a054-4176-9a44-19c5755c4fbe-home\") pod \"rate-test-model-ecdd13-kserve-59697cf67-nnxdp\" (UID: \"ce16b97f-a054-4176-9a44-19c5755c4fbe\") " pod="ai-tenant-e2e-shared-b-ecdd13/rate-test-model-ecdd13-kserve-59697cf67-nnxdp"
Jul 12 22:55:23.424363 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:23.424255    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce16b97f-a054-4176-9a44-19c5755c4fbe-model-cache\") pod \"rate-test-model-ecdd13-kserve-59697cf67-nnxdp\" (UID: \"ce16b97f-a054-4176-9a44-19c5755c4fbe\") " pod="ai-tenant-e2e-shared-b-ecdd13/rate-test-model-ecdd13-kserve-59697cf67-nnxdp"
Jul 12 22:55:23.424363 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:23.424282    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwpnf\" (UniqueName: \"kubernetes.io/projected/ce16b97f-a054-4176-9a44-19c5755c4fbe-kube-api-access-gwpnf\") pod \"rate-test-model-ecdd13-kserve-59697cf67-nnxdp\" (UID: \"ce16b97f-a054-4176-9a44-19c5755c4fbe\") " pod="ai-tenant-e2e-shared-b-ecdd13/rate-test-model-ecdd13-kserve-59697cf67-nnxdp"
Jul 12 22:55:23.424363 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:23.424305    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce16b97f-a054-4176-9a44-19c5755c4fbe-dshm\") pod \"rate-test-model-ecdd13-kserve-59697cf67-nnxdp\" (UID: \"ce16b97f-a054-4176-9a44-19c5755c4fbe\") " pod="ai-tenant-e2e-shared-b-ecdd13/rate-test-model-ecdd13-kserve-59697cf67-nnxdp"
Jul 12 22:55:23.424527 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:23.424357    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce16b97f-a054-4176-9a44-19c5755c4fbe-tmp-dir\") pod \"rate-test-model-ecdd13-kserve-59697cf67-nnxdp\" (UID: \"ce16b97f-a054-4176-9a44-19c5755c4fbe\") " pod="ai-tenant-e2e-shared-b-ecdd13/rate-test-model-ecdd13-kserve-59697cf67-nnxdp"
Jul 12 22:55:23.525157 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:23.525061    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce16b97f-a054-4176-9a44-19c5755c4fbe-tls-certs\") pod \"rate-test-model-ecdd13-kserve-59697cf67-nnxdp\" (UID: \"ce16b97f-a054-4176-9a44-19c5755c4fbe\") " pod="ai-tenant-e2e-shared-b-ecdd13/rate-test-model-ecdd13-kserve-59697cf67-nnxdp"
Jul 12 22:55:23.525339 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:23.525178    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce16b97f-a054-4176-9a44-19c5755c4fbe-home\") pod \"rate-test-model-ecdd13-kserve-59697cf67-nnxdp\" (UID: \"ce16b97f-a054-4176-9a44-19c5755c4fbe\") " pod="ai-tenant-e2e-shared-b-ecdd13/rate-test-model-ecdd13-kserve-59697cf67-nnxdp"
Jul 12 22:55:23.525339 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:23.525203    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce16b97f-a054-4176-9a44-19c5755c4fbe-model-cache\") pod \"rate-test-model-ecdd13-kserve-59697cf67-nnxdp\" (UID: \"ce16b97f-a054-4176-9a44-19c5755c4fbe\") " pod="ai-tenant-e2e-shared-b-ecdd13/rate-test-model-ecdd13-kserve-59697cf67-nnxdp"
Jul 12 22:55:23.525339 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:23.525219    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwpnf\" (UniqueName: \"kubernetes.io/projected/ce16b97f-a054-4176-9a44-19c5755c4fbe-kube-api-access-gwpnf\") pod \"rate-test-model-ecdd13-kserve-59697cf67-nnxdp\" (UID: \"ce16b97f-a054-4176-9a44-19c5755c4fbe\") " pod="ai-tenant-e2e-shared-b-ecdd13/rate-test-model-ecdd13-kserve-59697cf67-nnxdp"
Jul 12 22:55:23.525339 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:23.525240    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce16b97f-a054-4176-9a44-19c5755c4fbe-dshm\") pod \"rate-test-model-ecdd13-kserve-59697cf67-nnxdp\" (UID: \"ce16b97f-a054-4176-9a44-19c5755c4fbe\") " pod="ai-tenant-e2e-shared-b-ecdd13/rate-test-model-ecdd13-kserve-59697cf67-nnxdp"
Jul 12 22:55:23.525339 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:23.525254    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce16b97f-a054-4176-9a44-19c5755c4fbe-tmp-dir\") pod \"rate-test-model-ecdd13-kserve-59697cf67-nnxdp\" (UID: \"ce16b97f-a054-4176-9a44-19c5755c4fbe\") " pod="ai-tenant-e2e-shared-b-ecdd13/rate-test-model-ecdd13-kserve-59697cf67-nnxdp"
Jul 12 22:55:23.525746 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:23.525719    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce16b97f-a054-4176-9a44-19c5755c4fbe-tmp-dir\") pod \"rate-test-model-ecdd13-kserve-59697cf67-nnxdp\" (UID: \"ce16b97f-a054-4176-9a44-19c5755c4fbe\") " pod="ai-tenant-e2e-shared-b-ecdd13/rate-test-model-ecdd13-kserve-59697cf67-nnxdp"
Jul 12 22:55:23.525883 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:23.525760    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce16b97f-a054-4176-9a44-19c5755c4fbe-home\") pod \"rate-test-model-ecdd13-kserve-59697cf67-nnxdp\" (UID: \"ce16b97f-a054-4176-9a44-19c5755c4fbe\") " pod="ai-tenant-e2e-shared-b-ecdd13/rate-test-model-ecdd13-kserve-59697cf67-nnxdp"
Jul 12 22:55:23.525969 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:23.525922    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce16b97f-a054-4176-9a44-19c5755c4fbe-model-cache\") pod \"rate-test-model-ecdd13-kserve-59697cf67-nnxdp\" (UID: \"ce16b97f-a054-4176-9a44-19c5755c4fbe\") " pod="ai-tenant-e2e-shared-b-ecdd13/rate-test-model-ecdd13-kserve-59697cf67-nnxdp"
Jul 12 22:55:23.527735 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:23.527712    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce16b97f-a054-4176-9a44-19c5755c4fbe-dshm\") pod \"rate-test-model-ecdd13-kserve-59697cf67-nnxdp\" (UID: \"ce16b97f-a054-4176-9a44-19c5755c4fbe\") " pod="ai-tenant-e2e-shared-b-ecdd13/rate-test-model-ecdd13-kserve-59697cf67-nnxdp"
Jul 12 22:55:23.527888 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:23.527870    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce16b97f-a054-4176-9a44-19c5755c4fbe-tls-certs\") pod \"rate-test-model-ecdd13-kserve-59697cf67-nnxdp\" (UID: \"ce16b97f-a054-4176-9a44-19c5755c4fbe\") " pod="ai-tenant-e2e-shared-b-ecdd13/rate-test-model-ecdd13-kserve-59697cf67-nnxdp"
Jul 12 22:55:23.536202 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:23.536177    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwpnf\" (UniqueName: \"kubernetes.io/projected/ce16b97f-a054-4176-9a44-19c5755c4fbe-kube-api-access-gwpnf\") pod \"rate-test-model-ecdd13-kserve-59697cf67-nnxdp\" (UID: \"ce16b97f-a054-4176-9a44-19c5755c4fbe\") " pod="ai-tenant-e2e-shared-b-ecdd13/rate-test-model-ecdd13-kserve-59697cf67-nnxdp"
Jul 12 22:55:23.596112 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:23.596080    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ecdd13/rate-test-model-ecdd13-kserve-59697cf67-nnxdp"
Jul 12 22:55:23.735058 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:23.735034    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ecdd13/rate-test-model-ecdd13-kserve-59697cf67-nnxdp"]
Jul 12 22:55:23.736651 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:55:23.736626    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce16b97f_a054_4176_9a44_19c5755c4fbe.slice/crio-43b5a34cd0e7e7da32f122c972bab83b8ef21db308c4b0097d506a581c216e27 WatchSource:0}: Error finding container 43b5a34cd0e7e7da32f122c972bab83b8ef21db308c4b0097d506a581c216e27: Status 404 returned error can't find the container with id 43b5a34cd0e7e7da32f122c972bab83b8ef21db308c4b0097d506a581c216e27
Jul 12 22:55:24.273712 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:24.273671    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ecdd13/rate-test-model-ecdd13-kserve-59697cf67-nnxdp" event={"ID":"ce16b97f-a054-4176-9a44-19c5755c4fbe","Type":"ContainerStarted","Data":"4e7ed4f6b27f331ce92ab8635c19a079b206242cf06a7586a93456ccfcf885b3"}
Jul 12 22:55:24.273712 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:24.273721    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ecdd13/rate-test-model-ecdd13-kserve-59697cf67-nnxdp" event={"ID":"ce16b97f-a054-4176-9a44-19c5755c4fbe","Type":"ContainerStarted","Data":"43b5a34cd0e7e7da32f122c972bab83b8ef21db308c4b0097d506a581c216e27"}
Jul 12 22:55:24.291476 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:24.291418    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-ecdd13/rate-test-model-ecdd13-kserve-59697cf67-nnxdp" podStartSLOduration=1.291400555 podStartE2EDuration="1.291400555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 22:55:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:55:24.289428709 +0000 UTC m=+2696.602932063" watchObservedRunningTime="2026-07-12 22:55:24.291400555 +0000 UTC m=+2696.604903910"
Jul 12 22:55:28.384851 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:28.384823    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52pbf_1b1f17bc-a992-4c72-8fdc-a33a3c3649c4/ovn-acl-logging/0.log"
Jul 12 22:55:28.389827 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:28.389796    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52pbf_1b1f17bc-a992-4c72-8fdc-a33a3c3649c4/ovn-acl-logging/0.log"
Jul 12 22:55:33.596254 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:33.596215    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-ecdd13/rate-test-model-ecdd13-kserve-59697cf67-nnxdp"
Jul 12 22:55:33.596751 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:33.596279    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-ecdd13/rate-test-model-ecdd13-kserve-59697cf67-nnxdp"
Jul 12 22:55:33.608721 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:33.608696    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-ecdd13/rate-test-model-ecdd13-kserve-59697cf67-nnxdp"
Jul 12 22:55:34.345838 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:55:34.345806    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-ecdd13/rate-test-model-ecdd13-kserve-59697cf67-nnxdp"
Jul 12 22:56:20.999218 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:20.999186    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e321a4/rate-test-model-e321a4-kserve-59c8499d47-5btlv"]
Jul 12 22:56:22.059165 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:22.059121    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ecdd13/rate-test-model-ecdd13-kserve-59697cf67-nnxdp"]
Jul 12 22:56:36.020117 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:36.020047    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-e321a4/rate-test-model-e321a4-kserve-59c8499d47-5btlv" podUID="efcdf6e6-b6c2-4562-b2ae-1498db062c36" containerName="main" containerID="cri-o://cf7621ac6f876886937acfa34fc9e9a18433054e0c62aa3d863887c5799635f8" gracePeriod=45
Jul 12 22:56:36.268185 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:36.268159    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e321a4/rate-test-model-e321a4-kserve-59c8499d47-5btlv"
Jul 12 22:56:36.298800 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:36.298723    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efcdf6e6-b6c2-4562-b2ae-1498db062c36-tmp-dir\") pod \"efcdf6e6-b6c2-4562-b2ae-1498db062c36\" (UID: \"efcdf6e6-b6c2-4562-b2ae-1498db062c36\") "
Jul 12 22:56:36.298800 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:36.298764    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efcdf6e6-b6c2-4562-b2ae-1498db062c36-tls-certs\") pod \"efcdf6e6-b6c2-4562-b2ae-1498db062c36\" (UID: \"efcdf6e6-b6c2-4562-b2ae-1498db062c36\") "
Jul 12 22:56:36.299021 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:36.298803    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efcdf6e6-b6c2-4562-b2ae-1498db062c36-home\") pod \"efcdf6e6-b6c2-4562-b2ae-1498db062c36\" (UID: \"efcdf6e6-b6c2-4562-b2ae-1498db062c36\") "
Jul 12 22:56:36.299021 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:36.298838    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghmfn\" (UniqueName: \"kubernetes.io/projected/efcdf6e6-b6c2-4562-b2ae-1498db062c36-kube-api-access-ghmfn\") pod \"efcdf6e6-b6c2-4562-b2ae-1498db062c36\" (UID: \"efcdf6e6-b6c2-4562-b2ae-1498db062c36\") "
Jul 12 22:56:36.299021 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:36.298874    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efcdf6e6-b6c2-4562-b2ae-1498db062c36-dshm\") pod \"efcdf6e6-b6c2-4562-b2ae-1498db062c36\" (UID: \"efcdf6e6-b6c2-4562-b2ae-1498db062c36\") "
Jul 12 22:56:36.299021 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:36.298930    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efcdf6e6-b6c2-4562-b2ae-1498db062c36-model-cache\") pod \"efcdf6e6-b6c2-4562-b2ae-1498db062c36\" (UID: \"efcdf6e6-b6c2-4562-b2ae-1498db062c36\") "
Jul 12 22:56:36.299021 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:36.298987    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efcdf6e6-b6c2-4562-b2ae-1498db062c36-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "efcdf6e6-b6c2-4562-b2ae-1498db062c36" (UID: "efcdf6e6-b6c2-4562-b2ae-1498db062c36"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:56:36.299285 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:36.299163    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efcdf6e6-b6c2-4562-b2ae-1498db062c36-home" (OuterVolumeSpecName: "home") pod "efcdf6e6-b6c2-4562-b2ae-1498db062c36" (UID: "efcdf6e6-b6c2-4562-b2ae-1498db062c36"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:56:36.299285 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:36.299240    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efcdf6e6-b6c2-4562-b2ae-1498db062c36-model-cache" (OuterVolumeSpecName: "model-cache") pod "efcdf6e6-b6c2-4562-b2ae-1498db062c36" (UID: "efcdf6e6-b6c2-4562-b2ae-1498db062c36"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:56:36.299285 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:36.299249    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efcdf6e6-b6c2-4562-b2ae-1498db062c36-tmp-dir\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:56:36.299401 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:36.299288    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efcdf6e6-b6c2-4562-b2ae-1498db062c36-home\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:56:36.301208 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:36.301186    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efcdf6e6-b6c2-4562-b2ae-1498db062c36-dshm" (OuterVolumeSpecName: "dshm") pod "efcdf6e6-b6c2-4562-b2ae-1498db062c36" (UID: "efcdf6e6-b6c2-4562-b2ae-1498db062c36"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:56:36.301335 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:36.301284    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efcdf6e6-b6c2-4562-b2ae-1498db062c36-kube-api-access-ghmfn" (OuterVolumeSpecName: "kube-api-access-ghmfn") pod "efcdf6e6-b6c2-4562-b2ae-1498db062c36" (UID: "efcdf6e6-b6c2-4562-b2ae-1498db062c36"). InnerVolumeSpecName "kube-api-access-ghmfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 22:56:36.301510 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:36.301490    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efcdf6e6-b6c2-4562-b2ae-1498db062c36-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "efcdf6e6-b6c2-4562-b2ae-1498db062c36" (UID: "efcdf6e6-b6c2-4562-b2ae-1498db062c36"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 12 22:56:36.399753 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:36.399715    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efcdf6e6-b6c2-4562-b2ae-1498db062c36-tls-certs\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:56:36.399753 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:36.399746    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghmfn\" (UniqueName: \"kubernetes.io/projected/efcdf6e6-b6c2-4562-b2ae-1498db062c36-kube-api-access-ghmfn\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:56:36.399753 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:36.399756    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efcdf6e6-b6c2-4562-b2ae-1498db062c36-dshm\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:56:36.400001 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:36.399765    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efcdf6e6-b6c2-4562-b2ae-1498db062c36-model-cache\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:56:36.687326 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:36.687291    2560 generic.go:356] "Generic (PLEG): container finished" podID="efcdf6e6-b6c2-4562-b2ae-1498db062c36" containerID="cf7621ac6f876886937acfa34fc9e9a18433054e0c62aa3d863887c5799635f8" exitCode=0
Jul 12 22:56:36.687522 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:36.687332    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e321a4/rate-test-model-e321a4-kserve-59c8499d47-5btlv"
Jul 12 22:56:36.687522 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:36.687334    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e321a4/rate-test-model-e321a4-kserve-59c8499d47-5btlv" event={"ID":"efcdf6e6-b6c2-4562-b2ae-1498db062c36","Type":"ContainerDied","Data":"cf7621ac6f876886937acfa34fc9e9a18433054e0c62aa3d863887c5799635f8"}
Jul 12 22:56:36.687522 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:36.687428    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e321a4/rate-test-model-e321a4-kserve-59c8499d47-5btlv" event={"ID":"efcdf6e6-b6c2-4562-b2ae-1498db062c36","Type":"ContainerDied","Data":"1c21f0c90a6cc72dbf6d222e8abe8fa00a431f1adb4d1c83a73b8cb977e3328c"}
Jul 12 22:56:36.687522 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:36.687444    2560 scope.go:117] "RemoveContainer" containerID="cf7621ac6f876886937acfa34fc9e9a18433054e0c62aa3d863887c5799635f8"
Jul 12 22:56:36.697427 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:36.697407    2560 scope.go:117] "RemoveContainer" containerID="cf7621ac6f876886937acfa34fc9e9a18433054e0c62aa3d863887c5799635f8"
Jul 12 22:56:36.697718 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:56:36.697699    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf7621ac6f876886937acfa34fc9e9a18433054e0c62aa3d863887c5799635f8\": container with ID starting with cf7621ac6f876886937acfa34fc9e9a18433054e0c62aa3d863887c5799635f8 not found: ID does not exist" containerID="cf7621ac6f876886937acfa34fc9e9a18433054e0c62aa3d863887c5799635f8"
Jul 12 22:56:36.697782 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:36.697729    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf7621ac6f876886937acfa34fc9e9a18433054e0c62aa3d863887c5799635f8"} err="failed to get container status \"cf7621ac6f876886937acfa34fc9e9a18433054e0c62aa3d863887c5799635f8\": rpc error: code = NotFound desc = could not find container \"cf7621ac6f876886937acfa34fc9e9a18433054e0c62aa3d863887c5799635f8\": container with ID starting with cf7621ac6f876886937acfa34fc9e9a18433054e0c62aa3d863887c5799635f8 not found: ID does not exist"
Jul 12 22:56:36.709221 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:36.709198    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e321a4/rate-test-model-e321a4-kserve-59c8499d47-5btlv"]
Jul 12 22:56:36.711278 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:36.711258    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-e321a4/rate-test-model-e321a4-kserve-59c8499d47-5btlv"]
Jul 12 22:56:37.082239 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:37.082200    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-ecdd13/rate-test-model-ecdd13-kserve-59697cf67-nnxdp" podUID="ce16b97f-a054-4176-9a44-19c5755c4fbe" containerName="main" containerID="cri-o://4e7ed4f6b27f331ce92ab8635c19a079b206242cf06a7586a93456ccfcf885b3" gracePeriod=45
Jul 12 22:56:37.340868 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:37.340807    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ecdd13/rate-test-model-ecdd13-kserve-59697cf67-nnxdp"
Jul 12 22:56:37.408895 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:37.408862    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce16b97f-a054-4176-9a44-19c5755c4fbe-tls-certs\") pod \"ce16b97f-a054-4176-9a44-19c5755c4fbe\" (UID: \"ce16b97f-a054-4176-9a44-19c5755c4fbe\") "
Jul 12 22:56:37.408895 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:37.408894    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce16b97f-a054-4176-9a44-19c5755c4fbe-tmp-dir\") pod \"ce16b97f-a054-4176-9a44-19c5755c4fbe\" (UID: \"ce16b97f-a054-4176-9a44-19c5755c4fbe\") "
Jul 12 22:56:37.409141 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:37.408938    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce16b97f-a054-4176-9a44-19c5755c4fbe-dshm\") pod \"ce16b97f-a054-4176-9a44-19c5755c4fbe\" (UID: \"ce16b97f-a054-4176-9a44-19c5755c4fbe\") "
Jul 12 22:56:37.409141 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:37.409056    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce16b97f-a054-4176-9a44-19c5755c4fbe-model-cache\") pod \"ce16b97f-a054-4176-9a44-19c5755c4fbe\" (UID: \"ce16b97f-a054-4176-9a44-19c5755c4fbe\") "
Jul 12 22:56:37.409141 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:37.409119    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce16b97f-a054-4176-9a44-19c5755c4fbe-home\") pod \"ce16b97f-a054-4176-9a44-19c5755c4fbe\" (UID: \"ce16b97f-a054-4176-9a44-19c5755c4fbe\") "
Jul 12 22:56:37.409303 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:37.409227    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwpnf\" (UniqueName: \"kubernetes.io/projected/ce16b97f-a054-4176-9a44-19c5755c4fbe-kube-api-access-gwpnf\") pod \"ce16b97f-a054-4176-9a44-19c5755c4fbe\" (UID: \"ce16b97f-a054-4176-9a44-19c5755c4fbe\") "
Jul 12 22:56:37.409303 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:37.409242    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce16b97f-a054-4176-9a44-19c5755c4fbe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ce16b97f-a054-4176-9a44-19c5755c4fbe" (UID: "ce16b97f-a054-4176-9a44-19c5755c4fbe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:56:37.409303 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:37.409290    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce16b97f-a054-4176-9a44-19c5755c4fbe-model-cache" (OuterVolumeSpecName: "model-cache") pod "ce16b97f-a054-4176-9a44-19c5755c4fbe" (UID: "ce16b97f-a054-4176-9a44-19c5755c4fbe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:56:37.409457 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:37.409323    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce16b97f-a054-4176-9a44-19c5755c4fbe-home" (OuterVolumeSpecName: "home") pod "ce16b97f-a054-4176-9a44-19c5755c4fbe" (UID: "ce16b97f-a054-4176-9a44-19c5755c4fbe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:56:37.409515 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:37.409479    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce16b97f-a054-4176-9a44-19c5755c4fbe-model-cache\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:56:37.409515 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:37.409494    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce16b97f-a054-4176-9a44-19c5755c4fbe-home\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:56:37.409515 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:37.409504    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce16b97f-a054-4176-9a44-19c5755c4fbe-tmp-dir\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:56:37.411244 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:37.411223    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce16b97f-a054-4176-9a44-19c5755c4fbe-dshm" (OuterVolumeSpecName: "dshm") pod "ce16b97f-a054-4176-9a44-19c5755c4fbe" (UID: "ce16b97f-a054-4176-9a44-19c5755c4fbe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:56:37.411359 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:37.411252    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce16b97f-a054-4176-9a44-19c5755c4fbe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ce16b97f-a054-4176-9a44-19c5755c4fbe" (UID: "ce16b97f-a054-4176-9a44-19c5755c4fbe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 12 22:56:37.411359 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:37.411328    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce16b97f-a054-4176-9a44-19c5755c4fbe-kube-api-access-gwpnf" (OuterVolumeSpecName: "kube-api-access-gwpnf") pod "ce16b97f-a054-4176-9a44-19c5755c4fbe" (UID: "ce16b97f-a054-4176-9a44-19c5755c4fbe"). InnerVolumeSpecName "kube-api-access-gwpnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 22:56:37.510658 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:37.510610    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce16b97f-a054-4176-9a44-19c5755c4fbe-dshm\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:56:37.510658 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:37.510653    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwpnf\" (UniqueName: \"kubernetes.io/projected/ce16b97f-a054-4176-9a44-19c5755c4fbe-kube-api-access-gwpnf\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:56:37.510658 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:37.510669    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce16b97f-a054-4176-9a44-19c5755c4fbe-tls-certs\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:56:37.695738 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:37.695645    2560 generic.go:356] "Generic (PLEG): container finished" podID="ce16b97f-a054-4176-9a44-19c5755c4fbe" containerID="4e7ed4f6b27f331ce92ab8635c19a079b206242cf06a7586a93456ccfcf885b3" exitCode=0
Jul 12 22:56:37.695738 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:37.695693    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ecdd13/rate-test-model-ecdd13-kserve-59697cf67-nnxdp"
Jul 12 22:56:37.695738 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:37.695714    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ecdd13/rate-test-model-ecdd13-kserve-59697cf67-nnxdp" event={"ID":"ce16b97f-a054-4176-9a44-19c5755c4fbe","Type":"ContainerDied","Data":"4e7ed4f6b27f331ce92ab8635c19a079b206242cf06a7586a93456ccfcf885b3"}
Jul 12 22:56:37.696002 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:37.695754    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ecdd13/rate-test-model-ecdd13-kserve-59697cf67-nnxdp" event={"ID":"ce16b97f-a054-4176-9a44-19c5755c4fbe","Type":"ContainerDied","Data":"43b5a34cd0e7e7da32f122c972bab83b8ef21db308c4b0097d506a581c216e27"}
Jul 12 22:56:37.696002 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:37.695778    2560 scope.go:117] "RemoveContainer" containerID="4e7ed4f6b27f331ce92ab8635c19a079b206242cf06a7586a93456ccfcf885b3"
Jul 12 22:56:37.705744 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:37.705725    2560 scope.go:117] "RemoveContainer" containerID="4e7ed4f6b27f331ce92ab8635c19a079b206242cf06a7586a93456ccfcf885b3"
Jul 12 22:56:37.705997 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:56:37.705977    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e7ed4f6b27f331ce92ab8635c19a079b206242cf06a7586a93456ccfcf885b3\": container with ID starting with 4e7ed4f6b27f331ce92ab8635c19a079b206242cf06a7586a93456ccfcf885b3 not found: ID does not exist" containerID="4e7ed4f6b27f331ce92ab8635c19a079b206242cf06a7586a93456ccfcf885b3"
Jul 12 22:56:37.706060 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:37.706004    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e7ed4f6b27f331ce92ab8635c19a079b206242cf06a7586a93456ccfcf885b3"} err="failed to get container status \"4e7ed4f6b27f331ce92ab8635c19a079b206242cf06a7586a93456ccfcf885b3\": rpc error: code = NotFound desc = could not find container \"4e7ed4f6b27f331ce92ab8635c19a079b206242cf06a7586a93456ccfcf885b3\": container with ID starting with 4e7ed4f6b27f331ce92ab8635c19a079b206242cf06a7586a93456ccfcf885b3 not found: ID does not exist"
Jul 12 22:56:37.717647 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:37.717621    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ecdd13/rate-test-model-ecdd13-kserve-59697cf67-nnxdp"]
Jul 12 22:56:37.723280 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:37.723247    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-ecdd13/rate-test-model-ecdd13-kserve-59697cf67-nnxdp"]
Jul 12 22:56:38.245805 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:38.245777    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce16b97f-a054-4176-9a44-19c5755c4fbe" path="/var/lib/kubelet/pods/ce16b97f-a054-4176-9a44-19c5755c4fbe/volumes"
Jul 12 22:56:38.246189 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:56:38.246143    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efcdf6e6-b6c2-4562-b2ae-1498db062c36" path="/var/lib/kubelet/pods/efcdf6e6-b6c2-4562-b2ae-1498db062c36/volumes"
Jul 12 22:57:06.532328 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:06.532289    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"]
Jul 12 22:57:06.532898 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:06.532572    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5" podUID="3697d60a-84d8-4667-9298-40b51c9a26f9" containerName="istio-proxy" containerID="cri-o://691d2f3d800c7b9cf40914d3d53c815e42dbee40c4f6e4818f65b8d72ee9b302" gracePeriod=30
Jul 12 22:57:11.790923 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:11.790896    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"
Jul 12 22:57:11.886591 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:11.886550    2560 generic.go:356] "Generic (PLEG): container finished" podID="3697d60a-84d8-4667-9298-40b51c9a26f9" containerID="691d2f3d800c7b9cf40914d3d53c815e42dbee40c4f6e4818f65b8d72ee9b302" exitCode=0
Jul 12 22:57:11.886767 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:11.886604    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"
Jul 12 22:57:11.886767 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:11.886605    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5" event={"ID":"3697d60a-84d8-4667-9298-40b51c9a26f9","Type":"ContainerDied","Data":"691d2f3d800c7b9cf40914d3d53c815e42dbee40c4f6e4818f65b8d72ee9b302"}
Jul 12 22:57:11.886767 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:11.886684    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5" event={"ID":"3697d60a-84d8-4667-9298-40b51c9a26f9","Type":"ContainerDied","Data":"8c490d9ed1e25c1bfd239e69ea2e568ad36d38ebf69a69e997b94803179a9cef"}
Jul 12 22:57:11.886767 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:11.886709    2560 scope.go:117] "RemoveContainer" containerID="691d2f3d800c7b9cf40914d3d53c815e42dbee40c4f6e4818f65b8d72ee9b302"
Jul 12 22:57:11.896153 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:11.896134    2560 scope.go:117] "RemoveContainer" containerID="8240ea4fcd4635aa50215f46fa8ebe87823e2f9b389d493ed54f4d574755f226"
Jul 12 22:57:11.904505 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:11.904487    2560 scope.go:117] "RemoveContainer" containerID="691d2f3d800c7b9cf40914d3d53c815e42dbee40c4f6e4818f65b8d72ee9b302"
Jul 12 22:57:11.904835 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:57:11.904816    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"691d2f3d800c7b9cf40914d3d53c815e42dbee40c4f6e4818f65b8d72ee9b302\": container with ID starting with 691d2f3d800c7b9cf40914d3d53c815e42dbee40c4f6e4818f65b8d72ee9b302 not found: ID does not exist" containerID="691d2f3d800c7b9cf40914d3d53c815e42dbee40c4f6e4818f65b8d72ee9b302"
Jul 12 22:57:11.904915 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:11.904849    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"691d2f3d800c7b9cf40914d3d53c815e42dbee40c4f6e4818f65b8d72ee9b302"} err="failed to get container status \"691d2f3d800c7b9cf40914d3d53c815e42dbee40c4f6e4818f65b8d72ee9b302\": rpc error: code = NotFound desc = could not find container \"691d2f3d800c7b9cf40914d3d53c815e42dbee40c4f6e4818f65b8d72ee9b302\": container with ID starting with 691d2f3d800c7b9cf40914d3d53c815e42dbee40c4f6e4818f65b8d72ee9b302 not found: ID does not exist"
Jul 12 22:57:11.904915 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:11.904869    2560 scope.go:117] "RemoveContainer" containerID="8240ea4fcd4635aa50215f46fa8ebe87823e2f9b389d493ed54f4d574755f226"
Jul 12 22:57:11.905148 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:57:11.905130    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8240ea4fcd4635aa50215f46fa8ebe87823e2f9b389d493ed54f4d574755f226\": container with ID starting with 8240ea4fcd4635aa50215f46fa8ebe87823e2f9b389d493ed54f4d574755f226 not found: ID does not exist" containerID="8240ea4fcd4635aa50215f46fa8ebe87823e2f9b389d493ed54f4d574755f226"
Jul 12 22:57:11.905202 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:11.905159    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8240ea4fcd4635aa50215f46fa8ebe87823e2f9b389d493ed54f4d574755f226"} err="failed to get container status \"8240ea4fcd4635aa50215f46fa8ebe87823e2f9b389d493ed54f4d574755f226\": rpc error: code = NotFound desc = could not find container \"8240ea4fcd4635aa50215f46fa8ebe87823e2f9b389d493ed54f4d574755f226\": container with ID starting with 8240ea4fcd4635aa50215f46fa8ebe87823e2f9b389d493ed54f4d574755f226 not found: ID does not exist"
Jul 12 22:57:11.932908 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:11.932882    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3697d60a-84d8-4667-9298-40b51c9a26f9-istio-envoy\") pod \"3697d60a-84d8-4667-9298-40b51c9a26f9\" (UID: \"3697d60a-84d8-4667-9298-40b51c9a26f9\") "
Jul 12 22:57:11.933027 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:11.932938    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3697d60a-84d8-4667-9298-40b51c9a26f9-workload-certs\") pod \"3697d60a-84d8-4667-9298-40b51c9a26f9\" (UID: \"3697d60a-84d8-4667-9298-40b51c9a26f9\") "
Jul 12 22:57:11.933027 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:11.932992    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3697d60a-84d8-4667-9298-40b51c9a26f9-istiod-ca-cert\") pod \"3697d60a-84d8-4667-9298-40b51c9a26f9\" (UID: \"3697d60a-84d8-4667-9298-40b51c9a26f9\") "
Jul 12 22:57:11.933027 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:11.933019    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3697d60a-84d8-4667-9298-40b51c9a26f9-credential-socket\") pod \"3697d60a-84d8-4667-9298-40b51c9a26f9\" (UID: \"3697d60a-84d8-4667-9298-40b51c9a26f9\") "
Jul 12 22:57:11.933170 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:11.933074    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3697d60a-84d8-4667-9298-40b51c9a26f9-istio-podinfo\") pod \"3697d60a-84d8-4667-9298-40b51c9a26f9\" (UID: \"3697d60a-84d8-4667-9298-40b51c9a26f9\") "
Jul 12 22:57:11.933226 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:11.933120    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3697d60a-84d8-4667-9298-40b51c9a26f9-istio-token\") pod \"3697d60a-84d8-4667-9298-40b51c9a26f9\" (UID: \"3697d60a-84d8-4667-9298-40b51c9a26f9\") "
Jul 12 22:57:11.933226 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:11.933218    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3697d60a-84d8-4667-9298-40b51c9a26f9-workload-socket\") pod \"3697d60a-84d8-4667-9298-40b51c9a26f9\" (UID: \"3697d60a-84d8-4667-9298-40b51c9a26f9\") "
Jul 12 22:57:11.933334 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:11.933245    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjnvx\" (UniqueName: \"kubernetes.io/projected/3697d60a-84d8-4667-9298-40b51c9a26f9-kube-api-access-gjnvx\") pod \"3697d60a-84d8-4667-9298-40b51c9a26f9\" (UID: \"3697d60a-84d8-4667-9298-40b51c9a26f9\") "
Jul 12 22:57:11.933334 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:11.933248    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3697d60a-84d8-4667-9298-40b51c9a26f9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3697d60a-84d8-4667-9298-40b51c9a26f9" (UID: "3697d60a-84d8-4667-9298-40b51c9a26f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:57:11.933438 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:11.933408    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3697d60a-84d8-4667-9298-40b51c9a26f9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3697d60a-84d8-4667-9298-40b51c9a26f9" (UID: "3697d60a-84d8-4667-9298-40b51c9a26f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 12 22:57:11.933438 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:11.933419    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3697d60a-84d8-4667-9298-40b51c9a26f9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3697d60a-84d8-4667-9298-40b51c9a26f9" (UID: "3697d60a-84d8-4667-9298-40b51c9a26f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:57:11.933559 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:11.933460    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3697d60a-84d8-4667-9298-40b51c9a26f9-istio-data\") pod \"3697d60a-84d8-4667-9298-40b51c9a26f9\" (UID: \"3697d60a-84d8-4667-9298-40b51c9a26f9\") "
Jul 12 22:57:11.933653 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:11.933620    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3697d60a-84d8-4667-9298-40b51c9a26f9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3697d60a-84d8-4667-9298-40b51c9a26f9" (UID: "3697d60a-84d8-4667-9298-40b51c9a26f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:57:11.934419 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:11.934335    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3697d60a-84d8-4667-9298-40b51c9a26f9-workload-socket\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:57:11.934419 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:11.934361    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3697d60a-84d8-4667-9298-40b51c9a26f9-workload-certs\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:57:11.934419 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:11.934375    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3697d60a-84d8-4667-9298-40b51c9a26f9-istiod-ca-cert\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:57:11.934419 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:11.934389    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3697d60a-84d8-4667-9298-40b51c9a26f9-credential-socket\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:57:11.935052 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:11.935027    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3697d60a-84d8-4667-9298-40b51c9a26f9-istio-data" (OuterVolumeSpecName: "istio-data") pod "3697d60a-84d8-4667-9298-40b51c9a26f9" (UID: "3697d60a-84d8-4667-9298-40b51c9a26f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:57:11.935481 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:11.935455    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3697d60a-84d8-4667-9298-40b51c9a26f9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3697d60a-84d8-4667-9298-40b51c9a26f9" (UID: "3697d60a-84d8-4667-9298-40b51c9a26f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:57:11.935837 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:11.935814    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3697d60a-84d8-4667-9298-40b51c9a26f9-istio-token" (OuterVolumeSpecName: "istio-token") pod "3697d60a-84d8-4667-9298-40b51c9a26f9" (UID: "3697d60a-84d8-4667-9298-40b51c9a26f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 22:57:11.936012 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:11.935988    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3697d60a-84d8-4667-9298-40b51c9a26f9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3697d60a-84d8-4667-9298-40b51c9a26f9" (UID: "3697d60a-84d8-4667-9298-40b51c9a26f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 12 22:57:11.936163 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:11.936146    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3697d60a-84d8-4667-9298-40b51c9a26f9-kube-api-access-gjnvx" (OuterVolumeSpecName: "kube-api-access-gjnvx") pod "3697d60a-84d8-4667-9298-40b51c9a26f9" (UID: "3697d60a-84d8-4667-9298-40b51c9a26f9"). InnerVolumeSpecName "kube-api-access-gjnvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 22:57:12.034863 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:12.034829    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3697d60a-84d8-4667-9298-40b51c9a26f9-istio-podinfo\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:57:12.034863 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:12.034861    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3697d60a-84d8-4667-9298-40b51c9a26f9-istio-token\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:57:12.034863 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:12.034870    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjnvx\" (UniqueName: \"kubernetes.io/projected/3697d60a-84d8-4667-9298-40b51c9a26f9-kube-api-access-gjnvx\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:57:12.034863 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:12.034879    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3697d60a-84d8-4667-9298-40b51c9a26f9-istio-data\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:57:12.035134 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:12.034888    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3697d60a-84d8-4667-9298-40b51c9a26f9-istio-envoy\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:57:12.209081 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:12.209050    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"]
Jul 12 22:57:12.212792 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:12.212768    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-e321a4-openshift-default-864664c4c9-92ph5"]
Jul 12 22:57:12.247619 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:12.247590    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3697d60a-84d8-4667-9298-40b51c9a26f9" path="/var/lib/kubelet/pods/3697d60a-84d8-4667-9298-40b51c9a26f9/volumes"
Jul 12 22:57:32.614451 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:32.614417    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"]
Jul 12 22:57:32.614959 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:32.614727    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6" podUID="4b95dbe7-f37a-45ab-a0ee-f1a270b95f81" containerName="istio-proxy" containerID="cri-o://759fce98424c7f82071483773a2d6c1d132e00d981d34955e51c118dba740081" gracePeriod=30
Jul 12 22:57:33.290050 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:33.290010    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-75b966d6fb-284gh_a4fb8815-1f53-43ad-b617-0333ef625b60/authorino/0.log"
Jul 12 22:57:37.719672 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:37.719636    2560 patch_prober.go:28] interesting pod/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Jul 12 22:57:37.720073 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:37.719711    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6" podUID="4b95dbe7-f37a-45ab-a0ee-f1a270b95f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Jul 12 22:57:37.882287 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:37.882222    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"
Jul 12 22:57:37.958552 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:37.958507    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-istio-podinfo\") pod \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\" (UID: \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\") "
Jul 12 22:57:37.958738 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:37.958625    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-istio-data\") pod \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\" (UID: \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\") "
Jul 12 22:57:37.958738 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:37.958683    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bskfs\" (UniqueName: \"kubernetes.io/projected/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-kube-api-access-bskfs\") pod \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\" (UID: \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\") "
Jul 12 22:57:37.958738 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:37.958708    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-istio-token\") pod \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\" (UID: \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\") "
Jul 12 22:57:37.958875 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:37.958742    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-workload-certs\") pod \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\" (UID: \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\") "
Jul 12 22:57:37.958875 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:37.958770    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-istio-envoy\") pod \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\" (UID: \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\") "
Jul 12 22:57:37.958875 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:37.958792    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-workload-socket\") pod \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\" (UID: \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\") "
Jul 12 22:57:37.958875 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:37.958819    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-credential-socket\") pod \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\" (UID: \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\") "
Jul 12 22:57:37.958875 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:37.958867    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-istiod-ca-cert\") pod \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\" (UID: \"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81\") "
Jul 12 22:57:37.959140 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:37.959041    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4b95dbe7-f37a-45ab-a0ee-f1a270b95f81" (UID: "4b95dbe7-f37a-45ab-a0ee-f1a270b95f81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:57:37.959345 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:37.959228    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-workload-certs\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:57:37.959345 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:37.959240    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4b95dbe7-f37a-45ab-a0ee-f1a270b95f81" (UID: "4b95dbe7-f37a-45ab-a0ee-f1a270b95f81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:57:37.959525 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:37.959444    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4b95dbe7-f37a-45ab-a0ee-f1a270b95f81" (UID: "4b95dbe7-f37a-45ab-a0ee-f1a270b95f81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:57:37.959525 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:37.959459    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4b95dbe7-f37a-45ab-a0ee-f1a270b95f81" (UID: "4b95dbe7-f37a-45ab-a0ee-f1a270b95f81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 12 22:57:37.959841 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:37.959679    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-istio-data" (OuterVolumeSpecName: "istio-data") pod "4b95dbe7-f37a-45ab-a0ee-f1a270b95f81" (UID: "4b95dbe7-f37a-45ab-a0ee-f1a270b95f81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:57:37.961166 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:37.961140    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-istio-token" (OuterVolumeSpecName: "istio-token") pod "4b95dbe7-f37a-45ab-a0ee-f1a270b95f81" (UID: "4b95dbe7-f37a-45ab-a0ee-f1a270b95f81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 22:57:37.961271 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:37.961230    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4b95dbe7-f37a-45ab-a0ee-f1a270b95f81" (UID: "4b95dbe7-f37a-45ab-a0ee-f1a270b95f81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 12 22:57:37.961491 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:37.961469    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-kube-api-access-bskfs" (OuterVolumeSpecName: "kube-api-access-bskfs") pod "4b95dbe7-f37a-45ab-a0ee-f1a270b95f81" (UID: "4b95dbe7-f37a-45ab-a0ee-f1a270b95f81"). InnerVolumeSpecName "kube-api-access-bskfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 12 22:57:37.961576 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:37.961470    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4b95dbe7-f37a-45ab-a0ee-f1a270b95f81" (UID: "4b95dbe7-f37a-45ab-a0ee-f1a270b95f81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 12 22:57:38.005823 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:38.005790    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-r26w4_1cd6224b-33bc-41ce-bb56-c25600cc46f1/manager/0.log"
Jul 12 22:57:38.036481 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:38.036444    2560 generic.go:356] "Generic (PLEG): container finished" podID="4b95dbe7-f37a-45ab-a0ee-f1a270b95f81" containerID="759fce98424c7f82071483773a2d6c1d132e00d981d34955e51c118dba740081" exitCode=0
Jul 12 22:57:38.036656 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:38.036499    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"
Jul 12 22:57:38.036656 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:38.036553    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6" event={"ID":"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81","Type":"ContainerDied","Data":"759fce98424c7f82071483773a2d6c1d132e00d981d34955e51c118dba740081"}
Jul 12 22:57:38.036656 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:38.036616    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6" event={"ID":"4b95dbe7-f37a-45ab-a0ee-f1a270b95f81","Type":"ContainerDied","Data":"3661cffa9c8fbdfdcb94adad6b6a6fbada139b59ea725e23ae8ea83f7aaf7fe1"}
Jul 12 22:57:38.036656 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:38.036647    2560 scope.go:117] "RemoveContainer" containerID="759fce98424c7f82071483773a2d6c1d132e00d981d34955e51c118dba740081"
Jul 12 22:57:38.046758 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:38.046738    2560 scope.go:117] "RemoveContainer" containerID="158014843cb2b979de0474d3f87549b01c34fda43048dda0e60b2360296869db"
Jul 12 22:57:38.056366 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:38.056343    2560 scope.go:117] "RemoveContainer" containerID="759fce98424c7f82071483773a2d6c1d132e00d981d34955e51c118dba740081"
Jul 12 22:57:38.056683 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:57:38.056663    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"759fce98424c7f82071483773a2d6c1d132e00d981d34955e51c118dba740081\": container with ID starting with 759fce98424c7f82071483773a2d6c1d132e00d981d34955e51c118dba740081 not found: ID does not exist" containerID="759fce98424c7f82071483773a2d6c1d132e00d981d34955e51c118dba740081"
Jul 12 22:57:38.056771 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:38.056691    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"759fce98424c7f82071483773a2d6c1d132e00d981d34955e51c118dba740081"} err="failed to get container status \"759fce98424c7f82071483773a2d6c1d132e00d981d34955e51c118dba740081\": rpc error: code = NotFound desc = could not find container \"759fce98424c7f82071483773a2d6c1d132e00d981d34955e51c118dba740081\": container with ID starting with 759fce98424c7f82071483773a2d6c1d132e00d981d34955e51c118dba740081 not found: ID does not exist"
Jul 12 22:57:38.056771 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:38.056709    2560 scope.go:117] "RemoveContainer" containerID="158014843cb2b979de0474d3f87549b01c34fda43048dda0e60b2360296869db"
Jul 12 22:57:38.056956 ip-10-0-135-47 kubenswrapper[2560]: E0712 22:57:38.056937    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"158014843cb2b979de0474d3f87549b01c34fda43048dda0e60b2360296869db\": container with ID starting with 158014843cb2b979de0474d3f87549b01c34fda43048dda0e60b2360296869db not found: ID does not exist" containerID="158014843cb2b979de0474d3f87549b01c34fda43048dda0e60b2360296869db"
Jul 12 22:57:38.057017 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:38.056962    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"158014843cb2b979de0474d3f87549b01c34fda43048dda0e60b2360296869db"} err="failed to get container status \"158014843cb2b979de0474d3f87549b01c34fda43048dda0e60b2360296869db\": rpc error: code = NotFound desc = could not find container \"158014843cb2b979de0474d3f87549b01c34fda43048dda0e60b2360296869db\": container with ID starting with 158014843cb2b979de0474d3f87549b01c34fda43048dda0e60b2360296869db not found: ID does not exist"
Jul 12 22:57:38.057787 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:38.057768    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"]
Jul 12 22:57:38.060282 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:38.060263    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bskfs\" (UniqueName: \"kubernetes.io/projected/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-kube-api-access-bskfs\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:57:38.060372 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:38.060286    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-istio-token\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:57:38.060372 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:38.060296    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-istio-envoy\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:57:38.060372 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:38.060303    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-workload-socket\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:57:38.060372 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:38.060311    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-credential-socket\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:57:38.060372 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:38.060319    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-istiod-ca-cert\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:57:38.060372 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:38.060327    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-istio-podinfo\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:57:38.060372 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:38.060335    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81-istio-data\") on node \"ip-10-0-135-47.ec2.internal\" DevicePath \"\""
Jul 12 22:57:38.062962 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:38.062942    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-ecdd13-openshift-default-db45c5cdc-qhxm6"]
Jul 12 22:57:38.123359 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:38.123336    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-z8f69_aad212a3-b149-4734-b08e-0d2d90e21379/manager/0.log"
Jul 12 22:57:38.246323 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:38.246238    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b95dbe7-f37a-45ab-a0ee-f1a270b95f81" path="/var/lib/kubelet/pods/4b95dbe7-f37a-45ab-a0ee-f1a270b95f81/volumes"
Jul 12 22:57:38.351645 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:38.351617    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-c4vs6_83198b60-f427-4933-949e-78c94964ccb3/server/0.log"
Jul 12 22:57:38.465375 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:38.465346    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-bv65k_e56c0e2b-43b5-47df-9181-62cd21c51201/manager/0.log"
Jul 12 22:57:38.697915 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:38.697888    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56d7986bbd-8wmxx_af400fc7-0820-496b-83d4-6ffea9eedac1/copy-manifests/0.log"
Jul 12 22:57:38.709721 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:38.709694    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56d7986bbd-8wmxx_af400fc7-0820-496b-83d4-6ffea9eedac1/manager/0.log"
Jul 12 22:57:39.290454 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:39.290423    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29731590-6j5t9_73e2be03-630b-4a4e-a56f-63bf767afd29/cleanup/0.log"
Jul 12 22:57:39.395523 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:39.395497    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29731605-kdklg_d4ec29fe-5bff-44fc-8821-36f12c05d39f/cleanup/0.log"
Jul 12 22:57:39.508782 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:39.508748    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-c725b4f0-kclpj_11cf1a6b-3e3a-4d82-8d69-04166eeef07f/revoke-keys/1.log"
Jul 12 22:57:39.618045 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:39.618020    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-cbe7f53b-nn7b7_7b161689-298b-4034-a6e8-3c4a04a1046f/revoke-keys/0.log"
Jul 12 22:57:39.728307 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:39.728268    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-8a9e41-qcgnt_9d928aaf-ed14-4141-9689-2bef6c82893b/revoke-keys/0.log"
Jul 12 22:57:39.838508 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:39.838480    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-189e5d-vk2vj_d7aaf1b0-9be1-46a6-ac1e-3a170e5d79b8/revoke-keys/0.log"
Jul 12 22:57:39.944499 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:39.944410    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-55d0a565-6gzd9_e1daa9cc-c99c-4d25-9160-1f00d8a1683e/revoke-keys/0.log"
Jul 12 22:57:40.270585 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:40.270474    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-migrate-e6750e7c-5ccnn_c6653b93-7999-41af-9812-a21428c64eb3/revoke-keys/0.log"
Jul 12 22:57:40.377452 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:40.377422    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-10dc8c96-gngqr_8db14621-e88f-4e33-a062-40d12c147f12/revoke-keys/0.log"
Jul 12 22:57:40.486436 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:40.486407    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-3f30527e-kfzfg_6f1c44de-eda6-4069-8c1f-2078bdc2e4f9/revoke-keys/0.log"
Jul 12 22:57:42.118101 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:42.118066    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-75b966d6fb-284gh_a4fb8815-1f53-43ad-b617-0333ef625b60/authorino/0.log"
Jul 12 22:57:42.239104 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:42.239079    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-82qc5_de5d7d02-92fe-4232-bb9f-a1c710f5fab4/manager/0.log"
Jul 12 22:57:42.354138 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:42.354103    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-2nhpx_59f50f9c-506a-458e-b2cf-138d8094da03/manager/0.log"
Jul 12 22:57:42.466686 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:42.466611    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-8mm8s_54bb394f-0e12-49ce-bb8b-d91088eddb45/kuadrant-console-plugin/0.log"
Jul 12 22:57:42.583436 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:42.583405    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-jslgr_64dcd566-6879-4260-8bd0-17767ee9fb29/registry-server/0.log"
Jul 12 22:57:42.710544 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:42.710502    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-vkvst_03b1cf92-7bdc-4944-a52e-f6c0f8c5bd3c/manager/0.log"
Jul 12 22:57:42.953377 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:42.953345    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-xzccs_e78a3ec0-5fae-4c7f-a62e-43571c60c7da/manager/0.log"
Jul 12 22:57:43.627185 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:43.627156    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5747db99d8-nm6x2_b0b68ad9-67bc-4274-9319-6a9e5979d35e/kube-auth-proxy/0.log"
Jul 12 22:57:43.866135 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:43.866104    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-d2264_8b5f6930-2044-4c4e-b26d-65a6a273e4e2/payload-pre-processing/0.log"
Jul 12 22:57:44.089816 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:44.089784    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7cb4cc88b6-fscqr_bc370348-fbc2-4e67-96ce-a1129336bf21/router/0.log"
Jul 12 22:57:44.553598 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:44.553489    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-bm656_f737042d-b618-47e6-8815-6dbc10c4bf17/main/0.log"
Jul 12 22:57:44.776329 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:44.776301    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db5b2kv_cb7f33eb-f41d-45f1-8b64-7fa4acc07633/main/0.log"
Jul 12 22:57:52.175132 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:52.175104    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-b7t5r_700bcb2a-4c93-4abe-b403-9b404b3731e3/global-pull-secret-syncer/0.log"
Jul 12 22:57:52.370169 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:52.370137    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lbvpn_00bab1f2-a13f-47ff-b0c7-c04484a4c85e/konnectivity-agent/0.log"
Jul 12 22:57:52.414242 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:52.414209    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-47.ec2.internal_2f0d6da932fad75070caf7c5d75748da/haproxy/0.log"
Jul 12 22:57:57.259396 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:57.259365    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-75b966d6fb-284gh_a4fb8815-1f53-43ad-b617-0333ef625b60/authorino/0.log"
Jul 12 22:57:57.304516 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:57.304485    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-82qc5_de5d7d02-92fe-4232-bb9f-a1c710f5fab4/manager/0.log"
Jul 12 22:57:57.328348 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:57.328313    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-2nhpx_59f50f9c-506a-458e-b2cf-138d8094da03/manager/0.log"
Jul 12 22:57:57.353042 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:57.353013    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-8mm8s_54bb394f-0e12-49ce-bb8b-d91088eddb45/kuadrant-console-plugin/0.log"
Jul 12 22:57:57.391102 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:57.391070    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-jslgr_64dcd566-6879-4260-8bd0-17767ee9fb29/registry-server/0.log"
Jul 12 22:57:57.546004 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:57.545917    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-vkvst_03b1cf92-7bdc-4944-a52e-f6c0f8c5bd3c/manager/0.log"
Jul 12 22:57:57.644670 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:57.644641    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-xzccs_e78a3ec0-5fae-4c7f-a62e-43571c60c7da/manager/0.log"
Jul 12 22:57:59.289443 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:59.289408    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-248f2_25cdd4ee-5dbf-4ce5-b5dd-c4bf9a1f6afa/cluster-monitoring-operator/0.log"
Jul 12 22:57:59.459661 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:59.459632    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-28frw_b2356589-c594-4a1b-8b54-c45101644a44/node-exporter/0.log"
Jul 12 22:57:59.478403 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:59.478357    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-28frw_b2356589-c594-4a1b-8b54-c45101644a44/kube-rbac-proxy/0.log"
Jul 12 22:57:59.497379 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:57:59.497354    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-28frw_b2356589-c594-4a1b-8b54-c45101644a44/init-textfile/0.log"
Jul 12 22:58:01.579325 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:01.579296    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-tlbnn_7af611d3-24c8-4266-88d2-23b4523117e2/networking-console-plugin/0.log"
Jul 12 22:58:02.988040 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:02.988009    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xkcz8/perf-node-gather-daemonset-wwdtp"]
Jul 12 22:58:02.988412 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:02.988359    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b95dbe7-f37a-45ab-a0ee-f1a270b95f81" containerName="istio-proxy"
Jul 12 22:58:02.988412 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:02.988370    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b95dbe7-f37a-45ab-a0ee-f1a270b95f81" containerName="istio-proxy"
Jul 12 22:58:02.988412 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:02.988380    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b95dbe7-f37a-45ab-a0ee-f1a270b95f81" containerName="istio-proxy"
Jul 12 22:58:02.988412 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:02.988385    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b95dbe7-f37a-45ab-a0ee-f1a270b95f81" containerName="istio-proxy"
Jul 12 22:58:02.988412 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:02.988396    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3697d60a-84d8-4667-9298-40b51c9a26f9" containerName="istio-proxy"
Jul 12 22:58:02.988412 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:02.988401    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3697d60a-84d8-4667-9298-40b51c9a26f9" containerName="istio-proxy"
Jul 12 22:58:02.988412 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:02.988412    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3697d60a-84d8-4667-9298-40b51c9a26f9" containerName="istio-proxy"
Jul 12 22:58:02.988412 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:02.988417    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3697d60a-84d8-4667-9298-40b51c9a26f9" containerName="istio-proxy"
Jul 12 22:58:02.988688 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:02.988422    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efcdf6e6-b6c2-4562-b2ae-1498db062c36" containerName="main"
Jul 12 22:58:02.988688 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:02.988427    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="efcdf6e6-b6c2-4562-b2ae-1498db062c36" containerName="main"
Jul 12 22:58:02.988688 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:02.988442    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce16b97f-a054-4176-9a44-19c5755c4fbe" containerName="main"
Jul 12 22:58:02.988688 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:02.988447    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce16b97f-a054-4176-9a44-19c5755c4fbe" containerName="main"
Jul 12 22:58:02.988688 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:02.988500    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3697d60a-84d8-4667-9298-40b51c9a26f9" containerName="istio-proxy"
Jul 12 22:58:02.988688 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:02.988509    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce16b97f-a054-4176-9a44-19c5755c4fbe" containerName="main"
Jul 12 22:58:02.988688 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:02.988514    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="efcdf6e6-b6c2-4562-b2ae-1498db062c36" containerName="main"
Jul 12 22:58:02.988688 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:02.988519    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b95dbe7-f37a-45ab-a0ee-f1a270b95f81" containerName="istio-proxy"
Jul 12 22:58:02.993020 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:02.993002    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xkcz8/perf-node-gather-daemonset-wwdtp"
Jul 12 22:58:02.995394 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:02.995364    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xkcz8\"/\"default-dockercfg-kqqvs\""
Jul 12 22:58:02.995544 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:02.995445    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xkcz8\"/\"openshift-service-ca.crt\""
Jul 12 22:58:02.996495 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:02.996473    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xkcz8\"/\"kube-root-ca.crt\""
Jul 12 22:58:03.002511 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:03.002487    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xkcz8/perf-node-gather-daemonset-wwdtp"]
Jul 12 22:58:03.077993 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:03.077948    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ea9558cf-86a8-4b37-964e-78efd9518773-podres\") pod \"perf-node-gather-daemonset-wwdtp\" (UID: \"ea9558cf-86a8-4b37-964e-78efd9518773\") " pod="openshift-must-gather-xkcz8/perf-node-gather-daemonset-wwdtp"
Jul 12 22:58:03.077993 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:03.077995    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ea9558cf-86a8-4b37-964e-78efd9518773-proc\") pod \"perf-node-gather-daemonset-wwdtp\" (UID: \"ea9558cf-86a8-4b37-964e-78efd9518773\") " pod="openshift-must-gather-xkcz8/perf-node-gather-daemonset-wwdtp"
Jul 12 22:58:03.078223 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:03.078073    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc655\" (UniqueName: \"kubernetes.io/projected/ea9558cf-86a8-4b37-964e-78efd9518773-kube-api-access-jc655\") pod \"perf-node-gather-daemonset-wwdtp\" (UID: \"ea9558cf-86a8-4b37-964e-78efd9518773\") " pod="openshift-must-gather-xkcz8/perf-node-gather-daemonset-wwdtp"
Jul 12 22:58:03.078223 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:03.078142    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ea9558cf-86a8-4b37-964e-78efd9518773-lib-modules\") pod \"perf-node-gather-daemonset-wwdtp\" (UID: \"ea9558cf-86a8-4b37-964e-78efd9518773\") " pod="openshift-must-gather-xkcz8/perf-node-gather-daemonset-wwdtp"
Jul 12 22:58:03.078223 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:03.078172    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ea9558cf-86a8-4b37-964e-78efd9518773-sys\") pod \"perf-node-gather-daemonset-wwdtp\" (UID: \"ea9558cf-86a8-4b37-964e-78efd9518773\") " pod="openshift-must-gather-xkcz8/perf-node-gather-daemonset-wwdtp"
Jul 12 22:58:03.179004 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:03.178966    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ea9558cf-86a8-4b37-964e-78efd9518773-podres\") pod \"perf-node-gather-daemonset-wwdtp\" (UID: \"ea9558cf-86a8-4b37-964e-78efd9518773\") " pod="openshift-must-gather-xkcz8/perf-node-gather-daemonset-wwdtp"
Jul 12 22:58:03.179004 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:03.179003    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ea9558cf-86a8-4b37-964e-78efd9518773-proc\") pod \"perf-node-gather-daemonset-wwdtp\" (UID: \"ea9558cf-86a8-4b37-964e-78efd9518773\") " pod="openshift-must-gather-xkcz8/perf-node-gather-daemonset-wwdtp"
Jul 12 22:58:03.179203 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:03.179084    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ea9558cf-86a8-4b37-964e-78efd9518773-proc\") pod \"perf-node-gather-daemonset-wwdtp\" (UID: \"ea9558cf-86a8-4b37-964e-78efd9518773\") " pod="openshift-must-gather-xkcz8/perf-node-gather-daemonset-wwdtp"
Jul 12 22:58:03.179203 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:03.179131    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc655\" (UniqueName: \"kubernetes.io/projected/ea9558cf-86a8-4b37-964e-78efd9518773-kube-api-access-jc655\") pod \"perf-node-gather-daemonset-wwdtp\" (UID: \"ea9558cf-86a8-4b37-964e-78efd9518773\") " pod="openshift-must-gather-xkcz8/perf-node-gather-daemonset-wwdtp"
Jul 12 22:58:03.179203 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:03.179177    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ea9558cf-86a8-4b37-964e-78efd9518773-lib-modules\") pod \"perf-node-gather-daemonset-wwdtp\" (UID: \"ea9558cf-86a8-4b37-964e-78efd9518773\") " pod="openshift-must-gather-xkcz8/perf-node-gather-daemonset-wwdtp"
Jul 12 22:58:03.179203 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:03.179197    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ea9558cf-86a8-4b37-964e-78efd9518773-sys\") pod \"perf-node-gather-daemonset-wwdtp\" (UID: \"ea9558cf-86a8-4b37-964e-78efd9518773\") " pod="openshift-must-gather-xkcz8/perf-node-gather-daemonset-wwdtp"
Jul 12 22:58:03.179333 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:03.179139    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ea9558cf-86a8-4b37-964e-78efd9518773-podres\") pod \"perf-node-gather-daemonset-wwdtp\" (UID: \"ea9558cf-86a8-4b37-964e-78efd9518773\") " pod="openshift-must-gather-xkcz8/perf-node-gather-daemonset-wwdtp"
Jul 12 22:58:03.179333 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:03.179274    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ea9558cf-86a8-4b37-964e-78efd9518773-sys\") pod \"perf-node-gather-daemonset-wwdtp\" (UID: \"ea9558cf-86a8-4b37-964e-78efd9518773\") " pod="openshift-must-gather-xkcz8/perf-node-gather-daemonset-wwdtp"
Jul 12 22:58:03.179333 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:03.179311    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ea9558cf-86a8-4b37-964e-78efd9518773-lib-modules\") pod \"perf-node-gather-daemonset-wwdtp\" (UID: \"ea9558cf-86a8-4b37-964e-78efd9518773\") " pod="openshift-must-gather-xkcz8/perf-node-gather-daemonset-wwdtp"
Jul 12 22:58:03.186795 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:03.186771    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc655\" (UniqueName: \"kubernetes.io/projected/ea9558cf-86a8-4b37-964e-78efd9518773-kube-api-access-jc655\") pod \"perf-node-gather-daemonset-wwdtp\" (UID: \"ea9558cf-86a8-4b37-964e-78efd9518773\") " pod="openshift-must-gather-xkcz8/perf-node-gather-daemonset-wwdtp"
Jul 12 22:58:03.304296 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:03.304200    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xkcz8/perf-node-gather-daemonset-wwdtp"
Jul 12 22:58:03.444098 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:03.444064    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xkcz8/perf-node-gather-daemonset-wwdtp"]
Jul 12 22:58:03.447629 ip-10-0-135-47 kubenswrapper[2560]: W0712 22:58:03.447593    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podea9558cf_86a8_4b37_964e_78efd9518773.slice/crio-5aae6c6279d589033364d2411fe2e597596f769cd5b71032c1ace7de23b633ac WatchSource:0}: Error finding container 5aae6c6279d589033364d2411fe2e597596f769cd5b71032c1ace7de23b633ac: Status 404 returned error can't find the container with id 5aae6c6279d589033364d2411fe2e597596f769cd5b71032c1ace7de23b633ac
Jul 12 22:58:03.477062 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:03.477024    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-gf2x8_772a2a08-9010-4063-ba53-040e1e5881d1/volume-data-source-validator/0.log"
Jul 12 22:58:04.174014 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:04.173979    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xkcz8/perf-node-gather-daemonset-wwdtp" event={"ID":"ea9558cf-86a8-4b37-964e-78efd9518773","Type":"ContainerStarted","Data":"9561fb894c76145e1b96d3e8864748ed9567f427f557711f2ebeda0c0996a2c9"}
Jul 12 22:58:04.174411 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:04.174041    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xkcz8/perf-node-gather-daemonset-wwdtp"
Jul 12 22:58:04.174411 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:04.174058    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xkcz8/perf-node-gather-daemonset-wwdtp" event={"ID":"ea9558cf-86a8-4b37-964e-78efd9518773","Type":"ContainerStarted","Data":"5aae6c6279d589033364d2411fe2e597596f769cd5b71032c1ace7de23b633ac"}
Jul 12 22:58:04.190266 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:04.190207    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xkcz8/perf-node-gather-daemonset-wwdtp" podStartSLOduration=2.190188064 podStartE2EDuration="2.190188064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 22:58:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 22:58:04.187563048 +0000 UTC m=+2856.501066396" watchObservedRunningTime="2026-07-12 22:58:04.190188064 +0000 UTC m=+2856.503691418"
Jul 12 22:58:04.515974 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:04.515893    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jhqcn_e3e0d19f-0837-473f-a4eb-89df81150888/dns/0.log"
Jul 12 22:58:04.534917 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:04.534892    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jhqcn_e3e0d19f-0837-473f-a4eb-89df81150888/kube-rbac-proxy/0.log"
Jul 12 22:58:04.598244 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:04.598217    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g4qw4_05ebf682-f7b0-4267-8a22-46d0c41ecba9/dns-node-resolver/0.log"
Jul 12 22:58:05.151358 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:05.151327    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6b5b4b6c75-vtlnj_ede4fc85-270f-40d3-8e4e-8f0d47d673bd/registry/0.log"
Jul 12 22:58:05.215656 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:05.215630    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zkk8b_c946a409-971a-4760-b311-5278608e31f9/node-ca/0.log"
Jul 12 22:58:06.265185 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:06.265142    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5747db99d8-nm6x2_b0b68ad9-67bc-4274-9319-6a9e5979d35e/kube-auth-proxy/0.log"
Jul 12 22:58:06.352068 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:06.352036    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-d2264_8b5f6930-2044-4c4e-b26d-65a6a273e4e2/payload-pre-processing/0.log"
Jul 12 22:58:06.402310 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:06.402285    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7cb4cc88b6-fscqr_bc370348-fbc2-4e67-96ce-a1129336bf21/router/0.log"
Jul 12 22:58:07.168443 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:07.168416    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8965j_fd201c7d-6b11-430a-b079-9a16e001c434/serve-healthcheck-canary/0.log"
Jul 12 22:58:07.729359 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:07.729321    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mjg7l_05f8f52f-6343-4d4c-af3b-028de417ace6/kube-rbac-proxy/0.log"
Jul 12 22:58:07.748498 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:07.748471    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mjg7l_05f8f52f-6343-4d4c-af3b-028de417ace6/exporter/0.log"
Jul 12 22:58:07.768146 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:07.768120    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mjg7l_05f8f52f-6343-4d4c-af3b-028de417ace6/extractor/0.log"
Jul 12 22:58:09.815961 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:09.815932    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-r26w4_1cd6224b-33bc-41ce-bb56-c25600cc46f1/manager/0.log"
Jul 12 22:58:09.945918 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:09.945884    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-z8f69_aad212a3-b149-4734-b08e-0d2d90e21379/manager/0.log"
Jul 12 22:58:10.090116 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:10.090038    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-c4vs6_83198b60-f427-4933-949e-78c94964ccb3/server/0.log"
Jul 12 22:58:10.160956 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:10.160922    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-bv65k_e56c0e2b-43b5-47df-9181-62cd21c51201/manager/0.log"
Jul 12 22:58:10.189576 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:10.189552    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xkcz8/perf-node-gather-daemonset-wwdtp"
Jul 12 22:58:10.286115 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:10.286079    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56d7986bbd-8wmxx_af400fc7-0820-496b-83d4-6ffea9eedac1/manager/0.log"
Jul 12 22:58:10.302959 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:10.302935    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56d7986bbd-8wmxx_af400fc7-0820-496b-83d4-6ffea9eedac1/copy-manifests/0.log"
Jul 12 22:58:13.290441 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:13.290414    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5747db99d8-nm6x2_b0b68ad9-67bc-4274-9319-6a9e5979d35e/kube-auth-proxy/0.log"
Jul 12 22:58:13.360277 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:13.360245    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-d2264_8b5f6930-2044-4c4e-b26d-65a6a273e4e2/payload-pre-processing/0.log"
Jul 12 22:58:13.394314 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:13.394237    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7cb4cc88b6-fscqr_bc370348-fbc2-4e67-96ce-a1129336bf21/router/0.log"
Jul 12 22:58:14.809047 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:14.809021    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5747db99d8-nm6x2_b0b68ad9-67bc-4274-9319-6a9e5979d35e/kube-auth-proxy/0.log"
Jul 12 22:58:14.882966 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:14.882936    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-d2264_8b5f6930-2044-4c4e-b26d-65a6a273e4e2/payload-pre-processing/0.log"
Jul 12 22:58:14.930182 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:14.930153    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7cb4cc88b6-fscqr_bc370348-fbc2-4e67-96ce-a1129336bf21/router/0.log"
Jul 12 22:58:17.289175 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:17.289136    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-dqc2v_f91f26b5-1055-4423-a442-da2443b2d44e/kube-storage-version-migrator-operator/1.log"
Jul 12 22:58:17.289859 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:17.289842    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-dqc2v_f91f26b5-1055-4423-a442-da2443b2d44e/kube-storage-version-migrator-operator/0.log"
Jul 12 22:58:18.229247 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:18.229219    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-2t5r9_669e3fb7-aeeb-4746-b4ee-84c83cd5a972/kube-multus/0.log"
Jul 12 22:58:18.542054 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:18.541983    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kvvgp_912f085a-8942-4512-98f9-c8ba189509c2/kube-multus-additional-cni-plugins/0.log"
Jul 12 22:58:18.561412 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:18.561387    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kvvgp_912f085a-8942-4512-98f9-c8ba189509c2/egress-router-binary-copy/0.log"
Jul 12 22:58:18.582024 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:18.581995    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kvvgp_912f085a-8942-4512-98f9-c8ba189509c2/cni-plugins/0.log"
Jul 12 22:58:18.603820 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:18.603798    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kvvgp_912f085a-8942-4512-98f9-c8ba189509c2/bond-cni-plugin/0.log"
Jul 12 22:58:18.623485 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:18.623452    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kvvgp_912f085a-8942-4512-98f9-c8ba189509c2/routeoverride-cni/0.log"
Jul 12 22:58:18.642714 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:18.642689    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kvvgp_912f085a-8942-4512-98f9-c8ba189509c2/whereabouts-cni-bincopy/0.log"
Jul 12 22:58:18.662359 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:18.662333    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kvvgp_912f085a-8942-4512-98f9-c8ba189509c2/whereabouts-cni/0.log"
Jul 12 22:58:18.827386 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:18.827361    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-st67z_551bf86c-bd1a-40be-b977-41205da830c2/network-metrics-daemon/0.log"
Jul 12 22:58:18.845580 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:18.845552    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-st67z_551bf86c-bd1a-40be-b977-41205da830c2/kube-rbac-proxy/0.log"
Jul 12 22:58:19.666362 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:19.666323    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52pbf_1b1f17bc-a992-4c72-8fdc-a33a3c3649c4/ovn-controller/0.log"
Jul 12 22:58:19.681848 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:19.681796    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52pbf_1b1f17bc-a992-4c72-8fdc-a33a3c3649c4/ovn-acl-logging/1.log"
Jul 12 22:58:19.682002 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:19.681860    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52pbf_1b1f17bc-a992-4c72-8fdc-a33a3c3649c4/ovn-acl-logging/0.log"
Jul 12 22:58:19.697958 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:19.697931    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52pbf_1b1f17bc-a992-4c72-8fdc-a33a3c3649c4/kube-rbac-proxy-node/0.log"
Jul 12 22:58:19.717814 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:19.717788    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52pbf_1b1f17bc-a992-4c72-8fdc-a33a3c3649c4/kube-rbac-proxy-ovn-metrics/0.log"
Jul 12 22:58:19.737178 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:19.737152    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52pbf_1b1f17bc-a992-4c72-8fdc-a33a3c3649c4/northd/0.log"
Jul 12 22:58:19.756346 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:19.756306    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52pbf_1b1f17bc-a992-4c72-8fdc-a33a3c3649c4/nbdb/0.log"
Jul 12 22:58:19.778462 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:19.778436    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52pbf_1b1f17bc-a992-4c72-8fdc-a33a3c3649c4/sbdb/0.log"
Jul 12 22:58:19.833123 ip-10-0-135-47 kubenswrapper[2560]: I0712 22:58:19.833085    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-52pbf_1b1f17bc-a992-4c72-8fdc-a33a3c3649c4/ovnkube-controller/0.log"